Using a massive open online course (MOOC) to build a culture of evidence in Brazil

One of our fundamental goals at J-PAL Latin America & Caribbean (LAC) is building government capacity to generate and use scientific evidence in order to achieve more effective public policies. But are policymakers and civil servants in Latin America interested in learning more about these topics? Last year we saw evidence that in Brazil, the answer is yes.

In partnership with ENAP (the National School of Public Administration), J-PAL LAC’s Training team and Brazil team developed a MOOC (Massive Open Online Course) on Impact Evaluation of Social Programs and Policies. This course is a Portuguese version of J-PAL 101x: Evaluating Social Programs (also available in Spanish and English).

The partnership took off when J-PAL LAC approached ENAP—which had 340,000 students throughout Brazil enrolled in their online courses in 2017—to develop a Portuguese-language course relevant for a Brazilian audience, particularly government officials. The idea became a reality thanks to funds from J-PAL’s Government Partnership Initiative (GPI).

In 2017, J-PAL LAC took charge of developing new content and adapting existing material to the Brazilian context. J-PAL affiliated professors Claudio Ferraz, Bruno Ferman, Leonardo Bursztyn, and Marco Gonzalez-Navarro, along with invited professor Cristine Pinto and J-PAL LAC staff, recorded videos for the course. Evaluations led by J-PAL affiliates were used to develop examples, exercises, and case studies. ENAP hosted our staff and professors, recorded the ten lectures, edited the videos, and uploaded the course on its online education platform.

Watch the course trailer video:

The 30-hour course is free and open to both civil servants and the general public. The course was piloted over six weeks in November and December 2017 and the inaugural class had 5,790 students enrolled, most of them government officials.

Participants in the course learn why evaluations matter and how evaluations can be used to rigorously measure the impact of social policies and programs. Through a combination of lectures and case studies from real-world evaluations, topics of discussion include the benefits of measuring impact and methods of randomized impact evaluations; techniques for determining an appropriate sample size and evaluation design; threats that can undermine the integrity of evaluation results; and strategies for translating evaluation results into policy action.

The course is designed for people with a variety of backgrounds: managers and researchers from governments, foundations, and non-governmental organizations, as well as trained economists seeking to develop new knowledge in randomized evaluations. This made the course a good fit for Brazilian civil servants.

Luciana Ferreira de Almeida, who works at the Secretary of Budget and Planning at the State of Rio de Janeiro, took the course and also disseminated the opportunity among her colleagues in government.

"My interest in the course came from the perception that we have to create a culture of evaluation and evidence in the State to guide policy", Luciana said. "The government has the duty of directing public resources in the most effective way, strengthening and scaling up programs that really make a positive impact for people. However, very often the selection of programs is based on impressions rather than evidence, which results in waste of money in a State in fiscal crisis."

Approximately 25 percent of the enrolled cohort, or 1,488 students, completed the course—one of the highest completion rates to date among online courses offered by J-PAL—indicating that impact evaluation is a topic of interest among Brazilian civil servants.

Among the students who finished the course is Ronaldo Souza, director at the Secretary of Evaluation and Information Management (SAGI) at the Ministry of Social Development in Brazil.

"The course material is comprehensive and updated. The use of Brazilian and Latin American case studies get the theory closer to our reality," Ronaldo said. "The need to develop evaluations of social programs should be widely disseminated, not only to show positive impacts, but also to optimize the design, make corrections and scale up the access of programs based on this process."

This is just the beginning. In 2018, the online course will be offered regularly as part of ENAP’s academic calendar, and as a result, more government officials and citizens will have the opportunity to learn about the importance of measuring impact and using rigorous evidence to improve social programs.

We hope that the course contributes to fostering a culture of evidence and evaluation inside the Brazilian government. Watch the course preview and see more information about future courses.