Growing a global community for impact evaluation: J-PAL’s online courses

This is the third blog in a series illustrating stories of how J-PAL’s training courses have built new policy and research partnerships and strengthened existing ones to advance evidence-informed decision-making. The first blog in the series highlights two meaningful examples from J-PAL’s Evaluating Social Programs course participants, the second blog showcases how custom courses strengthened government partnerships in Indonesia and Egypt, and the fourth shares how training activities within J-PAL's evaluation incubators helped partners engage more deeply with research design.

J-PAL’s online courses expand the reach of our impact evaluation training materials to audiences worldwide and enable participants to access content on their own schedule. These offerings include Evaluating Social Programs online, which mirrors the content in our flagship in-person training. Through a series of integrated, interactive modules, the course provides participants with a thorough understanding of randomized evaluations and how they are designed in real-world settings. Participants dive into lectures by J-PAL affiliated professors and senior staff, case studies based on research projects from the J-PAL network, and opportunities to share ideas and collaborate via the course discussion forum.  

J-PAL currently offers the online version of our Evaluating Social Programs course in English, Spanish, and Portuguese, with plans to add content in French in the future. Each course is free to access, with the option to earn a certificate of completion for a small fee, open to anyone, and typically runs for several months. Since the first of these online courses launched in 2014, each year we incorporate improved and updated course content to reflect developments in the field and participant feedback on what is most useful in their work. 

These courses have reached tens of thousands of learners from 181 countries around the world since their launch. Over 11,000 participants have earned certificates of completion, and the worldwide course community continues to grow. This blog post zooms in on three examples of how these online courses helped participants around the world strengthen the culture of evidence use in their organizations and inform real-world evaluations. 

Applying lessons to education program evaluation in the Philippines

Juvhan Rebangcos enrolled in the Evaluating Social Programs online course in 2022 to build a monitoring and evaluation community of practice at Teach for the Philippines (TFP), where he is a data and impact assessment manager supporting the organization’s mission to ensure all Filipino children benefit from high-quality education. He noted that learning from the course equipped him “with the know-how of estimating social impact, which is critical in measuring and validating student outcomes,” and that the real-world best practices shared throughout the course are especially applicable to his and his colleagues’ work. 

TFP conducted a small pilot study to test the organization’s online, modular remedial literacy and numeracy program during the Covid-19 pandemic. After Juvhan completed Evaluating Social Programs online, he used what he learned in the course to refine the evaluation design, address challenges such as attrition and low attendance, and expand the sample size for TFP’s next evaluation, testing the impact of implementing the same program with face-to-face learning. He recalled that referring back to the course was his compass when managing the study earlier this year and also provided new ideas for future evaluations of TFP’s impact. 

He says, “Moving forward, we know that randomized evaluations will be a go-to option whenever we are testing a pilot, pursuing program innovations, or validating outcomes. Aside from the results we could use down the line, we can learn so much from our experience conducting them.” Juvhan plans to continue to apply lessons from the course to strengthen the culture of evidence use in his organization.

“The course prepared me for designing experimental setups, handling potential research threats, and navigating the ethics of our research activities. Most importantly, my course experience aided me in crafting data-informed, evidence-based, and compelling stories to demonstrate the eventual impact of our work as an organization.”
- Juvhan Rebangcos, Data and Impact Assessment Manager, Teach for the Philippines

Democratizing knowledge to inform decision making across Latin America

J-PAL’s Latin America and the Caribbean office (J-PAL LAC) offers two online courses in Spanish and Portuguese to democratize access for learners in the region. Evaluación de Impacto de Programas Sociales, in Spanish, is led by J-PAL LAC scientific director Francisco Gallego. Avaliação de Impacto de Programas e Políticas Sociais is offered in Portuguese, in partnership with the Brazilian National School of Public Administration. J-PAL LAC’s online courses fortify training for Spanish- and Portuguese-speaking partners, helping participants to engage with evidence for impactful policy change and fostering a culture of evidence and evaluation. Past participants highlighted the practical value of the material for designing, implementing, and evaluating programs in the region, as well as the courses’ effective combination of theory and practice. The courses also serve as an introduction to J-PAL’s work and a starting point for further partnerships.

Virgilio Pires, a labor inspector at the Brazilian Ministry of Labor, connected with J-PAL to explore evaluation opportunities after completing the course in 2021. This led to partnering with J-PAL affiliated researchers Jeanne LaFortune and José Tessada on a randomized evaluation measuring the impact of worker safety interventions, with support from J-PAL LAC’s Jobs and Opportunity Initiative Brazil. The evaluation will study the effect of virtual safety training and in-person safety inspections on worker well-being in the Brazilian manufacturing sector, as well as investigate the cost-effectiveness of traditional in-person visits compared to less expensive virtual training.

"The course was crucial in sparking my interest in public policy evaluation. From this course, I realized that it is possible to conduct an impact evaluation of projects in labor inspection… Thus, our proposal emerged to conduct an evaluation focused on preventing work-related accidents and diseases. Beyond providing knowledge on [impact evaluation], the course also presented J-PAL's work, which I was previously unaware of. We were fortunate enough to present the project to J-PAL, and [J-PAL affiliated researcher] Jeanne Lafortune accepted it because it aligned with research topics she was already exploring."
                                   - Virgílio Pires, Labor Inspector for the Brazilian Ministry of Labor 

Future directions: Tailoring hybrid learning for Francophone training partnerships

In 2022, J-PAL Europe developed a French-language version of the Evaluating Social Programs online course in partnership with the Morocco Employment Lab for a hybrid training with government partners in Morocco. Course development was spearheaded by J-PAL Europe's Director of Training Ilf Bencheikh, together with J-PAL affiliated researchers Elise Huillery, William Parienté and Roland Rathelot. The course, Evaluer les Programmes Sociaux, was first used in a hybrid format with the Moroccan Court of Audit in early 2023. Around thirty magistrates followed the online course modules over a period of three weeks. After completing the online course, the magistrates attended three half-day, face-to-face sessions where they explored certain aspects in more depth, interacted with the teaching team, and got answers to their questions. This partnership demonstrates the potential of tailored hybrid courses, which combine the accessibility of asynchronous learning for foundational modules with in-person content customized to a particular sector, geographic area, or level of technical detail. 

While Evaluer les Programmes Sociaux is not yet available for open enrollment, the J-PAL Europe team continues to tailor and incorporate course material into training partnerships, with upcoming events planned in Cote d’Ivoire. We hope to incorporate these materials into J-PAL’s open online course offerings in the future.

Another new online course, based on Evaluating Social Programs and carefully designed for the needs of education policymakers and practitioners in France, is currently being developed by J-PAL Europe’s Innovation, Data and Experiments in Education team. The course will launch in the coming months and join J-PAL’s suite of course offerings, marking a new area of innovation in online courses customized to support impact evaluation and evidence use by region and sector—check back on our online courses page for updates once the course is available. 

Join a global community of practice

Spencer Obiero, a student at Bates College from Nairobi, Kenya who completed J-PAL’s Evaluating Social Programs online course earlier this year, said that the course reminded him of a classic team of superheroes: “I like how the different professors, like the Marvel superheroes The Avengers, have been brought together to teach this course. It’s amazing!” 

Online courses, like superhero movies, are all about bringing together a large cast with diverse backgrounds to tackle critical and complex problems. With Evaluating Social Programs online, J-PAL combines regional expertise with a worldwide network to support the work of policymakers, practitioners, and program implementers in understanding and measuring impact. However, the core of the J-PAL’s online courses is the dedicated learners who continually share their thoughts and experiences from different contexts, improving the courses—and each other’s learning— with their ideas.

Interested in enrolling in one of J-PAL’s online training courses? Evaluating Social Programs is open for enrollment now through December 1 on the MITx Online platform, and you can visit our online courses page for a full list of upcoming offerings. To dive even deeper into topics in economics and data analysis, consider enrolling in one of the semester-long courses in the Data, Economics, and Design of Policy MicroMasters program.