The devil’s in the details (of adapting and scaling evidence-based programs)

One day in early spring 2019, I hopped off of a motorbike in a village in Western Côte d’Ivoire to speak with a group of teachers about an exciting new education program from India: Teaching at the Right Level (TaRL), or as it’s locally known in Côte d’Ivoire, the Programme d’Enseignement Ciblé (PEC).

I was seconded from my role at J-PAL to support the government-led scale up of the PEC program, and was trying to learn more about implementation at the school level. After sharing a round of introductions with the teachers, we began a series of focus group discussions.

“The children love PEC—they learn by playing and the activities help them quickly learn reading and math skills,” they shared with me. “For us teachers, dividing the children into level-based groups for a few hours a day allows us to tailor instruction to their level. PEC is a great idea for this reason.” 

I prodded them further on their feelings about the new remedial pedagogy they had been working with for the past year. 

“But there are some challenges, too. It’s too far for us to travel to the nearest district office to report learning results. And the administrative tasks are a burden on the school director, who also has her own class to teach.”

It was clear that while PEC was a good fit, further adaptation of certain components of the program were needed before scaling up. 

I had two objectives in Côte d’Ivoire: to help ensure successful completion of the pilot phase of the program, and to work with government officials and the NGO Pratham, the creators of TaRL, to further adapt the systems-level approach for scale-up in the Ivoirian context. Along the way, I learned a few lessons about scaling up evidence-based programs like TaRL within government systems. 

The details are critical when working with a government to scale a policy innovation.

Teaching at the Right Level (TaRL) is an approach developed by Pratham that aims to build foundational skills in math and reading for all children before exiting primary school. To date, TaRL has improved learning opportunities for over 60 million children in India and several African countries.

At the classroom level, the approach works by assessing children’s learning levels, grouping children based on learning needs, focusing on foundational skills using an engaging pedagogy, and tracking children’s progress.

When TaRL is implemented within government systems, Pratham helps ensure that teachers receive support through mentors and monitors. These elements have been critical to the success of the program, as shown in previous at-scale randomized evaluations, as they encourage and support leaders throughout education systems to keep the focus on foundational learning and quality implementation of the program. 

Because of the relative complexity involved in implementing an approach like TaRL, adapting the program to new contexts such as Côte d’Ivoire requires deep understanding of two types of details: (i) the methodology and theory of the approach and (ii) the local context and the capacity of local systems.  

For example, in many countries, teacher support is the responsibility of local administrators. One challenge when adapting teacher support systems across contexts is wide variation in local administrative structures from country to country. Some countries employ sub-district level coordinators (akin to a superintendent) to supervise and support school-level staff; others do not.

Much of our work with the government in Côte d’Ivoire focused on this question specifically: how to adapt the structure of the program so that mentoring and monitoring support is in place without overburdening an already resource-constrained system. 

Adapting approaches designed for scale like TaRL across contexts requires leveraging the expertise of officials and technical staff embedded in complex education systems. Because the nature of these systems can vary dramatically across contexts, questions of whether a program could work in a new area depend not just on evidence, needs, and political economy, but also on the deeply granular details of program implementation and system capacity required to scale a given policy innovation. 

Local leaders know the context-related details best. 

Arguments abound for why local leaders should be empowered to lead education reforms inspired from programs or policies in other contexts. Empowered local leaders can increase the political sustainability of new initiatives. They can also be more committed to the communities they serve, keeping their best interests in mind.

But there is another very practical reason: local leaders are vastly more likely to know the political and operational details, both formal and informal, of education systems and to predict areas where a program may need further adaptation.

In Côte d’Ivoire, one key challenge was finding local administrators with the time and skills to manage process monitoring data necessary for larger scale implementation of the program.

In district offices, there are “statistics assistants” who play a key role in data collection and administration at the beginning and end of the school year, but who are relatively idle for the rest of the year. This cadre of officials was thus a logical candidate for personnel to support the process monitoring components of the program as it scales up. Luckily, because the government (not an outside consultant or donor) was leading the implementation of the program, we didn’t have to read through org charts or have phone calls with district offices to figure out if this was a viable option. We simply asked our national government counterparts, who have extensive experience working with district offices, and they confirmed that this could work.

My colleagues at TaRL Africa, who work with governments and other partners across a variety of contexts, play this role often in making space to think through key implementation issues with government leaders. TaRL Africa brings its deep knowledge of program details, and government leaders bring deep knowledge of the education system. Together, they are a powerful force for adapting and scaling evidence-based education programs that will impact the lives of millions of children.

Improved capacity to leverage data for action can help to implement new ideas at scale.

High-quality data systems can be critical to monitor program implementation within government systems. My colleagues at J-PAL’s Latin America and the Caribbean office have documented the important role of relevant and timely administrative data to enable governments to monitor implementation as effective programs are scaled up.

A similar lesson is emerging from the projects supported by J-PAL’s Innovation in Government Initiative: developing user-friendly charts and dashboards with key stats on program implementation can help officials drive better program management decisions as programs scale.

The experience in Côte d’Ivoire has been similar. In the case of TaRL, information on children’s learning and progression provides important insights for administrators tasked with supporting teachers who implement the program in schools.

In late June, I met with the education chief from the district of Méagui in Western Côte d’Ivoire. During our meeting, I turned my laptop around to show him a simple data visualization on students’ progress in his district, which showed significant improvements in reading and simple math, but lagging progress in mastering operations like multiplication and division. After quickly looking over the graphs, he responded, “Wow, that is impressive progress. But there is still lots of work to do.” He turned to his staff and said, “Let’s focus on strengthening multiplication and division instruction in our next meeting.” 

Good data on important components of a program can be critical for decision making, especially as the program scales and as coordination across broad geographies becomes more complicated. A key focus of our work in Côte d’Ivoire, and of TaRL Africa’s work in a variety of contexts, is to build the capacity of education systems to use good data for decision making.

In many cases, these capacities exist already but are not being leveraged in a coordinated way to drive action. For example, going forward, the district-level statistics assistants in Côte d’Ivoire will be trained to analyze the data and meet regularly with education chiefs to review.

There is great potential to scale policy innovations at the intersection of global evidence, local knowledge, and program-specific expertise.

At J-PAL, we’ve shared a simplified framework for thinking through whether or not a given program or policy could work (or “generalize”) in another context. This framework breaks down the generalizability puzzle into four answerable questions:

1. What is the disaggregated theory behind the program?

2. Do the local conditions hold for that theory to apply?

3. How strong is the evidence for the required general behavioral change?

4. What is the evidence that the implementation process can be carried out well? 

It’s critical to think carefully about the first three questions, drawing from the global evidence base and policy priorities across contexts. My experience with TaRL in Côte d’Ivoire, as well as the experiences of many of my colleagues, suggests that the last question on implementation processes is equally important for bringing evidence-based innovations to scale. This question can only be answered with a deep understanding of the details. 

Organizations like TaRL Africa are harnessing the great potential of scaling evidence-based programs at the intersection of deep knowledge of global evidence, program details, and local contexts. More importantly, they’re enabling governments to lead these changes themselves.

Robert Rogers is a Senior Policy Associate at J-PAL and Manager of the Innovation in Government Initiative.