The long and winding road of evidence adoption

J-PAL’s mandate is to improve lives by ensuring that policy is informed by scientific evidence. We know that sharing a link to an informative academic paper with a senior policymaker, or even a more readable summary, has little chance of creating the engagement, momentum and follow-through needed to change lives on the ground.

So what does bridging the gap between evidence, policy and practice actually look like? How do you meet policymakers where they are, providing timely advice on their priorities without selling them a solution in a box that will self-destruct when jammed into a completely different context? How can we stand on the shoulders of academic and implementation giants, drawing on global evidence and implementation best practices, while still having our feet firmly planted in the context we are trying to support?

Over the last seven years, I have been fortunate enough to be a member of the J-PAL Africa policy team, where improving lives is at the center of our decision-making and how best to do it is part of an ongoing journey. As I prepare to transition from J-PAL to a new role at Teaching at the Right Level (TaRL) Africa, (a joint effort of J-PAL and our partner Pratham), I’ve been reflecting on these challenges. I wouldn’t say we’ve solved them yet, but we have learned a great deal and bridged important gaps from evidence to action.

Most of our work has been with African governments that are aiming to solve complex policy challenges with limited resources, while navigating often competing priorities from the donors they depend on. We tread carefully, focusing on where we can add value in these environments, as the risks of unintentional harm are high. Working together with team members who so deeply care about J-PAL’s mission has been a gift. There are three approaches we’ve adopted that I think have been critical to evidence adoption and contributing to improved lives.

1. Use randomized evaluations as a piece of the puzzle, not the whole picture.

Policymakers face a multitude of daily challenges and decisions. In most instances, there is not an evaluated solution for each of these challenges that falls within the policymaker’s geography. Using geography as a constraint can limit the advice evidence-to-policy translators can provide—but this doesn’t mean that new products or programs must be developed from scratch.

At J-PAL Africa we use the generalizability framework described in Mary Ann Bates and Rachel Glennerster’s The Generalizability Puzzle, to help policymakers make better-informed decisions and better-designed programs. This involves combining global lessons from rigorous evidence and economic theory with local knowledge and data to provide helpful advice.

This enables us to meet policymakers where they are and address their immediate challenges. Through this approach we can leapfrog untested ideas, and start with ideas that are grounded in solid evidence and theory.

In 2015, when the Zambian Ministry of General Education (MoGE) expressed concern about the low levels of foundational skills among students in upper primary school, we shared evidence from India on the effectiveness of Teaching at the Right Level (TaRL). We connected these policymakers to Pratham (the pioneers of the approach), and co-designed a program informed by global evidence but rooted in local conditions.

This was not a matter of copying and pasting a program from one country to another. The process of designing Catch Up, Zambia’s TaRL program, was a collaborative process that brought together local actors, global evidence, extensive implementation data, and scale-up knowledge. Leveraging a 15-year journey of rigorous evaluation in India meant that the Zambian MoGE did not need to start from scratch.

In 2021 alone, six years after its launch, Catch Up reached over 200,000 children. While the journey to refine the program is ongoing, that 15-year head start has certainly accelerated progress.

2. Get the implementation details right.

While the gap between evidence and policy is an important one to fill, an even larger one may be the gap between policy and practice. Many institutions are getting it “right” by writing the need for evidence-informed programming into terms of references or project appraisal documents. However, the broad bulleted lists of evidence-informed strategies in these documents rarely make it to the beneficiary intact—they are often diluted through the many steps it takes to transform strategy guidance into action.

There are a number of factors that may contribute to this dilution. One is that those who craft the terms of reference request that evidence be used, but often do not provide sufficient guidance or support on how to use evidence. Another is that implementers, whether governments or NGOs, are generally inclined to continue the status quo, evaluated or not. The costs of this strategy are lower; the human and material resources of past approaches often already exist and are much easier to tweak than to overhaul. 

Finally, collecting the necessary process and outcome data to aid learning and adaptation in a new context has the potential to reveal what is not working, which means exposing program challenges to the donor and potentially diminishing chances of future funding. This can make implementers cautious about applying global lessons and evidence, as the process required to ensure the programme is locally grounded through collecting more data could inadvertently jeopardize funds. Donors who recognize this incentives mismatch and ensure that implementers are not penalized as a result of learning and iteration can help avoid this.

Closing the gaps between evidence generation and use

At J-PAL Africa we want evidence to contribute to improving programs—and ultimately lives. Our experiences suggest that effective evidence adoption does not happen organically. There are often several gaps between evidence creation and use. J-PAL Africa is working with many governments, implementing partners, and multilateral institutions to fill these gaps by operationalizing broad evidence-based lessons by connecting implementing partners with right academic and technical expertise, developing user-friendly evidence-use tools and co-developing and designing program plans.

An example of this work is in Northern Nigeria, where we are providing technical support to the Katsina State government to design a life skills program to increase school participation for adolescent girls.

In order to support Katsina State to design a truly evidence-based program, we developed an evidence synthesis outline of what we know—and don’t know—about different life skill courses’ content and delivery; supported a needs assessment to better understand the challenges to girls’ education and empowerment in Katsina; will connect local policymakers to leaders of programs that have been found to be effective; and are planning a series of design workshops with the state government and local stakeholders. These efforts, combined, aim to help the government to design a contextually-appropriate and evidence-based life skills program.

There is a great need for more of these evidence translation “services.” Including stakeholders whose job it is to prioritize evidence use and learning when developing new initiatives or programs is critical to help shift if, and how, research is used. Organizations like J-PAL Africa are well placed to do this kind of work as we are not invested in a particular program or policy agenda, but rather we are invested in using both global evidence and local knowledge and data to design effective programs—and we have developed a skill set for how to do so.

3. Stay a long-term knowledge partner.

The process for evidence adoption is a long and winding one, and complete contextualization can take time.

One of J-PAL Africa’s unique offerings is that we are invested in being a long-term knowledge partner. For governments in Africa, support often comes in the form of projects with defined start and end dates, or once-off injections of technical advice concluded by a 100-page report deliverable with unrealistic assumptions and impractical recommendations.

The challenges policymakers face are complex. They do not have end dates. There are no quick fixes. J-PAL is committed to working hand-in-hand with governments to strengthen and refine programs so that they achieve their intended goals. This is an iterative process that cannot be achieved in a single step, but we are eager to walk this path together.

In our work with policymakers and partners we have focused on building the capacity to reflect on what we are learning and adapting so that we can achieve better outcomes for the people whose lives we aim to improve. Our in-depth engagement in the delivery of different evidence-informed programs has pushed us to grapple with the realities of these programs more deeply, prompted us to engage with different kinds of data to investigate challenges thoroughly, and enabled us to develop useful and practical learning agendas for the partners and programs we support.

This is reflected, for example, in our work with the MoGE in Zambia to further integrate Catch Up into government systems. In-depth qualitative research on what teachers find helpful to deliver effective TaRL classes, alongside in-depth interviews to understand the existing support mechanisms for teachers, has revealed several ways in which the program could continue to be refined to ensure sustainability. Together with the MoGE and partners VVOB, TaRL Africa, and Pratham, we plan to identify innovations to be trialled which could reduce training and mentoring costs. These long-term engagements are all part of the evidence adoption and scale-up process.

Looking to the future

Even as I prepare for a new position as Programs and Partnerships Director for TaRL Africa, I have never been more convinced of the important role J-PAL plays in this sector. I have learned so much, and also been made aware of how much I don’t know.

The J-PAL Africa Policy team consists of a group of translators, connectors, coordinators, and collaborators. We approach this work with a passion for constructive partnership, a deep appreciation for the importance of the implementation details, a focus on applying what we know and learning about what we don’t, and a hunger for continual improvement. We believe this is important work, but it is severely under-funded. Our ability to support this kind of work has only been made possible by generous, flexible core funding that is not project specific.

I have experienced such extraordinary generosity and kindness from our partners. I am forever grateful to all those who replied to my cold-call emails, and to busy government officials who answered their door and pointed me in the right direction. While J-PAL Africa may no longer be in my email signature, the beliefs that there is strength in collaboration, that continual learning and adaptation are necessary for success, and that decisions should be centered on who you’re aiming to help, are strongly entrenched and ones I will carry forward with the team at TaRL Africa.