Increasing the use of data and evidence in real-world policy: Stories from J-PAL’s Government Partnership Initiative

At J-PAL, we often talk about the importance of using evidence to design and implement effective policies and programs, and many governments are working to make it a priority. But what does this look like in practice? How can we best support governments in finding and applying the evidence they need to make informed decisions?

We manage the Government Partnership Initiative (GPI), a J-PAL fund that supports governments in using evidence in policy design and decisions. This takes many forms, from temporarily placing J-PAL staff in a government office to support them in scaling up an evidence-informed program to developing new platforms to help governments use their administrative data more effectively. While J-PAL manages GPI’s day-to-day operations, the initiative is funded through generous support from longstanding J-PAL partner Community Jameel, as well as several other foundations and individuals dedicated to maximizing the positive impact that governments can have.

Since it launched in 2015, GPI has contributed to at least two scale-ups of evidence-informed policies in social protection and education in India and Zambia and improved systems for evidence use in Chile, India, and Peru. Our experience illustrates some promising strategies for closing the gap between evidence and policy in government. We’ve found that relatively small investments dedicated to putting evidence into practice can help some of the biggest players in poverty reduction use evidence to invest their limited funds more effectively.

This post is the first in a series on practical lessons learned from the first two years of GPI partnerships. Before we get into details, let’s start at the beginning:

Why create a fund to increase governments’ use of data and evidence?

Developing country governments support social programs that reach millions of people, with budgets that dwarf those of foreign aid agencies and foundations. Like many institutions, they sometimes spend money on ineffective programs or lose program resources due to implementation challenges.

By identifying ways to improve existing programs and pointing policymakers to more effective approaches, evidence from rigorous impact evaluations can help governments allocate scarce resources to programs that can have a bigger impact on the lives of the poor.

Since J-PAL was founded, we have built many long-term partnerships with governments to generate and use rigorous evidence in policy decisions. J-PAL's network of affiliated researchers and offices has conducted over fifty randomized evaluations in collaboration with governments, in addition to dozens of long-term capacity building and policy partnerships (for example, see GPI Co-Chair and J-PAL Executive Director Iqbal Dhaliwal's recent op-ed about J-PAL South Asia's decade of work with governments in India).

When governments decide to use data and evidence to improve policy, the results can be powerful. Of the 300 million people reached by scale-ups of programs evaluated and found effective by J-PAL affiliated professors, over 80 percent were reached by programs implemented or commissioned by governments.

What is the Government Partnership Initiative?

Recognizing that increasing the effectiveness of public spending is one of the biggest levers we have to reduce poverty worldwide, and given our positive experience working with governments over the past decade, we launched GPI in 2015 with a mission to build long-term partnerships between researchers and governments to increase the use of evidence in policy.

GPI provides funding to support three types of activities:

1. New randomized evaluations and pilot studies designed to answer governments’ priority questions; 
2. Technical assistance to governments to scale up an effective program based on existing evidence; and 
3. Technical assistance to governments to create systems and institutions that can help make evidence-informed policymaking more the norm.

What has GPI supported so far?

GPI has awarded US$2 million over the past two years to support 28 partnerships in 15 countries. This represents just 24 percent of the US$8.3 million requested by the 67 governments and researchers that have applied for funds.

Two measures of GPI's progress are our partnerships’ contributions to scale-ups of evidence-informed policies, and to improved systems for data and evidence use. We also track the number of policy-relevant research projects that are funded by or emerge from partnerships supported by GPI.

In all the partnerships GPI supports, J-PAL affiliated researchers and staff could not contribute to meaningful policy change without the leadership and dedication of our government partners, who are leading the way in making evidence-informed policy a reality, as well as the generous support of many funders and implementing partners.

SCALE-UPS

Most of our partnerships are still ongoing, with the potential to inform and improve policy in 2018 and beyond. But to date, the GPI portfolio has already contributed to at least two governments scaling evidence-informed policies that reach at least 48 million people.

A national funds flow reform in India

In India, GPI provided funds to place a J-PAL South Asia staff member in the Ministry of Rural Development to support them in scaling up a financial reform in a national social protection program for the poor, the Mahatma Gandhi National Rural Employment Guarantee Scheme (MGNREGS).

In 2012, J-PAL affiliated researchers Abhijit Banerjee (MIT), Esther Duflo (MIT), Clément Imbert (University of Warwick), and Rohini Pande (Harvard), collaborated with Santhosh Mathew, an official in the Government of Bihar, to conduct a randomized evaluation of a financial reform that disbursed MGNREGS funds to local governments on a reimbursement basis, rather than in advance. The reform reduced losses due to corruption and inefficiency by 24 percent, saving the government roughly US$6 million. The study was funded by the International Growth Centre and the International Initiative for Impact Evaluation.

This study formed the basis for the collaboration between J-PAL and the Ministry of Rural Development, where Santhosh Mathew began serving as Joint Secretary in 2013, marking an opportunity to leverage learning from the study nationwide. He collaborated with J-PAL affiliated researchers to share the research results and emerging policy insights with key central government ministries. The GPI-funded staff member supported this work by writing policy memos and coordinating meetings to generate buy-in for the scale-up, and developing a policy implementation plan with detailed steps, workflows, and timelines.

The research and policy outreach informed the Indian Union Cabinet’s approval of a national reform of MGNREGS funds flow in 2015. The reform allowed beneficiary payments to be made through a new National Electronic Fund Management System directly from the central government to the beneficiary’s bank account. The Cabinet note cites the evaluation as part of the rationale for this decision—highlighting the reduced leakage and improved service delivery generated through fund-flow reforms.

The reform has been scaled up in 21 states and one union territory, where the MGNREGS program reached over 48 million households last year. The authors of the Bihar study examined the rollout of the national reform and found that program expenditures fell by 18 percent, similar to the effect found in the original study.

“I could not have done this on my own. It’s the powerful research but most important—far more important—is the support over the last six years that I got from the J-PAL team. They have been a dedicated team whose only job is to help people like me to push so that when I need a new document it’s redone. When I need to make a presentation, it’s they who do it for me.”
– Santhosh Mathew, former Joint Secretary, Ministry of Rural Development, India

Teaching at the Right Level in Zambia

In Zambia, GPI provided funds to J-PAL Africa to support the Ministry of General Education in designing and piloting a remedial education program based on randomized evaluations from India, Ghana, and Kenya. GPI funds supported learning journeys between India and Zambia and helped support independent process monitoring of the government’s pilot program in 80 schools.

The Zambian remedial education program, named “Catch Up,” is based on the Teaching at the Right Level (TaRL) pedagogical approach. The program aims to improve the basic literacy and numeracy skills of primary school students by grouping them according to their initial level of knowledge and providing them with instruction tailored to their current learning level.

Pratham, a large education-focused NGO based in India, developed and refined the TaRL approach over the last two decades. Pratham has collaborated with J-PAL affiliated researchers since the early 2000s to evaluate of several different versions of the approach in India. Results showed that tailoring instruction to the level of the child was a highly effective and cost-effective way to improve student learning, and Pratham has since scaled it up in many states in India.

Some of the GPI funds in Zambia supported learning journeys, enabling Zambian officials to travel to India to see Pratham implement the TaRL approach firsthand. GPI funds also supported Pratham staff travel to Zambia to train the ministry on implementing the TaRL model. Zambia's design additionally drew on global evidence on adjusting teaching to the level of the child, including from the TCAI program in Ghana and from a Kenya program that tracked students by ability.

As a result of this work, the Zambian government launched an 80-school pilot of the Catch Up program, trying out multiple versions to identify the Catch Up model that was best for their school system. The GPI grant supported a collaboration between Innovations for Poverty Action (IPA), J-PAL Africa, and UNICEF to conduct an independent process monitoring of the pilot program to assess the viability of a broader scale-up to more schools across the country.

In August 2017, after the completion of the pilot and process monitoring, the Ministry decided to scale the program to approximately 1,800 schools across Zambia over the next three years. The evidence-based program will reach approximately 280,000 students in grades 3-5 with support from USAID’s Development Innovation Ventures and the USAID Zambia Mission. Several additional partners provided support to the pilot and/or scale-up, including UNICEF, IPA, Pratham, VVOB, and Zambia Education Sector Support Technical Assistance.

GPI funds helped meet a critical policy window. The Ministry was interested in finding a remedial education (or "Catch Up") program, but did not know about effective approaches from randomized evaluations or have the flexible funding to adapt and monitor a Catch Up program to their context. By meeting this need, the Ministry was able to test the program for itself, monitor the implementation, and make decisions about how to scale it up. Other partners (such as USAID) were also watching the pilot and only decided to invest after they saw that this program could be successfully implemented in Zambia.

IMPROVED SYSTEMS FOR EVIDENCE USE

While individual evaluations and scale-ups are important, GPI's ultimate goal is to institutionalize the use of evidence in policymaking, making it more the norm rather than the exception.

GPI grants have contributed to several improved systems for data and evidence use that encourage, require, or incentivize greater generation and/or use of evidence in the policy process. Below are just three examples:

1. In Chile, the Ministry of Economy’s Innovation Fund worked with J-PAL LAC to create new system for reviewing potential investments requested by other departments within the government. The system requires applicants to include a theory of change and review of existing evidence from past impact evaluations when seeking funds. They also launched a new public bid for impact evaluations of the Innovation Fund’s investments and reserved a portion of program funds for evaluation.

“In the fund, this system created the perception that it is necessary to base yourself on evidence to make public policies.”
– Antonio Martner Sota, Research Coordinator, Innovation Fund for Competitiveness, Ministry of Economy, Development, and Tourism

2. In India, GPI is supporting staff working on J-PAL South Asia’s institutional partnership with the Government of Tamil Nadu to create an evidence-informed approach to policymaking across its departments. To date, the government has supported 15 policy-driven research projects across nine departments, using over US$4 million of its own funding and collaborating with more than 34 academic researchers. Recently, the state issued new guidelines requiring all programs in sectors of strategic importance with budgets greater than US$22 million undergo impact evaluations. The state’s Planning Department has also created a dedicated fund for program evaluation of US$1.5 million per year.  Using funds from GPI, J-PAL South Asia staff also provide capacity-building support to a data analytics unit set up by the Government to allow them to make real-time decisions based on the administrative data they already collect.

3. In Peru, the Ministry of Women and Vulnerable Populations refined their approach to reducing domestic violence to focus more on violence prevention after J-PAL LAC created an evidence repository summarizing key findings from impact evaluations on reducing violence. As a result of this partnership, the Ministry is also currently preparing a randomized evaluation with J-PAL affiliated researcher Erica Field to evaluate the impact of a community program to prevent domestic violence. 

"With this repository it was clear that one of the most important themes in the area of violence was prevention of violence. Our focus changed based on this evidence.”- Director, Monitoring and Evaluation, Ministry of Women and Vulnerable Populations

ONGOING PARTNERSHIPS

GPI has twenty ongoing partnerships, many of which have the potential to inform and improve policy in the future. For example, six of the 16 institutional partnerships supported by GPI have already contributed to an improved system for data and evidence use or led to a new evaluation on a government’s priority question.

Research partnerships often represent the first time a government or department is investing in evaluation, and often lay the groundwork for future collaborations that move beyond a single project. For example, the partnership described above with the Ministry of Rural Development in India began with a single evaluation in Bihar and contributed to a national reform after several years of collaboration.

What are we learning about building a culture of evidence use in government?

Our experience with these partnerships has created a rich base of internal knowledge on how to most effectively work with governments to apply evidence to policy in meaningful ways. We’re working to summarize and share this knowledge in an effort to further expand GPI’s impact in 2018. Stay tuned for our next post in this series to learn some early insights from our interviews with our government partners.

For more information about GPI, including opportunities for foundations and individuals to support this important work, please contact [email protected].

This is the first of two posts about GPI. Read the second post in this series.