Informing shifts in policy: Reflections on a long-run impact evaluation of a community block grant program in Indonesia

Creating good policies is a complex and dynamic process. A program can be evaluated, found to improve people’s lives, and be scaled up. However, in the longer run, these programs interact with other policies, contexts change, and policymakers are in constant need of new information and evidence to adjust their programs and address new challenges.

This presents an opportunity for researchers and policymakers to collaborate on long-term evaluations to understand how program impacts and contexts change over time, and how policies can be adjusted according to the evolving needs of the people. Yet long-term evaluations are rare after a successful program has been expanded; Governments often decide to expand these programs to control areas that did not originally receive the program and assume that progress continues.

In the case of the Generasi program (the National Community Empowerment Program—Healthy and Smart Generation), the Government of Indonesia (GoI) took a different approach. Based on the results of a 2009 evaluation of the program, as they scaled up Generasi they prioritized expansion to the least developed regions of the country first, rather than expanding it to the control areas from the original study—such as West Java, East Java, and North Sulawesi, which had better baseline access to infrastructure and health care services. This created an unprecedented opportunity for a long-term randomized evaluation of the original  intervention.

The beginning: Using block grants to incentivize better health and education outcomes

Generasi was an incentivized community block-grant scheme developed to help Indonesian villages improve the quality of and access to health and education services. The program emerged as part of the PNPM-KDP (Kecamatan Development Program)—a community-driven development program that has been in place in Indonesia since 1998.

As part of the program, facilitators worked with villagers to conduct a social mapping and participatory planning exercise to decide how the block grant would be spent, with the goal of improving twelve key health and education indicators. In addition, the program had an incentive component, in the form of a performance bonus—the value of the block grant villages received the following year varied based on their performance in fulfilling the twelve target indicators.

To allow room for a rigorous, randomized evaluation of Generasi, the National Development Planning Agency (Bappenas) and the Ministry of Home Affairs (MoHA) worked closely with J-PAL affiliated researchers and the World Bank to incorporate random assignment into the original selection of Generasi locations. The first evaluation was conducted from 2007-2009 by J-PAL affiliate Ben Olken (MIT), together with Junko Onishi and Susan Wong.

The short-term randomized evaluation found that Generasi improved health and education indicators across a range of dimensions: primary school enrollment improved; weight check frequency increased; and malnutrition and stunting decreased.

Furthermore, the most disadvantaged areas, which often lacked sufficient health personnel and infrastructure, performed better and benefitted the most from Generasi. These results were particularly strong in East Nusa Tenggara (NTT) subdistricts.

Based on the results of this evaluation, in 2010, the GoI decided to scale up the incentivized block grants and revisited its program to target stunting and nutrition outcomes further. In total, the scale-up reached 4.9 million women and children in approximately 499 subdistricts from 2010-2018.

Long-Term Evaluation: What changed after nine years of Generasi program implementation?

After the expansion of Generasi, another policy window surfaced in 2016 that provided an opportunity for the researchers involved in the first evaluation to again collaborate with the GoI on a longer-term evaluation of the program. This evaluation was facilitated by two main factors:

1. Following the original evaluation, the government expanded the Generasi program to more disadvantaged regions of the country instead of villages in the control areas. Based on a recommendation from the short-term evaluation, the government prioritized the expansion of the program to the least developed regions in the country, which lagged behind in terms of facility quality and overall access to public health services.
2. The Generasi program was, by design, slated to end in 2018. The government planned to gradually replace Generasi by the enactment of the Village Law in 2014—a massive decentralization effort that significantly increased direct transfers to villages. This offered an opportunity to evaluate how the program interacted with other government policies and to build the evidence base on whether a similar incentive mechanism should be used by the Ministry of Village to manage the village funds.

The long-term evaluation found that Generasi’s initial impact on stunting—concentrated in NTT province—and improvements in school enrollment had not been sustained over time. Researchers found evidence suggesting that these results could be explained by overall, countrywide improvements in health and education since the end of the previous evaluation in 2009 (i.e. through new programs such as the enactment of Village Law and the launch of an integrated National Health Insurance system in 2014). With the general improvements in what used to be weak performing indicators in 2009, there was less room for improvement on many of the Generasi targets evaluated in 2016.

However, the evaluation also showed that one of Generasi’s most meaningful accomplishments was the continued revitalization of the Posyandus—integrated community health posts which provide free basic maternal and child services at the grassroots level—which was accomplished through information dissemination, community participation, and the targets/incentive system. Families in Generasi communities remained more likely to complete infant weight checks and attend maternal health and parenting classes at these community health centers. In the least developed areas, Generasi continued to be effective in getting people to use community health services, increasing both immunization and the uptake of Vitamin A supplements.

These findings were presented to academics and policymakers at the J-PAL SEA Stunting Conference in 2018. Based on the short-term and long-term findings, the researchers recommended that future GoI health-related programming should consider increasing its efforts to sustain Posyandu and ensure mothers continue to bring in their children for weight/height measurement, attend early childhood education programs, and gain basic maternal and infant health services after the exit of Generasi.

The World Bank incorporated insights from the evaluation in designing their new Investing in Nutrition and Early Years (INEY) program to reduce stunting. Other J-PAL researchers at the J-PAL SEA Stunting Conference in 2018 recommended to the GoI that it expand the national conditional cash transfer program, which was shown to be effective in reducing stunting over the long-term.

Learnings: Takeaways from maintaining a nine-year collaboration with government partners

When Generasi was first introduced and scaled, it had meaningful impacts on nutrition and education outcomes in the least developed areas of Indonesia. As time continued and as the government invested in more social protection programs, health care services, and infrastructure to ensure equity across regions, it became apparent that a comparatively small performance-based community block grant may no longer be as relevant. While the Generasi program was still more effective in improving certain indicators in the most disadvantaged areas, country-wide improvements to infrastructure and health personnel meant that there were fewer areas where the Generasi program could demonstrably affect growth.

Nevertheless, the results of the longer-term evaluation provided significant value in shaping future programming. The randomized evaluation confirms that the national prevalence of stunting remained virtually unchanged between 2007 and 2013. Additionally, it showed that one of Generasi’s greatest accomplishments was the sustained revitalization of the Posyandu, which was accomplished through program facilitation, community participation, and a targets/incentive system. Therefore, future GoI health-related programming needs to consider how to sustain the Posyandu and ensure that mothers continue to bring their children for weight/height measurement, PAUD attendance, and basic maternal and infant health services after Generasi exits.

Complementing the result of the randomized evaluation, the qualitative study also showed that community participation can help revitalize village-level Posyandus. These findings have been used by the World Bank to design their new program (Investing in Nutrition and Early Years) that aims to reduce stunting. The learnings were also taken up by the Ministry of Village, Disadvantaged Regions, and Transmigration in their guide for villages to tackle stunting in places where lack of infrastructure and access to health care remain key challenges. Both documents prioritized additional investments in Posyandu and early childhood education as part of national guidance for village governance, which is in line with the findings of the long-term evaluation of Generasi.

Based on our experience supporting this long-term collaboration with key government stakeholders, our team at J-PAL SEA along with partners from World Bank and Bappenas has identified several takeaways:

• It’s important to adapt to new priorities and programs to increase the policy-relevance of evaluations. Maintaining interest in a long-term evaluation can be challenging. Other than having a key champion within the government who understands the value of rigorous evidence, researchers should take their policy partner’s concerns into consideration and adapt the study to reflect changes in policy priorities. In 2014, some of the initial targets of the Generasi program were revised to include nutrition counseling and school participation for people with disabilities, which were priorities for the GoI. Incorporating outcomes and survey questions that could generate evidence on the government’s growing concern about stunting allowed the study to maintain its relevance and generated collective support to run the randomized evaluation for almost a decade.
• Conducting qualitative research in conjunction with a randomized evaluation can provide significant value in informing what works and why. The Generasi impact evaluation was supported by qualitative research to understand people’s experiences with the program and generate hypotheses about which parts of the program were working and which were not. From informant field interviews, the research team was able to generate several potential hypotheses about why the health impacts did not persist in 2016 and evaluate them by analyzing the data from the randomized evaluation. As confirmed by one of our interviewees, the qualitative research of the Generasi program conducted by the World Bank also greatly informed the design for their new program, which targets stunting in Indonesia, especially on the importance of citizen engagement and community empowerment to support the government program.
• Between data collection periods, maintaining close engagement with government partners is essential. Staying engaged with government partners allowed researchers to tailor findings into useful recommendations to inform policy. Although the results were still forthcoming, the J-PAL research team and affiliated researchers maintained frequent communication with government partners to understand their concerns, as well as to present the progress of the study and preliminary results as soon as they were available. These engagements allowed the research team to stay up to date on policies that may have interacted with the program being evaluated, understand any changes in priorities, and maintain momentum and advocate for the importance of conducting the evaluation. As mentioned by one of our champions in the government, convincing different institutions to maintain an evaluation over a long period can take significant political capital—providing preliminary results helps people in the government to set their expectations, maintain interest in the results, and plan accordingly.

The evaluation of the Generasi program would not have been possible without the generous financial support from the Australian Government and partnerships with the World Bank, Bappenas, and Ministry of Home Affairs, and all of these actors’ deep commitment to evidence-informed policymaking.