The role of administrative data in improving social protection programs: A success story from Indonesia

Data collection is one of the most costly and time-consuming aspects of conducting impact evaluations. However, past research by J-PAL affiliated researchers has demonstrated that conducting a rigorous evaluation using preexisting administrative data is not only possible, but can potentially expedite and reduce the cost of the data collection process.

Governments collect rich data on program beneficiaries in the process of running their social assistance programs. When this administrative data is collected rigorously and made accessible, policymakers and researchers have the opportunity to evaluate social programs more effectively and strive for continuous improvement in their design and delivery.

To promote and guide the use of administrative data in impact evaluations, J-PAL’s Innovations in Data and Experiments for Action Initiative (IDEA) published a free, online handbook on Using Administrative Data for Research and Evidence-based Policy. The chapter on “Using Admin Data to Improve Social Protection in Indonesia” explores the collaboration between the Government of Indonesia and researchers to use government administrative data to evaluate some of the largest social programs in the country.

J-PAL SEA hosted a talk as part of the IDEA Handbook webinar series to dig deeper into this collaboration story. The event was moderated by Indonesian economic journalist Ade Safrina Nasution and featured chapter authors Vivi Alatas (CEO, Asakreativita), Sudarno Sumarto (SMERU Research Institute; Policy Advisor, TNP2K), Rema Hanna (Harvard; Co-Scientific Director, J-PAL SEA), and Ben Olken (MIT, Co-Director, J-PAL; Co-Scientific Director, J-PAL SEA).

The essential “dance” to improve access and availability of administrative data in Indonesia

As a prominent economic researcher, Vivi Alatas has extensive experience in evaluating social programs with the Government of Indonesia. The government has always been interested in understanding a program’s effectiveness in eradicating poverty, but leading up to this big question also lies the more preliminary program design concerns such as, “How do we define ‘poverty’? And how do we correctly identify beneficiaries who are eligible to receive social assistance?” Here, administrative data act as a powerful tool to address the question of who should receive a program.

Administrative data helps researchers and policymakers to design an impact evaluation. The essential component of impact evaluation is estimating what would have happened to beneficiaries had a program not been implemented, also called the counterfactual. Deciding how to replicate this counterfactual is a crucial step that affects the rigour of evaluation results.

This is where administrative data plays a part, as it allows researchers to better replicate the counterfactual by comparing characteristics of beneficiaries and program implementation across different areas without additional data collection effort. For example, when designing an evaluation for a nationwide policy, administrative data collected on a national scale helps shed light on whether beneficiaries are selected in a similar manner across provinces and program components are rolled out uniformly. With regularly updated high-quality administrative data, we can understand not only whether or not a program had an impact, but also whether there are remaining implementation challenges to address at a cheaper cost compared to conducting a nationwide survey.

Despite the important role administrative data can play, its use for program evaluation in Indonesia is still limited in part due to the mismatch of research interests and desired result timelines between researchers and policymakers. Closer collaboration between researchers and practitioners in program monitoring and evaluation planning is therefore instrumental in producing more usable evidence. This synchronization of steps between the two parties is the essential “dance” that can promote evidence-informed decision-making through improved administrative data collection.

A successful collaboration story: Long-term partnership to evaluate Indonesia’s social protection programs

In 2012, researchers worked with the Government of Indonesia to utilize administrative data in implementing the Raskin ID card program—a card that would state the eligibility of a beneficiary, and hence increase the transparency of the program. The card was found to reduce the program leakage by 33-58 percent. 

When the Raskin program was reformed in 2017, it was Indonesia’s largest social assistance reform in twenty years. Researchers again worked together with the government to evaluate the impact of this reform by utilizing a set of administrative data called SUSENAS (the national socio-economic survey), a biannual national sample survey administered to over 250,000 households.

The collaboration to evaluate Raskin ID cards led to a long-term administrative data use partnership between researchers and the Government of Indonesia, including the updating of UDB (Unified Database), one of the largest social assistance databases in the world. Sudarno Sumarto, senior research fellow at SMERU and policy advisor at TNP2K, shared this success story.

The Raskin collaboration allowed researchers and the government to work together to explore other questions related to optimizing social assistance programs. For example, they wanted to understand the best way to update a program beneficiary list by looking into whether eligibility surveys would alter people’s behavior.

By randomizing additional questions in the targeting survey on asset ownership (e.g. of TVs, cell phones, motorcycles, and goose-neck toilets), researchers were able to identify whether or not respondents were more likely to alter their answer in order to be eligible for the program, such by not disclosing some of their assets, or even changing their purchasing decision entirely. The latter may have important welfare consequences if people choose a lower quality product or do not buy essentials. Results from the study suggest that households may have responded by hiding assets but not actually changing their consumption.

At the moment, the challenge is to ensure that the administrative data is used not only by the national government, but also by local governments. Some local governments worry that the national administrative data may not reflect the actual conditions in their regions. According to Sudarno, one step to solve this issue is to conduct a regular discussion between the national government and the relevant stakeholders. Furthermore, empowering the Regional Coordination Team for Poverty Reduction (TKPK) to work with their local government to update the social assistance database is also essential in promoting the use of administrative data in local government decision-making.

Significant improvement, but still a long way to go

The Government of Indonesia is a world leader in incorporating evidence into policymaking. The Covid-19 pandemic has shown how important social protection programs are to protect vulnerable groups from economic shocks. When the pandemic required the government to move quickly to alter available programs to new needs, they were able to rely on a past evaluation to inform this decision. Continuous investment in research will be helpful for future shocks as well.

Encouraging the use of administrative data in more research may be the key to unearthing more innovative solutions. One way to do this is by creating a forum where researchers and policymakers can meet to identify the important policy questions together, determine the data required to answer those questions, and set standards and criteria for how data is used and collected to provide protection and anonymity to each subject.

This article is based on the IDEA Handbook Webinar, “The use of admin data to improve social protection in Indonesia,” on January 26, 2021.