Institutionalizing a culture of evidence-informed policymaking in Tamil Nadu
Governments worldwide need high-quality data and research to inform policy, though few have systematic processes for generating and incorporating such evidence into their decisions. Since 2014, the Government of Tamil Nadu has partnered with J-PAL South Asia to institutionalize the use of evidence in their decision making. This partnership is centered around four main activities: (i) engaging in policy research dialogues to apply insights from existing research to local challenges; (ii) designing and testing new innovations with the potential for scale; (iii) strengthening government capacity to generate and consume evidence; and (iv) leveraging administrative data for use in decision making. To date, this work has spanned nine government departments and involved more than twenty J-PAL affiliated professors and their collaborators.
The Problem
While there is demand among governments for high-quality research to inform individual policy decisions, governments worldwide rarely systematically formalize the generation and use of this evidence.
State governments in India have the responsibility not only to create and implement state-level policies, but also to implement all policies conceived and funded by the federal government. However, there rarely are institutional systems in place that create room for innovating on program design, piloting and testing new policy solutions, and incorporating this evidence back into decisions.
The systematic generation and use of rigorous impact evaluations can significantly improve state-level policy decisions, especially in the long term. Formal collaborations with evidence-to-policy catalysts like J-PAL can help develop systems and processes that capitalize on this opportunity.
In 2013, S. Krishnan, the former Planning Secretary of the Government of Tamil Nadu (hereafter “GoTN”) and Iqbal Dhaliwal, then Deputy Executive Director of J-PAL, began discussing ways to collaborate on evidence use in policy. The ultimate objective was to partner not just on one individual research project or policy scale up, but rather to change the culture of policy and decision making itself such that rigorous evidence is one of the key factors considered when choosing programs or policies.
The Innovation
J-PAL established an institutional partnership with the Government of Tamil Nadu to support data and evidence use in government policymaking.
To work toward the above goal, the partnership between GoTN and J-PAL began with a series of policy-research dialogues in 2014 between J-PAL affiliated researchers, staff, and senior officials of various state departments, including education and health.
These dialogues created space for J-PAL affiliated researchers and staff not only to better understand the government’s key priorities and implementation constraints, but also to map insights from existing evidence to challenges faced by their departments. Simultaneously, J-PAL led a field-based needs assessment to gather data on local implementation challenges across multiple districts in order to construct preliminary hypotheses for testing.
As these dialogues were taking place, GoTN and J-PAL formalized the partnership through a Memorandum of Understanding (MoU) signed in 2014 by the state’s former Chief Minister. This MoU leveraged an innovations budget that the government had recently established to encourage departments to pilot and rigorously test innovative programs, creating an opportunity to use evidence from impact evaluations to inform critical policy issues. The state innovation fund allows departments to submit proposals for innovative ideas to tackle persistent challenges, accompanied by a monitoring and evaluation plan to learn whether these ideas worked or not.
Importantly, the MoU helped institutionalize a framework for evidence use and partnership that would persist beyond political and bureaucratic changes. Today, the partnership is anchored in the state’s Planning and Development Department and overseen by a Steering Committee and an Advisory Committee, with active engagement by senior government and J-PAL representatives.1
In practice, J-PAL affiliated researchers and staff work with officials across departments to explore problems and co-create innovative interventions for evaluation. When a research proposal is agreed upon, J-PAL and the partnering department present it to the Steering Committee. The Committee assesses the proposal’s feasibility, scope, and scalability.
Following their approval, the Planning Department and the relevant sectoral department, in partnership with J-PAL, work to secure funding for the underlying intervention and for the evaluation, through the state innovation fund if applicable, or the department’s own funding if available. The evaluation and funding mechanism encouraged many departments to participate. Research and policy progress is then overseen by the Advisory Committee, which meets roughly three times per year to review ongoing projects, discuss interim and final research results, and guide the adoption of insights from these research collaborations.
To date, five impact evaluations have been commissioned (of which three have been completed) that address a range of topics, from reducing the burden of non-communicable diseases to understanding the economic effects of sleep among the urban poor.2 In addition, ten formative and pilot studies have been completed, which have helped identify questions for further evaluation and provided quick insights on key challenges facing the government. This work has spanned nine government departments and involved more than thirty researchers, including more than twenty from J-PAL’s network.
From Research to Action
The Government of Tamil Nadu and J-PAL have built a culture of data and evidence use to improve decision-making.
The partnership between GoTN and J-PAL has helped the government use rigorous evidence in policy making through four main channels:
- Engaging in policy research dialogues to apply insights from existing global and local evidence to address local challenges;
- Designing, piloting, testing, and scaling up new innovations;
- Strengthening government capacity to generate and consume evidence; and
- Leveraging administrative data for use in decision making.
Combined, these activities are helping institutionalize a strong culture of evidence use in decision making within the Tamil Nadu Government, leading to more effective policies and programs that support people experiencing poverty.
Applying research insights
Policy research dialogues have generated sustained demand for evidence among government champions and created a formalized channel for researchers and decision makers to engage on ideas for effective programs and policies, drawing from the global evidence base. This has enabled the government to draw insights from programs and policies evaluated in other contexts to inform interventions to address similar challenges in Tamil Nadu.
In many instances, sharing evidence on a particular topic has helped inform the government’s decision-making process for whether to initiate a new program or modify an existing program. J-PAL affiliates and staff have engaged in numerous consultations on the design of new policies, such as the Housing and Habitat Policy with the Housing and Development Department and the Safe and Ethical Artificial Intelligence Policy with the e-Governance Agency in 2019.
Affiliates have also advised on improvements to existing interventions with the School Education Department, reviewing pedagogy strategies to improve learning outcomes; and with the Micro, Small and Medium Enterprises Department, reviewing targeting strategies to better identify high performing entrepreneurs and small firms from 2017–18.
In parallel, insights from scoping studies, process monitoring, and interim observations from ongoing evaluations are routinely shared back with government officials by researchers and project teams, creating feedback loops that allow for rapid adjustment and innovation. For example:
- A key finding from the first wave of a long-term elderly panel survey concluded in 2019 highlighted a growing proportion of elderly-only households, suggesting the need for policy interventions to address emerging challenges of access to geriatric health services, financial security, and mental health support.3 Based on this data, the government announced a plan in 2019 for a new intervention that will pilot elderly resource centers.
- Similarly, a 2015-16 scoping study of health data infrastructure related to maternal and child health outcomes found that large numbers of pregnant women were registered into the system too late into the pregnancy term, which made it difficult for the department to ensure they were receiving timely ante-natal checks and identify high risk pregnancies early on.4 Based on this feedback and consultations, the Health Department decided to strengthen their Health Management Information System (MIS) and update guidelines to an existing conditional cash transfer scheme by changing registration timings to encourage earlier registration.
Designing, piloting, testing, and scaling new innovations
In addition to raising awareness of the global evidence base, policy research dialogues have provided a forum for the government and J-PAL to co-design research projects that test the effectiveness of key policy innovations that have the potential for scale. These research findings feed into policy decisions and inform future research and data initiatives. For example:
- Following a policy dialogue on evidence related to early childhood education and nutrition, the Social Welfare Department identified a need to strengthen a program delivered through the Integrated Child Development Services (ICDS) scheme.5 Acknowledging the scheme’s resource constraints, researchers launched a study in 2016 to evaluate the impact of adding a locally hired, low-cost early childhood care and education (ECCE) facilitator.6 They found the intervention to be highly cost-effective in improving both learning and nutritional outcomes, including reducing the rate of severe malnutrition. Findings from this study prompted conversations between the Social Welfare Department and Education Department to develop a coherent strategy for strengthening early childhood education, resulting in a follow-on study and pilot in 2019 to integrate kindergarten classes within government schools that were co-located with day care (anganwadi) centers. Prior to school closures triggered by Covid-19 lockdowns, J-PAL affiliated researchers were tracking the implementation of the new program and working closely with the government to study the impact of the new program.7 Researchers are now repurposing the baseline data to understand the (unequal) effects of the pandemic on children in Tamil Nadu, as well as to test remedial measures to mitigate pandemic-induced learning losses and improve existing programs.8
- As early as 2014, combatting and managing non-communicable diseases has been a policy priority of the Government, especially given the state’s highly urbanized and aging population. To encourage the adoption of preventative behaviors, J-PAL affiliated researchers designed and tested a new intervention in which individuals with diabetes or hypertension, or who were at risk of these conditions, were identified through mass screening camps before being enrolled in a three-month preventative health program.9 Participants received a pedometer combined with either a messaging intervention or an incentive plan which rewarded them with a small recharge to their mobile phone talk time for meeting their step (walking) targets. The study found that a combination of monitoring via pedometers and incentives had the greatest impact on increasing the percentage of days that participants met their step targets, and that the effects persisted even beyond the intervention period. This led to further research on different incentive strategies and prompted the government to explore the potential of delivering such a model through the primary health care system. Insights from the study encouraged discussions around NCDs and the idea of performance-based incentives, which were included as part of a plan for a large World Bank program to strengthen health systems reform.
Strengthening capacity within government for evidence generation and use
J-PAL has continuously engaged with various government departments to help strengthen public officials’ capacity to generate and consume data and evidence. Between 2014 and 2020, J-PAL conducted fifteen customized workshops and training courses for staff, including personnel in the Departments of Evaluation and Applied Research (DEAR), Economics and Statistics (DES), and Social Welfare, as well as for trainees of the Indian Administrative Service.10
These capacity building efforts led DES and DEAR to adopt digital data collection platforms and integrate new quality assurance mechanisms to improve data usability in their work. As a result, the government is undertaking longitudinal surveys to provide information on the change in status and emerging issues among the population. They were also able to use digital data collection methods for the first time, integrating a system of independent validations to improve data reliability.
Recognizing the growing need for evidence in decision-making, the government formalized an engagement with a list of pre-approved research and policy organizations in 2017 that it can engage with to commission high-quality process and program evaluations. Also in 2017, it issued state-wide guidelines on program evaluation requiring departments to undertake rigorous impact evaluations of all large-scale programs exceeding US$20 million in implementation costs.
The government has also allocated a dedicated budget for evaluations and associated institutional structures, including the formation of an advisory board in 2015 that consists of external academic researchers who provide advisory inputs to DEAR on the methodology and interpretation of results from studies taken up or commissioned by them.
Leveraging administrative data for effective decision-making
Finally, the partnership has set up systems to streamline data generation and supported improvements in data analytics. Notably, the government established a Data Analytics Unit in 2016 to support the analysis and use of administrative data for effective decision-making. J-PAL launched an Innovations in Data and Experiments for Action (IDEA) initiative and lab in 2019 to further leverage administrative data for use in research and generate quick insights to inform policy making. For example:
- Analysis of large volumes of crime data have enabled creation of predictive tools and data visualizations that can be leveraged to improve policing. For instance, improved data visualization on reported crime (such as timing of road accidents and burglaries) helped police identify the best options for patrolling. Other predictive tools have been developed to automate and improve the matching of datasets, such as linking datasets of missing persons with datasets of the unidentified deceased. Ultimately, a predictive tool and dashboards were integrated into a centralized management information system maintained by the crime records bureau.
- Analysis of demand and usage of medical equipment in health insurance claims data improved identification of gaps and patterns and informed public health expenditures on necessary medical equipment.
Together, the combined efforts of drawing upon global knowledge to address local challenges, innovating and testing new solutions, strengthening government capacity to generate and use evidence, and improving data systems have contributed to a strong culture of evidence use within policymaking in Tamil Nadu.
This long journey has not been without its challenges. Research lifecycles and innovations take time, often years, and patience. In a context where political priorities may shift with election cycles, and transfers of officials are routine within a year or two, change is constant. With every change, the need for innovation and evaluation as well as trust in the working partnership has to be reestablished.
For a project to be successful, it has to have complete ownership by the researcher, as well as the policymaker and their concerned department. From the researcher perspective, this involves significant upfront investment to collaboratively work with the government partner to both identify the problem(s) to be addressed and co-design the potential interventions for testing. Researchers also have to be flexible to refine, adapt, and adjust the study design based on practical implementation constraints and policy timelines, which often require immediate and quick responses. To address these policy needs, researchers should engage with government counterparts through every step of the study by sharing preliminary and interim observations, which serve as useful and independent feedback to the department, even as the final impact evaluation results may be forthcoming.
On the policymaker side, individual evidence champions are critical to a project’s success, as is the broader support and buy-in from others within the department. The role of individual champions within partner departments is crucial not just in early stages of ideation and proposal development, but also throughout the course of the evaluation, in managing field constraints, protecting the randomization design, and securing cooperation of field-level officials who may perceive the evaluation as an audit of their performance. Beyond the individual champion, it is essential to also involve other key stakeholders from the department throughout project ideation and planning. Importantly, if the department is able to articulate the need for an evaluation and dedicate funding to see it through (for instance, by leveraging department funds for monitoring and evaluation), this helps create strong buy-in for ensuring projects continue and plans do not need to be revisited, even if the original champion leaves.
Moreover, not all studies go according to plan. A few evaluations initiated through this partnership could not be completed as planned due to operational constraints in program implementation, poor take up by intended recipients of the study, or concerns around scalability of the intervention. In each instance, the challenges were documented in detail and where possible, new areas of study were identified. These studies equally contributed to our collective understanding of the realities associated with translating innovations on the ground and the associated challenges that one needs to be mindful of.
Despite these challenges, this long-standing partnership provides a formal structure for departments to engage with J-PAL on an ongoing and long-term basis to integrate the use of randomized evaluations to address key policy questions. In its 2020-21 budget speech, the government specifically referenced its partnership with J-PAL, noting that through this collaboration, “more studies [will be] taken up to enhance the quality of policy making.”
Ultimately, the people involved in this partnership, and their enthusiastic engagement with one another has been critical to its success. In addition to dozens of government officials at all levels across departments and the J-PAL affiliated researchers and their collaborators in India and internationally, J-PAL South Asia has also set up a dedicated partnership team in Tamil Nadu to provide long-term, continuous institutional support for individual research projects as well as the overall partnership. This has been critical given the large number of stakeholders involved and the routine transfer of key government officials. Finally, donor core funding for J-PAL South Asia’s policy work allowed for rapid hiring of staff, necessary travel, policy research dialogues, scoping studies, and evidence to policy support.
In June 2021, J-PAL Director Esther Duflo (MIT) was appointed to the GoTN’s newly established Economic Advisory Council. She joins council members Raghuram Rajan (former central bank governor), Arvind Subramanian (former chief economic advisor to the federal government), Jean Dreze (development economist), and S. Nararyan (former Union Finance Secretary). These appointments represent a commitment by the government to tackle a number of pressing economic concerns—from reducing the State’s overall debt burden to promoting the economic health of all citizens—and signal a continued focus on evidence use in policy decisions.
Abdul Latif Jameel Poverty Action Lab (J-PAL). 2022. "Institutionalizing a culture of evidence-informed policymaking in Tamil Nadu." J-PAL Evidence to Policy Case Study. Last modified January 2022.