Informing Policy with Research in Brazil
- Politicians and electoral candidates
- Elected Official Performance
- Information
Despite the widespread growth in the number of randomized evaluations conducted, there has been little rigorous research on their potential to influence actual policy decisions. Researchers partnered with the National Confederation of Municipalities (CNM) in Brazil to evaluate the impact of access to research results on policymakers’ demand for research findings and interest in adopting evidence-informed policies. Policymakers were willing to pay for research findings and subsequently updated their beliefs. Those who were invited to an information session on a simple, evidence-informed policy to increase tax compliance were more likely to implement that policy within the next two years.
Policy issue
Recent decades have seen an explosion of program evaluation research.1 More than 7,500 studies have been registered on the American Economic Association’s registry for Randomized Control Trials (RCT) since its launch in May 2013. Such research has generated wide-ranging knowledge on what types of policies work–and do not work–for achieving outcomes including education, health, and income. Yet, little is known about whether policymakers value findings from program evaluations and whether those findings can influence their beliefs and, ultimately, the policies they implement. In short, can information from research influence policy choice?
Context of the evaluation
In Brazil, municipal governments are headed by directly-elected mayors who have substantial discretion over policy decisions in their municipalities. They are responsible for delivering key public services (including education, health, and sanitation) that are mostly funded by transfers from the Brazilian government. At the same time, about 15 percent of municipal government revenues are generated by local tax collection. Among the mayors included in this evaluation, almost 90 percent were men and about 60 percent had at least a bachelor’s degree.
The Brazilian National Confederation of Municipalities (CNM) coordinates among and advocates for Brazil’s municipalities; over 80 percent of Brazilian municipalities are members of CNM. CNM organizes a variety of meetings every year that are attended by thousands of municipal leaders (including mayors, vice-mayors, local legislators, and municipal secretaries). Researchers leveraged major CNM meetings in 2016 through 2018 to conduct two experiments to understand the effect of research on policy-making.
Details of the intervention
Researchers partnered with CNM to conduct two randomized evaluations to understand policymakers’ interest in research findings, as well as the impact of providing information about the effectiveness of a certain policy on mayors’ policy adoption decisions.
Demand and beliefs experiment: Researchers conducted a two-round evaluation to measure policymakers’ willingness to pay for study findings, as well as the impact of those findings on their beliefs. They shared results from randomized evaluations of selected Early Childhood Development (ECD) interventions, which varied in both sample size and location (Jamaica, the United States, Michigan, and Colombia). The policymakers’ initial beliefs about the impact of ECD policy interventions varied widely. The research team attended 14 CNM meetings in 2017 and 2018, where a total of 764 municipal leaders from 579 municipalities participated in the evaluation. Participants were each given 100 tickets to enter a lottery for a free trip to the United States, which could be exchanged to ‘purchase’ research findings. At the start of the experiment, participants shared their beliefs of the impact of an ECD intervention in their municipalities and in two of the randomly-selected study locations.
- Round 1: Researchers offered each participant the option to bid some of their lottery tickets to purchase the findings from one randomly-selected study. The results were revealed to participants whose bids were above a price that was randomly determined for each participant.
- Round 2: Researchers gave each participant a fresh supply of 100 lottery tickets and allowed them to bid for the results from each of the three remaining studies. Researchers then randomly selected one study and price, and revealed the results to participants whose bids were above that price.
After each round, participants who purchased study findings were again asked for their beliefs on the impact of the ECD program. Altogether, this experiment was designed to shed light on the policymakers’ interest in learning about research, the types of research they value most, and their willingness to change their prior beliefs after being exposed to new evidence.
Policy-adoption experiment. To test whether sharing research findings can influence policy implementation, researchers invited a randomly-selected 881 of the 1,818 mayors who attended CNM’s largest national conference in 2016 to a research information session. The session described the impacts of sending reminder letters to taxpayers; this is a low-cost and easy-to-implement policy that increased tax compliance, for example, in Peru by 10-20 percent.2 Researchers contacted all mayors, as well as bureaucrats in charge of tax policy within each of the municipalities, 15 to 24 months after the conference to identify the extent to which the policy recommendations were adopted.
Results and policy lessons
Overall, results suggest that policymakers are interested in research findings and are willing to review their prior beliefs and act based on evidence.
Demand for research findings: Policymakers were willing to pay an average of 45 lottery tickets (equivalent to about US$36) to learn the findings of a study. Almost all participants (98 percent) were willing to bid at least one lottery ticket. Policymakers were willing to pay more for studies with large sample sizes and when they were involved in implementing a similar policy in their municipalities. However, participants did not value studies conducted in low- and middle-income countries more than studies in the U.S. Participants were consistent in that the more they were willing to pay for the findings, the more likely were they to change their prior beliefs. Furthermore, if participants already held positive beliefs toward ECD programs, their willingness to pay for additional information on implementing ECD programs increased. College-educated and politically left-leaning mayors placed more weight on study results, and older mayors placed less weight on them.
While the results from the demand and beliefs intervention suggest an overall interest in research findings among the sample of Brazilian mayors, it is important to note that the studies shown to the mayors only came from three countries, and the effects on the mayors’ beliefs were recorded over a short period of time. It remains uncertain whether the effects of this intervention persisted over a longer timeframe.
Policy-adoption experiment. Over one-third (38 percent) of the mayors invited to the research information session chose to attend the session. The younger and college educated mayors were more likely to attend the sessions. Participation in the research information session increased the probability that municipalities were implementing taxpayer reminders 15-24 months later by 10 percentage points, a 33 percent increase relative to the comparison group.
In sum, this study shows how access to research findings may be a limiting factor in adopting policies that are shown to be effective.
This paper was cited in a Center for Global Development paper on how to spend the billions of pounds the U.K. allocates to “economic development R&D” every year and in a “Science Meets Parliament” report provided to Canada’s parliament by prominent researchers.