Measuring the impact of information on mayors' policy decisions: Evidence from Brazil

In recent decades, we’ve seen enormous growth in the number of program evaluations to measure the impacts of various government policies, NGO programs, and other implementers’ work.

Our own experience, along with the work of evidence-to-policy organizations like J-PAL, EPoD, and IPA, suggests that there is demand for evidence among policymakers (see J-PAL’s case studies for some examples). However, we have little empirical evidence on how much policymakers in general are interested in and able to act on research findings.

We leveraged a unique collaboration with Brazil’s National Confederation of Municipalities (Confederação Nacional de Municípios or CNM) to generate experimental evidence on these questions.

Over 80 percent of Brazil’s municipalities are members of CNM, which coordinates among and advocates for their interests at the state and federal level. Throughout the year, CNM organizes a variety of conferences and conventions that bring together thousands of municipal officials from across the country.

We attended 15 of CNM’s convenings over the course of 2016 to 2018 to conduct two related randomized evaluations: one to measure policymakers’ interest in research findings (our “demand/beliefs” experiment), and one to measure the impact of providing information on their policy decisions (our “policy adoption” experiment).

Measuring demand for information and changing beliefs

Our first step was to determine what information we would provide to policymakers in our study. Through surveys and focus group discussions with mayors, we learned that mayors were interested in learning from research on pre-school education, preventive healthcare, management practices, and budgetary issues. 

Working with CNM, we determined that Early Childhood Development (ECD) and taxpayer reminder letters were within the control, familiarity, and interest of the municipal officials in our study. Both interventions have been studied using multiple rigorous evaluations, including studies in Latin America.

We focused on ECD for our demand/beliefs experiment, which took place at 14 CNM meetings across Brazil in 2017 and 2018. We recruited over 900 randomly selected participants, including mayors, local legislators, and officials from 657 municipalities.

Participants used tablets to guide themselves through a 30-minute questionnaire designed to elicit three findings:

• Willingness to “pay” for the results of research on these programs, a proxy for how valuable they found the information (using lottery tickets as currency, a common research practice);
• Whether they updated their beliefs when provided with new information; and
• How such updating was mediated by the attributes of the research findings.

Measuring policy adoption

Our policy adoption experiment took place at CNM’s 2016 Novos Gestores conference, a large convention for mayors that covers a variety of public policy areas including education, tourism, and local taxes. We randomly selected about half of the mayors who had registered to participate in the conference, 881 mayors in total, to receive an invitation to attend a research session on the topic of increasing local tax revenues.

To accommodate all the invited mayors, we organized several identical sessions, each of which was attended by about 40 mayors. The sessions lasted 45 minutes and were led in Portuguese by an experienced instructor–in most cases a Policy Manager from J-PAL’s Latin America and the Caribbean office.

During a 30-minute lecture, the instructor described a specific intervention, taxpayer reminder letters, and the findings from a set of rigorous impact evaluations that have found letters to increase on-time tax payment. The last 15 minutes of each session were reserved for questions from the audience. At the end of the session, we provided mayors with a policy brief summarizing the presentation and listing the cited papers.

We then conducted in-depth phone surveys with session participants and municipal tax officials 15 to 24 months later to determine whether mayors had implemented similar taxpayer reminder letters in their municipalities.

Results 

Overall, we found that policymakers were interested in research findings and were willing to update their prior beliefs and act based on evidence.

In the demand/beliefs experiment, almost all participants (98 percent) were willing to “pay” at least one lottery ticket to learn the findings from a randomized evaluation of an ECD intervention. Policymakers seemed to find information from studies with larger samples more useful, but did not value studies conducted in low-income countries differently from studies conducted in higher-income countries.

The political climate in Brazil at the time of the experiment was very polarized along party lines. Yet, we found little evidence of confirmation bias or politically-motivated reasoning. ECD is not a highly-charged political topic in Brazil, so policymakers may respond differently when presented with evidence on a more controversial policy.

Turning to our policy adoption experiment, we found that informing mayors about the effectiveness of taxpayer reminder letters increased their adoption of the policy.

Participation in our research session was relatively high; 38 percent of mayors who were randomly invited chose to attend. We found that being invited to the research session increased the likelihood that municipalities were implementing reminder letters by 4 percentage points (a 13 percent increase relative to the comparison group).

For municipalities whose mayor actually attended the session, their attendance increased the likelihood of implementing reminder letters by 10 percentage points, a 33 percent increase.

What do these findings mean?

We carefully selected the interventions used in this study to ensure that the research shared was relevant to participants and relatively easy and low-cost to implement. It remains to be tested whether providing evidence on costlier or more complex interventions could have similar effects on policy decisions.

We also focus on how policymakers respond to a very small set of policy-research attributes, namely sample size, location, and political leaning. We look forward to seeing future research on how other attributes are relevant to their decision-making, and on how policymakers at different levels of government or policymaker type (e.g., bureaucrats vs. elected officials) might respond differently.

That being said, our results are encouraging on a number of fronts.

First, they suggest that research addresses a constraint facing policymakers: limited information about the effectiveness of tax reminder letters appears to be one reason that the mayors in our study had not adopted this relatively straightforward tool.

In addition, the mayors appeared to be fairly sophisticated consumers of accessible research, for example by paying more for studies—such as those with a large sample size—that subsequently affected their beliefs more.

This highlights the important role played by evidence-to-policy organizations in making research information directly and easily available to policymakers. As our results demonstrate, with appropriate research-to-policy translation, local government leaders in Brazil processed the research findings and had the political will to implement the policy.

For more details on this study, see the J-PAL evaluation summary.