We need evidence-based public policies in Brazil
This post was first published as an op-ed in Nexo Jornal and has been translated from the original Portuguese.
The main challenge when it comes to public policy is knowing what works and why. Brazilian policy debate too often is built on belief-based arguments and ideology and relies little on data or rigorous evidence.
The adoption of policies based on empirical evidence is growing worldwide. Unfortunately, Brazil still lies very far behind when it comes to evaluating its public policies and using evidence to support them. Several recent policy discussions on reducing the age of criminal responsibility, labor reform, subsidies to automobile manufacturers, and political campaign financing reform, have been carried out with limited attention to evidence and evaluation.
In the column I am heading for Nexo, I will discuss current issues related to public policies, focusing on economic aspects and empirical evidence from Brazil and the world. The column will tackle a variety of topics, such as poverty reduction, education quality, corruption control, business productivity, health care quality, and crime and violence reduction.
The main challenge when it comes to public policy is knowing what works and why. Brazilian debate too often is built on belief-based arguments and ideology and relies little on data or rigorous evidence.
What do I mean by rigorous evidence? When talking about public policy impacts, we need to talk about cause and effect. That is, we need to attribute some action to a policy that causes people to change behavior after an intervention. The main challenge of empirical work is to isolate this relationship between cause and effect. In controlled laboratory studies, this analysis is done using experiments. In the social sciences, on the other hand, isolating this cause-effect relationship is more difficult. How can we do that?
First, we need to bring the lab into the real world. A growing number of studies use randomized evaluations to identify the effects of public policies when some individuals (or units, such as schools or towns) are given access to a specific policy and others are not. The decision of who gets what is done randomly—i.e. similar to a lottery or draw. When we choose the beneficiaries through a lottery, we make sure that the people who are given the policy are similar—except in the policy they have received—to those who have not received it. Thus, we can assign any measured program effects to the policy in question.
The impact evaluation of Progresa (Education, Health, and Food Program) in Mexico provides an example of this methodology. Progresa was a program similar to the Brazilian social welfare program, Bolsa Família. It consisted of an income transfer to low-income families, conditional on their compliance with education and health requirements. However, unlike the Brazilian program, beginning with its implementation in 1997, it had an evaluation built in.
In 1998, 506 towns in seven states were selected through a lottery for the evaluation. Of these, 320 towns were selected to receive the program, while 186 towns served as the comparison group. Low-income households in the towns chosen to start the program received a monthly transfer of 90 Mexican Pesos (approximately US$4.50 or R$7). Researchers conducted a survey in 14,500 households with approximately 89,000 individuals to analyze the baseline conditions for these families. After that, the same information from the initial survey was collected every six months for two years. The surveys measured income, consumption, labor, and indicators of education, fertility, and health.
A few years later, data showed that the program was highly effective in reducing school dropout rates (especially among young people) and improving health indicators. Conditional income transfer programs are now present in most Latin American countries, in part, thanks to the Progresa impact evaluation. In addition, the evaluation served to persuade subsequent governments to continue the program, even when new parties came to power.
In other situations, the government itself uses lotteries to implement programs when there is high demand for a service, but the resources or vacancies of the program are limited.
In Brazil, for example, the Controladoria Geral da União (CGU)—a federal agency tasked with preventing and investigating the misuse of public funds—has been auditing city governments since 2003 and selects, through a lottery each year, 60 cities to be audited. This lottery creates a natural experiment that allows us to compare audited city governments with those not audited in order to evaluate if the audits are effective in reducing corruption in municipal governments.
The CGU case offers an example of what can be done when, despite its advantages, it is impossible to bring the laboratory to real life. Often, policies are implemented based on other motivations and randomly selecting beneficiaries is not feasible, either for ethical or political reasons. How then can we assess these policies?
Several real-world situations resemble laboratory experiments, where we have people or places that receive a policy while others do not. Economists call such situations natural experiments. In these cases, we can often use the units that did not adopt the policy as a comparison group. What is important is that there is a comparison group that allows us to measure the effects of the desired policy—rather than using before and after comparisons, which are routinely made by politicians or the media.
Another example of these ideal natural experiments occurred in Miami in 1980. Within a few months’ time, approximately 125,000 immigrants arrived in the city by boat, fleeing Cuba. Economist David Card, a professor at the University of California, Berkeley, used neighboring towns as a comparison group to account for what happened to the wages of local workers when there was a large influx of low-skilled workers from outside. To the surprise of many economists, and a controversy that exists to this day, he found no effect on local wages as a result of this large migratory movement.
Brazilian researchers are largely aware of the numerous methodologies at their disposal to evaluate public policies, and there is sufficient data available in Brazil to conduct high-quality studies.
However, most public policies implemented in the country are not evaluated, and, when they are, the results are not used. For example, a recent study in Rio de Janeiro found the Renda Melhor Jovem program demonstrated positive results for reducing school drop-outs. Regardless, the program was discontinued by the state government and the evaluation was never discussed.
It is time for Brazil to bring empirical evidence to the center of its political debates and for public policies not to be implemented or discontinued without society knowing their real impacts. I hope this column contributes to that discussion.
