Leonardo Shibata, J-PAL LAC '16, on the evolution of impact evaluation in Brazil

In our Alumni Spotlight series, we explore the journeys of former J-PAL members who have continued to contribute to policy and research in various fields. Leonardo Shibata, a former J-PAL Research and Training Manager in Brazil, now works at the Inter-American Development Bank (IDB). He reflects on his experiences at J-PAL and how they shaped his current work in this edition of the series.

When was the first time you heard about J-PAL? Why did you decide to work here?

I heard about J-PAL through a job posting. I had just returned from completing my master's in Public Health at Johns Hopkins University and was trying to re-enter the job market when I was put in contact with a specialist from the IDB (Inter-American Development Bank) through a professor. At the time, IDB had no vacancies, but I was informed that a partner organization did—J-PAL, for a research and training manager position. There was already a policy manager, so I was to become the second person to work for J-PAL in Brazil. I did some research to understand what J-PAL was, to see if I would be interested or not, and if J-PAL would be interested in me. I learned that it was a research center founded by economists and was initially apprehensive, but at that time, there were not many people in Brazil were familiar with impact evaluations of public policy, and the movement was just beginning. My profile made more sense in this context: although I was not an economist, my background in public health allowed me to understand the methodology of a randomized evaluation. I had also worked in the Brazilian federal government and the position was embedded in the government.

What did you do when you worked at J-PAL? What challenges did you face?

As a research and training manager, I had two main functions. One was to prospect impact evaluations in the federal government by contacting managers, looking at government policies, and making a preliminary assessment of what could be evaluated using randomized evaluations, and the other was to train federal government employees so they could understand what an impact evaluation is and how to apply it to public policy. In Brazil, we had little knowledge about the subject and I even witnessed resistance — some didn't want an outsider checking if their policy was working or not, fearing they would lose control of the narrative. Another challenge came from explaining randomization and what it meant, as most government officials objected to it, wanting to offer a program to the entire population or set up clear criteria for offering to specific groups. A lot of work had to be done to explain how randomization could be done without negative effects.

I also encountered many obstacles due to the political dynamics in Brazil at the time. I started at J-PAL at the end of 2015/beginning of 2016, during President Dilma Roussef’s impeachment. J-PAL's arrival in Brazil was at a difficult time, especially given that the objective was to work with the federal government, which was undergoing constant  changes with new ministers and entire new teams. Many times, we started conversations that had to be repeated in a few months due to personnel changes. To surpass this, we came up with different strategies. One of them was working with career civil servants, people who would still be there regardless of political changes. This worked out well: even years later, some of these civil servants still remember J-PAL when I meet them. Finding government departments that are less visible and not usually targeted, that in general can be considered more protected, is also a good strategy. This happened with J-PAL's partnership with the Brazilian National Public Administration School (ENAP) and the establishment of a course in impact evaluations for public policies. By the time it was formalized I had already left J-PAL, but I helped structure the course along with Carolina Araujo (the policy manager at the time) and Anna Mortara (who replaced me). For J-PAL colleagues that are going through turbulent situations, my advice is to focus on career civil servants and parts of the government that should not be too affected by changes.

How do you see the evolution of J-PAL's role in Brazil and worldwide since you left the organization?

From my perspective, the first thing I have noticed is that the course done in partnership with ENAP can be considered very relevant: it means that J-PAL is managing to train generations of public servants in Brazil. So far, we have had more than eight years of government employees who have at least heard of impact evaluations, not to mention the ones who have chosen to dive in, specializing and taking it to their areas of expertise. The other thing I see is that talking about impact evaluation in the federal government is much easier now, there is a lot less friction and resistance. For instance, in October 2023 the Brazilian Health Ministry published a book on impact evaluation, focused on health policies in the Brazilian Unified Health Service. I perceive this as a partial result of J-PAL's work, along with other organizations, of course, but compared to where J-PAL was in Brazil when I first started there, people know about the organization much more often and are also more frequently aware of what an impact evaluation is.

How do you think your time at J-PAL contributed to your current position at the IDB? What skills did you learn at the organization that help you to this day?

The vision of impact evaluation. Thinking about policies from start to finish, from planning to evaluation. Normally, people forget to evaluate, focusing on the present, and only think about evaluating later, when there are no longer conditions to do so because data was not collected or the required articulation between the government departments and researchers was not made—or any other number of reasons. My time at J-PAL gave me the vision that evaluation must be thought of from the very start. It is not something extra, but something that will tell us whether there is an impact or not. 

What advice would you give to someone interested in working at J-PAL or another organization focused on research and public policy?

My main advice, assuming the person has the technical knowledge (the very first step would be to study!) is that any given institution is made up of people. It is very important to build personal relationships. J-PAL's work of having people embedded in the government makes a big difference: being there to have a coffee or grabbing lunch is very important. The J-PAL professional, when in a meeting or interacting with the government, is representing the research center and if they cannot build relationships, they are harming the center's entire work. Therefore, in addition to having a solid base of technical/theoretical knowledge, it is important to build good relationships with the organizations you are interacting with. 

Diversity is also very important and something that must be taken beyond talk. At many organizations, we can see they are making advances with the hiring of minorities, but there is still a lack of Black and Indigenous hires. Sexual orientation seems to be handled relatively well, but trans people still suffer—it can be much more difficult for them. Having a diverse group of employees will allow us to make connections across the board and can help us better reach our goals.