Laura Costica, J-PAL '14, on applying impact evaluations in humanitarian settings

The Alumni Spotlight series highlights J-PAL alumni who are making an impact across industries and around the world. Formerly a research manager at J-PAL Africa, Laura Costica ‘14 now manages monitoring, learning, and evaluation at the IKEA Foundation. She describes her journey from J-PAL to the IKEA Foundation in this post in the Alumni Spotlight series.

What first inspired you to build a career in the development and impact evaluation space?

I was inspired by research being done by J-PAL affiliates even before J-PAL was created. During my masters, my development econ professor would present the latest research papers at the time. There were some, like Esther Duflo’s paper about reserving seats for women on village councils in India, that showed me that there are ways to look at real-world problems and try to figure out how to solve them using the tools of economics. That's when I decided that this is what I wanted to do. I always had J-PAL in the back of my mind as the place where I wanted to work, but I didn't have any work experience when I finished my masters, so I started by going to India to do an internship. 

That was an eye-opening experience. It's very different when you learn from the books about how development works and when you see it in practice. After about two years of working with local organizations in India I felt ready to apply to J-PAL/IPA—I ended up in a role in Nigeria, managing an evaluation of a project working to prevent maternal and under-five mortality and morbidity. The learning curve was steep because there was no prior presence in Nigeria for either J-PAL or IPA. We had to uncover a lot of things and even deal with the sudden insurgence by a terrorist group in one of the two project states. Having to navigate implementing the evaluation in these challenging circumstances, especially being the only staff based in Nigeria, was foundational for me. When my assignment to coordinate the baseline survey in Nigeria came to an end, I moved to J-PAL Africa, where I supported impact evaluations in different African countries.

Could you describe your day-to-day work at J-PAL as a Research Manager for J-PAL Africa?

I joined to provide research management support to the Urban Services Initiative. USI was funded by the Gates Foundation to identify and evaluate solutions to improve access to services in African cities, for example water, sanitation, transportation, housing. You ask about the day to day work and what I can say is that no day was similar to the next. I find the initial stages of any project really fascinating—starting a new evaluation is like a puzzle waiting to be solved! I really enjoy diving into the details of each intervention, especially exploring with partners how they think activities will lead to the desired impact, what variations they would like to test and how an impact evaluation can inform their future work. For a few projects, I also planned intervention pilots and developed implementation manuals to standardize the client's experience. I also worked with researchers on designing the evaluation, identified which outcomes to measure, drafted survey instruments, trained enumerators, and observed interviews to find out whether people understood the questions and were able to respond. Throughout my time at J-PAL, I learned that what I really like is getting a project started. There are many unknowns at the beginning, and I like the challenge of figuring out how to make things work.

You left J-PAL to pursue impact evaluation coordination at the World Bank. What motivated you to pursue this specific path? How did your time at J-PAL inform your time there? 

While I was still at J-PAL Africa, I had started supporting a World Bank evaluation in Nigeria led by J-PAL affiliated professors Abhijit Banerjee (MIT) and Eliana La Ferrara (Harvard). When my work visa for South Africa was not renewed due to a change in immigration policy, I left J-PAL Africa and transitioned to consulting. That first project with the World Bank was an exciting evaluation of the educational entertainment TV series MTV Shuga, which aimed to prevent the spread of HIV among young people. My portfolio grew over time and I worked on evaluations in Mexico, Brazil, Honduras, and another one in Nigeria. I was based in Mexico City for over five years and I used everything I learned while working at J-PAL on a daily basis. The ins and outs of designing, setting up, and implementing impact evaluations are very similar between the different organizations working in this field—but I think J-PAL has had a big role to play in setting the standard.

Many of the impact evaluations you oversaw at the World Bank seem to be focused on improving outcomes for youth around the world. Are there any key learnings you took away from that scope of work?

Understanding how people make decisions and how they can be nudged into making better decisions is a personal interest of mine. Most of the projects I worked on were with youth, and mainly focused on behavior change, like the MTV Shuga project. One key learning is that you have to understand the barriers to behavior change and how adopting a new behavior might be made easy. For example, one project tried to get people living in slums in Kenya to use toilets rather than defecating out in the open. There were already fee-based public toilets and we asked, “How can we help create a habit of going to the public toilet?”. We set up a system of giving discount vouchers for a set period of time and then tracked whether people used them and if they continued going to the toilets after the discounts. We really had to understand why people behaved in a certain way and what could trigger them to adopt healthier habits.

Another key learning is that we aren’t always rational and that telling stories can help connect with people at an emotional level. A lot of my work with the World Bank was related to educational entertainment, or edutainment interventions. There’s a difference between (A) giving someone a pamphlet explaining all of the benefits of using a condom and (B) weaving the message into a story using characters and situations that one can identify with. Parents can envision a different future for their children with the help of visual stories, like in a project we did with Impact(Ed) International (the My Better World series even won an Emmy). Children learned to read and do simple math through playing games on mobile phones. Communication strategies that go beyond traditional approaches can be more impactful, and narratives are particularly powerful.

You now manage monitoring, learning, and evaluation at the IKEA Foundation. What are your role’s main issue area priorities?

After eight years of working on impact evaluations, I reached a point where I felt I wasn’t growing anymore. I was always curious how some organizations develop interventions that reach a large enough scale for an impact evaluation, and I wanted to support others to start walking on the path of evidence-based and evidence-generating programming.

This is why I decided to join the IKEA Foundation, where I support our Refugee Livelihoods and Employment and Entrepreneurship portfolios. The work is varied and entails working with many different organizations. A large part of the role involves helping applicants develop a clear theory of change that tells a story about what the project is trying to accomplish, and to identify indicators that can shed light on whether that theory of change is playing out as intended. I look at the strength of evidence supporting an intervention, identify projects that are at a stage where they could benefit from an evaluation, and commission external parties to conduct them. I'm still involved in randomized control trials but have a different role than when I was at J-PAL—I now do more on early conceptualization and link implementers and researchers and facilitate conversations so that they result in evaluations which both answer questions relevant to implementers and also contribute to the evidence base.

The monitoring, learning, and evaluation (MLE) space is always expanding. How have you seen MLE approaches be applied to new fields, like in the humanitarian space, for example?

Most refugees are in protracted displacement situations, where they've been living in their host countries for five, ten, or more years. In the last decade, the number of refugees has been increasing, but funding has not really kept up with the needs. There's now a general understanding that development solutions are needed to support people in these situations to make a living without being dependent on humanitarian aid. Interventions that were previously targeting nationals are now being tested and evaluated with refugee populations. Impact evaluations in this space are growing. For example, the IKEA Foundation funds quite a number of them looking at how to support refugees to become micro-entrepreneurs because most don't have the right to work, so it's more difficult to find jobs. The graduation approach is also being replicated with refugee populations. There’s a growing number of evaluations in the protracted displacement space, but even in the humanitarian space there is some interesting evaluation being done by the World Food Programme. Surveying displaced populations comes with a different set of challenges, including increased participants’ mobility and a lot of languages, but there are researchers who are willing to put in that extra effort.

I hope by showing that refugees can contribute to the economy, governments will be persuaded that it's actually in their populations' best interest to integrate these refugees. Many projects are not only targeting refugees, but a mix of refugees and host communities members. From a government’s perspective, the fact that their constituencies are also being supported helps open the door to conversations about the rights of refugees. But I think the big question is, does enabling refugees to work and contribute to the economy have a positive impact overall or is there truth to the narrative that they are displacing locals? I think it's the former—that refugees are a net contributor to the economy, especially if they're allowed to do so. 

We recently came up with an idea to start a fund that is dedicated to evaluating interventions for forcibly displaced populations. A strong focus of the IKEA Foundation’s work in this space is evidence generation. This is because we want to accelerate progress toward identifying solutions that support economic self-reliance and demonstrate that the integration of refugees into social programs by governments and NGOs is the way forward. We are already providing funding for a number of ongoing impact evaluations through our partners, but measuring results takes years. Last year we recognized that we have an opportunity to make a big push in evidence generation by funding evaluations of interventions supported by other donors. We’re very excited about upcoming initiatives in this space—J-PAL and IPA are natural partners for this work because they have prior experience supporting evaluations in this sector.

What skills or experiences gained from working at J-PAL have you found useful in your own career?

It’s the non-technical skills that I find most useful in my current role. For example, the ability to act as a liaison between implementers and researchers, and to help them find common ground between their respective interests. An implementer would ask “does my project work?”, whereas a researcher would be more interested in what this can tell us about human behavior in general and what is applicable in other settings. Being able to understand both parties has been very helpful. Then, in the very early stages of a program’s design, when there is no decision to evaluate and there's no researcher involved, I’ve found that having a research mindset has enabled me to think creatively about how a project might be evaluated and work with the partners to see if that's feasible given the realities on the ground. Finally, being able to explain why evaluations are important or present results in simple terms to non-specialist audiences is also a very useful skill I learned while working at J-PAL.

What advice do you have for development professionals and researchers interested in working with a large philanthropy like the IKEA Foundation?

One piece of advice would be to invest time in understanding how monitoring is done and work with the implementing partners of an evaluation to think through what data they can collect to understand early if the theory of change is playing out as intended. I do this on a daily basis in my current role, and I think it's really important to not focus only on the evaluation, but also help partners create the frameworks and tools to measure indicators that they are excited to track so that they can learn continuously and not only when evaluation results are available.

Secondly, if you’re working at a foundation, you need to be able to dialogue with applicants and grantees, and it's really important to have a good understanding of how development projects are created and what their implementation entails. Otherwise, you're that person in HQ that absolutely has no idea about the realities on the ground. Remember to stay humble and ask questions that show genuine curiosity and you will be surprised how much you’ll learn.

To close, I’d say that at J-PAL I learned the power of experimentation, measurement, and iteration. I am now fortunate enough to have a role where I can influence other organizations to adopt this modus operandi. My hope is to contribute to a culture of data-informed programming and to support the identification and scaling of what works.