How to integrate qualitative methods into RCTs

Economists have long used qualitative methods to inform the design and help interpret the results of RCTs. However, this qualitative work is often implicitly understood rather than explicitly discussed in research papers. This practice has started to change in recent years. For example, Chris Blattman’s study on improving police access in gang-controlled neighborhoods in Colombia used in-depth interviews to understand the nature of clandestine organizations in Medellin and to develop and refine their hypotheses. Seema Jayachandran, in her work on women’s empowerment in India, used in-depth interviews and machine learning to create a validated index of women’s agency. In Dean Karlan’s research on a sexual health messaging service in Uganda, focus group discussions (FGDs) were used to explore user perceptions of the service and uncover mechanisms. To support the further integration of qualitative methods into randomized evaluations and provide more structured guidance, J-PAL has developed new research resources and training materials for researchers and practitioners.

Research staff working on RCTs often use qualitative methods in the research process—and more formal guidance can help research teams apply best practices and lessons learned from previous research in economics and other disciplines. We hope to strengthen research staff capacity with two new resources on qualitative methods: The first resource, "Integrating qualitative methods into randomized evaluations," focuses on the theoretical motivation for the use of qualitative methods in experimental research and provides guidance for incorporating qualitative methods at different project stages. The second resource, "Implementing qualitative methods in the field," offers practical applications for a number of tools and strategies to guide teams looking to collect qualitative data in the field. 

Alongside these new research resources, J-PAL has collaborated with Innovations for Poverty Action (IPA) and Busara to introduce new sessions on qualitative methods for our Research Staff Trainings and Conducting Experimental Research course. J-PAL’s regional offices are also working to advance these efforts. In this blog, we discuss some of our ongoing evaluations that incorporate qualitative methods and learn what the future of qualitative RCT research may hold. 

Examples from J-PAL Southeast Asia

In recent years, the J-PAL Southeast Asia (SEA) office has incorporated qualitative methods into different stages of several ongoing studies. These methods helped contextualize research designs to local settings, offer insights into how interventions are being implemented, and clarify issues that might influence preliminary results of ongoing RCTs. For example, a study by researchers, including J-PAL affiliated researchers Arya Gaduh and Gumilang Aryo Sahadewo, evaluating the impact of integrating technology and interactive, student-centered teaching on math learning outcomes used in-depth interviews with teachers to understand their perspectives and experiences during the program. Qualitative data revealed key challenges in implementing the interventions, which helped interpret the impact evaluation results. Another study implemented by J-PAL SEA and led by Gumilang Aryo Sahadewo and co-authors evaluated the impact of home-based growth charts on child growth. Through qualitative interviews, the authors explore how caregivers monitor their children's development and use the charts, uncovering insights into both the usage of the charts and the challenges families face in providing nutritious food for their children. 

From our experiences conducting qualitative in-depth interviews in the field, we have learned several important lessons that offer practical guidance for researchers and practitioners interested in incorporating qualitative methods into field surveys. One that stands out is to clearly determine the key objectives for each question and discussion prompt in the interview protocol. This helps interviewers adjust their probing strategies to align with both the instrument’s objectives and the informant’s responses. We cover this, and other lessons learned, more extensively in our new resource on implementing qualitative methods in the field.

Examples from J-PAL Middle East and North Africa

The J-PAL Middle East and North Africa (MENA) team has made it a priority to increase the team's capacity to systematically integrate qualitative methods into our research projects. J-PAL MENA has begun a series of trainings on qualitative methods for research staff covering topics such as the value of integrating qualitative methods into randomized evaluations, how to move towards a mixed methods approach, and planning for incorporating qualitative methods into ongoing projects. The training also covers qualitative sampling and data collection techniques. The trainings will explore considerations for implementing qualitative methods in the MENA context, accounting for unique social conditions that could affect the research process. 

Following these trainings, research staff apply the tools learned in their project work. For example, a study on the impact of childcare subsidies on female labor force participation by J-PAL affiliates Bruno Crépon, Abdelrahman Nagy, and Carolin Krafft used FGDs to understand the social context of Egyptian society in Upper Egypt. The FGDs results provided a set of gender and social norms insights that helped determine which aspects of the interventions were most relevant to the community.

Future opportunities for qualitative research in randomized evaluations

In the context of RCTs, qualitative methods are most commonly used for exploratory research to inform the design of interventions and outcomes and after statistical analysis of endline data is complete to contextualize quantitative results. However, the examples discussed in this blog post and in our new research resources show how economists, often in collaboration with researchers from other disciplines, sometimes integrate qualitative methods more systematically into randomized evaluations and may present qualitative findings alongside quantitative results in journal articles. In the future, we may see more researchers who conduct randomized evaluations incorporating qualitative data collection and analysis into pre-analysis plans, academic papers, and replication materials. 

With the proliferation of AI technologies such as large language models, it is becoming easier to perform large-scale natural language processing of qualitative data. This can make it more feasible to use qualitative data to measure outcomes in randomized evaluations. For instance, imagine an education study that aims to increase the quantity and quality of girls’ classroom participation. A traditional quantitative evaluation might ask teachers to count the number of times each student asks or answers a question each day and then compare these participation counts between intervention and comparison  groups. Today, it would be possible to use machine learning to analyze audio recordings of all class sessions to capture more reliable and complex measures of student participation. In addition to measuring the number of times girls and boys participate, AI-powered coding of audio transcripts could measure the duration, vocabulary, tone, and sophistication of student questions and answers as well as the nature of the teachers’ reactions. Statistical analysis of such qualitative data could then lead to a much richer understanding of the intervention’s impact. 

As more researchers begin to incorporate qualitative methods into randomized evaluations with more integrated and innovative research designs, we are excited to continue to strengthen our qualitative research resources, trainings, and collaborations. 

To read more on how some J-PAL staff and affiliated researchers are using qualitative methods in their work, please see J-PAL North America’s five part blog series on strengthening randomized evaluations through incorporating qualitative research. For more detailed guidance on integrating qualitative methods into randomized evaluations and implementing qualitative methods in the field, please check out these new research resources.