Strengthening randomized evaluations with qualitative research, Part 3: Creating moves to opportunity

In part three of our qualitative research blog series on incorporating qualitative research into randomized evaluations, we learn more about how researchers conducting the Creating Moves to Opportunity (CMTO) project embedded qualitative research methods into their study and what factors made conducting high-quality, interdisciplinary research feasible.

Since 2018, MDRC, J-PAL affiliates Peter Bergman, Raj Chetty, Nathaniel Hendren, Lawrence Katz, and Christopher Palmer, along with sociologist and qualitative research expert Stefanie DeLuca, have been conducting a randomized evaluation of Creating Moves to Opportunity, a housing mobility program in Seattle and King County, Washington. The study, conducted with approximately 1,300 families, aims to understand CMTO’s impact on helping families move to neighborhoods with lower rates of poverty and more opportunities for upward income mobility.

In Phase One of the study, qualified low-income families with at least one child under fifteen were drawn from the Housing Choice Vouchers waitlist and offered an opportunity to enroll in the study. Participants were then randomly selected to receive CMTO services—including customized search and landlord engagement assistance from family and housing navigators and short-term financial assistance—or receive the housing authorities’ standard services. 

Results from Phase One of the study found that CMTO-participating families were more likely to move to higher-opportunity neighborhoods than families who only received standard services. To help interpret these results and understand participating families’ experiences, the research team conducted qualitative analyses to complement the quantitative results. Led by DeLuca, a team of research staff carried out in-depth interviews with 161 participating families (from both the treatment and control groups) using an approach that borrowed some elements typically used in quantitative research to ensure the collection of high-quality, representative data.

“Using the program administrative data, we pulled a stratified random sample of families participating in CMTO to interview and achieved an 80 percent response rate. Those are two relatively unusual things to do in interview studies, but it helped ensure that the particular patterns that emerged from the data were representative of the families in the program.”

According to DeLuca, having a research team with the capacity and flexibility to follow up and be on-site when necessary was critical to the success of the qualitative research component of CMTO.

“Achieving an 80 percent response rate took more than just well-spaced regular phone calls. It required a research team with the capacity to be on-site and commit to really connecting with the study participants, especially with the door-knocking component of recruitment. It makes a huge different for people to see you as interested in their stories, not as telemarketers. With interviews taking up to two to four hours, being on-site also meant we could conduct follow-up interviews in case the interviewer wasn’t able to get through everything the first time. It also allowed us to collect ethnographic observations of the neighborhood and see the rhythm of the household, meet family members and friends—even run errands with families when needed.

We also used pairs when possible for interviews and had team members memorize the interview guide. With one team member conducting the interview without looking at a script and another paying attention to anything that got missed and noting that at the end, we had the benefit of an organic, more natural conversation alongside systematic data collection. While team members trained coders and designed codebooks, the bulk of the coding was also done by people who weren't interviewers, so the data wasn’t coded through the lens of assumptions and preconceived notions that someone present at the interview might have. Finally, we had several reliability procedures in place, including having the interviews coded by at least two different people, and a third person to note inconsistencies to be resolved.”

Participant interview findings helped to highlight the aspects of the program that led families to move to higher-opportunity neighborhoods.

“Through the interviews, we were able to better understand what was happening from the participants’ point of view and what mechanisms were at play. Previously, many believed that interventions focused on providing more information and monetary resources would encourage families to move to neighborhoods with more opportunities for upward income mobility. But what we see from CMTO is that those two types of resources are not sufficient to explain the success of this program. What came through so clearly in the narratives of the participants was how important it was to have that support from the housing navigators who could boost confidence and provide customized assistance based on the specific needs of the families.”

Ultimately, for DeLuca, incorporating qualitative research into randomized evaluations is about providing opportunities to check assumptions and to see a bigger picture of what might be driving the impact of a program or policy.

“The joining of disciplines like economics and sociology for randomized evaluations can provide an opportunity to see something you otherwise wouldn’t see. Sociologists are often trained to think about barriers to social mobility and wellbeing and tend to focus a bit less on the decision-making process of individuals. In contrast, economists tend to emphasize decision-making quite a bit and might give less attention to the context in which the decisions are being made. By bringing the two together, researchers can leverage the theories and tools of each discipline and, hopefully, conduct a more policy-relevant and consequential study.”

Part one of this four-part blog series highlights the value of incorporating qualitative methods into randomized evaluations and outlined specific tips for researchers. Part two talks about how qualitative research helped motivate and shaped the central question and hypothesis for a study on racial concordance between physicians and patients. Part four discusses how qualitative research helped the Oregon Health Insurance Experiment research team make sense of some of the study’s results. Part five highlights the value of qualitative research in providing a deeper understanding of mothers' experiences in the Baby's First Years study.