Strengthening randomized evaluations with qualitative research: Baby’s First Years mothers' experiences

In part five of our blog series on incorporating qualitative research into randomized evaluations, we feature the perspectives of researchers conducting the Baby’s First Years (BFY) study. BFY is a J-PAL-funded research project assessing the impact of poverty reduction on family life and infant and toddlers’ cognitive, emotional, and brain development. In this post, the researchers reflect on the value of qualitative research in both providing a deeper understanding of each participant’s background and context and painting a fuller picture of mothers’ experiences in the study.

Randomized evaluations are great for telling us the answers to the questions we ask; they are less useful for helping us answer questions we didn’t know to ask.

As researchers from the BFY study, we recognized that survey and debit card transaction data would show how mothers spent their monthly cash gifts, but not why they did so—that is, what meaning and motivation lay behind spending decisions and what power dynamics or negotiations within families may surround these decisions. Therefore, we pursued a multi-method approach to understanding the impact of monthly unconditional cash transfers distributed to families with low incomes for the first several years of their focal child’s life.

In 2018 and 2019, the BFY study recruited 1,000 mothers in four US cities whose household incomes were below the federal poverty line when they gave birth. Mothers agreed to be randomly assigned to receive a large (US$333) or small (US$20) cash gift on a debit card each month until their child was around four years old.

Our qualitative companion study, BFY: Mothers’ Voices, invited a stratified random sample of eighty mothers to participate in repeated, in-depth qualitative interviews each year for the duration of the study. Both BFY and BFY: Mothers’ Voices are currently in the field, surveying and interviewing mothers by phone to find out how they and their now-three-year-old children are doing.

Our qualitative study has two distinct contributions to offer from typical survey data and related methods in understanding the impacts of the cash transfers. First, the qualitative approach starts from a position that we, as researchers, do not necessarily know all the important questions to ask or the response categories that make sense. So, while in BFY: Mothers’ Voices we use an interview guide to think through topics and ways to word questions, we also ask open-ended and follow-up questions to allow mothers to lead us where they think it is important to go. If we ask about the joys of motherhood, and a mom shares a story about her children playing together, or she starts talking about a recent stressful event, each answer is equally welcome and insightful.

Qualitative research opens the door to a fuller appraisal and exploration of mothers’ experiences that consider their dreams and fears in ways that are not pre-supposed to fit into validated scales. There are some things we simply cannot make sense of if we only ask our own questions, provide our own answer options, and, therefore, stay rooted in our own perspectives and experiences.

Nina, a 26-year-old, Black, high-gift-group mother of four from New Orleans, described using the first installment of the BFY money shortly after she gave birth (as is standard, we use pseudonyms to protect mothers’ identities). “I could have just cried because it was a total relief. Because first of all we went in the hospital flat broke. We was flat broke in the hospital.” You can hear not just Nina’s relief but also her joy: “We got food, a lot of food. We put food in the house. I even went and got the kids a gift. That's how happy I was. I was like, ‘Let me get the children something.’ So, I even got them a toy at the store. I got some cleaning supplies to make sure it was really sanitary for when I brought [my daughter] home.” Multiple mothers described wanting to bring their babies home from the hospital to clean houses—these are moments that are rich with the symbolism of having a fresh start.

Our open-ended interviews give us a chance to capture cash uses we didn’t anticipate. Because money is a distinct intervention in that it can be used in nearly an infinite number of ways, it’s just not possible to ask about them all on a survey. As a simple example, the BFY survey queried mothers about purchasing common goods for infants, such as highchairs and car seats. But, as with Nina, in the interviews we learned about purchases mothers made intentionally to benefit their children—such as cleaning supplies—that we did not know to ask about as child-related items. This changes our conceptualization of what counts as a “child-related purchase.”

The qualitative approach also provides a rich understanding of the background and context for each person. This contextualized approach may be particularly important in studying a topic like unconditional cash transfers, since mothers’ financial allocation decisions are likely embedded in the complex context of their childhood experiences, parenting values, family material circumstances, dreams for their children’s futures, and conceptualization of the cash transfer and what they see as its “proper” use.

When we center this complex context in our analyses, we may understand socioeconomic behavior differently. Following the typical frameworks of economics, buying a fast-food kid’s meal when you’re on a tight budget might make little sense, since you don’t have the money to spare and the burger and treat will not be an investment in developing your child’s human capital. To a mom who is scraping by, the skip in her child’s step and the smile on her child’s face as they leave the restaurant may be great reasons for such a financial allocation decision. And her motivation for doing so could be found in her experiences as a child, either relishing similar memories of her own or wishing to give her children more happy moments than what dominated her upbringing. We could learn how the deprivation parents want their children to avoid may extend beyond the food, clothing, and shelter parameters that often constitute material hardship in research to include social and emotional experiences as well.

Scholars’ existing measures of financial disadvantage don’t typically capture the small, daily expenses that can also be central to helping parents feel like they are doing right by their children. The qualitative work in BFY allows our team to think differently about how the cash gift affects allocations that support children—expanding the important but also narrow expenditure types captured in the survey to include popsicles that offer a toddler joy or cleaning supplies that help a mom feel she is providing a safe place for her baby. These are things that we may discover are important ingredients in child investment and can have ripple effects for children’s development.

Part one of this blog series highlights the value of incorporating qualitative methods into randomized evaluations and outlined specific tips for researchers. Part two talks about how qualitative research helped motivate and shaped the central question and hypothesis for a study on racial concordance between physicians and patients. Part three looks at how Creating Moves to Opportunity randomized evaluation embedded qualitative research methods into its study design. Part four discusses how qualitative research helped the Oregon Health Insurance Experiment research team make sense of some of the study’s results.