Strengthening randomized evaluations with qualitative research: Baby’s First Years household measurement

Building on J-PAL North America’s qualitative research blog series on incorporating qualitative research into randomized evaluations, this post features the perspectives of researchers conducting the Baby’s First Years (BFY) study. BFY is a J-PAL-funded research project assessing the impact of poverty reduction on family life and infant and toddlers’ cognitive, emotional, and brain development. In this post, the researchers describe what we can gain from triangulating with qualitative and quantitative data on household rosters and how it should encourage us to be cautious in interpreting our results.

In the Baby’s First Years study, we bring together qualitative and quantitative data to examine how seemingly simple and objective facts—like who lives in a household—are captured by different types of data collection and what this means for how we should interpret results.

A mother of four in Louisiana told us that her family was recently evicted, leaving them to stay with family and friends. During her interview, she and her children were living with her cousin’s family, though during the week, her seven-year-old stayed with kin closer to school. Mom said that after an argument with her children’s father the day before, he went to stay with his brother. Depending on the day, therefore, there could be up to three adults and six children in this household. Two weeks later, in the survey, she described a household of herself and her children, as well as another adult (neither her cousin nor her children’s father). With her variable housing circumstances, there may not be one right answer about this mother’s household membership. 

In Minnesota, we met a mom who lived with her baby at her parents’ house, where her adult cousin stayed part-time, but she and her child also spent about half the nights of the week at her child’s father’s house. While we can choose one as the primary household to report on, doing so obscures the reality of the adults and resources to which this child has access—there are more than either household’s roster would capture, but they are also not available full time.

In 2018 and 2019, the BFY study recruited 1,000 mothers in four US cities whose household incomes were below the federal poverty line when they gave birth. Mothers agreed to be randomly assigned to receive a large ($333) or small ($20) cash gift each month until their child was around four years old. Our qualitative companion study, BFY: Mothers’ Voices, invited a stratified random sample of 80 mothers to participate in repeated, in-depth qualitative interviews each year for the duration of the study. Recently released results  from the BFY study showed that babies in the high-cash-gift group were more likely to have brain activity patterns that previous research has associated with future learning and cognitive skills. 

When we collected survey data and conducted interviews with moms when the focal children were around one year old, both instruments asked moms about who else lived in her household (for other research on household measurement, see Clark 2017; Waller and Jones 2014). 

In addition to providing descriptive information about families, such data is key to calculating measures core to the study. This includes the number of adults in the household who could contribute financially to or invest their time and care in children, and the number of children in the household in need of adult investments and attention. Additionally, who lives in the household and their relationships feed into key measures of economic well-being that matter for safety net programs, like where the household falls relative to the federal poverty threshold—a calculation based on income and family size. As we explore the mechanisms driving the recent brain activity findings, understanding these issues—who is in the house with the child, who is contributing to and drawing on household resources—all come into play.

During the survey, we asked mothers about each of the people living in their household, defined as “anyone who has been living with her and is related to her baby through blood, marriage, domestic partnership, or adoption.” In the qualitative interviews, we asked broadly “Who all lives here?” and then followed up about adults and children to make sure everyone was captured. In the interview, we also asked mothers about people who lived with her only part-time. 

Our biggest takeaway was that the household roster data collected through surveys and semi-structured interviews were largely consistent:

However, differences revealed interesting new information. Across both survey and interview data, 56 people were mentioned in only one of the two studies. Twenty-one individuals listed in the quantitative study were not identified in the qualitative study, only one of whom was a child. Thirty-five individuals described in the interviews were not identified in the survey, with fourteen (40 percent) being children. We did not see a strong relationship between length of time between the survey and interview and there being discordance in household rosters.
 
For about one in five of the individuals missing in either the survey or interview, we identified a life event such as a baby being born, a release from incarceration, a move, or a change in relationship status that explained the difference in household rosters. For another one in five, their part-time residence in the household may explain the discrepancy between interview and survey (although the survey did capture some individuals reported in the interviews as part-time residents). This means that for most individuals who were only reported in one of the two data collection instruments, there was not a readily identifiable reason for them to be missing. While the survey was more likely to miss household residents, both types of data collection missed some people. Yet both largely produced similar accounts. Why might these differences occur, and what do they mean?

No data collection is perfect, and we are not arguing that surveys or interviews are the “right” way to collect household rosters or other such data. Rather, pairing data collection methods helped reveal what we were seeing and missing with each instrument. While collecting a household roster in this detailed way claimed a substantial amount of survey “real estate,” it also had results largely concordant with those produced through open-ended responses. 

Both the interviews and the discrepancies we see in household rosters offer insight into the fluid nature of relationships within and across households; there is more instability than point-in-time estimates allow us to see. Triangulating with multiple data collection methods can bolster our confidence and reveal when seeming instances of measurement error are actually illuminating household and family complexities that our research must take into account. Greater understanding of supposedly simple information about households can help us explore the mechanisms that drive RCT impacts.