Nurturing the null: Approaching research design with all possibilities in mind

Null results—when a study does not find significant impacts on chosen outcomes—can provide valuable insights for research and policies alike. However, it can be difficult for stakeholders to identify and leverage these insights. In J-PAL's null results blog series, we highlight randomized evaluations that yielded null results in order to elevate their lessons learned and inform future research. In this blog post, Manasi Deshpande and Rebecca Dizon-Ross shared how they designed a multi-step evaluation to determine if their null findings were due to a failure of their research design or a flaw in their Theory of Change.

There are several reasons why researchers may find null results. There may be issues with their study design, such as flaws in their measurement strategy or inadequate power to detect a result. There may be issues with program implementation where an intervention was not carried out as it was intended. Alternatively, it may be that the program simply did not have its intended impact. These point to a breakdown in the intervention’s Theory of Change.

Our (Manasi and Rebecca) study looked at the impact of an informational intervention about Supplemental Security Income (SSI) eligibility on parents’ behaviors. SSI provides monthly payments to individuals with low incomes who have disabilities. Eligibility for SSI differs for children and adults, leading many children to lose their SSI benefits when they turn eighteen years old as they no longer qualify under the adult criteria. We provided parents with information about the potential loss of their children’s SSI benefits upon turning eighteen and assessed whether receiving this information impacted parents’ investment in their child’s education and job preparation (i.e., “human capital”).

We hypothesized that an increase in the perceived likelihood of losing benefits would lead parents to invest more in the human capital of their children through income effects (i.e., parents invest less because they think their children have less “need” for income in the future) and substitution effects (i.e., parents invest less because they anticipate their children’s future earnings will be taxed away at a high rate in the form of lower benefits). However, our study revealed that this was not the case. Parents did come away with a greater understanding that their children might not receive benefits in adulthood, but that understanding had no impact on parents’ investments in their children’s human capital.

By conducting two additional surveys, we were able to attribute this null finding to a shortcoming of our theory. These added steps—an expert survey and a mechanisms survey—enabled us to better understand our null and the true factors driving parents’ investment decisions.
 

Using an expert survey to assess whether others shared our predictions
Our initial Theory of Change was built around the hypothesis that understanding the likelihood of their child losing SSI benefits would lead parents to change the extent to which they invested in their children’s education and job preparation. To assess whether this belief was held by others as well, we fielded a survey to experts on the Social Science Prediction Platform. We found that others shared our hypothesis, allowing us to have a more informed prediction of the intervention’s potential impact. 

When we found the null results, this survey helped us see that our final estimated results were different from not only our own assumptions but also wider expert priors.
 

Fielding a mechanisms survey to better understand our results
Because we had been confident in our Theory of Change, our null result prompted us to question why we did not find our predicted impacts. We, therefore, conducted a follow-up survey to identify the mechanisms underlying our findings. We asked about parents’ beliefs in various ways to check that respondents had, in fact, understood the information presented to them. For example, we asked a hypothetical insurance question that required participants to understand the probability of their child receiving SSI in adulthood. We also put in place a time lag between sharing information and sharing resources parents could use to allow enough time for parents to absorb the information. 

We found three primary explanations for parents' behavior regarding SSI benefits and investments in their children:

1. Many parents planned to compensate for lost SSI income by increasing their own work. 
2. Parents didn’t solely base their decisions on financial returns; many prioritized helping their children "realize their potential" over financial goals. 
3. Many parents may have already reached the limit of the investments they can make given time, money, and bandwidth constraints. 

The mechanisms survey helped us identify what assumptions in our Theory of Change did not hold true. We were, therefore, able to update our Theory of Change with a more informed understanding of the drivers of parents’ decision-making.

By conducting these additional steps, we were able to pinpoint the cause of our null result and still deepen our understanding of human behavior. We hope that future research can leverage these learnings to find effective strategies to impact human capital investment in this population.