Strengthening randomized evaluations with qualitative research, part 4: Oregon health insurance experiment

For the final part of our blog series on incorporating qualitative research into randomized evaluations, we spoke with Associate Professor of Social Work and co-author of the Oregon Health Insurance experiment, Heidi Allen, about how in-depth interviews with study participants helped the research team interpret some of the study’s results.

In 2008, Oregon held a lottery to select an additional group of low-income, uninsured adults into its Medicaid program. Around 90,000 applied for 10,000 openings, providing an opportunity for J-PAL North America Scientific Co-Chair Amy Finkelstein, J-PAL affiliated professor Kate Baicker, Allen, and coauthors to conduct a randomized evaluation to understand the impact of providing health insurance to the uninsured.

The study found that participants who received Medicaid coverage experienced increased health care use, reduced financial strain, and improved mental health and self-reported health. To better understand the causal mechanisms behind these quantitative results, a staff of interviewers led by Allen conducted 120 structured interviews with study participants who recently gained Medicaid coverage. 

The interviews were especially useful in helping researchers understand the quantitative results that were more unexpected. For example, many in the broader health policy community believed that providing health insurance to low-income individuals would result in a decrease in emergency department use. However, the results of the study showed that Medicaid coverage actually increased the use of health-care services across the board, including the emergency department. Through the interviews with participants, researchers started to uncover some of the potential factors contributing to the sustained increase in emergency department use.

“There were several things driving this increase. First, we learned that there were many individuals who should have been seeking care but were not. Some of these individuals described themselves as healthy despite having serious, chronic health problems or were generally hesitant to seek care due to cost concerns. Receiving health insurance removed a financial access barrier that may have prevented them from going to the emergency department in the past. On the other hand, we also started hearing how some primary care providers actually told their patients to go to the emergency department. One study participant shared that her primary care provider directed her to the ER since her blood sugar was dangerously high. All these stories helped us understand why we saw an increase in emergency departments.”

According to Allen, another finding that was surprising was that many study participants believed they had emergency-only coverage. Initially, this made little sense to the researchers since the coverage study participants received was a comprehensive health insurance plan. The researchers decided to probe a little deeper to understand the root cause of the confusion.

“We learned that one of the very first things many people did when they got their insurance card was call a dentist and say, ‘I just got the Oregon Health Plan, can I make an appointment?’ And the receptionist at the dental clinic would tell them that the Oregon Health Plan was emergency-only coverage, which is the case for dental care. But hearing this led many study participants to believe that their insurance only covered emergency care for health services too. I don't know that this drove any large effects in the study, but it was really interesting to hear that there was an unexpected kind of administrative complexity that I wouldn't have guessed prior to actually talking to people.”

Study participants also reported significant improvements in health. However, these findings were not coupled with clinically significant improvements in objective health measures like hypertension, diabetes, obesity, and behaviors like smoking. This posed another puzzle for researchers, but some potential explanations were found in participants’ stories.

“Initially, we thought when participants received health insurance, they’d gain access to a provider who could diagnose their health issues, and with this diagnostic information, they could start treatment that led to improved health outcomes. The interviews helped us understand that the pathway of receiving health insurance to improved health wasn’t as simple as we had thought. Having a good patient-provider relationship is a key factor in whether a patient’s health improves. For many participants, it took seeing multiple providers before they found someone they really liked working with. Once participants established a good relationship with a provider, they often prioritized what they were going to work on. And they might have more immediate needs like a broken ankle that needs surgery, so they may not address their diabetes or their obesity or smoking first.”

For Allen, conducting this qualitative follow-up research was critically important to making meaning out of the quantitative results, especially the findings that were surprising. These conversations with patients helped researchers understand where Medicaid was working for people and where people ran into barriers.

“The Oregon study constantly drives me to think about how we can make Medicaid work better for people, how we can improve the places where we lose efficiency and effectiveness in this causal chain from getting coverage to having positive health outcomes. There are probably multiple points where we could make slight modifications and see meaningful improvements. I think what's been helpful about our qualitative work is that it provides a pathway to think through real people's experience, rather than just partisan expectations of the experience. The interviews also drove home the humanity behind the numbers which isn’t to be taken lightly.”

Part one of this four-part blog series highlights the value of incorporating qualitative methods into randomized evaluations and outlined specific tips for researchers. Part two talks about how qualitative research helped motivate and shaped the central question and hypothesis for a study on racial concordance between physicians and patients. Part three looks at how Creating Moves to Opportunity randomized evaluation embedded qualitative research methods into its study design. Part five highlights the value of qualitative research in providing a deeper understanding of mothers' experiences in the Baby's First Years study.