Strengthening randomized evaluations with qualitative research, part 2: Matching provider race to increase take-up of preventive health services

For the second blog in our blog series on incorporating qualitative research into randomized evaluations, we spoke with US Health Care Delivery Initiative Co-Chair Marcella Alsan about her research team’s experience using qualitative research methods in the scoping and analysis phases of their randomized evaluation.

In 2017-2018, Alsan, Owen Garrick, and Grant Graziani conducted a randomized evaluation to test the impact of financial incentives and race concordance (between patients and health care professionals) on the take-up of preventive health care services by Black men.

Before embarking on the project, the research team felt that it was important to get to know the population they would be working with for the study. Led by Bridge Clinical Research and its president and CEO, Owen Garrick, the research team conducted a series of focus groups with Black men recruited from Oakland barbershops to inform the study’s design.

“We walked into every single barbershop that we could find within a ten-mile radius of the clinic where our study would take place and recruited Black men from those barbershops to conduct several focus groups before actually enrolling anyone in a pilot. Led by Dr. Owen Garrick, the focus groups were carried out in a way that encouraged participants to speak as freely as possible—we didn’t ask for personal information and had a scribe take notes instead of recording the session. We wanted to ensure that we would learn as much as possible about the participants’ lived experiences.

It was incredibly helpful to have Bridge Clinical Research and Dr. Garrick as a partner in this study. They were deeply invested in the science behind the evaluation and also helped bridge the divide that existed between academia and the population that we're trying to serve with our research.”

Insights from the focus groups affirmed that the topic of the diversity of health care professionals was relevant to the lived experiences of the study population. The focus group conversations also helped the research team iron out critical design features of the randomized evaluation.

“Since the outcome of the study was take-up of preventive health care like BMI measurement and flu shots, procedures that do not require a medical doctorate, there was an open question around whether we should use nurses or physicians in the study. The focus groups showed that the men had a strong preference for physicians since they felt like physicians were not readily available to them as patients. Specifically, they felt like they were often paired with less qualified staff and that that was disrespectful. These conversations made the decision very easy since the aim of the study was to examine what fostered trust and respect between Black men and health care professionals.”

In addition to answering key study design questions like whether the intervention would use nurses or physicians, the focus groups helped refine other logistical details of the study. For example, based on participant feedback that advertising flu shots could suppress demand for the clinic’s services, the research team decided not to mention flu shots and only to mention cardiovascular screenings on the flyers advertising the clinic.

Ultimately, the research team conducted the study in two phases. In the first phase, study participants—Black men recruited from Oakland, California barber shops—arrived at the clinic and received a tablet with information about their visit. Researchers randomly assigned study participants to see Black or non-Black doctors and a financial incentive of $0, $5, or $10 if they chose to get the flu shot.

The participants then received a list of services and were told that their assigned doctor would provide all preventive services they selected. Participants were then asked to choose the preventive health care services they would like to receive. In phase two, participants met with the doctor whose image was displayed on the tablet and had another opportunity to decide which preventive services to receive.

The study found that seeing a Black male doctor significantly increased demand for all preventive health services. Those who saw a Black doctor were 11 percentage points more likely to choose to get their blood pressure measured (a 15 percent increase) and 16 percentage points more likely to choose to get their BMI measured (a 27 percent increase).

Seeing a Black doctor had an even larger impact on the take-up of more invasive procedures like diabetes screenings, cholesterol screenings, and flu vaccines. The qualitative data collected during the randomized evaluation shed light on potential mechanisms driving these effects.

“We used word clouds to compare the open-ended comments on the feedback forms from patients and found that the comments from patients who saw both Black and non-Black physicians were remarkably similar. This suggested that participants were happy with the care that they were getting from both Black and non-Black physicians. The excellent feedback of the non-Black doctors also helped rule out discriminatory behaviors towards patients as a factor driving the observed effects.

Where we did see differences were in the written notes physicians took during the visits. We found that the type of information that Black physicians versus non-Black physicians could extract was completely different. These findings bolstered our hypothesis that the driving mechanism was better communication between race concordant physicians and patients and not discrimination on the physicians’ part.”

Alsan feels that, when possible, researchers should try to incorporate qualitative methods into their research for several reasons.

“There are many times where, as researchers, we can lose touch with what the real questions, constraints, and realities are. In general, I think qualitative data can be incredibly fruitful. Not only does it let us understand what kind of variation we want to introduce or how we might leverage administrative data, but qualitative research can lead us to ask a whole new set of research questions that weren't even on the agenda.”

Alsan also believes that qualitative research approaches can provide opportunities to generate buy-in from key stakeholders, including the communities the research aims to serve.

“If you're doing work in criminal justice or health care or education, we ultimately want doctors, nurses, patients, sheriffs, prosecutors, teachers, and principals to read our work. One way to get these key individuals invested in our research is to ask them to be a part of the discussion when we pose research questions and interpret the results. If they feel like they’ve helped shape the research, then they might be more open to the policy recommendations that come out of it.”

Part one of this four-part blog series highlights the value of incorporating qualitative methods into randomized evaluations and outlined specific tips for researchers. Part three looks at how Creating Moves to Opportunity randomized evaluation embedded qualitative research methods into its study design. Part four discusses how qualitative research helped the Oregon Health Insurance Experiment research team make sense of some of the study’s results. Part five highlights the value of qualitative research in providing a deeper understanding of mothers' experiences in the Baby's First Years study.