Beyond causality: Additional benefits of RCTs for improving health care delivery

The medical community has long valued medical randomized controlled trials (RCTS) for their ability to tell us whether medicines cause any improvement (or worsening) of health and safety. Take the Covid-19 vaccines: manufacturers have used RCTs to determine the safety and efficacy of their vaccines.

RCTs are also well-known for being able to determine the effect of a health care program or policy, but they are not a one-trick-pony. As we (Alsan and Finkelstein) highlight in our new Milbank Quarterly article, RCTs have (at least) four additional benefits that are useful for addressing pressing health policy questions.

RCTs empower us to answer the questions we want to answer

RCTs allow researchers, practitioners, and policymakers to study the questions that they want to study, rather than having to limit themselves to what they can study based on naturally occurring variation that already exists.

One example is the 2017 Oakland study on diversity in the physician workforce. Researchers were interested in the effect of a physician’s race concordance between patient and physician on their demand for health services, among Black men. Identifying this effect with observational data would be difficult because most individuals choose their primary care doctor, and many of the most vulnerable individuals do not have a primary care doctor at all. Additionally, a lack of diversity in the physician workforce makes it hard for patients to find doctors that look like them.

Researchers overcame these challenges by randomizing Black male patients to receive either a Black or non-Black doctor and studied how such assignment affected demand for preventative care. The study found that Black male patients who were randomly assigned to a Black provider were eighteen percentage points more likely to undertake preventative care.

RCTs can test theory

A critique of many studies–including some RCTs–is that they don’t illuminate “why” a program or policy had or didn’t have an impact. In some cases, RCTs can be designed to test different theories and pinpoint reasons why a program or policy was effective or not.

One example of researchers designing an RCT in this manner is a 2010 study on the impact of immunization camps and incentives. A nonprofit, Seva Mandir, aimed to increase child immunization rates in rural Udaipur, India. The researchers designed the evaluation to determine why immunization rates were so low. One potential barrier was that clinics that offered vaccines were often closed. Another possibility was that it was challenging for families to go to the clinics for the five courses of immunizations and they were not being incentivized enough to go.

The researchers investigated this problem by randomly assigning the community into three groups: a comparison group, a treatment group with an intervention aimed at combating the closed clinics issue, and a second treatment group that also received incentives. Because of this study design, researchers were able to determine that both were significant barriers to child immunization.

RCTs can credibly examine indirect effects

Sometimes a program may be intended to serve one group of people, but it may have positive or negative indirect, or “spillover,” effects on other groups. Large-scale RCTs can be used to credibly measure these spillovers.

Two recent, nationwide RCTs involving the Centers for Medicare & Medicaid Services revealed that there are substantial impacts of Medicare policy on the treatment of non-Medicare patients. In one, a different way of reimbursing for hip and knee replacements for Medicare patients was randomly introduced in some metropolitan areas and not in others. Yet both Medicare patients and privately insured patients were impacted similarly. These results suggest that one insurer’s reforms can have broader effects on the treatment of patients as a whole.

RCTs can facilitate fruitful collaboration

RCTs often necessitate researchers to connect with providers to design and implement a study, strengthening research designs. The collaboration between researchers and the organizations that implement the programs or policies that are being evaluated are invaluable. Implementing organizations can provide advice about study design and study interpretation.

This proved to be true for the Oakland physician-patient racial concordance study discussed earlier. I (Alsan) and my co-authors collaborated with implementing organizations to run focus groups with Black men that indicated that advertising the study as including injections, such as flu vaccines, could deter patients. This information led to modifications to the study advertisements, which ultimately improved recruitment and the study overall.

Interested in learning more about RCTs?

The additional benefits of RCTs beyond causality are numerous and can help us learn how to improve health policy. Read more about these benefits demonstrated by recent RCTs in the paper, Beyond Causality: Additional Benefits of Randomized Controlled Trials for Improving Health Care Delivery.

For the health policy and medical community excited to learn about lessons learned from randomized evaluations in health care delivery, recordings of our recent virtual convening, “HCDI at 8: Building on Eight Years of Randomized Evaluations to Improve Health Care Delivery” are now available.