Reflecting on a decade of impactful research at J-PAL North America: Ensuring rigorous research design and effective implementation
In part two of J-PAL North America’s ten-year anniversary blog series, we dive into study design and implementation and look at past studies on provider race concordance, strategies to increase SNAP take-up, and hiring discrimination in large employers to distill key lessons on ensuring rigorous research design and effective implementation to build an evidence base on poverty reduction strategies.
To generate reliable results that can be translated into policy action, randomized evaluations must be rigorously designed to detect appropriate outcomes and implemented to ensure reliable analyses. Throughout ten years of this work, we’ve provided funding, training, and technical support for studies that have changed the way that randomized evaluations are used to inform policy.
A well-designed randomized evaluation can investigate complex and nuanced topics
Over the past decade, our team at J-PAL North America has worked with researchers in the network to refine policy-relevant research questions and design robust randomized evaluations, including on theoretical topics that had not yet been evaluated through experimental research. A 2019 study by Marcella Alsan (Harvard University), Owen Garrick, and Grant Graziani tested whether increasing representation among physicians of color would improve health outcomes for patients of color. They found that when a Black man saw a Black male doctor, they were more likely to take up preventive health services, especially more invasive tests like a finger prick or an injection.
The study was motivated by policy statements from leading medical organizations calling for increased diversity among physicians, which were based on the theory that doctors with similar backgrounds may be more efficient at building trust and communicating with patients. Alsan had also conducted previous research linking the US Public Health Service Experiment in Tuskegee to a decrease in the lifespan of Black men. The researchers were interested in designing a randomized evaluation that would allow them to draw reliable causal conclusions about the effects of Black physicians on Black patients.
Through funding from J-PAL's Health Care Delivery Initiative and feedback sessions with J-PAL scientific leadership and affiliated researchers, the study design was refined. The final design included two phases: in the first phase, patients were shown a photo of their doctor and then selected what preventive services they would receive; in the second, patients met with the doctor and were given the option of changing which services to receive. The two-stage design allowed the researchers to determine whether there were factors within the interaction, beyond race, that impacted the patient's health. They found that after seeing a photo of their doctor, Black men selected to receive preventive services at the same rate regardless of the race of their doctor. However, after meeting with the doctor, Black men who were randomly assigned to see a Black doctor were much more likely to select every preventive service, particularly invasive services. This result indicated that better communication drove the increased take-up of preventative services.
The study has now been widely cited by others interested in understanding the health of diverse communities, including research identifying the impact of Covid-19 on hospital admissions, expanding the evidence base on physician race concordance, and unpacking racial bias in health algorithms.
Randomized evaluations can be designed to quantify the extent of an outcome or phenomenon
In a 2021 study funded by J-PAL North America, co-authors Patrick Kline (University of California, Berkeley), Evan Rose (University of Chicago), and Christopher Walters (University of California, Berkeley) aimed to identify acts of discrimination by major employers in the United States. The researchers sent more than 83,000 fictional resumes, randomly varying demographic information, to job postings of Fortune 500 companies to determine whether certain characteristics would lead to different follow-up contact rates. Employers were less likely to contact resumes with distinctively Black names than resumes with distinctively white names.
The researchers found that discrimination was concentrated in a subset of companies and more prevalent in certain sectors: one-fifth of firms accounted for the majority of discrimination documented in the study, bias was more common in customer-facing industries like retail, and companies with decentralized hiring practices were more likely to discriminate.
Randomization was key to generating robust evidence about how the behaviors of a subset of large firms have wide-reaching influence on the racial and gender discrimination job-seekers across the country experience. By ensuring that applicant qualifications and job histories were equal on average across protected demographic groups, researchers knew that any differences they observed in contact rates were due to employers responding differently to job seekers with that demographic information. Further, by including many postings from the same parent companies in their study, the researchers harnessed randomized evaluation to document patterns of systemic discrimination, which by their nature affect more people and are of greater concern to policymakers seeking to combat discrimination.
J-PAL North America has since built off the experiences of this firm discrimination study, as well as the provider race study and others like them, to drive the direction of our work. This has included the development of our current planning period to prioritize racial equity in research and promote equitable and inclusive research practices.
Strong research management is critical to ensuring an effective evaluation
Even the most thoughtfully designed studies require care and effort to be implemented successfully. Dedicated research management focused on implementation, from enrolling participants to conducting randomization to adapting to unexpected circumstances, is an essential catalyst for rigorous studies.
A randomized evaluation by Matthew Notowidigdo (University of Chicago) and Amy Finkelstein (MIT), in partnership with the non-profit Benefits Data Trust (BDT), found that sending informational mailers to eligible older adults nearly doubled enrollment in the Supplemental Nutrition Assistance Program (SNAP) compared to households that didn’t receive any additional information. Pairing the informational mailers with an offer of help compiling required documents and submitting applications tripled SNAP enrollment. The research team was supported by John Tebes (a Ph.D. student at the time, now an assistant professor at Notre Dame and a member of J-PAL’s research network) and Laura Feeney (who was at the time a J-PAL North America research manager and is now our co-executive director) who provided research management and oversight.
With this support from J-PAL, the team ran a successful, high-quality evaluation with trustworthy results that BDT harnessed in decisions about how to maximize their impact. Pauline Abernathy, BDT’s chief strategy officer, noted:
“The J-PAL North America team provided critical support in determining the right research questions and guided the design and implementation of a randomized evaluation of our SNAP outreach and enrollment assistance. The positive results of the evaluation have been critical to BDT’s growth. The evaluation quantified our impact and documented that behaviorally informed outreach from a trusted source is sufficient for some people while others also need application assistance. This underscored the importance of tailoring outreach and assistance to the specific person and helped us scale our impact.”
J-PAL North America’s support of research implementation is now a core function of our work
Over our ten-year history, J-PAL North America has sought to build research management structures and knowledge-sharing practices that strengthen randomized evaluations. Our staff have directly aided the implementation of over 30 randomized evaluations. These include studies supported through Research Management Support, a program that provides short-term, customized assistance to researchers in the J-PAL network during the earliest phases of a new project, as well as studies that are run entirely in-house by J-PAL staff and researchers. Our annual Research Staff Training, run in collaboration with J-PAL Global and Innovations for Poverty Action, trains research staff on how to implement and analyze randomized evaluations, drawing on practical lessons learned from years of work at J-PAL.
In addition, we’ve drawn on our ten years of experience to develop a suite of research resources, in collaboration with J-PAL Global, that share best practices and key considerations for strengthening the design, implementation, and analysis of randomized evaluations—including our most recent health care evaluation toolkit—which garner over 140,000 views annually. Leveraging this suite of resources and services, we look forward to supporting the next decade of policy-relevant research.
In J-PAL North America’s ten-year anniversary blog series, we reflect on some of the most impactful randomized evaluations and bodies of research that our organization has supported over the past decade. We also celebrate the tremendous contributions of our researcher network and the policymakers and practitioners who have made this research possible. Part one kicks off the series with reflections from our scientific leadership. Part three dives into effective collaboration between researchers and practitioners. Part four discusses how credible evidence can identify effective strategies to reduce poverty, regardless of the impact estimate. Part five digs into how policy can be informed at scale.
