Three lessons from J-PAL North America’s ten year convening

2023 marks the Abdul Latif Jameel Poverty Action Lab’s (J-PAL) twentieth year of evidence-based policymaking and J-PAL North America’s tenth. To honor these milestones, J-PAL North America convened a two-day event in September—one of several regional celebrations to honor J-PAL’s anniversaries worldwide. 

J-PAL North America’s anniversary event—What Works To Reduce Poverty: 10 years of research and evidence-based policy to advance economic mobility—celebrated the office’s progress over the past decade by collaborating with a network of incredible partners. We hosted over 100 guests and speakers from across the region, including staff, funders, service providers, government officials, researchers, and peer organizations, without whom J-PAL North America wouldn’t be where it is today. While attendees specialized in various social policy areas, all shared a common goal of using rigorous evidence to reduce poverty.

Our jam-packed agenda showcased promising evidence-based interventions that address different pillars of economic mobility. Multidisciplinary panels discussed scaling evidence-based programs like high-impact tutoring, improving government services through evaluation, generating evidence to address homelessness, and advancing racial equity with rigorous research. Researchers shared evidence on effective programs and strategies, including sectoral employment job training, summer jobs programs, and diversifying the health care workforce. Kim Janey, former acting Mayor of Boston, delivered a keynote address on her recent work at EMPath, and Laura Arnold, Arnold Ventures’ co-founder, and Esther Duflo, J-PAL’s co-founder and director, discussed the role of research in shaping policy. Across this breadth of topics, speakers touched on common themes about how to generate evidence and inform policies effectively and equitably. 

Elevating participant voices through qualitative data and community engagement 

Across several sessions, the importance of community engagement in rigorous research, centering the voices of those with lived experiences, and incorporating qualitative methods into study design from the outset, was reinforced. When discussing her landmark study on physician-patient racial concordance in health care delivery, Marcella Alsan (Harvard) noted that researchers had initially planned to evaluate an intervention advertising flu shots. However, after focus groups with the target population revealed concerns that this approach might actually lower clinic use, the team pivoted to a pop-up clinic to better meet the needs of the community. 

In the Improving Government Services Through Evaluation panel, J-PAL affiliated professor Elizabeth Linos (Harvard) urged fellow economists to view community engagement as an empirical research question to be tested with rigorous evaluation methods: “When does it work best? How do we establish trust? How do we convene groups that are representative of the communities that we’re in?” Kim Desmond, Chief of Race and Equity at the City of San Diego, reminded the audience that, “the moment we say we’re speaking for an entire community, we begin the process of oppressing them. By speaking about someone else's lived experience, we may unintentionally remove the textured authenticity and significance of their reality.

...To fully comprehend the impact of historic trauma, we must ensure governments listen to the community experiencing that trauma.” 

Kim Janey shared how she approaches centering the experiences of program participants in her current role as CEO at EMPath Boston, which supports people experiencing poverty through mentorship, comprehensive support services, and advocacy. “For far too long, these experiences [of living in a shelter] have had a lot of stigma. By centering these individuals and helping them develop their voice by treating them as equal partners in this work, we can learn more about what individuals need and engage in essential narrative changes.” 

Qualitative methods, when incorporated into the study design, allow researchers to answer important questions, such as: what mechanisms are driving the effects of these programs and policies? If replicated in a different context, will the program have the same effects? Is the study asking the right question? 

Community engagement helps ensure that projects reflect the strengths and needs of community members, leading to more equitable measurement, analysis, and policy conclusions.

Building partnerships and sustaining buy-in

Building policy-relevant evidence requires significant coordination and trust. Speakers reflected on the challenges that can arise when different stakeholders, such as researchers, implementers, and policymakers, have different incentives. 

Governments and service providers want to know if their program works or if another program might work in their community, but may be wary of randomized evaluations. Generating buy-in is key to our work at J-PAL North America. Kim Desmond provided an insightful reframing of this challenge, informed by several randomized evaluations under her belt as a government official. “Set a precedent that we can do hard things in government and don’t be scared of RCTs. I like to think of it as ‘who needs our help?' and 'how can we be fiscally responsible when addressing complex challenges by using insights from behavioral science and evidence-based research to provide that help?'”

Policymakers, Laura Arnold noted, often feel pressure to act quickly to address problems in their communities. This desire for speed can feel at odds with the life-cycle of randomized evaluations and can therefore pose a threat to effective evidence-based policymaking. J-PAL affiliated professor Lawrence Katz (Harvard) affirmed this tension when discussing his research in sectoral employment job training, which consistently found positive impacts on worker employment and earnings, with the effects largely driven by workers gaining access to higher-wage and higher-quality jobs. The outcomes of interest in Lawrence’s work are primarily long-term outcomes such as earnings, job stability, and credentialing, meaning that researchers and policymakers looking for quick solutions often don’t know the true effect of a program for many years.

At the same time, both Elizabeth Linos and J-PAL affiliated professor Sara Heller (University of Michigan) reflected that academic incentives push researchers away from conducting replication studies, since replication work is often not viewed as furthering the academic literature. However, Elizabeth urged her colleagues to reframe replications as an effort by different researchers to contribute to a portfolio of evidence, understanding variability in effects in different contexts and minimizing the risk of scaling up ineffective policies. Put differently, a replication study may answer the government’s question about whether something will work while simultaneously expanding the evidence base through program changes, as demonstrated by recent work on summer youth employment programs and tutoring.

Despite these differences, there are myriad opportunities for working together by identifying common goals. Elizabeth concluded that,

“Ultimately, it’s a process of building trust. This is not taught in grad school or built in a day, but it is critical for getting this work done.” 

Racial equity as a priority

Finally, intentionally prioritizing research that addresses racial equity emerged as an urgent and prominent theme throughout the convening. Elizabeth Linos noted that, “Until recently, many papers would end with ‘this is the average effect across the study population and future work will examine heterogeneous effects across subgroups.’ We’ve hit the limit on being able to say that.

...Our work should be adequately powered to detect and understand differences in treatment effects across different racial identities from the beginning of the study.” 

Our tutoring panel noted the racial and ethnic disparities in our education system. Marcella discussed racial inequity in the US health care system. J-PAL affiliated professor Ingrid Gould Ellen (New York University), who is evaluating cash transfer programs for people experiencing homelessness in the San Francisco Bay Area, remarked that, “homelessness is a fundamentally racialized phenomenon. Addressing homelessness is advancing racial equity.” 

Well-designed randomized evaluations go beyond quantifying discrimination and racial inequity. Both of our inaugural evidence champions, Carrie Cihak and Gustavo Bobonis, called on future research to design and evaluate programs and policies aiming to foster racial equity and economic mobility.

In the final panel, J-PAL North America’s Racial Equity Scientific Advisor Damon Jones (University of Chicago) called upon the audience to change the way we think about policy research to consider the historical implications of race in social policy. This means rethinking the questions we’re attempting to answer with randomized evaluations and centering race at the outset of study design discussions, as opposed to retrospectively disaggregating impacts by race at the study's conclusion. Damon is leading the work of J-PAL North America’s Racial Equity Advisory Committee. Over the next decade, J-PAL North America will prioritize research that addresses racial inequities, not just designs that aim to measure them.

Looking forward: The future of J-PAL North America 

Our September convening was an incredible opportunity to reflect on J-PAL North America’s first ten years and begin creating a roadmap for the next ten. Over the next decade, J-PAL North America will focus on generating policy-relevant evidence through randomized evaluations at a deeper level and on a more significant scale. 

We envision launching hundreds of new evaluations, shifting the paradigm of what can be studied with randomized evaluations, prioritizing research that addresses racial equity, and supporting the scale-up of a new generation of evidence-based programs. We aim to partner with governments at all levels to institutionalize a culture of evidence-based policymaking that goes beyond individual studies. 

J-PAL North America has a bright future, thanks in large part to the incredible partners who make our work possible and help further our mission of reducing poverty in North America by ensuring policy is informed by rigorous evidence.