October 2019 North America Newsletter
Good afternoon,
What do you do when your data explodes? This month we are thrilled to share resources that build on what our network has learned (like triple-shrink-wrapping data samples) running randomized evaluations and hope you find them useful.
When implementing evaluations, challenges arise that can be impossible to predict. Challenges like when a FedEx truck carrying data for the Oregon Health Insurance Experiment quite literally blew up (pictured left). Our Evaluation Toolkit has resources for every stage of running a randomized evaluation, synthesizing the lessons our network has learned. I hope what we have learned from our successes, our mistakes, and the unexpected across hundreds of evaluations will be useful for you.
Our policy insight featuring lessons from two randomized evaluations on workplace wellness explores why such programs have had limited impacts in two different contexts. I appreciate how the synthesis highlights the value of randomized evaluations in health care policy, given that observational studies would have overstated each program’s impact.
Our new case study—and this Twitter thread—for the National Center outlines how to use the generalizability framework to assess whether an evidence-based community health worker intervention would work in a new setting. This case study builds on the Generalizability Puzzle article Rachel Glennerster (former J-PAL Executive Director, now Chief Economist at the UK’s Department for International Development) and I wrote.
Mary Ann Bates
Executive Director, J-PAL North America
New Evaluation Toolkit guides research teams in launching randomized evaluations
J-PAL North America recently launched an Evaluation Toolkit on how to design and implement an impact evaluation. This suite of resources is drawn from the experience of J-PAL staff and affiliated researchers who have run more than 80 randomized evaluations in North America. With more than 14 (and counting) resources, like guides for using administrative data and conducting power calculations, we hope this toolkit can help you employ this complex but powerful evaluation methodology to answer critical social policy questions.
The limited impact of workplace wellness programs on health and employment
The US workplace wellness industry’s revenue has tripled in size since 2010 to $8 billion, partly bolstered by policies such as the Patient Protection and Affordable Care Act that encouraged firms to adopt wellness programs. Through J-PAL North America’s Health Care Delivery Initiative, we funded two randomized evaluations of different workplace wellness programs to better understand their impact on employees’ health habits. Our new policy insight synthesizes the findings from these two evaluations. We highlight that workplace wellness programs had limited impact on health habits and no impact on employees’ health and employment-related outcomes, contrary to previous observational studies. This could be because the programs are not effective or because workplace wellness programs are not effectively targeting the types of employees who stand to benefit from Workplace Wellness programs, such as those with poor health habits.
Case study: will an evidence-based program work here?
In the summer of 2017, J-PAL published The Generalizability Puzzle in the Stanford Social Innovation Review, which outlined a practical approach for using evidence to help decisionmakers answer a critical question: can a given program be replicated or adapted to work in a new setting? This August, J-PAL staff published a case study building on the generalizability framework for The National Center for Complex Health and Social Needs. The resource outlines how to use the generalizability framework to assess whether an evidence-based community health worker model might work in an outpatient care center in rural Indiana. While this example refers to a health intervention, we hope this framework can help inform decision-making across many policy areas from education to criminal justice.