January 2024 North America Newsletter
For both researchers and implementing partners using a randomized evaluation to evaluate their interventions, finding that a program or policy had positive effects is viewed as a success. In an ideal world, an effective policy is re-tested across different contexts with continual positive results and then scaled to reach even more people. Conversely, finding that an intervention did not have the intended effects on the outcomes we measure–known as a null result–can often be seen as a failure. However, null results and the lessons they teach us can be just as informative as positive results.
In a new piece in Health Affairs Forefront, J-PAL North America staff and I discuss the importance of embracing “failure” in evidence-based policymaking, drawing on examples from evaluations of health care delivery interventions. We demonstrate that null results push us to challenge our assumptions about programs and policies, spark ongoing, iterative progress, and ultimately help leaders make informed choices about what policies and programs to invest in.
In this month’s newsletter, we spotlight new research results that further underscore the role of null findings in scientific progress: a follow-up on a randomized evaluation of the Camden Coalition’s care management program and an evaluation of a “food-as-medicine” program. Each demonstrates the importance of testing various theories behind a null result—such as whether the finding was due to flawed implementation or an unsound theory of change. These examples also demonstrate how randomized evaluations can credibly and accurately identify the true effect of a program.
Ultimately, we at J-PAL North America aim to empower academics and policymakers alike to continue their investment in ongoing learning and be open to what results—positive, negative, or null—can teach us about how best to serve our communities. I invite you to read our article for more details and reach out to me with your thoughts and questions on the value of null findings in evidence-based policymaking.
Matt Notowidigdo
Co-Scientific Director, J-PAL North America
On the blog: Researching racial equity and administrative data bias
Administrative data (data already collected by organizations for operational purposes, such as educational records and crime data) has transformed the capability of researchers to measure the impact of programs and policies. However, data and measurement bias can influence the validity of research results. A new post on the J-PAL blog discusses how subjectivity and structural inequity cause bias in administrative data, provides examples in educational contexts, and identifies potential solutions to address data and measurement bias.
J-PAL North America joins the Learning Engineering Virtual Institute
In October 2023, J-PAL North America was thrilled to join the Learning Engineering Virtual Institute (LEVI). LEVI’s mission is to double the rate of middle school math learning within five years, with a focus on students from low-income backgrounds. As the LEVI’s Evaluation Hub, J-PAL North America is supporting seven organizations to rigorously evaluate learning products that harness the potential of AI and machine learning. Learn more about this multiyear collaboration and the LEVI Evaluation Hub on the J-PAL blog.
New research results: A ‘food-as-medicine’ program did not impact diabetes management
Food-as-medicine programs, where health care providers supply healthy food or meals to patients with diet-related chronic diseases, aim to bolster food security and health. In a recent article in JAMA Internal Medicine, researchers in the J-PAL network presented evaluation results from one such program, which provided nutritious food and education to low-income patients with diabetes. Researchers found patients in both the intervention and comparison groups saw improved blood sugar levels over time, but there was no additional improvement for those in the program, despite high program engagement. In a StatNews opinion piece, the researchers further unpack the results and call for more research on strategies to combat food insecurity and improve health.
Featured Evaluation Summary
New research results: Health care hotspotting in the United States
In a 2020 study, researchers evaluated the impact of a care management program for high-need, high-cost patients. They found no impact on the rate of six-month hospital readmissions. In a newly published follow-up to the original study, in which the researchers paired Medicaid claims data with a subset of the original population, researchers found that the program did increase outpatient care, a key component of care coordination. These results suggest that care coordination programs on their own, even when implemented as intended, are not enough to reduce hospitalizations for patients with complex needs.
Featured Research Resource
Upcoming LEVER activities for state and local governments
Are you part of a government agency interested in expanding your data and evidence use? J-PAL North America and Results for America’s LEVER program provides customized support for state and local governments to create a culture of evidence and data use in decision-making, ensuring programs are best-positioned to advance economic mobility and racial equity. Registration is now open for our February 15 convening to all government officials and jurisdictions who are interested in learning from peer government agencies that have previously implemented rigorous evaluations. Stay tuned for the upcoming Evaluation Incubator and Evaluating Social Programs opportunities to open on February 6.