Translating research into action in 2019: J-PAL North America’s year in review

As 2019 comes to a close, we reflect back on key accomplishments, lessons learned, and the partnerships that have made it possible to advance J-PAL North America’s mission of reducing poverty in the region. From compelling new evidence and reviews of over 160 rigorous evaluations, to the Evaluation Toolkit and building the capacity of policymakers, all the accomplishments we have spotlighted have been made possible through our partnership with the broader evidence-based policymaking community. Together with our committed network of state and local governments, implementing partners, funders, researchers, and study participants—without whom none of this would be possible—we have continued working to advance the evidence-based policymaking movement. And we’ve learned a lot along the way.

Photo: Shutterstock.com

Catalyzing rigorous research

At J-PAL North America, we are passionate about tackling critical questions on poverty alleviation through rigorous research. We want to know what works and why and focus on the most policy-relevant questions. This year, we launched twelve new randomized evaluations and nine pilot studies. In addition to catalyzing new studies, completed studies by J-PAL affiliates produced compelling results that shed light on some of today’s biggest policy questions. Several completed studies are highlighted below.

Research suggests that growing up in high-poverty neighborhoods negatively affects a number of important life outcomes. This August, preliminary results from a rigorous evaluation of the Creating Moves to Opportunity (CMTO) project found that access to housing mobility services increased the share of families who moved to higher-opportunity neighborhoods by 40 percentage points. These results suggest that improving housing choice may be an effective strategy to reduce housing segregation.

To evaluate whether workplace wellness programs, which have dramatically increased in popularity in recent years, actually promote healthy behaviors and reduce medical spending, J-PAL affiliated researchers conducted randomized evaluations of two programs in the United States. Results showed that the programs improved a small subset of health behaviors, but had no impact on health, health care spending and utilization, or employment-related outcomes. This was possibly due to the fact that employees most likely to benefit (those with poor health habits) participated at a lower rate.

Health care spending accounts for almost one-fifth of the US economy. In 2014, Medicare spent over $4 billion on high-cost diagnostic imaging alone—research suggests that up to 30 percent were unnecessary. To reduce inappropriate imaging, beginning in January 2021, Medicare will no longer reimburse providers for high-cost scans unless they are ordered using a clinical decision support (CDS) system. With the impending mandate in mind, a randomized evaluation was conducted on the impact of a CDS system on the number of high-cost medical imaging orders in Wisconsin and Illinois. The CDS system reduced the number of scans targeted by the software, but did not impact the total number of low- or high-cost scans.

To address the many barriers to completing community college, Chicago nonprofit, One Million Degrees (OMD), provided wraparound support to low-income community college students using a model that targets academic, professional, personal, and financial needs of students. Early findings from a randomized evaluation found that program participants experienced a 25 percent increase in full-time enrollment, a 35 percent increase in persistence, and a 47 percent increase in full-time persistence. The effects of OMD predict positive program effects on degree attainment.

Photo: Shutterstock.com

In 2019, J-PAL North America also focused on launching a body of research in new critical areas, including housing and homelessness, crime and violence prevention, and work of the future.

To expand the base of rigorous evidence on strategies that reduce and prevent homelessness and promote housing stability, we are supporting randomized evaluations on the impact of several interventions, including targeted financial assistance, eviction diversion, and rapid re-housing, on homelessness. Similarly, our crime and violence prevention work supports randomized evaluations of strategies that aim to increase the equity and functionality of the criminal justice system. One recent study assesses the impact of various types of text message reminders on rates of failure to appear in court.

Technology is rapidly changing the nature of work in North America and has potentially profound implications for workers and the labor market. Launched in April of this year, the Work of the Future Initiative supports randomized evaluations of strategies and innovations that address these changes in a way that benefits workers and the overall economy. A research agenda identifies four key areas where research is needed, including programs designed to reskill workers, strategies to improve traditional and non-traditional educational institutions, interventions to facilitate worker transitions between occupations and industries, and more.

Photo: J-PAL

Translating Research into Action

In addition to catalyzing new research, J-PAL North America works to ensure rigorous research conducted by our affiliated network helps policymakers make evidence-based decisions. We do this by synthesizing research findings, identifying policy insights that decision-makers can apply, and promoting promising programs for scale. Our staff members have also traveled to over ten states and presented at over eight conferences to share evidence with leading decision-makers.

Recent years have seen widespread buzz and excitement around the potential for technology to help students learn. But which programs actually deliver? To shed light on the effectiveness of various education technology interventions, J-PAL North America published the Education Technology Evidence Review, which summarized key findings and policy lessons from 126 rigorous studies that assessed the impact of technology-based nudges, social psychology interventions, online and in-person instruction, and educational software, and more. Key lessons include that educational software programs designed to help students develop particular skills have shown enormous promise in improving learning outcomes, particularly in math, while initiatives that expand access to computers and internet alone generally do not improve K-12th grade students’ grades and test scores.

More than 500,000 people experience homelessness on a given night and 1.4 million people pass through emergency shelters in a given year in the United States. The last several decades have seen rigorous evaluation of interventions like the Housing First approach resulting in significant policy changes. To draw key policy lessons from this body of research, J-PAL North America published an Evidence Review on Reducing and Preventing Homelessness that highlights key findings from forty rigorous evaluations on policies that seek to provide housing stability and identifies important questions for future research to address. Key insights include that comprehensive interventions providing a range of assistance can prevent homelessness among families at risk of losing their homes, permanent supportive housing increases housing stability for individuals with severe mental illness and veterans experiencing homelessness, and Housing Choice Vouchers help low-income families avoid homelessness and maintain housing stability.

With the goal of translating research into policy action, J-PAL North America’s Evidence to Scale team also systematically reviewed our evidence base and identified three programs to prioritize for scale-up, with the aim of improving many more lives in the years to come.

Photo: J-PAL

Building Researcher and Practitioner Research Capacity

One of J-PAL North America’s core activities is capacity building—equipping researchers and policymakers with the tools to be better producers and users of evidence. In 2019, our team developed research resources, including the Evaluation Toolkit, and built the capacity of state and local governments through multiple trainings on rigorous evaluations of programs and policies.

Drawing from our own lessons learned and guidance from researchers and research organizations, J-PAL North America developed an Evaluation Toolkit, which provides practical guidance for designing, implementing, and communicating about evaluations. The Toolkit provides information relevant to both those who are only beginning to think about conducting a randomized evaluation to those that are already in the weeds of one. In addition to written research resources, J-PAL North America expanded our Short-Term Research Management (STReaM) program: this year we supported six projects.

This year, J-PAL North America worked closely with state and local leaders to build evaluation capacity. In partnership with the Pew-MacArthur Results First Initiative, J-PAL North America staff members conducted a training series for state and local policymakers interested in rigorously evaluating their programs and policies. Topics covered included concepts related to measuring impact and practical design considerations that go into randomized evaluations of programs and policies. To strengthen partnerships and share lessons on communicating research results, the State and Local Innovation Initiative also hosted the Building Evidence, Advancing Policy, Impacting Lives Convening, which brought together over seventy government leaders, researchers, and funders. Eight state and local governments presented on randomized evaluations currently under development, received feedback from research and policy experts on their research plans, and discussed ways to communicate research results to produce meaningful policy impact.

As illustrated by the policy-relevant studies in The Lessons of Administrative Data Brief, administrative data can help reduce research costs, offer opportunities for long-term follow-up on intervention impacts, and improve research accuracy. In 2019, J-PAL North America provided practical guidance to researchers and policymakers interested in using administrative data, through resources like our Administrative Data Catalog and Evaluation Toolkit. This work complements the Innovations in Data and Experiments for Action (IDEA) Initiative, an effort led by J-PAL Global to increase the use of administrative data in research.

Thank you for your partnership in 2019 and for helping to make this work possible. Together, we are improving the lives of those who experience poverty in North America, and we look forward to continuing this important work together in 2020.