Reflecting back and looking forward: A decade of generating evidence with J-PAL North America

This guest post is authored by Lawrence Katz. Katz is the Elisabeth Allison Professor of Economics at Harvard University, a J-PAL Affiliated Professor,  and a Research Associate of the National Bureau of Economic Research. Katz served as J-PAL North America’s co-scientific director from 2013–2022. 
 

In my career as an economist, I’ve spent decades studying labor market issues and social problems in the United States. I will always view my time as co-scientific director of  J-PAL North America as one of my career highlights. Serving as co-scientific director was a real opportunity to contribute to expanding the infrastructure and networks for building bodies of credible evidence, to inform and improve policymaking, and to enhance the scientific rigor of economics and the broader social sciences.

Bridging a gap: linking economic theory to evidence

Amy Finkelstein first invited me to partner with her in building a North American regional office and the team at J-PAL back in 2013. I was delighted to join Amy in this exciting venture, and we moved forward with the amazing Mary Ann Bates (who later became the Executive Director) as our initial dedicated staff member.  J-PAL had catalyzed many randomized evaluations in the international development space since its launch in 2003, but fewer evaluations by academic researchers were being conducted in the North American geographic context. Our own previous experience with randomized evaluations, specifically Amy’s work on the Oregon Health Insurance Experiment and my work on the Moving to Opportunity housing mobility project, illustrated how randomized evaluations could support and enrich economic theory with rigorous experimental evidence and thereby impact policy and practice in the region. 

At the time, many academic economists viewed running randomized evaluations as an extremely valuable tool to improve the quality of evidence, to test economic theories, and to evaluate policies. However, many economists found randomized evaluations logistically difficult to organize and execute with a lack of infrastructure and support for such projects, even at leading research universities. There was a glaring need for guidance and assistance on setting up data protocols, gaining access to appropriate administrative data sets, managing ethical and practical issues related to human subjects research, and making key experimental design decisions, and assessing statistical power. We wanted the J-PAL North America office to highlight the opportunities that randomized evaluations could provide in generating evidence while playing an active role in building infrastructure to address the challenges academics faced in running them. 

My policy work as chief economist at the US Department of Labor from 1993 to 1994 demonstrated to me that results from transparent randomized evaluations were easy for policymakers to understand and were more convincing than related observational studies. As such, we envisioned a North American team that would build partnerships to design well-powered randomized evaluations that could create rigorous, credible, and clearly interpretable results, and disseminate these results to improve policy.

Growing interest in evidence-based policy 

Since joining J-PAL in 2013, I have seen a flourishing of interest in creating evidence-based policy, particularly for evaluations that address poverty alleviation and tools for increasing economic opportunities and upward mobility. When I initially started working on randomized evaluations in the 1990s, the notion that researchers could connect study participants to national administrative datasets and passively learn about their longer-term outcomes did not appear to be logistically feasible. At that time, randomized evaluations were limited by data availability and the costs of tracking participants through surveys over time. In the last ten years, J-PAL North America has supported well-designed evaluations through the development of research resources and evaluation incubators, where data collection and strategies to integrate administrative datasets are often at the heart of the design process. Growth in researcher capability to link randomized evaluations to large administrative datasets has generated bodies of rigorous evidence in spaces where previously only observational studies had existed. In the mid-2010s, we actually were able to link study participants from the Moving to Opportunity study that began in 1994 to national datasets. In this way, randomized evaluations have come to complement historic observational studies by providing new insight and evidence, ultimately creating a more informed policy landscape.

An example of randomized evaluations building on observational work is research on workplace wellness programs. Bodies of observational research had suggested the positive impacts of workplace wellness programs on both health and productivity outcomes for employees. However there has always been a concern with self-selection—are employees who are healthier and more productive more likely to choose to participate in their workplace wellness programs? Using randomized evaluation, J-PAL affiliated researchers (Damon Jones, David Molitor, and Julian Reif) were able to investigate the impact of workplace wellness programs at the University of Illinois, where randomization eliminated the issue of self-selection. Through using administrative datasets to measure health and employment outcomes, researchers found that the workplace wellness program had effectively no impact on employee clinical health and employment outcomes. Due to the high quality of randomized evidence, they had sufficient statistical power to rule out previous observational studies, whose results reported opposite, positive effects on employees from workplace wellness programs. Given this new, rigorous evidence, the researchers were able to better inform future policy.

In the last decade, there has also been growing policy interest in understanding how individuals interact with policy, beyond traditional economic policy levers of change, and toward questions concerning behavioral science. Previously, most economists focused on only traditional economic factors like incentives, taxes, and transfers when informing policy. Now, economic thinking considers how individuals perceive and interact with the incentives they face, including how to make poverty alleviation tools more accessible to individuals in often stressful situations with limited bandwidth.

Over time, randomized evaluations have evolved to include investigations of behavioral science in the evidence-based policy space. In the housing sector, this is not just subsidizing housing through housing vouchers, but actually helping individuals find housing in a place where they want to live, as in our Creating Moves to Opportunity (CMTO) demonstration in Seattle and King County, Washington State. Currently, J-PAL-affiliated researcher Elizabeth Linos is interested in not just the take up of programs but in the infrastructure for implementing successful public programs and management. 

Building networks for change

Over the past decade, J-PAL North America has developed robust networks of diverse stakeholders—including funders, policymakers, researchers, and practitioners—that support the generation and dissemination of rigorous evidence.

A clear example of the J-PAL North America network in action is the Creating Moves to Opportunity (CMTO) project, which seeks to understand how to facilitate moves to better neighborhoods. In this project, a range of organizations, policymakers, and other experts collaborated to design an innovative housing mobility intervention. The intervention was designed with an embedded randomized evaluation that built on participant perspectives about the key challenges facing low-income households when trying to move to new higher-opportunity neighborhoods with Housing Choice Vouchers. Different areas of expertise contributed to designing an evaluation that remains relevant and individualized to local public housing authorities. These include a qualitative sociological perspective from Stefanie DeLuca and knowledge about how participants use information technology to find housing from Peter Bergman. Partnerships with policy outreach groups like the Center on Budget and Policy Priorities helped to foster bipartisan support for expanding housing mobility programs using evidence generated from the CMTO project and the MTO research.

The CMTO project is a micro-study of how coordinated collaboration between academics from different backgrounds, implementing organizations, and participants can contribute to running a successful randomized evaluation and thereby moving the needle in a related policymaking space.

Excitement for the future

In just ten years, there has been remarkable progress in the evidence-based policymaking space in North America. Moving forward, there are many exciting opportunities to continue to evolve the evidence-building landscape through growing new capabilities and expanding the J-PAL network in the North American context.

The pandemic has shown the potential power of government resources and created excitement for not just evaluating cash transfers, but for thinking about hands-on and comprehensive assistance like AMP Up Boston to improve economic mobility. The research published in leading journals in economics is now much more focused on empirical evidence and the implications of the evidence for economic theory than when I started in the profession four decades ago, and I see opportunities for innovation with rigorous empirical methods by expanding the use of big data and algorithmic and machine learning approaches. There is an opportunity to get better at combining insights from big data with community perspectives and participant voices.

I also am hopeful about the potential for expanding where evidence-based policy and research can take place and by whom. J-PAL North America has started widening and diversifying its network across institutions and researcher backgrounds, yet there are still many opportunities for progress in this space. Diversity, equity, and inclusion efforts like the Economics Transformation Project are doing important work in targeting high schoolers to create a more diverse and inclusive pipeline of scholars in economics. I have been very proud of our staff alumni who go on to hold policy positions or attend graduate school; this is another way we can integrate evidence-based policy into other organizations.

I cannot anticipate what the next ten years will look like at J-PAL North America, but I am excited for its future, given the tremendous, growing network of scholars and an amazing staff.