Evaluating effectiveness and equity in climate mitigation and transportation policy: A webinar recap

In December 2023, J-PAL North America’s Leveraging Evaluation and Evidence for Equitable Recovery (LEVER) program hosted a webinar in collaboration with Results for America highlighting how state and local jurisdictions can use randomized evaluation to accelerate climate action and tap into federal funding opportunities. The webinar, “Building Better Policy: Evaluating Effectiveness and Equity in Climate Mitigation and Transportation Programs,” featured a panel discussion led by Erin Graeber, J-PAL North America Environment, Energy and Climate Change Sector Lead. Panelists included Louise Geraghty (J-PAL North America ), Chelsea Powell (Results for America), Peter Christensen (University of Illinois Urbana-Champaign), Carrie Cihak (King County, WA), Terry Sullivan (King County, WA), Ryan Dooley (King County, WA), and Rachel Brown (King County, WA). 

A new frontier for climate mitigation policy 

Chelsea Powell, Associate Director of Local Practice at Results for America, kicked off the panel, speaking to the urgent need for effective climate mitigation and transportation decarbonization policies and outlining federal funding trends. She emphasized that the recent passage of the Inflation Reduction Act and Bipartisan Infrastructure Law provide “a momentous opportunity to introduce equitable decarbonization and mobility interventions.” With these new funding opportunities, however, jurisdictions face new challenges around defining policy priorities and allocating resources to the most effective climate mitigation and transportation decarbonization programs. Chelsea highlighted that there is not enough research available on what works in climate policy and that state and local jurisdictions have a key role in generating evidence in this space.

Terry Sullivan, King County Climate and Energy Program Manager, echoed these difficulties, explaining that local governments face “challenges in effectively leveraging this funding,” and need support in prioritizing policies to advance multiple climate goals at the same time.

Rigorous evaluation to support innovative policy

At both federal and local levels, governments are setting ambitious climate goals that require transformational changes to transportation, energy, and economic systems. Meeting these goals will require jurisdictions to work in new ways, experimenting with strategies that may not yet have an evidence base. Carrie Cihak, King County Evidence and Impact Officer, called for more shared learning among jurisdictions to build a knowledge base and ultimately drive more rapid change. “We are often measuring—or just estimating—inputs and outputs rather than really rigorously understanding whether our actions are making a difference in our communities,” she said, emphasizing the need to conduct rigorous, randomized evaluations in order to understand the causal impact of programs and whether they are making a difference in King County communities.  

J-PAL affiliated researcher Peter Christensen (University of Illinois Urbana-Champaign), noted that many researchers focused in the climate arena are interested in evaluating the underlying mechanisms—often behavioral mechanisms—of climate programs that aren't well understood. Peter emphasized that better understanding the human behavior aspects of climate and transportation interventions, such as how individuals take-up or engage with programs, “could potentially have a very large impact on emissions reductions, poverty alleviation, or equity impacts of climate programs.”

Rachel Brown, Research Associate at King County Metro, added that incorporating program evaluation early on can help both clarify program mechanisms and ensure policies are being implemented equitably. She encouraged governments to ask more questions about who is bearing the burden of new policies, “who are we asking to make changes to their lives and live more sustainably?” 

Evaluation in action: King County, Washington 

Evaluation can carry negative connotations for many government offices. Common concerns are that it is resource intensive, too retrospective to be practical, and that work products will be negatively judged. King County teams are working to “flip the script” by pairing co-equal teams of researchers and program staff to integrate evaluation into existing and new work. Panelists spoke to their experience working with J-PAL affiliated researchers on two programs:

Carrie noted that integrating evaluation activities has enabled program teams to get real-time impact information, better understand what is and isn’t working, and adjust policy roll out accordingly. Rachel shared her experience working with J-PAL researchers to define target metrics and develop data collection instruments. In response to concerns government staff may have about their work being scrutinized, Rachel encouraged framing evaluation as part of a “continuous learning process” and emphasized that, regardless of evaluation findings, “the learning is in and of itself a good outcome.”

Ryan Dooly, King County Transportation Planner, spoke to his experience working with researchers while developing a program. He emphasized that governments do not need fully formed ideas to benefit from researcher input and that researchers can help teams iterate on a program theory of change and define the right outcome metrics. He noted that “being able to work with [researchers] to ensure that our program was well established by the time we started evaluating” was extremely helpful. 

Carrie explained that integrating randomized evaluation has required a mindset shift from tracking outputs to causal impact. She urged government teams to get clear and specific about the intended impact of a policy and let that drive the research design. Carrie concluded by calling on government staff to not underestimate the value of their experience, “as human beings, we’re all researchers, and your expertise is necessary and valid for creating a successful research partnership.”  

Collaboration, capacity building, and knowledge sharing opportunities 

State and local jurisdictions are at the forefront of climate action and are uniquely positioned to generate evidence for effective and equitable decarbonization policies. Peter spoke to the valuable perspective governments provide, “for state and local jurisdictions that are thinking about doing something completely different, of experimenting, really consider your leadership impact. As researchers, we are interested in learning about what is best with you and helping you, through rigorous evaluation, amplify the message.” 

Through the LEVER program, J-PAL North America and Results for America provide capacity building tools for governments to learn more about building evaluation into program design and using rigorous evaluation to tap into funding applications.  

Upcoming LEVER Opportunities:

• Virtual Convening: Pathways to Implementing Evaluations: On February 15, 2024, J-PAL North America  and Results for America will provide a space where government leaders and decision makers can connect with other government agencies to share their experiences and challenges in their evaluation journey and building effective government-research partnerships. Register today.
• Evaluation Incubator: A call for applications will open February 6 for government agencies to receive in-depth support to turn policy-relevant research questions into well-designed randomized evaluations. Selected applicants will receive funding, technical support (including project management, research support, and training as needed), and connections with researchers in the J-PAL network.
• Evaluation capacity building opportunities: Visit the LEVER webpage to learn more about a range of courses, trainings, and workshops tailored to the unique needs of governments with varying levels of familiarity with data and evaluation.
• LEVER email updates: Fill out the LEVER interest form to stay up to date on opportunities and receive additional information about LEVER programming.