Ideas to Implementation: Scoping evaluation possibilities to strengthen Climate Action Plan implementation in San Diego County

In J-PAL North America’s Ideas to Implementation blog series, our 2024 LEVER Evaluation Incubator partners share their experience on the different steps in developing a randomized evaluation of their innovative programs. Through the Evaluation Incubator, the County of San Diego’s Office of Evaluation, Performance, and Analytics (OEPA) and Planning and Development Services (PDS) teams worked together with J-PAL staff and researchers from the University of California, San Diego (UCSD) to scope potential randomized evaluations to strengthen implementation of the County’s extensive Climate Action Plan. We spoke with Ariel Hamburger, PDS Planning Manager, and Courtney Hall, OEPA Principal Data and Research Analyst, to learn more about their experience in this process.

The goal of the scoping exercise was to highlight areas where randomized evaluation can strengthen San Diego County’s Climate Action Plan implementation. The scoping process involved a systematic review of select Climate Action Plan actions to understand available evidence, evaluation feasibility, and implementation priorities. The OEPA and PDS teams brought a mix of subject matter and technical expertise to the Evaluation Incubator. PDS led the development of the Climate Action Plan and is one of six departments that make up the County’s Land Use and Environment Group. OEPA works across County departments to integrate evidence-building capabilities to inform policy, budget, and operational decisions. 

Tell us a little about the goals of the LEVER Evaluation Incubator project. Why did San Diego County initially apply for the program? 

Courtney: This is the second LEVER Evaluation Incubator that the OEPA team has participated in. We found Evaluation Incubator support for our evaluating rental subsidies program to be extremely helpful, and we were interested in collaborating with J-PAL North America again to think more deliberately about opportunities for evaluation within the Climate Action Plan. It was also an opportunity to work more formally and build our relationship with the PDS team. 

What were your primary questions about how to set a learning agenda for the Climate Action Plan at the start of the scoping process? 

Courtney: OEPA is committed to building a culture of evaluation across County operations. Our main goal was to utilize J-PAL’s technical expertise, UCSD’s academic research expertise, and PDS’s expertise in Climate Action Plan implementation to think strategically about areas that would benefit most from evaluation. 

Ariel: The actions within the County’s Climate Action Plan are based on guidance from ICLEI (Local Governments for Sustainability), which identifies actions that are known to reduce greenhouse gas emissions, so we came in with a clear idea of policy areas to prioritize. The piece that was less known was how randomized evaluations can help build out those actions. Climate Action Plan actions really run the gamut and involve program development, policy design, education, and more. Being able to think about evaluation for some of these specific actions is helpful from both a process and program evaluation perspective so we can improve implementation. Building this relationship with the OEPA team has been very valuable.

What were the most valuable aspects of conducting the Climate Action Plan scoping exercise? 

Ariel: PDS did a lot of scoping work when we originally developed the Climate Action Plan. One of the first things we do when developing a new program is look at evidence-based or practice-based research. We also talk to fellow jurisdictions who have run similar programs to the ones we are thinking of developing. Bringing different perspectives to this process through the Evaluation Incubator was very valuable. We approach things in terms of what is possible and feasible for our jurisdiction. Bringing additional brainpower to look at the academic literature on different programs has been really helpful. It makes our research more comprehensive.  

Courtney: OEPA came into the Evaluation Incubator with a lot of evaluation expertise, but not a lot of content knowledge about climate actions. This was a useful process for us to gain a sense of research in the climate space. We are also nerds for spreadsheets, so going through the literature in an organized and systematic way with research tied back to each Climate Action Plan action has been a valuable way to identify evidence gaps. This will hopefully enable us to generate evidence to strengthen the implementation of our Climate Action Plan and help other jurisdictions in the climate field as well. As the Evaluation Incubator comes to a close, having additional content knowledge on climate programming will help us support PDS and continue to engage in evaluation thinking. 

Was there anything that surprised you about the process? 

Ariel: Having an in-person training with J-PAL, UCSD, PDS, and OEPA teams was unexpected and an awesome bonus. It helped solidify the relationship between PDS and OEPA that Courtney mentioned. I don’t think anything necessarily surprised me in terms of research findings from the scoping exercise, but there were other unintended benefits. For example Gordon McCord (UCSD), a researcher in the J-PAL network, connected us with another UCSD professor, and we are intending to work with them to get additional student brainpower on some other Climate Action Plan actions. 

Courtney: Yes, the in-person training was a nice add-on and way of spreading the learning we were doing as a smaller team to other folks within the County. Hopefully this will have a bigger impact beyond this project as we continue to explore evaluation possibilities within the Climate Action Plan. The theory of change workshop during the training was particularly helpful. For instance, mapping out the theory of change (as part of the scoping process) for the Sustainable Operations in Land Stewardship (SOILS) program helped us understand program elements so we can identify what’s reasonable and feasible for randomized evaluation. Ultimately, this process will help us design an evaluation that yields results without disrupting the intent of the program. Having different perspectives and stakeholders in the room also helped illuminate some of the nuances of the SOILS program I had not really understood based on higher-level conversations. As we are designing a randomized evaluation, we can hopefully avoid making assumptions that hinder evaluation planning. 

How are you planning to use this information going forward? 

Ariel: The root cause analysis, which is a component of the theory of change framework, was also very helpful for us. Team members don’t always have time to think about why people have not made a particular behavior change and what the impetus would be to really motivate a targeted change in behavior. Taking the time to do that sort of work will definitely be something we do moving forward. 

Courtney: Working through the Climate Action Plan in such detail to identify opportunities for evaluation will also help us think about evaluation opportunities for years to come. It is nice to think ahead strategically instead of addressing things as they come up. It's a real long-term benefit. 

Ariel: That’s a really good point. Scoping Climate Action Plan actions ahead of time will enable us to not piecemeal things together later because we took a more systematic approach upfront.

What advice would you give to other jurisdictions that are interested in developing learning agendas or building evaluation plans for their own Climate Action Plans or programs?

Courtney: I would advise other jurisdictions to form cross-sector partnerships as soon as possible. We had the luxury of being able to consider the Climate Action Plan as a whole right after it was adopted by the board (in September 2024). That is not necessarily the case for every jurisdiction, but having all the different stakeholders (researchers, San Diego County staff and evaluators) in the same room was a huge benefit and made the scoping process more effective. Even if there were just a subset of these groups in the room, it would not have been as useful as having everyone thinking about this together—each group has unique knowledge that is essential to bring to the conversation. We are hoping this process will really increase the likelihood of conducting successful, meaningful, and useful evaluations. 

Ariel: I would add that,  as much as possible, jurisdictions should involve the community that is going to be affected by the action they are evaluating and hear from them about what they think is important to evaluate. For instance, with the SOILS project, it would have been ideal to have some agriculture folks at the table representing that perspective. So if people have a longer timeline or those relationships already in place, I think that is a really important piece.