Ideas to Implementation: Developing a Theory of Change to evaluate mobile medications for opioid use disorders in Pierce County, Washington

In J-PAL North America’s Ideas to Implementation blog series, our 2024 LEVER Evaluation Incubator partners share their experience on the different steps in developing a randomized evaluation of their innovative programs. In part one of the series, we sat down with Pierce County Human Services (PCHS) to discuss evaluation considerations for its mobile medication for opioid use disorder (mobile MOUD) program. Through an engaging and comprehensive session led by J-PAL staff, PCHS developed a Theory of Change to gain an in-depth understanding of the program and facilitate planning for a randomized evaluation.

A Theory of Change is a conceptual framework that maps the causal link between the program being implemented and the final outcome(s). A Theory of Change is used to better understand each step a program must go through, and the assumptions that must hold to do so, before having the intended impact on the target population. It forms the starting point for developing a randomized evaluation by identifying the outcomes that need to be assessed in order to determine if a program is operating as expected and can also be used to gather further insights into a program’s structure and goals. 

To start off, please tell us about the program you are planning to evaluate and your evaluation goals.

Sarah: We are planning to evaluate our mobile medications for opioid use disorder (MOUD) program. After talking to treatment providers and individuals seeking services, we identified a need for more accessible treatment options for people in rural areas of the county that are not just centralized in Tacoma. With this evaluation, we want to see which groups and services are most impacted by having a mobile MOUD program. If there is a large impact, it will help inform future decisions on how to make MOUD services more accessible for rural communities, such as opening a brick and mortar treatment access point.

What was the process like for developing your Theory of Change? Was this concept new to you?

Anika: There were varying levels of expertise with a Theory of Change on our team, so having a session led by J-PAL staff that took us through the essential components of the Theory of Change was very helpful in setting up a base understanding for everyone to build on.

Sarah: I had heard of a Theory of Change, but was unsure of how it applied to research. The session with J-PAL staff was very in-depth—we dug into the program, allowing us to ask ourselves questions like ‘What is this project about?’ ‘How does this program actually look like for the people we are trying to reach?’ We broke our program down into a granular level, giving us an opportunity to think about things we hadn’t thought of before, such as how this program might affect fixed-site MOUD services or how we can establish this program to reduce the stigma associated with accessing these services.

Were there any parts of the Theory of Change that were helpful to think through or that surprised you?

Trevor: It was helpful to think through the threats and assumptions for each step of our Theory of Change. Oftentimes, we build out our plan for an evaluation or program and it can be easy to forget about the parts which might not work as intended. Taking the time to think about those potential issues was particularly helpful, as that is not usually at the top of the list when discussing a program.

Sarah: I agree with Trevor that the threats were important to think through. We tend to be optimistic about our programs, so thinking more deeply about the threats allows us to preemptively prepare for them, fostering a stronger, more robust program. 

I would also add that working through our Theory of Change allowed us to hone in on what aspects of our program were fixed and flexible. For example, we had always thought of using vans to deliver the MOUD program, but when discussing the inputs of our Theory of Change, we realized this input is flexible and the van could be an RV instead. Finding those areas of flexibility and rigidity will also help us down the line when thinking about finding a provider for this program.

Do you see any other use cases for the Theory of Change now that you have developed it?

Anika: Developing our Theory of Change makes me want to implement this in all of our projects, not just the ones where we embed an evaluation. This session allowed our team members to build new skills and can now apply this deliberate and thoughtful process in other areas of work. It can be hard to do, given resource constraints, but this practice can be very valuable in gaining a deeper understanding of our programs. 

As we are in the process of finding a provider for the mobile MOUD services, our Theory of Change will help us write our Request for Proposals and make sure we are including the most important requirements for our providers. 

And since we thought about the threats to our program and evaluation, when these questions come up from leadership, we can show that we have thought about this and have a plan in place to mitigate these threats.

Margo: Currently in Pierce County, mobile treatment programs have a lot of interest and buzz. If we can show that we are being thoughtful about our program by using a Theory of Change—that we are not just jumping into it—it makes our work, and the evidence we do find, more credible and useful to inform future iterations of the program.