BAE Incubator partner series, part two: Compass Family Services on taking evaluation to the next level

The Bay Area Evaluation (BAE) Incubator supports Bay Area service providers with implementing and evaluating cash transfer programs to assess their impact on homelessness and housing stability. In part two of our BAE Incubator partner series, Compass Family Services reflects on their experiences as an Incubator partner. Nicole Moler, Impact Analyst, discusses how the BAE Incubator reinforces Compass’s culture of evidence and tells prospective Incubator partners to “go for it.”

Compass Family Services is a community-based organization serving San Francisco families experiencing or at risk of homelessness. It has supported vulnerable San Franciscans for over 100 years. As a member of our Impact + Learning team, my main goal is to learn; to better understand the impact of our work and to iterate on programs to better serve our families. However, before the BAE Incubator, we didn’t have the expertise to understand our impact with the rigor of a randomized evaluation. 

When we applied to the BAE Incubator last summer, we hoped to take our evaluation capacity to the next level. Our goal was to take all of the information we had—all of the assumptions about what we thought we knew—and really test it using rigorous data. 

Our staff is very excited about the idea of cash transfers, and many believe that it will be the best thing for our families in rapid re-housing. We’ve had the opportunity to give small amounts of funds to some clients before and have seen the difference it can make but haven’t had the data to back up our observations. We are excited to explore randomized evaluation because we want to really know if cash transfers will work at a larger scale. At the same time, Compass has a desire to make sure the evaluation is done carefully, ethically, and rigorously. This is where J-PAL’s expertise comes in.

As a naturally skeptical person, I wasn’t exactly sure what to expect from the BAE Incubator. Moreover, while we were excited to dig deeper into rigorous evaluation, we were also a bit overwhelmed by the prospect of taking on a randomized evaluation given the many moving pieces. Because of these concerns, I have really appreciated that the Incubator has been an intentional, steady process. From the beginning, the focus on simply determining feasibility made me feel comfortable. Since then, each subsequent component has assured me that we are building the right blocks. 

The training we received through the BAE Incubator provided the opportunity to get in the weeds about some of the more technical concepts related to both randomized evaluations and evaluation in general. We even shared some training materials with other programs at Compass, which helped reinforce the data-driven culture of evidence we’re building. We also took advantage of the opportunity to attend J-PAL’s week-long Evaluating Social Programs course, which sparked several additional RCT ideas. (For another time, perhaps!)

The Incubator also includes regular, small group technical assistance sessions. When we meet with J-PAL staff members Amanda and Anisha, we have the chance to dig into questions specific to our design. They’ve helped us build confidence in both the feasibility of our project and our capacity to carry it out. They acknowledge that there can and will be challenges, but also that there are often ways to address them. The technical assistance has helped us think through potential unintended consequences of this project so that we can address them proactively. While there are times I feel as though we can’t do this, the next time we meet, I am reassured that we can. 

Through this engagement, we’ve really been able to sharpen our technical skills. We’re also learning how to ask the right questions to advance our evaluation goals, particularly when working directly with program staff. These tools help us not only think through the randomized evaluation we’re designing but also enhance our other work. We’ve been able to plant evaluation seeds across Compass’s many programs and spark conversations with staff about rigorous data.

For any organizations thinking about applying to the J-PAL North America Housing Stability Evaluation Incubator, I say go for it! They have a deliberate, transparent, and thoughtful process focused on determining feasibility. And if a randomized evaluation of your program is not feasible, that’s okay! They’ll still help you add tools to your toolbox. J-PAL staff provide space for organizations to test out ideas and ask questions. Don’t be afraid: trust the process.

Part one of this blog series explores existing evidence on cash transfers and highlights the need for further evaluation in the context of homelessness reduction and prevention in the United States. In parts three and four, Abode Services and Hamilton Families reflects on their experiences as an Incubator partner

The Homelessness and Housing Stability team at J-PAL North America will be accepting Letters of Interest for the Housing Stability Evaluation Incubator from August 1–October 17, 2022. Public housing authorities, continuums of care, nonprofit organizations, and other service providers working to reduce and prevent homelessness are encouraged to apply.