Inside an evaluation: The Creating Moves to Opportunity partnership in Seattle and King County

Preliminary results from the Creating Moves to Opportunity (CMTO) project are fresh off the press, and there’s a lot to be excited about. In this post, we dig deeper into the planning and research management activities that underlie the success of the CMTO project to date, highlighting the role that J-PAL North America and other partners have played on this project. We hope these reflections can provide helpful guidance to researchers and their partners when collaborating on research projects to come.

Housing mobility programs, which help low-income families overcome barriers to moving to higher-opportunity areas, may be a promising strategy for reducing residential segregation and promoting economic mobility, as outlined in a recent paper from J-PAL-affiliated researchers.

The paper summarizes preliminary findings from the ongoing Creating Moves to Opportunity (CMTO) study in Seattle and King County, Washington. To date, researchers have found that the housing mobility services offered to program participants have increased the share of families who move to higher-opportunity neighborhoods by 40 percentage points relative to a rate of 14 percent among families with standard services (a 286 percent increase).

It’s hard to overstate the implications of these results. The findings suggest that barriers to housing choice are a major driver of housing segregation, dispelling the belief among some that low-income families with Housing Choice Vouchers are living in low-opportunity neighborhoods principally because of preferences for living in such neighborhoods. The study results have already attracted interest from local housing practitioners, who are now considering implementing similar programs in their own jurisdictions.

Part of the power of these findings comes from the relative simplicity and rigor of the research methodology used: a randomized evaluation. The project randomly assigned families who applied for a housing voucher to either a mobility services group or a group receiving standard services. As such, any difference in housing outcomes between the two groups can be attributed to the mobility services program.

While randomized evaluations can paint a straightforward picture of a program’s impact, the much more challenging (and often untold) story is in designing and implementing the evaluation itself. Running a randomized evaluation requires a great deal of effort, and as such necessitates having an equal partnership between the implementing partners and the research team, healthy communication among all those involved, careful monitoring of study implementation, and a strong commitment to learning throughout the research process.

Setting expectations for an equal partnership

From its initial conception, CMTO was built from the aligned interests of researchers and housing practitioners. Inspired by research led by J-PAL affiliates Raj Chetty, Nathaniel Hendren, and Lawrence Katz (Harvard University, J-PAL) demonstrating that moving to lower-poverty neighborhoods significantly increased the college attendance rate and future earnings of young children, public housing authority officials partnered with a team of researchers to explore how they might be able to act on the results. The results suggested two complementary policy approaches:

1. invest in strategies to improve neighborhoods with lower measures of opportunity (place-based initiatives); and
2. help families with younger children move to higher-opportunity neighborhoods (mobility-based initiatives)

CMTO is focused on the second of those two approaches: learning if mobility services can help families with vouchers move to areas of higher opportunity. However, the Seattle and King County Housing Authorities are currently working on other place-based initiatives to help invest in neighborhoods with lower measures of opportunity.

The Seattle and King County Housing Authorities were the first housing authorities among the larger CMTO group to take those initial conversations and turn them into an active research project, dedicating significant staff time and resources to developing a randomized evaluation of a new mobility program.

The academic team was also interested in how measuring the impact of a housing mobility program on housing mobility might be able to contribute to the larger academic literature on the drivers of housing segregation. Helping families overcome barriers to housing choice, as through the CMTO services, could produce valuable insight on the degree to which barriers to finding housing in higher-opportunity areas prevent families from exercising true choice.

As such, the study was designed in a way that would answer both important policy questions (whether or not to scale up a new experimental program) and important academic questions (what is driving residential segregation in the US), with the Seattle and King County Housing Authorities and the research team playing equal roles shaping the design of the study.

The CMTO study in Seattle and King County demonstrates the value of centering research partnerships on aligned interests and common understanding. See Assessing viability and building relationships in the J-PAL North America Evaluation Toolkit for additional lessons on building a successful research partnership.

Establishing clearly defined roles

An often-overlooked component of running a research study is the project management activities required to keep a project on track. The CMTO study in Seattle and King County is an incredibly collaborative project, involving stakeholders from Seattle Housing Authority, King County Housing Authority, the research team, the non-profit service provider, J-PAL North America, MDRC, MEF Associates, and Opportunity Insights.

Each partner on the project brings a unique expertise, with perspectives from sociology, Housing Choice Voucher program operations, housing policy, social service provision, microeconomics, communications, program evaluation, and research management represented on the team. Many other research projects involve a similar number of stakeholders, but what has made this project particularly unique has been the degree of communication and coordination across the different partners, which has enabled the team to maximize the contributions of each perspective.

The sheer number of different stakeholders and the many moving parts across different working groups and partner organizations necessitated having a central intermediary who could be tuned into the constraints and needs of each organization. J-PAL North America has filled this role on the Seattle and King County project, with project management led by Kristen Watkins.

Kristen plays a key role in clearly delineating who is responsible for what, who should be consulted for certain kinds of questions, and ensuring that all major project activities are on track according to the project timeline. The project management role also involves a good deal of communicating across stakeholders, and having someone with a foot in both the policy door and the academic door can be helpful for translating between researchers and practitioners.

See Formalize research partnership and establish roles and expectations in the J-PAL North America Evaluation Toolkit for additional lessons on project management of a randomized evaluation.

Implementing and monitoring a research study

When running a randomized evaluation, another key to success is thinking carefully about how to set up the study in a rigorous, ethical, and efficient manner. This often involves navigating programmatic complexities when considering how to embed study enrollment, random assignment, data collection, and monitoring into the normal operations of the program being evaluated.

MDRC has led the research management of this project, working with the Housing Authorities to draft study protocols, build data systems and processes, and inform key decisions for designing the evaluation. MDRC is also carefully monitoring the study to ensure that study enrollment operates smoothly and that all study procedures are adhered to.

See Define intake and consent process and Design and iterate implementation strategy in the J-PAL North America Evaluation Toolkit for additional lessons on implementing a randomized evaluation.

The value of formative and qualitative evaluation

Although preliminary results were only just released this week, learning has been central to CMTO from the initial conception of the program.

In embarking on this project, the Seattle and King County Housing Authorities launched a completely new mobility program. To do so, the team relied heavily on feedback from voucher-holding families, frontline staff, external researchers, landlords, and existing mobility programs from across the country, ensuring that the new program would be designed to fit the needs of families in  particularly tight housing markets such as the Seattle metropolitan area. A central part of the formative design phase was running a pilot of the new programs to fine-tune the proposed interventions and test the recruitment and enrollment process for families.

The study is also relatively unique among economics working papers in that it includes qualitative components in its analysis, led by Stefanie DeLuca, a sociologist at Johns Hopkins. Professor DeLuca and her team conducted in-depth interviews with a randomly selected sub-sample of 114 families in both the treatment group and control group. The interviews were critically important in adding much-needed context to the “black box” of the quantitative results, helping to explain why the intervention worked, not just if it worked. Overall, the interviews suggested that what made the CMTO program successful was its ability to customize services to the needs and circumstances of each family. 

Including qualitative components in a randomized evaluation also helped inform the research design. In February 2019, the CMTO project team drew on some initial interviews with participating families and implementation research, led by MDRC, that collected feedback from the staff who run the program and interact with families. The team then used those learnings to design the second phase of the study, which will test new iterations of the existing services to determine whether different approaches to service delivery will be more or less effective in supporting moves to areas of higher opportunity. Insights from this second phase of the research are critical for identifying policy solutions that are ultimately effective and scalable.  

Looking to the future

The Seattle and King County study originated out of a desire to act on the findings of prior research. In turn, the research team and the public housing authorities are now working to act on the findings of this study. For example, the study results stand to inform the roll-out of the federal Housing Voucher Mobility Demonstration Program, which aims to expand mobility programs across the country.

And, of course, the research in Seattle and King County is still ongoing. The next phase of the study was launched in July 2019 and will continue through 2020.  Additionally, the project will evaluate whether families persist in their new neighborhoods.

With a shared commitment to collaboration and learning, the team looks forward to how CMTO results can continue to inform and influence the housing mobility landscape in the future.

To learn more about CMTO or partner with us, please visit our project website or contact CMTO project manager Jacob Binder.

This is the first post of a two-part blog series on the CMTO study. In the second post, we sat down with Lawrence Katz—Co-Scientific Director of J-PAL North America and one of the authors on the CMTO study—to collect his reflections on the preliminary results, how this study builds upon his previous research, and how these and future results may inform housing policy moving forward.