Bringing together policymakers and practitioners to shift cultural norms toward evaluation

Implementing randomized evaluations and evidence-based policy often involves a culture shift—from fear of failure to embrace of learning, from program-centric to person-centric, and from services-oriented to outcomes-oriented.

As Linda Gibbs, a principal at Bloomberg Associates and senior fellow at Results for America, shared with an audience representing government, social services, and academia, “When you do this, you have to know some things aren’t going to work. You’ve got to acknowledge it and embrace it as part of the learning.”

With an emphasis on the risks and rewards of evidence-based policy, Gibbs’ keynote set the stage for J-PAL North America’s State and Local Innovation Initiative Year Two Convening.

The two-day event in Santa Clara County, California brought together researchers, service providers, and policymakers representing nineteen different jurisdictions. The convening created a space for this cohort and others facing complex issues, especially homelessness, to share knowledge, troubleshoot, and collectively brainstorm about the potential for evaluation and data-driven policy.

Shifting culture is a challenge worth pursuing to better serve constituents and program participants.

The need for culture shifts emerged as a recurrent theme throughout the event. During a panel on day one, Michael Wilkening, undersecretary at the California Health and Human Services Agency, noted that when it comes to evidence-based policymaking, building the right technology to facilitate the use of data and evaluation is relatively simple.

The trickier part is changing mindsets on why and how we should use evidence—for example, by considering outcomes beyond a particular agency.

“We should be thinking about person-centered, not program-centered.… We shouldn’t be siloed in our thinking because the people we’re serving aren’t siloed,” Wilkening explained.

Like many efforts to use data, coordinating services across agencies and data systems can pose a challenge. But as Joe Simitian, Santa Clara County supervisor for District Five, reminded us in his welcome address, “We’ve got to be rigorous if we want to prove we really care about the people we say we care about.”

Building a culture of evaluation can create potential for impact beyond a single study.

The convening highlighted how government-research partnerships are helping jurisdictions develop a culture where rigorous evaluation becomes the norm rather than the exception.

Anita Ravishankar, a fellow at The Lab @ DC, presented alongside Commander Ralph Ennis of the Metropolitan Police Department in Washington DC about a completed randomized evaluation of the impact of body-worn cameras. Commander Ennis explained how the evaluation is poised to have a lasting impact on the police department’s approach to evaluation of its programs.

“The study opened the door to us using this methodology in almost everything we do now. If we’re going to spend money on something, we’re going to look to see if it will be well spent or not,” said Commander Ennis.

The study of body-worn cameras also contributed to a culture of transparent evaluation by engaging the broader community through school visits, public hearings, and an interactive website.

Cultures of evaluation can help tackle seemingly intractable social issues such as homelessness.

Day two of the convening turned its focus to homelessness, but the theme of cultural shifts towards the use of evidence continued. In a panel on Santa Clara County’s efforts to measure the costs of and find solutions to homelessness, Jennifer Loving, chief executive officer of Destination: Home, reflected on a monumental shift: “For thirty years we’ve looked at homelessness as a charity model. […] Not only is that wrong, but it’s cost us a billion dollars. We can create a culture where people say yes to finding the solution, but we can only do it if we can prove it.”

Loving was referencing the largest, most comprehensive study on the costs of homelessness in the US, undertaken in response to local policymakers’ demand to understand spending on homelessness across multiple systems in Santa Clara County.

The study, which found that the county spent $3 billion to serve people experiencing homelessness from 2007 to 2012, has served as both a catalyst and blueprint for a shift toward data-driven policy over several years.

Building on this momentum, Santa Clara County partnered with Abode Services and the University of California San Francisco in 2015 to develop California’s first Pay for Success project. The ongoing multi-year randomized evaluation will examine the impact of permanent housing and supportive services for people experiencing chronic homelessness.

Santa Clara County is now partnering with the University of Notre Dame’s Wilson Sheehan Lab for Economic Opportunity and J-PAL North America to design a randomized evaluation of rapid rehousing for single adults experiencing homelessness.

In addition to Santa Clara, Baltimore, MD and King County, WA are partnering with J-PAL North America to develop evaluations to build evidence on tackling homelessness.

J-PAL affiliate Bill Evans, professor of economics at the University of Notre Dame and co-founder of the Wilson Sheehan Lab for Economic Opportunity, summarized the current evidence on reducing and preventing homelessness in his keynote, “State of the Streets.” He also highlighted critical open questions, including challenges these jurisdictions are working to address.

Bill Evans summarizes the rigorous evidence on addressing homelessness in his talk, "State of the Streets."

Evaluation can point toward continuous improvement rather than judgment.

Looking forward to continuing to build a culture of evaluation, Irena Asmundson, program budget manager at the California Department of Finance, captured a guiding principle: "The thing that’s going to stick with me is that RCTs are not about judgment; they’re about improvement. I think my next step is finding researchers who already have evidence that can help us evaluate what we're doing."

For a full recap of the convening, see the MIT News article.

Learn more about J-PAL North America's State and Local Innovation Initiative.