Building research partnerships to understand the promises of Summer Youth Employment Programs, part two

Summer Youth Employment Programs (SYEPs) are municipal programs that provide qualifying youth and young adults ages 14 to 24, often from households with low incomes, with a paid, part-time job and related supplemental services during the summer months. Rigorous research, like that showcased in J-PAL North America’s Evidence Review, shows that participating in an SYEP can have many benefits for youth both during and beyond the summer. 

As the result of a long-term existing partnership, in 2017 J-PAL-affiliated researchers Judd Kessler (University of Pennsylvania) and Sara Heller (University of Michigan) worked with the Department of Youth and Community Development (DYCD) in New York City (NYC) to evaluate a component of the NYC SYEP that provided letters of recommendations to youth participants. 

We interviewed Judd, Sara, and Julia Breitman, Assistant Commissioner of Youth Workforce Development at the DYCD to shed insight into their partnership. In part two of this blog series, Julia, Judd, and Sara discuss integrating evidence in policy impact, and the benefits of using data.

J-PAL North America offers government-facing  opportunities to generate evidence, one of which is called the Evaluation Incubator. Through this opportunity, governments receive training, technical assistance, flexible funding, and connections to expert researchers to develop randomized evaluations. What do you think are the benefits for state and local governments to pursue evaluation opportunities, such as the Evaluation Incubator? 

Sara: Conversations with researchers can help governments assess whether the basic assumptions they are making are true, or help them to articulate the questions they might need answered to better achieve their goals. Researchers can help them get answers. That can involve technical assistance to better understand the data they may already collect, help in figuring out how to collect or access new data they may need, and a better understanding of what other jurisdictions have found and how that might apply to their own setting. Researchers can also provide ideas about new policy initiatives or tweaks to old ones that may help governments achieve their goals, and help determine if the money they are spending is really accomplishing what they want it to. Plus, after a study is completed, having strong evidence in hand can help raise money to expand successful programs.

Judd: Evaluations give governments the opportunity to see what is working—and what is not— and provide insights about how to make improvements to their programs. We are now in conversation with DYCD about rolling out letters of recommendation for all SYEP participants based on the findings of our research. This add-on would not have been possible if not for the evaluation and our research partnership.

What are some ways that you see DYCD and other similar government agencies uniquely positioned to generate and use evidence? What are the benefits for city government agencies to pursue evaluation opportunities?

Julia: It's important for agencies to collaborate and share data, such as education, health, and criminal justice data. Having a researcher bring all the data together to show the impact of a program is not something that agencies can often do alone. We sometimes need a third party to make us collaborate and put our data together. At DYCD, this has allowed us to work together and discover which programs weren’t making the impact we hoped they would, and use our  very scarce public resources in a better way.

How do you think evidence generation can fit into policy influence, and do you have any advice for other researchers that are interested in having a policy impact?

Sara: One thing I have learned from partnering with policymakers is that the evidence I generate doesn’t just sit in a journal on the shelf. There is already an invested partner who cares about the results and is likely to act on them. It turns out that it’s a lot easier to influence policy when you start with policymakers who already understand the design and are invested in the question. So one lesson for researchers who want to have a policy impact is that the extra work and difficulty that comes with doing randomized evaluations can have a real payoff in terms of public impact. Also, it takes time to build up these relationships. It may be valuable to provide technical assistance on various issues that won’t ever be a research paper to help establish a shared language, learn the institutional details, and build trust. And you may have to give up on some great ideas that just prove too complicated for your partners, or that they aren’t interested in. You have to learn to be flexible and compromise.

Judd: I have done a lot of research over my time as a graduate student and faculty member, and this partnership with DYCD has had the most policy impact, in part because a strong partnership between policymakers and researchers means policymakers can actually run with research  findings. For example, our results about mortality from our first SYEP study had a big impact on the way that politicians and policymakers in the city think about SYEP and its value. Since the research findings came out, NYC has been trying to expand the program and make it universal so that everyone who wants a job can get one. In terms of advice, when thinking about good research partnerships with governments or private firms, you need to find the intersection between an idea of academic interest, an idea that is feasible to implement with a randomized evaluation, and something that the organization is interested in learning. It’s hard to find something at the intersection of those three, but often that is the key to designing a research study that can have policy influence. It also helps to be very clear when you're talking to organizations about what research projects may be feasible and make sure that you focus on what is important to the partner in your research design.

What are some ways that you think researchers can build trust and strong relationships with local governments and communities to foster evidence building and positive policy impact?

Judd: The trust gets built over time through partnerships that develop and grow. With DYCD, first they let us work with their historic lottery data, and they saw us follow through on the research design in a way that kept the data secure and anonymous. That initial research project showed them that our intentions were good and we were interested in uncovering things that could improve the program. I think that helped them trust us to do our own randomized evaluation with the letters of recommendation. The letters project was more involved, it involved contacting supervisors and youth ourselves, and that was something that they trusted us to do after seeing the work that we had done in our prior research. We also knew that we could trust DYCD to be a good research partner. We knew we could rely on them if we needed additional data or support in getting data from other city and state agencies. It takes time, but building trust takes small steps, and eventually you can take on more ambitious projects.