Building research partnerships to understand the promises of Summer Youth Employment Programs

Summer Youth Employment Programs (SYEPs) are municipal programs that provide qualifying youth and young adults ages 14 to 24, often from households with low incomes, with a paid, part-time job and related supplemental services during the summer months. Rigorous research, like that showcased in J-PAL North America’s Evidence Review, shows that participating in an SYEP can have many benefits for youth both during and beyond the summer. 

As the result of a long-term existing partnership, in 2017 J-PAL-affiliated researchers Judd Kessler (University of Pennsylvania) and Sara Heller (University of Michigan) worked with the Department of Youth and Community Development (DYCD) in New York City (NYC) to evaluate a component of the NYC SYEP that provided letters of recommendations to youth participants. 

We interviewed Judd, Sara, and Julia Breitman, Assistant Commissioner of Youth Workforce Development at DYCD, to shed light on their partnership. In part one of this blog series, Julia, Judd, and Sara discuss their partnership and the development of their research question.

What motivated DYCD to evaluate New York’s SYEP? 

Julia: The Summer Youth Employment Program is an institution in NYC, serving countless New Yorkers since the program’s inception in 1963. While SYEP has always been a foundational experience for NYC youth and is the city’s largest workforce program, we wanted to see if there was an economic impact to this program. SYEP had never been formally evaluated, and while we instinctively knew that the program changed lives, we were looking for a formal evaluation partner. Judd came in at the perfect time for us. The project initially started by looking at the economic impact of the program, followed by questions about public safety. Now, we can say, without a doubt, that this is how many lives were saved through participation in the program, and we have the data to back it up. It was a really interesting partnership that led us to change the focus of the program, the groups of young people we focus our resources on, and how services are delivered. 

Why were you interested in working with J-PAL North America to support this evaluation, and what drew you to partner with DYCD?

Judd: My partnership with DYCD started when I first became an assistant professor at the University of Pennsylvania. The first project we did together was using the existing randomization in the lottery for spots in the NYC SYEP, evaluating the effect of the program on youth outcomes. DYCD was a phenomenal partner in that project—they were eager to learn what the impact of the program was and not afraid of doing research that might uncover limitations of the program. We found that SYEP had tremendously positive outcomes in terms of incarceration and mortality: SYEP was keeping youth out of prison and keeping them alive. What was surprising was that the program did not have positive impacts on youth participants' future labor market outcomes. That was a question that interested me as a researcher and a finding that troubled DYCD. They wanted the program to help youth with future job prospects as well, which is what led to the project that J-PAL North America funded. We built a letter of recommendation software, so supervisors could easily create letters of recommendation for randomly selected youth. Given our interest in this project and how well it fits with J-PAL North America’s mission, we thought J-PAL would be interested as well.

What about this partnership stood out to you?

Sara: DYCD is a really extraordinary partner. They are a public agency that is genuinely interested in better understanding the impact of their programs and improving what they do. They took the lessons of Judd’s initial SYEP study to heart, and spent a lot of time thinking carefully about program changes that could amplify the strengths and address the weaknesses of what they were doing. Working with an agency like that is an amazing privilege. A group of staff who have a real interest in learning and improving is perhaps the single most important thing for researchers to accomplish policy-relevant research. The staff’s commitment to learning and self-reflection leads to great communication and allows us to understand the relevant institutional details, and helps us improve our ideas. 

Can you share how your partnership with Sara and Judd began?

Julia: We were working with Judd first, when the team was looking at the economic impact of the program. Sara joined Judd to look at the criminal justice aspects of the program. The initial study showed that the economic impact is great during the summer, but the long-term economic impact is not as strong as we could have hoped for. This was understandable, given that it is a six-week summer program. The evaluation showed that the program does what it always intended to do, keeps kids safe and off the streets; it keeps them motivated and allows them to explore careers and meet mentors they would have otherwise not been able to. It especially helps young people already involved in the justice system or at risk of becoming so.

How did the team (of researchers and DYCD) arrive at a research question that was of mutual interest and feasible to assess through a randomized evaluation?

Sara: Both Judd and I have done prior studies of SYEPs, and we both had a very basic question come out of that work: why doesn’t the training and work experience that SYEPs offer improve future labor market outcomes for participants? Often, we just have to relegate these kinds of follow-up questions to the discussion section of our initial papers. But in this case, the existing relationship with DYCD meant we could approach them with the question. At first, it didn’t seem totally plausible to them that letters of recommendation—a few pieces of paper—would do anything. But they trusted us enough to help us find out. In the end, they were excited to understand why letters of recommendation seem to matter and for whom. DYCD is now working to incorporate what we learned about the value of skill signals into their programming so that future young people can benefit from the study’s findings.

Judd: DYCD recognized the value of randomization in generating research results that can help improve the program, and being fair when allocating a scarce resource. Supervisor time is a scarce resource: many supervisors oversee dozens of youth, and asking them to write letters is time consuming. The fair way of allocating this resource is through randomization, so it was appropriate to conduct a randomized evaluation in this context.