Investing in students early: The effects of a STEM pipeline program for underrepresented high school students

Sarah Cohodes (Teachers College, Columbia University), Helen Ho (Harvard University), and Silvia Robles (Mathematica) sit down with J-PAL staff to discuss the newly released results of their randomized evaluation of a STEM (Science, Technology, Engineering, and Mathematics) program hosted by a prestigious university. 

Can you provide an overview of the programs that you evaluated?

Silvia: We evaluated three summer programs that targeted rising high school seniors who were academically high-achieving and had a strong interest in STEM fields. These programs offered rigorous STEM courses at the host university. 

One was a six-week residential program, where participants lived on the host institution’s campus and took a simulated first-year college course schedule. The second program tried to compress that six-week experience into one week and only offered a single enrichment course. The third program did most of its activities online and included a short campus visit. 

Sarah: The programs also specifically targeted individuals from backgrounds that tend to be underrepresented in STEM. There was a very broad definition of what that could be, including Black, Indigenous, or Latinx students; students who would be first-generation college graduates; or  people who attended high schools where there wasn't strong college or STEM-preparation. As women also tend to be underrepresented in STEM, the program made an effort to ensure half of participants identified as female.*

*Note: During the study period (2014-2016), the gender identity categories available on the application were male and female. These categories may not have fully encompassed the gender identities of applicants.   

What was your motivation for studying these programs? 

Silvia: I participated in one such program when I was in high school and it was a very formative experience. It certainly changed the trajectory of my career, and I wondered if others who participated in similar programs were seeing similar impacts. So, in my second year of graduate school, I started working with the host institution to implement a randomization evaluation. 

Sarah: When Silvia came to me with this project, I was particularly excited by the buy-in of the partner. It’s rare to find an organization that is willing to randomize access to their programs, and it was very compelling that Silvia had worked to find a randomization design that worked for them. It was also important and unique that this program was trying to target the students who could benefit the most, especially in an era with a diminished focus on affirmative action. 

Helen: I joined the project after the first two years. I was excited to study what interventions could promote diversity and equity in education, given the importance of education in accessing economic opportunity and the historic and present-day biases that make education inequitable. It’s also exciting to work on a study that can track outcomes through college graduation, since college persistence and graduation are so important.

What considerations did you face when randomizing students into the various treatment groups?

Silvia: When working with partners, there is always a balance between research needs and program needs. In a randomized evaluation, it’s ideal to have basic individual random assignment: after applicants are screened for suitability for the programs, students would be assigned randomly to the programs and control group. That method did not sync with the program office’s vision because they were concerned that someone who was a less strong fit for the program would get a place over a student who was well-suited, or that a well-suited candidate would have a high likelihood of not receiving any program offer. 

Sarah: We came up with a compromise where we ranked the students in terms of their suitability for the six-week residential program, and then we divided students into groups based on that ranking. We then randomized within those groups, with the top ranked students randomized within the three programs and the lower-ranked students randomized between a program and the control group. 
  
What were the key outcomes and results from the study? 

Silvia: One of the headline results was a 2–8 percentage point jump between treatment and control groups in college graduation rates, from a baseline of 53 percent in the control group. The six-week program immediately increased four-year graduation rates, while all the other programs seemed to increase five-year graduation rates by even larger margins. We also saw a large shift (between 5 and 20 percentage points) in the rate at which students majored and graduated in STEM fields.

Sarah: What’s particularly interesting is that the higher graduation rates weren’t necessarily caused by a shift in college enrollment rates; those were relatively similar across treatment and control groups. Instead, the program changed what types of colleges students attended, primarily by changing their college application strategy. Students offered seats in the programs were more likely to apply to a larger number of higher-ranked schools. In contrast, those in the control group may have only applied to one or two well-ranked, but not top, institutions. Students who applied to those highest-ranked schools tended to get in and attend those schools. These types of universities tend to have very high graduation rates, which was reflected in the students’ college success. 

Helen: It’s important to note that this study focused on students with high academic performance and strong interests in STEM. These students scored well above average in standardized testing, yet the control group’s college graduation rates were still not much higher than the national average. Our study shows that a well-timed, pre-college intervention can boost college graduation for these underrepresented students and disrupts the commonly held, but mistaken, belief that the lack of diversity in STEM is due to a lack of interest from underrepresented students. Our results show that there is room to support underrepresented students in maintaining their focus on STEM. 

What are the broader implications of the results?

Silvia: We’re always talking about a “leaky pipeline,” but there hasn’t been much research to date that shows that you can actually plug those leaks. Now we have that evidence and can encourage people to invest in all students who, no matter their background, have the potential to be very successful. By shifting students to different colleges from those they would have attended otherwise, the programs may set students on a different path.

Helen: The investments provided by the summer programs are those to which privileged students tend to have more access, such as college application guidance, encouragement and mentorship, and out-of-school academic enrichment. These investments can culminate into access to another privileged resource: higher-ranked universities. Providing resources that are disproportionately accessed by privileged students to underrepresented students can make real progress toward achieving equity. 

Sarah: Another interesting application of the findings from our paper would be investing resources more equitably across colleges. We think the reason why going to higher-ranked schools had such a massive impact on graduation rates is because those schools are disproportionately well-resourced, and other schools cannot offer the same support. 

We should support high school students early and encourage them to apply to highly-ranked schools, but we should also ensure no matter where they go, they are offered the resources to thrive.