Pursuing evidence-based policymaking during crisis

In 2017, the Puerto Rico Department of Education (PRDE) faced multiple difficult policy decisions. A team of researchers led by J-PAL affiliated researcher Gustavo J. Bobonis worked with J-PAL North America to establish a collaboration to support the PRDE in applying an evidence-based approach to their policymaking process. Four years later, we reflect on the evolution of our research-practice partnership, how we pivoted in the face of national and international emergencies, and our efforts to improve academic achievement and reduce inequality in student learning in Puerto Rico. 

Forging a research partnership to inform education policy

What began as an exploratory collaboration with the PRDE has grown into a robust partnership that comprises a group of J-PAL affiliated researchers and other experts, including Marco Gonzalez-Navarro, Daniela Scur, Orlando Sotomayor, and Rubén Gaztambide-Fernández. Working with its local partner, the Universidad del Sagrado Corazón in Puerto Rico, the team recruited an Evaluation and Research Officer, Emily Goldman, to work with the PRDE to identify and support the implementation of high-priority projects where rigorous evaluation could help answer key policy questions.

From its earliest days, the research-practice partnership has been a joint effort focused on improving learning outcomes, reducing inequality in student academic achievement, and providing high-impact professional development tools to the PRDE’s leaders. At the same time, we recognized that the collaboration arrangement would need to be flexible enough to respond to the PRDE’s emerging needs.

For example, the research team has provided technical support with department-wide monitoring projects and with protocols to support online assessment during the Covid-19 pandemic. The events of the past years, including Hurricanes Irma and Maria, recurring earthquakes, and the Covid-19 pandemic, have tested this partnership’s relevance and durability. By all accounts, however, the partnership has thrived. One key aspect of its success has been its flexible nature, which has enabled it to quickly respond to emergencies and to the PRDE’s evolving priorities, while continuing its ongoing evaluation projects to inform policy decisions.

The most illustrative example of this partnership is the envisioning, implementation, and evaluation of a system-wide school principal leadership management training program entitled the Professional Development Academy for Leadership Management in Education (EDUGESPRO, for its Spanish initials). Early on, the PRDE identified professional development as a crucial lever that could improve students’ academic achievement. The team has been working ever since in close partnership with the PRDE’s Institute for Professional Development and its leader, Operations Manager Dr. Damarys Varela Vélez, to develop and evaluate state-of-the-art professional development programs for PRDE staff.

The team conducted a territory-wide survey of school principals’ leadership practices during the 2018-19 academic year and identified areas for improvement. In particular, we found that principals could benefit from strengthening strategies employed to support teachers’ data-driven planning, track their students’ and schools’ performance, and set challenging but informed goals.

Using this information, as well as best practices from the field of educational leadership, the partners designed the leadership training intervention. EDUGESPRO is a large-scale principal training program seeking to improve academic leadership and management practices in order to, in turn, improve students’ academic achievement. The program began in the summer of 2019 and will be offered to all school principals in the island’s public education system over a three-year period.

Pivoting in the face of a global pandemic

After conducting the first series of training sessions during the fall of 2019, a series of earthquakes and subsequent tremors resulted in the temporary closure of many PRDE school facilities. We quickly convened the EDUGESPRO team to chart a course forward and gave the participating principals a short recess from the training program as they worked to ensure educational continuity in their communities. We did not foresee, however, that on March 16, 2020, the PRDE would have to close all school facilities and announce that all learning would be remote until further notice due to the Covid-19 pandemic.

As these events unfolded, we realized that EDUGESPRO’s training program had the potential to support school principals during the pandemic. Therefore, while many other PRDE programs announced extended breaks, we worked with the training provider to get the entire program online in the span of three weeks. Training resumed during April 2020, and the first cohort of EDUGESPRO concluded training in the summer.

Participation rates in the program actually increased after the program became fully available online. We found that the online version eliminated the logistical challenge that attending day-long trainings posed for busy school principals, as they could log into the combined synchronous and asynchronous sessions from their workspaces. The higher participation rates assuaged our fears that principals might not attend the online programming due to the more limited opportunities to interact with peers and opened up a range of possibilities for future online professional development training initiatives.

Our EDUGESPRO team meets weekly to rapidly follow-up on program developments in order to ensure the project’s success. Not only did the program migrate online; it also became a source of emotional support for principals navigating a public health crisis. At the EDUGESPRO graduation ceremony, participating principals spoke of how this community of practice supported them as they navigated school leadership during the Covid-19 pandemic. In program-curated WhatsApp groups and discussion forums, principals shared best practices, advice, and the occasional motivational quote, video, or meme.

The importance of these communities in a time of extreme stress and limited peer support was exemplified by one participating principal’s graduation address, in which she shared that, after heart surgery, she had asked hospital nurses to set up a computer monitor so that she could participate in the online sessions from her hospital bed.

The Covid-19 pandemic increased the urgency and complexity of decisions that the PRDE must make, but the collaborative research partnership’s flexible working arrangement has helped the PRDE to navigate a number of these challenges. For example, we had scheduled a territory-wide survey of school principals as part of the EDUGESPRO evaluation for May 2020. At the PRDE’s request, we included a survey module that could provide the PRDE with rapid data on school-level responses during the Covid-19-related lockdowns.

The research collaboration is also expanding. Other PRDE units noticed the utility and positive results of the team’s work and have since requested and received technical support from the research team on a series of initiatives intended to limit learning loss stemming from Covid-19 school closures.

Leveraging preliminary results and charting the way forward

Despite the many unforeseen interruptions and adjustments, preliminary quantitative results from an evaluation of EDUGESPRO’s impact during the program’s first year show gains in many leadership and management outcomes among program participants, and the size and significance of these improvements will become clearer as more principals participate in the program. The second cohort of school principals will begin the year-long training in March 2021.

Looking ahead, the partners will identify other areas in which program impact evaluations could provide important public policy guidance. The partnership’s expanded scope has enabled us to work on additional professional development initiatives and to support the PRDE’s efforts to address their public policy priorities. Having endured multiple crises, the team remains more committed than ever to using rigorous evidence to inform policy decisions for the benefit of all students in Puerto Rico’s public education system.