Implementation challenges of an impact assessment: The ATEMA project in Puerto Rico

The Puerto Rico Education Department, the University of Puerto Rico, J-PAL North America, and J-PAL affiliated researchers are working together to evaluate the impact of the Technology Application in Mathematics Teaching (Aplicación de la Tecnología en la Enseñanza de las Matemáticas, or ATEMA) program on math achievement, as measured through standardized tests and performance on the Khan Academy platform’s activities. ATEMA offers mentoring to math teachers to use Computer Assisted Learning and is being implemented by the University of Puerto Rico, led by Project Director Olgamary Rivera and Research Manager Alina Amador. We discussed their major challenges and lessons learned from implementing ATEMA and why implementation is critical to the evaluation process. 

What is the ATEMA project and what are your roles in it?

Olgamary: ATEMA is a set of learning strategies driven by technology, specifically the Khan Academy platform. The platform includes videos and exercises for students to apply mathematical concepts. ATEMA provides a guide for teachers to incorporate this platform into their classes in order to support each student individually around mathematical concepts. We also hired and trained mentors to assist teachers on how to use the platform. The project targets students in fourth to eighth grade.

Alina: There are several key partners in this project, including the research team, the Department of Education, mentors, and teachers. My job is to keep each stakeholder up to date with decisions made at every level.

What were the key stages implementing this project?  

Olgamary: The first part was to select educational materials from the Khan Academy platform that were well aligned to the relevant syllabi. We then recruited twelve mentors and taught them how to use the Khan Academy platform, along with how to assist teachers in integrating the platform into their classrooms. In August 2021, we hosted a training course for teachers to introduce them to the program and platform. We then matched teachers and mentors according to their schedules. The plan was for teachers and mentors to meet for thirty minutes each week to discuss strategies for integrating the platform into the classroom and using the educational material. 

What major challenges have you faced in implementing the program?

Alina: Scheduling the participants was the biggest challenge during the first semester because there were several changes in the school system that made it difficult for teachers to join the weekly meetings. We had to focus on specific cases for several months to ensure a robust ATEMA participation rate and we had to be flexible with the teachers who often faced constraints beyond our control.

We also had issues with the Student Information System (the database where all Puerto Rican students are registered). The platform’s data comes from different companies, so access to information is not integrated and we had to work with each company directly to update the system. 

Olgamary:  Puerto Rico went through several natural events such as Hurricane Maria in 2017, the 2020 earthquake, and the Covid-19 pandemic, so we had approximately two years of academic setbacks. We had cases where schools did not have math teachers, or teachers had to change the subject they taught at the last minute. There were changes in school schedules. Strikes led to teachers quitting so other teachers covered additional classes and grades. These challenges made it difficult for some teachers to participate in the mentoring process. 

We also ran into technological challenges. There was one school where the internet lines were cut by mistake, so some teachers used their cell phone internet as a hotspot for students to work on Khan Academy. At the beginning of the program, our goal was to have students work for two hours per week on Khan Academy. When we began to experience these limitations, we recommended that each student simply practice as much as they could. 

Due to Covid-19, several research projects had difficulties with participation rates. What strategies did you use to reach your participation goal? 

Alina: At first we emailed teachers, held meetings with school principals, and talked to project facilitators to motivate teachers, but these strategies didn't work the way we expected. What worked was calling the teachers directly to tell them about the project. 
Due to the participation concerns, the researchers wanted to gather preliminary results at a smaller scale. We set up “El Reto de Khan,” (“the Khan Challenge,”) in schools that did not participate in the first ATEMA cohort. Researchers analyzed pre- and post-test results from 300 students before and after using the Khan Academy platform to see if using the platform improved students' academic performance. Although the researchers underestimated the initial scope of academic setbacks, the results were promising. 

Olgamary:  We also offered a make-up session for teachers who were unable to attend the initial training. At the beginning, many teachers thought that participating in the project meant having more work. Instead, we wanted to help them organize and prepare their class with exercises that they didn't have to create from scratch. By the end of the semester, many teachers were able to get their students to work for more than two hours per week on the platform.

 It looks like the key to overcoming implementation challenges was a strong coordination strategy, remaining flexible, and the willingness to learn and adapt. What has been the most satisfying part of working on this project?

Olgamary:  I think this project has the potential to bring many changes to the Education Department, especially teaching. We know we had some setbacks, but that has not taken away my desire to continue with this project and help teachers use this tool. After working with a teacher, I always feel satisfied because I know that we are planting a seed in each one of that teacher’s students that will continue to grow. 

Alina: There are two inspirations that keep me going in this job. One is the excellent team that is working on this project. This project would not be possible without Olgamary and all of her hard work ensuring ATEMA was implemented as well as it was. The second is that one of my priorities is to work for the Puerto Rican people, and with this project I feel that I am supporting a new generation of Puerto Ricans. I am glad to support principals and teachers who are doing their best to help youth. Seeing the photos, speaking with teachers, and hearing that the communities we are supporting value ATEMA fills me with pride.