The Impact of Teaching Teachers to Use Computer Assisted Learning on Student Math Performance in the United States
- Students
- Student learning
- Computer-assisted learning
Mastery learning, where students must demonstrate proficiency in a topic before progressing, is recognized as a highly effective way to learn yet is often challenging for teachers to implement in a classroom setting. Computer Assisted Learning (CAL) is a promising approach to facilitate more individualized learning. Researchers conducted two randomized evaluations to test the impact of coaching teachers on integrating CAL in their elementary and middle school mathematics instruction in two large, diverse school districts in the United States. The intervention led to improved math performance in one evaluation and mixed results in the other, with test-score gains observed in classrooms with higher weekly CAL practice time.
Policy issue
Mastery learning, where students must demonstrate proficiency in a topic before progressing, is recognized as a highly effective way to learn. However, it is difficult to implement mastery learning in a classroom setting because students learn at different paces and opportunities for individualized attention are often limited. Falling behind on foundational concepts is a particularly pronounced issue in subjects like mathematics, where early skill gaps can lead to long-term disengagement and poor performance. Computer Assisted Learning (CAL) can help support individualized learning, but getting students to engage with CAL and teachers to integrate CAL in their instruction can be challenging. Can providing teachers with proactive, continuous support to use CAL effectively improve student outcomes in math?
Context of the evaluation
This study took place in elementary and middle schools in the United States, specifically within the Metro Nashville Public Schools district in Tennessee and the Arlington Independent School District in Texas. In Nashville, the focus was on students in grades 6–8 in seven low-performing schools. In Arlington, all grade 3–8 math teachers in the district were invited to participate in the program. Both school districts serve primarily Black and Hispanic students.
Details of the intervention
Researchers partnered with Khan Academy, one of the most popular CAL programs in the world, and two school districts to conduct two evaluations to assess the impact of integrating CAL on student math performance. Khan Academy includes interactive videos, practice, quizzes, and recommendations for learning. Student accounts can be linked to those of parents and teachers, allowing teachers to assign content according to their curriculum and individual student needs, and enabling both parents and teachers to track student progress. Teachers can use Khan Academy either within or outside of the classroom.
Previous research has suggested that while CAL can be effective, student engagement is critical and implementation varies. The intervention in the present study, known as the Khoaching with Khan Academy program, provided teachers with coaches who assisted teachers in getting started with CAL, offered guidance on tracking student progress and helped teachers integrate CAL into their curriculum. Coaches helped teachers set goals, troubleshoot issues, and discuss best practices for using Khan Academy. Coaches also assisted in monitoring student progress, setting up weekly assignments, and identifying students needing extra attention.
The study included two main components:
Nashville Evaluation: As a proof of concept of the potential of CAL, researchers randomly assigned 3,183 students in grades 6–8 to practice one of two math topics using Khan Academy for a week. Teachers were invited to participate in the one-week program and allocated time for students to take a computer-based pre-test covering both topics. After the pre-test, students were directed to their Khan Academy account to begin working on their randomly assigned topic. Teachers were also asked to provide at least one hour of in-class practice time throughout the week, during which students were to work through lessons and exercises on their assigned topic in Khan Academy. Then, students took the computer-based post-test. In total, 1,806 students took the pre-test, 1,130 of whom also completed the post-test.
Arlington Evaluation: Researchers randomly assigned 224 grade 3–8 math teachers (covering 10,979 students) to receive coaching to implement Khan Academy in their classrooms either during the 2021–2022 or 2022–2023 school year. Teachers in the intervention group, implementing in the 2021–2022 school year, had the opportunity to attend a professional development session focused on the importance of a mastery learning approach and recommendations for facilitating at least an hour per week of CAL practice. Then, coaches, who were trained undergraduate and graduate students, provided support to teachers with CAL implementation via weekly 30-minute Zoom meetings initially, and then transitioned to email support as appropriate.
Results and policy lessons
Overall, the CAL program was effective at increasing math performance, but only when teachers effectively implemented student practice time.
Nashville Evaluation: On average, students who were randomly assigned to practice a particular topic performed 0.22 standard deviations better on that topic than students who were not assigned to that topic. Effects varied by grade level, with grades 6 and 8 improving an average of 0.30 standard deviations and grade 7 showing no statistically significant changes in performance, perhaps because the grade 7 topic was more difficult, as suggested by less progress being made on that grade’s CAL assignments. Higher average classroom practice time with CAL was associated with larger post-test score increases, with students in classrooms averaging 25 minutes or more of practice time per week experiencing test score improvements.
Arlington Evaluation: On average, across the full grade 3–8 study participants, there were no statistically significant differences in math performance, as measured by state standardized assessments, between students whose teachers received the Khoaching with Khan Academy intervention and those who did not. However, this masks important variation by grade levels: test scores improved by 0.122 to 0.171 standard deviations among grade 3–6 students, who practiced on average five times more than grade 7–8 students.
These findings suggest that CAL program effectiveness depends on integrating it with classroom instruction and allocating sufficient time for students to practice and pursue mastery of concepts, though it is important to note that this study did not randomly test different amounts of practice time. Qualitative interviews suggest that teachers who implemented high-practice time tended to have higher levels of principal support, buy-in to the program, and monitoring of student learning.