The Impact of an Online Math Learning Platform on Test Scores and Attitudes Towards Math in Brazil
- Primary schools
- Students
- Teachers
- Student learning
- Digital and mobile
Technological approaches have been increasingly used to improve education quality. However, the evidence regarding their efficacy remains mixed. Researchers conducted a randomized evaluation to measure the impacts of the Khan Academy’s online platform on math scores in Brazilian public schools. Despite improving student’s attitudes towards math, the program had no impact on students’ test scores. Results varied based on the school’s infrastructure and program implementation, highlighting that implementation challenges may hinder success of educational-technology programs in low-and middle-income countries.
Policy issue
To improve education quality and learning outcomes, governments, civil society organizations, and private sector actors have increasingly adopted educational-technology, or ed-tech, programs. Examples of such policies include expanding access to computers and the Internet in public schools and the development of computer assisted learning programs for students. Despite the significant number of studies in the topic, it is still unclear what types of approaches work best to improve educational outcomes, especially in low income contexts.
Different types of programs show different results. Interventions that use ed-tech to complement traditional teaching activities, for example, generally find positive effects on learning.1 In contrast, when computer assisted learning is used to substitute the regular classes, the impacts are mixed.2 What are the determinants of such a program’s success? Can they be effective in contexts of low technology infrastructure?
Context of the evaluation
Khan Academy is an online interactive platform offering free instruction and practice in mathematics as well as other subjects, such as science, computer programming, history, and economics. The platform, originally created for the United States, offers contents in a personalized environment, adapting the user’s experience to identify strengths and tackle learning gaps. Founded in 2008, Khan Academy currently reaches millions of learners around the world.
In 2014, the Lemann Foundation, a Brazilian non-profit focused at enhancing the quality of public schools in Brazil, translated and adapted a Portuguese version of Khan Academy’s math platform. Through the program Khan Academy in Schools, the organization began supporting its use in Brazilian public schools, which are mostly attended by children coming from lower income families. In addition to complementing the school curricula with high quality, tailored content and exercises, the math platform has the goal of improving student’s math performance by shifting the students’ perception regarding math through presenting the content in an interactive and friendly way.
Details of the intervention
In partnership with the Lemann Foundation, researchers conducted a randomized evaluation to measure the impacts of the Khan Academy in Schools program on primary school students’ math performance in Brazil. The evaluation focused on 5th and 9th grade classrooms of 157 public primary schools in five cities across the country that voluntarily agreed to participate in the program. The research team randomly assigned half of the 5th and 9th grade classrooms in each school to receive the Khan Academy program, while the other half did not receive the program and served as the comparison group. Schools in the comparison groups, however, did receive the program for other grades (i.e. 3rd, 4th, and 6th).
For the 2017 academic year, classrooms in the Khan Academy program carried out one of their weekly 50 minutes math classes at the school’s computer lab using the online platform. Teachers in these classrooms received a one-day training session to learn to use the platform as well as a series of support visits from Lemann Foundation staff. The platform also gave teachers access to a detailed feedback report on students’ performance, indicating their strengths and weaknesses
The Lemann Foundation provided computers for all schools in the evaluation that had fewer than one computer for every two students, including in comparison schools. The program was either implemented with students using each computer individually, or rotating use between two students. In the latter, each student used the computer during half of the class, and worked on other math activities assigned by their teacher during the remainder of the class.
To measure impacts on student learning, the research team collected math test scores from standardized tests that the Ministry of Education’s Basic Education’s Evaluation System (SAEB) administers at the end of the school year for all schools every two years. In addition, they conducted baseline and endline surveys to capture students’ attitudes towards math, demographics, and computer access and usage.
Results and policy lessons
Though improving student’s attitudes towards math, the Khan Academy at School program had no significant effect on student’s math performance. However, implementation details mattered: evidence suggests that the program improved math scores in schools with better implementation while there were negative effects for schools with poor implementation.
Impact of the program on attitudes towards math: Students offered the program scored 0.06 standard deviations (SDs) higher than the comparison group on a measure about their feelings regarding math. In particular, 5th grade and 9th grade students scored 0.062 SDs and 0.057 SDs higher respectively.
Impact of the program on test scores: On average, there were no differences in math proficiency between students in classrooms with access to the Khan Academy at School program and those in the comparison group. This suggests the modest gains in attitudes did not translate into higher test scores.
Importance of implementation quality: Researchers found suggestive evidence that the lack of impact on test scores might be due to infrastructure problems faced by schools, such as poor internet connection, and the different ways the program was implemented. For instance, students who had access to individual computers increased their test scores by 0.081 standard deviations. However, students who had to share computers with another student scored 0.76 SDs lower than the comparison group. Similar difference are found between students in schools that faced and those that did not face infrastructure problems. Implementation challenges potentially reduced the total amount of time students were exposed to math content, which may have limited the effectiveness of the program.
These results show how implementation challenges, such as issues with internet connectivity and lack of computers, can affect the performance of education-technology interventions, and must be taken into account when considering scaling up such programs, especially in low- and middle-income countries.