Limiting Learning Loss using Phone-based Programming during Covid-19 in Botswana
- Children
- Students
- Student learning
- Nudges and reminders
- Tracking and remedial education
School closures—due to viruses, teacher strikes, natural disasters, and more—can result in large learning losses for students. In these situations, low-income families who lack internet access need “low-technology,” inexpensive solutions to support their children’s learning. Working in Botswana, researchers rapidly evaluated a phone-based remote learning program aimed at keeping children engaged with math during the Covid-19 pandemic. Students who received weekly SMS messages and phone calls to review math exercises increased their math skills after twelve weeks, while students who received only SMS messages did not.
Policy issue
As a result of the Covid-19 pandemic, education systems around the world have halted in-person activities, leaving over one billion students out of schools.1 Evidence from both high- and low-income countries suggests that school closures–which also occur during summer holidays, public health crises, and weather shocks–often result in large learning losses for students.2 With children unable to attend school, existing educational inequalities become starker, with high-income families using alternative instructional resources—such as books, computers, internet, and smart phones—which many low-income families cannot access.3 To address the exacerbated learning crisis, policymakers worldwide have been looking for rapid, cost-effective, low-tech solutions, such as SMS messages, to reduce learning losses at scale. However, can these types of interventions actually impact children’s learning? If so, how cost-effective and scalable are they?
Context of the evaluation
In Botswana, where this evaluation took place, primary school enrollment is around 90 percent. However, learning levels are low and stagnating.4 A recent survey of basic literacy and numeracy showed that 32 percent of students in grade five could not do subtraction and 44 percent could not read a story in English. In an effort to improve learning levels in Botswana, a number of organizations, led by the Ministry of Basic Education, have been supporting the scale-up of an education program, Teaching at the Right Level (TaRL), to all primary schools. This program, which has been found effective in several contexts, involves regrouping students by their actual learning level rather than the grade-level curricula. By March 2020, TaRL had reached over 15 percent of primary schools in the country.
Although the direct effects of the Covid-19 pandemic were initially minimal in Botswana–with limited reported cases and deaths through July 2020–pre-emptive social distancing measures have been adopted by the government, which severely impacted educational and social services. Schools across the country were closed from March to June 2020. Shortly after reopening, a new wave of Covid-19 cases prompted a subsequent school closure. Similar waves of reopening and closing are expected in the coming months. As of July 2020, schools were running a double-shift system—half of the students attend school in the morning, while the other half attend in the afternoon–drastically reducing the total time in school for each student. Despite the government’s efforts to keep students engaged—through learning programs on national television and radio stations—ninety-nine percent of parents surveyed reported demands for additional remote educational activities for their children.
Details of the intervention
In partnership with Young 1ove and the Botswana Ministry of Basic Education, researchers conducted a randomized evaluation to assess the effectiveness of a low-tech intervention on reducing students’ learning loss during school closures. Given the Covid-19 pandemic’s demand for real-time policy response, the research team was able to conduct the evaluation and generate rigorous results in around two months.
Using phone numbers collected from primary schools for students in grades three through five, the research team identified 4,500 households who were willing to receive remote learning support via mobile phone. Each household was randomly assigned to one of three groups:
- Weekly SMS message only: Households in this group received a weekly SMS containing several simple math problems (“problems of the week”).
- Weekly SMS message + phone call: In addition to the SMS messages, households in this group also received a 15-20-minute phone call, in which facilitators answered any questions related to the “problems of the week” and provided one additional practice question. The calls required both parents and children to be present as a way to encourage engagement and maintain accountability.
- Comparison group: Households in this group did not receive any type of phone learning support from this program.
After four weeks, facilitators conducted phone surveys with parents and students from half of all households to collect information on their engagement in the educational activities, parents’ perceptions of their child’s learning, and student learning outcomes, as measured through standardized test scores.
On the fifth week, researchers randomly selected a subset of the households from the SMS only and SMS + phone calls groups to receive an additional support program: targeted instruction. In this subsequent intervention, students received text messages tailored to their individual learning levels over a period of four weeks. For example, students who knew addition received subtraction problems to push them to a higher level of learning, and students who knew multiplication were sent division problems. Through weekly phone surveys, facilitators collected students’ answers to the “problems of the week,” which were used to target subsequent weekly SMS messages. After twelve weeks, the survey team conducted a second round of surveying and data collection to evaluate these additional components of the intervention.
Results and policy lessons
SMS messages combined with phone calls had positive impacts on students’ learning. SMS messages alone did not have significant effects, suggesting that some level of direct instruction is necessary. Both programs increased parents’ sense of self-efficacy in supporting their children’s learning.
Impacts on students’ learning: SMS messages combined with phone calls had large impacts on students’ learning. Students assigned to receive SMS messages and phone calls saw a reduction in innumeracy of 31 percent. Their average numerical operation increased by 0.12 standard deviations, in addition to a 0.11 standard deviation increase in higher-order competencies, including solving fractions.
Effects on learning gaps: The programs were also effective in closing previous learning gaps. When comparing students who had previously received learning support in math–measured by participation in the Teaching at the Right Level program–with those who had not, researchers found that the second group experienced higher gains in learning from the remote phone interventions after four weeks. A similar trend was observed when comparing girls and boys: boys, who started with lower numeracy levels, benefitted more from the SMS and phone support than girls. This suggests that low-tech learning interventions have the potential to close gaps in learning between higher- and lower-performing students.
Parental engagement: Parents were more engaged in their children’s learning. The more intensive SMS + phone calls program helped parents engage with their children’s math education and subsequently update their beliefs. In households that received SMS-only and SMS + phone calls, parents were 7 and 12 percentage points more likely, respectively, to spend time engaging with their child on educational activities after four weeks–a 10 to 18 percent increase in parental engagement, from 67 percent in the comparison group. Additionally, parents in treated households improved the accuracy of their perceptions of their children’s learning levels. After four weeks, parents in the SMS-only group were 7 percentage points more likely to correctly estimate their child’s learning level, while those in the SMS + phone calls group improved their perception by 11 percentage points–a 23 and 35 percent increase from a comparison group average of 31 percent, respectively. Accurate perceptions are important as they enable families to target learning activities to their children’s level more effectively. Finally, after twelve weeks, parents in both groups reported higher self-efficacy in supporting their children’s learning, but only parents in the SMS + phone calls group accurately reported their children’s learning level.
Targeted vs. non-targeted remote instruction: Targeted instruction had significant, positive effects on students' math skills, increasing them by around 0.1 standard deviations, while non-targeted instruction did not. Researchers suggest that targeted instruction based on phone-based assessments could bring greater benefits, although they couldn’t distinguish the impacts between targeted and non-targeted instruction.
Cost-effectiveness and scalability: Researchers estimate that the total costs, including fixed costs, of sending weekly SMS messages equated to US$5 per child assigned to be texted, while SMS + phone calls costed US$19 per child assigned to be texted and called. In the SMS + phone call intervention, every US$100 spent led to a 0.63 to 0.89 standard deviations increase on learning. The cost-effectiveness of the SMS + phone calls intervention compares favorably to other interventions in the education literature, including another math tutoring program that yielded a 0.19 to 0.31 standard deviation increase in math achievement test scores at a cost of US$2,500 per participant. It is important to highlight that a portion of these costs are fixed costs to set up the program, including costs to collect phone numbers, set up new infrastructure, conduct training, and collect routine monitoring data, suggesting likely even lower costs when scaling up. Although more expensive, the SMS + phone calls intervention seemed more effective in promoting student learning. These results show that low-technology interventions, such as SMS messages combined with phone calls, can be a cost-effective and scalable way to prevent learning losses during school closures, especially among low-performing students.