Improving Educational Outcomes through Online Tutoring during Schools Closures in Italy during the Covid-19 Outbreak
- Secondary schools
- Students
- Student learning
- Self-esteem/self-efficacy
- Aspirations
- Mental health
- Socio-emotional development
- Soft skills
- Coaching and mentoring
- COVID-19 response
- Online learning
- Tailored instruction
The COVID-19 outbreak forced governments worldwide to close schools for an extended period of time resulting in massive learning losses and adverse psychological effects for children, especially the most vulnerable and those from low socioeconomic backgrounds. Researchers designed a free online tutoring program delivered by volunteer university students to test whether it could mitigate these effects. They found that the program had a positive outcome on students' learning outcomes, socio-emotional skills, and psychological well-being.
Policy issue
Over 190 countries closed schools at the peak of the COVID-19 outbreak.1 School closure has created massive learning losses for children, leading to less time spent receiving schooling,2 lower achievement, and worse test scores.3 The pandemic has also had adverse psychological and social effects for children and adolescents, leading to higher depression and lower development of socio-emotional skills.45 Students from lower-income families were disproportionately affected, as they often had limited access to technology, received less support from parents and lower quality of remote learning from schools. The combination of these effects risks having long term consequences on the human capital of the cohorts affected by school closures, as it hinders the full development of the skills needed to contribute to a productive and cohesive society.6
While many countries have tried to mitigate learning losses by switching to remote instruction and using different types of platforms, the implementation of these tools has varied substantially, even within the same country and largely based on income levels. Schools in wealthier areas have shown higher prevalence of learning through live interaction with teachers and online participation of students. 7 High-income students also tend to have access to better homeschooling inputs, including technology, help from parents,8 and online learning resources,91011 which exacerbates educational inequalities.
Policymakers have explored numerous avenues to address these challenges, including tutoring, a program often adopted for remedial education. Taking many forms, tutoring generally consists of supplemental one-on-one or small group instruction sessions aimed at improving students’ academic performance.11 Existing evidence12 has pointed toward tutoring as an effective strategy for addressing learning losses and helping students catch up, that has consistently led to large improvements in learning in certain contexts. Yet, little evidence exists on whether a remote delivery model, as it was made essential during a global pandemic, could achieve similar results.
Context of the evaluation
In 2020, Italian schools were closed from the beginning of March until the summer – more than a third of the entire school year. To facilitate the implementation of remote learning, the government allocated 85 million euros for buying tablets that students could temporarily borrow, improving internet connection and online platforms of schools, and offering digital training to teachers. Yet, not all students in need received support in time due to bureaucratic delays. Additionally, the response of teachers to remote teaching varied greatly, with some taking longer to adjust to remote teaching, as prior to the pandemic, less than 50 percent of teachers had used digital tools in their daily lectures. By June 2020, more than 96 percent of teachers in the sample were providing live online classes, 85 percent also provided about an hour of recorded content per week, and almost all assigned some homework every week.
In response to the educational crisis, researchers partnered with pedagogical experts to rapidly design and implement an innovative online tutoring program: TOP (“Tutoring Online Program”). The program targeted middle school students (grade 6 to 8) from disadvantaged backgrounds in terms of socioeconomic status, linguistic barriers, or learning difficulties, who were identified by school principals among those lagging behind during distance learning. Thirty-two percent of selected students had a learning disability, 22 percent were immigrants, and 81 percent needed help in more than one subject. The majority of tutors were female (70 percent), born in Italy (98 percent), moved by a desire to help others when applying to TOP (83 percent), had previous experience as volunteers (83 percent), and were pursuing a STEM (34 percent) or economics/business major (28 percent).
Details of the intervention
Researchers conducted a randomized evaluation to assess the impact of TOP on students’ learning outcomes, socio-emotional skills, aspirations, and psychological well-being. Shortly after the announcement of schools’ closure, researchers contacted all middle school principals in Italy asking them to identify up to three students per class who might benefit from additional support during the months of remote learning and ensure they had internet connection and a computer or tablet. In the meantime, they recruited volunteer tutors amongst the students enrolled in graduate and undergraduate programs at three large universities in Milan. Then, researchers invited the identified students and their parents to participate in the free program. A total of 1,059 students from 76 middle schools from all over the country applied to the program and researchers randomly assigned a tutor to 530 of them, with the remaining 529 students serving as a comparison group. Researchers processed applications on a rolling basis to ensure that students could start as soon as possible. Tutoring was offered fully online and consisted of help with the homework assigned by school teachers in three subjects: math, Italian, English. Tutoring activities began in April 2020 and lasted for five weeks, until the end of the school year in June.
The tutor training was the most expensive component of the program and limited the number of tutors recruited to 530, though 2,000 volunteers had applied. Tutors received an online self-training module, regular group meetings, and on-demand one-to-one sessions with expert educators. The majority of students received 3 hours of tutoring per week, with a subset of those needing help in more than one subject randomly assigned to receive 6 hours per week.
To measure the impact of the program, researchers administered a standardized test very close in format, though shorter, to the national standardized one to participating students and conducted surveys of parents, students, tutors and teachers to measure students’ academic performance, aspirations, socio-emotional skills, and psychological well-being. Additionally, the surveys enabled researchers to measure the impact of the program on tutors’ ability to empathize with students and views on the role of hard work and effort to achieve success in life. Researchers collected data in April, before the tutoring began, and in June.
Results and policy lessons
Overall, researchers found that the online tutoring program improved students’ academic performance, socio-emotional skills, and psychological well-being.
Student effort, attendance, and behavior: students enrolled in TOP exerted more effort in their homework compared to their peers, according to reports by students, parents and teachers. Additionally, teachers reported that students in TOP were 9.4 percentage points more likely to attend class regularly, a 16 percent increase with respect to their peers in the comparison group. Students in TOP were also 8.1 percentage points (or 10 percent) less likely than their peers to report difficulty in following the classes online and using their school’s online platform. Students in TOP also behaved better during school hours, with the fraction of students for which teachers reported behavioral problems being 6.4 percentage points lower relative to the comparison group, where behavioral problems were reported for 83 students. Finally, students in TOP were 4.9 percentage points more likely to enjoy the subjects of math, literature or English relative to comparison students (amongst which only 28 percent reported enjoying the subjects).
Academic performance: For students in TOP, standardized test scores improved by 0.26 standard deviations relative to the comparison group. Students who had participated in the tutoring sessions, answered more questions correctly, a 9 percent increase relative to an average of 53 percent correct in the comparison group. The effects are particularly strong for math, the subject in which most students received tutoring. Teachers’ assessments of overall learning also improved for students in TOP compared to their peers, by 0.18 standard deviations.
Aspirations, socio-emotional skills and psychological well-being: The program did not have an impact on student or parent aspirations. Students in TOP reported a greater belief in their ability to control the outcome of events in their lives (0.19 standard deviations), fewer symptoms of depression (by 0.16 standard deviations), and parents reported higher happiness levels for their children (by 0.16 standard deviations), relative to the comparison group.
Different effects based on students’ characteristics: Children from lower socioeconomic background, whose parents both worked outside the home, saw a larger improvement in learning outcomes. Additionally, students with learning disorders seemed to benefit more from the program and saw their performance increasing by over 0.5 standard deviations. This is potentially because this is a group that might have struggled the most with the learning methods and materials that schools provided during distanced learning. Additionally, the improvement in wellbeing was entirely driven by immigrant students, for whom the effect was an increase of 0.77 standard deviations. Researchers hypothesized that this group might have a less dense network of friendship and so felt more isolated during the lockdown and benefitted from the regular interaction with a tutor. Turning to parental employment, having a tutor improved students’ performance significantly more for the children of blue-collar mothers compared to white collar mothers.
Impact on tutors: Researchers found that participating in TOP increased tutors’ empathy by 3.4 percentage points, 0.27 standard deviations increase compared to volunteers who were not assigned a student to supervise. Researchers did not detect a meaningful effect on tutors’ perceptions of the role of hard work to achieve success in life.
The researchers are currently piloting TOP in the Dominican Republic and having discussions with several other countries in Latin America to expand the program in the fall of 2021. In Italy, the program will be scaled to 3,000 children during the school year 2021-22 as part of a wider partnership with the Ministry of Education.