Improving Student Learning through Classroom Technology Adoption in Pakistan
- Students
- Teachers
- Student learning
- Technology adoption
- Pedagogical innovation
- Technology
Around the world, many students who are enrolled in school are struggling to learn, and the variability in teacher capacity and performance can be one barrier to high quality learning. Researchers partnered with the Government of Pakistan to conduct two parallel randomized evaluations to test the impact of providing supplemental teaching materials on the quality of student learning and teachers’ effectiveness. Providing grade 8 classrooms with televisions with pre-recorded video lectures improved student learning outcomes and attendance. Conversely, providing grade 6 students with individual eLearn Tablets equipped with learning materials, but that could also be used for other purposes, worsened their test scores.
Policy issue
Around the world, many students who are enrolled in school are struggling to learn and teacher capacity and performance often varies, acting as one barrier to high quality learning. In particular, high teacher absenteeism and limited incentives that encourage teachers to perform well have limited student learning and achievement in many contexts. In addition, working within existing government structures to improve student knowledge and success in high-stakes exams can be challenging.
Two common approaches policymakers around the world have used to try and improve student learning are: 1) to bypass teachers entirely and encourage independent learning through the use of tutors, assistants, or technology; and 2) encourage existing teachers to become more effective through extensive training and monitoring. However, these solutions have typically only been successful with careful non-governmental organization supervision that exceeds the governmental supervisory capacity in most lower-income countries.
Can providing teachers and students in government schools with high-quality explanations of concepts in the official science and math curriculum through modern-tech educational tools improve student learning and standardized test scores?
Context of the evaluation
In Punjab, Pakistan, most middle school students’ achievements in government schools are considered low. Middle school students take a standardized exam called the Punjab Examination Commission (PEC) that covers English, Islamic Studies or Ethics, Mathematics, Science, and Urdu. Student test scores signal completion of middle school and are required for entering a government secondary school. Nationally in 2014, only 16 percent of those enrolled in middle school obtained grade-level proficiency in math or science, demonstrating that most students were not ready to enter secondary school. Additionally, many school principals believed that a lack of teacher qualifications was a barrier to student learning. For example, the most common middle school science teaching methods in Pakistan have remained stagnant since independence, with a focus on rote learning and textbook memorization over conceptual understanding.
The Provincial Government of Punjab developed eLearn, a program intended to improve student learning in government schools. The schools in this study were all single gender government middle schools that charged at most minimal tuition fees. Overall, the sample schools in the study were similar to the average school in Punjab based on infrastructure and test scores.
Details of the intervention
Researchers partnered with the Government of Pakistan to conduct two parallel randomized evaluations in Punjab, Pakistan to test the impact on the quality of student learning and teachers’ effectiveness of providing supplemental teaching materials either directly to students or for teachers to share with the classroom.
The eLearn program consisted mostly of video lectures developed and presented by expert teachers to explain a particular math or science concept. All videos directly mapped to the units of the official curriculum, were less than thirty combined hours of content designed to be spread over the entire school year, and were designed to improve student learning by providing expert content to enhance existing teachers. The videos used extensive visual aids to reinforce important concepts and had follow-up questions to assess comprehension. Teachers received a two-day in-service training focused on: 1) using the multimedia content; and 2) steps to incorporate multimedia into a more effective blended teaching practice. The two randomized evaluations tested different versions of the eLearn program, eLearn Classroom and eLearn Tablets, in separate schools in Punjab during the same academic school year. The interventions are as follows:
- eLearn Classroom: Researchers randomly selected thirty schools to receive the eLearn Classroom module. Within those schools, grade 8 math and science teachers were equipped with video lectures and 40-inch LED television screens in the classrooms. Teachers could watch the videos to prepare for lectures and project them on the screens during class time. Twenty-nine schools were randomly selected not to receive any eLearn program, serving as the comparison group.
- eLearn Tablets: Researchers randomly selected nineteen middle schools to participate in the eLearn Tablets program. Within those schools, grade 6 students and science teachers received personal eLearn Tablets, but the teachers did not have a way to display the material to the entire class at once. The Tablets were equipped with pre-loaded self-paced video-content, simulations, questions, and digital textbooks. Students obtained training and guidelines on how to use the tablets independently. Students could load non-academic materials such as game applications and access to movies on the tablets. Fifty-two schools did not receive the eLearn Tablets, serving as the comparison group.
The intervention took place in the Lahore, Multan, and Rawalpindi districts of Punjab Province during the 2016 – 2017 Pakistani school year. Researchers gathered information from head teachers, relevant grade math and science teachers, as well as randomly selected students to measure student achievements. They also administered exams covering math and science concepts in the established government curriculum, as well as additional higher order conceptual and problem-solving skills that do not rely on rote memorization. Additionally, researchers used administrative data to obtain subject level PEC exam results for grade 8 students in eLearn Classroom schools as well as publicly available teacher attendance data.
Results and policy lessons
Overall, the eLearn Classroom intervention improved student achievement and students’ attendance for grade 8 students. Conversely, providing grade 6 students with individual eLearn Tablets worsened their test scores.
Technology Usage in the Classroom: Ninety-five percent of teachers assigned to either the eLearn Classroom or eLearn Tablets reported using the technology at least twice a week, and around 70 percent of teachers and 80 percent of students reported finding the new technology to be “very useful” for student learning.
Student Achievement in eLearn Classrooms: Grade 8 students in classrooms that received the LED televisions and eLearn Classroom learning curriculum improved their PEC score by 0.22 standard deviations and increased their combined math and science scores by 0.27 standard deviations relative to students in the comparison group. The greatest increase in test scores were in schools with the lowest test scores taken at the beginning of the study, likely those with the most acute teacher capacity problems, and among students with the lowest baseline scores, those likely to be far below the grade level and least likely to self-learn content without teachers. The researchers posit that the teachers learned from the high-quality videos and adapted their teaching practices accordingly, so these improvements in student learning were from a combination of students watching the lectures and their teachers improving their teaching capacity. Additionally, students in the eLearn Classroom intervention were about 5 percentage points more likely to pass the government standardized grade 8 PEC exam relative to the comparison group’s mean passing rate of 92 percent.
Student Achievement in eLearn Tablets: Receiving personal eLearn Tablets worsened grade 6 students’ combined math and science scores by 0.42 standard deviations relative to their peers that did not receive tablets.
Student and Teacher Attendance: The eLearn Classroom increased students’ attendance by 4 percentage points relative to the comparison group’s average attendance of 85 percent. The eLearn Classroom also increased teachers’ attendance by 1 percentage point relative to the comparison group’s teacher attendance rate of 94 percent, which is in contrast to the concern that providing video lessons could lead to more teacher absenteeism if teachers used the videos as virtual substitutes. The eLearn Tablets, however, had no effect on either students’ attendance or teachers’ attendance in school.
Cost Effectiveness: Researchers estimate the eLearn Classrooms would cost US$31 per student if scaled to 100 schools and US$11 per student for 1000 schools. The eLearn Tablets, however, were substantially more expensive per student, harder to scale, and ineffective.