The Use and Misuse of Computers in Education: Evidence from a Randomized Controlled Trial of a Language Arts Program
- Students
- Student learning
- Digital and mobile
- Technology
Though many countries have expanded primary school access over the past decade, concerns persist about education quality in low- and middle-income countries: surveys find low learning levels even for children who regularly attend school for many years. In Colombia, researchers evaluated the impacts of a national computer distribution program (Computers for Education) on student test scores, time spent on learning, and attitudes towards education. The program had no effect on math or Spanish test scores, hours of study, or perceptions of school. Researchers found that, in practice, teachers only used the computers to teach computer usage skills, rather than the range of subjects for which they were intended.
Policy issue
Over the past decade, many developing countries have expanded primary school access, energized by initiatives such as the United Nations Millennium Development Goals, which call for achieving universal primary education by 2015. However, there are major concerns about the quality of education in developing countries—with surveys of achievement showing low levels of learning even for children who regularly attend school for many years. While much is now known about how to get children into school, much less is known about how to improve school quality in a cost-effective way.
The use of information and communication technologies (ICT) is becoming a major consideration as developing countries focus on improving the quality of their educational systems. Several countries are determinedly expanding the supply of computers in their educational institutions in the belief that schools will benefit from the use of new technologies and that students need to be exposed to them early in life. Despite this growing adoption of and demand for ICTs in education, relatively little systematic research and hard data is reported on the methods of ICT usage in the classroom. Even less is known about the impact on educational outcomes, social behavior, or employment, and worker productivity. While there is much attention for and interest in ICT programs in the education literature, robust evaluations are still too scarce to provide general conclusions regarding their effectiveness.
Context of the evaluation
Colombia is a relatively typical middle-income, Latin American country. Similar to many middle-income countries, school attendance in Colombia is close to 100 percent for younger children but declines substantially after the age of 13. Average attendance is 92 percent among 15-year-olds, 90 percent among 16-year-olds, and 80 percent among 17-year-olds. The drop is considerably faster for low-income individuals: by age 17 the attendance rate falls to 65 percent in this group. Yet, even though attendance rates are on a par with the rest of the region, Colombia’s educational performance is low compared to the rest of Latin America and falls short of its potential relative to income level.1
Details of the intervention
In an effort to improve the quality of its educational system, the Computers for Education program was created in 2002 by the Minister of Communications and has now been scaled up across the country. As an alliance between the public and private sector, its major purpose is to refurbish donated computers, install them in schools, and train teachers to use the computers in specific subjects, especially Spanish. Since its creation, the program has received over 114,500 computers and refurbished nearly 73,665 that have been installed in 6,386 public schools in 1,018 municipalities. To date, the program includes over 83,000 teachers and more than 2 million students. Moreover, the program creates partnerships between schools and local universities which are responsible for the design of the pedagogical strategy and participate in a 20 months training component for the participating teachers. These training include teaching methodologies using computers to strengthen students’ reading and writing skills.
In order to evaluate the program, 97 interested and eligible schools were randomly assigned into treatment and comparison groups. The lottery was performed at the beginning of August 2006, and the list of schools was given to the Ministry of Communication for the implementation of the program. A baseline survey was conducted immediately after the randomization, and a follow-up survey was conducted two years later. Both the baseline and the follow-up questionnaires followed the same format. Students were asked about socioeconomic characteristics, school outcomes (attendance, hours of study, grades, and dropout spells), and attitude towards the school, including the use of computers. Students also completed a shortened version of the national Colombian exam. Teachers were asked about their education and experience, their knowledge of computers, and their use of computers in class. The sample contains 3,889 comparison students and 4,327 treatment students enrolled in grades three through nine at the time of the follow-up survey. Due to the high attrition rate of 37 percent during the follow-up exam, the final sample is comprised of 5,201 students.
Results and policy lessons
The effects of the program were estimated by directly comparing the average test scores and survey responses in the treatment and comparison groups. Even though the program was successful in increasing the number of computers in treatment schools—on average these schools have 13.4 computers compared to 5.1 in comparison schools—the program had surprisingly little effect on test scores. In Spanish, the subject targeted by the program, treatment students did not perform significantly better than the students in the comparison group. Additionally, no improvements were seen in math. When considering the effects on specific subgroups of students, such as gender and age, there are no significant differences between treatment and comparison students either. Moreover, Computers for Education had little effect on a number of other academic variables including hours of study, perceptions of school, and relationships with their peers.
Surveying both teachers and students reveals that even though the number of available computers in treatment schools rose significantly, the program increased the actual use of the computers among students and teachers by a surprisingly small amount. Despite the program’s focus on using computers for teaching a range of subjects, they were only used to teach computer usage skills. Only 3 to 4 percent of students in both treatment and comparison schools reported using the computers in language class, which was one of the focal points of the program. Additionally, the extra computer use reported by teachers was concentrated in the lower grades, with teachers in higher grades reporting almost no computer use in treatment and comparison schools.
The results of this evaluation provide a sobering example of the potential limits of ICT interventions aimed at improving the methods that teachers use in the classroom. While previous academic studies have shown that computers in schools can significantly improve student test scores, the results of this study highlight the importance of implementation and training, specifically addressing the incorporation of available computers into the educational process and curriculum.
In J-PAL's comparative cost-effectiveness analysis, the Computers for Education program did not have an impact on test scores. For more information, see the full comparative cost-effectiveness analysis.