Complement or Substitute? The Effect of Technology on Student Achievement in India
- Primary schools
- Students
- Student learning
- Technology
- School-based inputs
Though improving education quality in low- and middle-income countries is a priority for many policymakers, little is known about how to improve school quality in a cost-effective way. Researchers in India attempted to measure the impact of a daily Computer Assisted Learning program on student test scores when delivered as either a substitute for or supplement to the status quo curriculum (delivered during or after school, respectively). While the program improved math scores for students when delivered as a supplement, it actually had a negative impact on math scores when delivered as a substitute for the status quo curriculum. This evaluation demonstrates the importance of considering the way a program is implemented relative to the status quo, rather than focusing just on the program components in isolation.
Policy issue
Over the past decade many low- and middle-income countries have expanded primary school access, energized by initiatives such as the United Nations Millennium Development Goals, which call for achieving universal primary education by 2015. However, there are major concerns about the quality of the education in low- and middle-income countries—with surveys of achievement showing low levels of learning even for children who attend school regularly for many years. While much is now known about how to get children into school, much less is known about how to improve school quality in a cost-effective way.
The use of information and communication technologies (ICT) is becoming a major consideration as low- and middle-income countries focus on improving the quality of their educational systems. Several countries are determinedly expanding the supply of computers in their educational institutions in the belief that schools will benefit from the use of new technologies and that students need to be exposed to them early in life. Despite this growing adoption of and demand for ICTs in education, relatively little systematic research and hard data is reported on the methods of ICT usage in the classroom. Even less is known about the impact on educational outcomes, social behavior, or employment and worker productivity. While there is much attention for and interest in ICT programs in the education literature, robust evaluations are still too scarce to provide general conclusions regarding their effectiveness. This study focuses on understanding how new technologies and teaching methods both interact with existing resources and differentially affect students with diverse needs and abilities.
Context of the evaluation
Although enrollment rates have increased significantly in India, a recent countrywide survey of rural children (ASER, 2007) found that only 58.3 percent of children in fifth grade could read at the second grade level. Similarly, only half of 9-10 year old children who were attending school could demonstrate basic numeric problem solving skills. Computer-assisted learning programs are generating excitement in India, where the high-tech industry is successful. These programs are designed to reinforce materials that students have already learned using novel, interactive tools, but schools have little guidance on how, or even if, they should use computer based educational assistance.
Details of the intervention
This study measures the impact of the same Computer Assisted Learning (CAL) program implemented in two different ways, thereby illustrating how a pedagogically similar program can produce drastically different results when context and implementation methods are slightly altered. Designed in two phases and conducted in four communities in Western India, all schools were part of the Gyan Shala project of the Education Support Organization, an NGO that develops new systems for delivering basic, primary education to children of low-income families.
In the first phase, a treatment group of 392 second and third graders in 23 schools participated in a pull-out CAL program for one year. These students received one hour of CAL instruction per day during school hours in lieu of the normal curriculum. In the second phase of the study, a new treatment group of 682 second and third graders in 37 schools participated in a supplemental CAL program for one year. These students received one hour of CAL instruction per day after school, as a supplement to the normal curriculum during school.
Both groups of students were compared against control students of the same age in their schools who received no CAL instruction. Student improvements were measured by comparing students’ math and language test scores in April of the academic year prior to the study with students’ math and language test scores in April of the academic year of the study.
Results and policy lessons
The research design aimed at disentangling the effects of these variations of the CAL program in different settings. Results are found in the following areas:
Pull-out vs. Supplemental Method: Students who participated in the supplemental CAL program had math scores 0.28 standard deviations higher, on average, than students in the control group. Students who participated in the pull-out CAL program, however, had 0.57 standard deviations lower math scores than students in the control group. It is surprising that the intervention which assigns one hour a day to computerized learning, away from other instructional methods, has a largely negative effect on all types of students. The supplemental CAL program, for example, also demonstrated more positive results for low-performing students than for high-performing students, an effect which was not observed in the pull-out program. This difference in treatment effect for stronger and weaker students (ranging from small and insignificant changes for most students, but gains of 0.47 standard deviations for the poorest performing students) seems to reflect the design of the program which emphasized reinforcement of material that the students had already learned rather than self-paced discovery of topics not covered in class. Thus, the method and context of implementation of the CAL program may play a significant role in determining its impact.
Cost Effectiveness of CAL: The cost-effectiveness of the out-of-school CAL program was between $3.22 and $4.59 per tenth of a standard deviation in test scores, putting it on par with other student performance interventions in low- and middle-income countries. The urban CAL program was shown to be more cost-effective than other math-based CAL programs. It was just as cost-effective as a girl’s scholarship program, cash-incentives for teachers, and textbooks. It was less cost-effective than a remedial education program and some teacher training programs.
Policy Relevance: These results emphasize the importance of considering both the relative productivity of learning environments and the effects that those difference will have on different types of students. In this study, the intervention proved effective only as a complement to the current school curriculum. Decision-makers must consider not solely whether a program works, but rather how well the program works relative to what students would otherwise experience. As the current evaluation demonstrates, the format of an implementation can mean the difference between on the one hand providing needed assistance to weak students, or, on the other, generally causing all students to learn less than they would have learned without the intervention.