Evaluating the Impact of Online Education in Russia
- Students
- Student learning
- Computer-assisted learning
- Technology
- School-based inputs
More instructors are needed to meet the demand for expanding the science, technology, engineering, and mathematics (STEM) workforce globally. Researchers conducted a randomized evaluation to test the impact and cost-effectiveness of fully online and blended STEM classes in Russia on student test scores. Students in online and blended courses achieved similar scores to students in traditional in-person courses at a lower cost.
Policy issue
There is demand for more STEM workers globally. To meet this demand, higher education institutions are searching for ways to respond to rising costs linked to bringing in talented teachers and accommodating more graduates. China, India, Russia, and the United States—the four countries in the world producing the most STEM graduates—have launched initiatives actively seeking to increase the cost-effectiveness of providing STEM education at scale.
Blended or fully online teaching has been recognized by experts in education and economics as a way of increasing access to quality higher education. Previous studies on blended or online instruction suggest that they can deliver learning gains at a similar or slightly lower level compared to in-person programs. Globally, several national online education platforms have emerged with the capacity to leverage resources from the top universities, such as top instructors and their lectures, to support resource constrained universities. However, prior research has mainly examined the effects of fully online and blended programs on a small scale and there remains questions about the potential impact and cost savings of a nationwide implementation. Can online courses serve as a scalable and cost-effective alternative to in-person courses?
Context of the evaluation
In 2015, concerns arose about the quality and cost-effectiveness of higher education in Russia. In response, the Ministry of Higher Education and Science (MHES) took an active approach by introducing comprehensive online university courses to the public. This initiative was made possible through the collaboration with a non-profit organization called OpenEdu.
Using curriculum from eight top Russian universities, any Russian university can pay a small fee to gain access to fully online or blended courses for their students on OpenEdu, taught by more experienced professors. Both in-person, online, and blended courses can meet Russia’s Federal State Education Standards, including most courses on the OpenEdu platform, meaning all students are exposed to standardized course content.
Three universities participated in the evaluation. Researchers selected universities based on their willingness to integrate online courses from OpenEdu and replace an in-person course for a second-year cohort of STEM students. These universities ranked in the bottom half of 458 higher education institutions in Russia. The participating cohorts included 98 mechanical engineering students from the first university, 140 civil engineering students from the second, and 87 mechanical engineering students from the third.
Details of the intervention
Researchers conducted a randomized evaluation to compare the effect of in-person, online, or blended STEM classes on student achievement in Russia. Researchers partnered with three resource-constrained universities that required students in selected majors to take Engineering Mechanics (EM) or Construction Materials Technology (CMT). The researchers randomly assigned second-year students from the selected majors to complete the courses under one of three conditions:
- In-person group (101 students): Students attended the EM or CMT classes fully in-person with the same instructors that usually teach at their university. Students attended a lecture with up to 300 other classmates and then participated in smaller discussion groups of up to 30 students.
- Online group (124 students): Students took the same course but fully online through the OpenEdu platform, with content produced and delivered by professors based at highly ranked federal university. Students completed course assignments online and did not participate in discussion groups.
- Blended group (100 students): Students took the same course, but in a blended format where lectures were held online through OpenEdu as in the online group, while discussion groups were in-person with the same professors as in the in-person group.
The researchers considered three student outcomes: final exam grade, average grade on course assignments, and satisfaction with the course. Additionally, they assessed the cost-effectiveness of scaling different instructional models.
Before the classes started, students completed a pretest to gauge their understanding of course content and a questionnaire about their sociodemographic characteristics. After the course, students reported their satisfaction with the course and completed a final exam. Additionally, researchers collected data on the instructors’ sociodemographic background, research experience, and teaching experience.
Results and policy lessons
Students from in-person, online, and blended courses scored similarly on the final exam. In terms of average assessment scores, students in the in-person and blended groups performed equally well, whereas those in the online group performed better. While blended and in-person students had similar satisfaction levels with their course, online students were less satisfied than in-person students. The online and blended model demonstrated cost savings.
Final exam scores: Across all three groups, students received comparable final exam scores.
Average grade across all course assessments: The average assessment score varied by teaching model: students who took the course online scored 7.2 percentage points higher than students in the in-person and blended groups (for an overall score of 81.6 percent up from 74.4 percent), who had similar average assessment scores. However, researchers suggest this difference is likely because online students had three chances to submit their weekly assignment for grading, compared to only one chance for in-person and blended students.
Student satisfaction: Online students were 5 percentage points (down from 63 percent to 58 percent) less satisfied with their course experience than in-person and blended course students.
Cost-effectiveness: If institutions opted for a blended instead of in-person model, they could teach 3.4 percent more EM students and 2.5 percent more CMT students at the same price. If the institutions used online instruction, they could teach 18.2 percent more EM students and 15.0 percent more CMT students.
Chirikov, Igor, Tatiana Semenova, Natalia Maloshonok, Eric Bettinger, and René F. Kizilcec. 2020. "Online Education Platforms Scale College STEM Instruction with Equivalent Learning Outcomes at Lower Cost." Science Advances 6, no. 15 (April). doi: 10.1126/sciadv.aay5324