The Impact of a Public Sector Management Program on Student Learning Outcomes in India
- Primary schools
- Secondary schools
- Teachers
- School administrators
- Student learning
- Productivity
- Take-up of program/social service/healthy behavior
- Labor
- Commitment devices
- Information
- Administrative reform
- Technology
- Tracking
- School-based inputs
Improving state capacity for public-service delivery in low- and middle-income countries remains a major development challenge. In India, researchers evaluated the impact of a public sector management program involving comprehensive assessments of schools, detailed school ratings, and customized school improvement plans, on school functioning and student learning. Researchers found that the assessments were near-universally completed and the ratings were informative. However, the intervention had no impact on either school functioning or student outcomes.
Policy issue
Improving state capacity for public-service delivery in low- and middle-income countries remains a major development challenge. Providing incentives for front-line government staff to improve their performance has been found to be effective in many settings, but such programs are not easily scaled and sustained due to political and monitoring constraints. A popular approach to improving service delivery in the education sector has been to try to improve management practices within school bureaucracies. However, even though these programs are widely used and come with associated fiscal and personnel costs, we know very little about how school management reforms really affect school functioning and student learning at scale. Can a comprehensive school management program improve teacher effort, classroom processes, and student learning?
Context of the evaluation
According to the last government Census in 2011, the state of Madhya Pradesh in India had a lower literacy rate than the national average. Despite a well-defined formal structure for school accountability, including academic resource coordinators and community-based school management committees, the performance of the public education system was weak and declining at the time of the evaluation. For instance, student learning levels were low in 2016, with only 31 percent and 15.3 percent of Grade 5 students in government schools being able to read a text at Grade 2 level and do division, respectively. Furthermore, teacher absenteeism had worsened to 26.2 percent by 2010, compared to 18.2 percent in 2003.
To address this problem, the Government of Madhya Pradesh (GoMP) made school improvement a high priority and subsequently developed a comprehensive school management program with technical support from Absolute Return for Kids (ARK), a leading international education services provider headquartered in London. The program, titled MP Shaala Gunvatta program (MP School Quality Assurance program or MPSQA), consisted of three main components:
- Independent customized school assessments and corresponding ratings of school quality, to identify strengths and weaknesses
- School-specific improvement plans with actionable steps
- Regular follow-up by supervisory staff to monitor progress and provide guidance and support
The program aimed to involve school inspectors, staff, and parent representatives in the assessments and the creation of improvement plans. The program also developed online tools to make it easier for senior education officials to access school assessment results and progress reports. The government designed the program to be scalable across Madhya Pradesh, with plans to eventually reach about 100,000 schools over multiple scaling phases.
Details of the intervention
In partnership with the GoMP, researchers tested the impact of a comprehensive school management program on school functioning and student outcomes. Working closely with the GoMP in five districts, researchers identified 308 academic clusters, the lowest unit of school administration in the state that have around 40 schools on average, and randomly assigned all schools within a subset of 153 randomly selected clusters to the following groups:
- MPSQA program (51 clusters including 1,774 schools): The government introduced the MPSQA program described above into all schools in this group. Every academic cluster had resource coordinators that assessed each school’s quality in domains like pedagogy, school leadership, community engagement, and more with the help of an external evaluator, such as a retired headteacher. Schools used these assessments to develop custom school improvement plans, and the resource coordinators followed up quarterly to review schools’ progress on implementing the plans.
- Comparison group (102 clusters including 3,661 schools): Schools did not receive the MPSQA program; schools and cluster resource coordinators continued operating under the status quo.
To measure the impact of the program, researchers leveraged both administrative data and primary data on both learning and school processes. Researchers collected data on teacher absence and school functioning from a random subset of schools through three rounds of data collection between September 2015 and February 2016. In order to evaluate the program’s impact on student achievement, researchers leveraged administrative data and also conducted independent assessments in March 2016.
Results and policy lessons
Researchers found that the MPSQA program was successful in conducting high-quality external school assessments. However, these assessments and the subsequent creation of school improvement plans did not change management or teaching practices or student learning outcomes.
Implementation of school assessments: Researchers found that 93 percent of schools in the treatment group had completed school assessments and that the assessments were of high quality. Schools had also made improvement plans and uploaded them on the program website.
Support and monitoring by officials: Researchers did not find a sustained improvement in the support provided by supervising officials in treated schools. The officials also did not increase their monitoring of these schools, be it in the frequency of their visits or the content of their inspections.
Teacher effort and student engagement: Researchers found no evidence of improved pedagogy, teacher effort, or student engagement within schools receiving the program. The program did not affect teacher and student absence, both of which remained high, or teachers’ instructional time, use of textbooks and workbooks, or the likelihood of checking student homework.
Learning outcomes: Across both school-administered tests and independently-administered tests, researchers found that the program did not have any impact on student learning outcomes either in the short run (3-4 months after the intervention) or in the long run (15-18 months after program rollout).
To understand potential reasons behind the program’s ineffectiveness, the researchers conducted extensive qualitative interviews with teachers, principals, and field-level supervisory staff. They found that many school officials treated the program only as a data collection exercise rather than as a reflective exercise in self-evaluation and improvement. As such, schools were mainly concerned with ensuring that the required paperwork on school assessments and improvement plans was submitted on time, following which program delivery effectively ceased.
Following the evaluation, the program continued to be scaled up to about 25,000 schools across the state, as originally planned by the GoMP. The Phase II expansion was part of a larger rollout of a new national school management program, Shaala Siddhi. Shaala Siddhi was similar to the original MPSQA program but gave schools direct responsibility for self-diagnosis and suggesting improvements. As a result, schools now had to conduct self-assessments (that were verified by external assessors) and submit more detailed improvement plans.
In addition to these changes, the GoMP attempted to respond to the researchers’ findings by emphasising the need to focus on teacher behaviours and student outcomes over administrative compliance. Specifically, in the trainings provided to schools on executing School Improvement Plans, the GoMP laid out three focus areas: (i) teaching and learning, (ii) student learning outcomes, and (iii) developing a close collaboration with parents and the community.
Researchers conducted a non-experimental evaluation of this newer model in the Phase II expansion in Madhya Pradesh. However, they again found no impact on student learning outcomes.
The national school management program has since been rolled out to over 600,000 schools in India and is expected to cover 1.6 million schools eventually.