By Silvia Montoya, Director, UNESCO Institute for Statistics; Maurice Walker, Research Director, Education Policy and Practice, Australian Council for Educational Research; and Luis Crouch, interim Chair of the UNESCO Institute for Statistics Governing Board
The COVID-19 pandemic has disrupted education around the world, forcing countries to respond by providing remote learning, adapting curricula and assessments, and developing ways to support the health and well-being of students, teachers and families.
But the concern is that, despite these efforts, children’s learning has been severely affected.
Before the onset of COVID-19, available data from across Africa showed that most children could not read or understand a simple age-appropriate text. Only 29% of African learners in the early grades were proficient in reading compared to 57% of those in developing countries. However, these estimates are imputed from a small sample of countries. The reality is that most low- and lower-middle-income countries do not carry out robust learning assessments.
The Monitoring Impact on Learning Outcomes (MILO) project, developed by the UNESCO Institute for Statistics (UIS), is a direct response to that need, providing a way for countries to measure progress against SDG indicator 4.1.1b: the percentage of students at the end of primary school who achieve a minimum proficiency level (MPL) in reading and mathematics. The project aimed to investigate changes in learning outcomes, and the effectiveness of emergency teaching/learning strategies, in six sub-Saharan African countries: Burkina Faso, Burundi, Côte d’Ivoire, Kenya, Senegal and Zambia.
The MILO project developed an Assessments for Minimum Proficiency Levels (AMPLs) toolkit, which was specifically calibrated to the MPL global definition. This toolkit provides an affordable methodological option for countries to measure learning and to develop capacity to regularly generate, analyse and report learning data, as well as monitor progress towards SDG 4. The toolkit has been initially created to monitor indicator 4.1.1b.
The MILO project addressed four overarching goals:
- Evaluate the impact of COVID-19 on learning outcomes by reporting against SDG 4.1.1b
An assessment of reading and mathematics was administered to students at the end of primary school in mid-2021. Results were then compared to those obtained prior to the pandemic in national assessments (e.g. the National Assessment System for Monitoring Learner Achievement grade 7 in Kenya or the 2016 National Assessment Survey in Zambia) and the PASEC cross-national assessment administered in Francophone African countries.
- Comparisons between reading proficiency levels before the pandemic and in 2021 showed there was no difference in the proportion of students who met the MPL in Burkina Faso, Burundi, Côte d’Ivoire, Senegal and Zambia. In Kenya, a comparison was not possible.
- In Burundi, Côte d’Ivoire, Senegal, Kenya and Zambia, no differences were found either in the proportion of students who met the MPL in mathematics, although there was some evidence of learning loss for boys in Kenya. By contrast, in Burkina Faso there was a statistically significant increase in the proportion of students who met the MPL from 18% in 2019 up to 24% in 2021.
Several factors may explain this apparent stagnation, including the possibility that learning outcomes would most likely have continued to improve, as a recent UIS information paper has shown, had it not been for the pandemic:
- Students on track to achieving the MPLs may have been less impacted by COVID-19 school closures.
- The low proportions of students meeting the MPLs in historical assessments make decline difficult to observe on that scale.
- Students may already had recovered from any learning losses by the time the assessment was carried out.
- Mitigation strategies set out by schools may have lessened the impact on reading and mathematics outcomes but not necessarily for other, unassessed, school subjects.
- Identify the impact of different distance learning mechanisms put in place to offset the learning disruption generated by COVID-19
Contextual data was also collected to help understand how the COVID-19 disruption affected learning and to identify ways to support students in future. The information collected highlighted the range of methods used in the six countries to offset the impact of the pandemic. Methods included virtual lessons and use of digital tools, community engagement, and increased communication between schools and their students and families.
Apart from Burundi, the other five countries experienced school closures and put in place plans or policies to provide directions for teaching and learning, as well as health and wellbeing, in response to the disruption. Still, across the MILO countries, only 21% of students attended schools where the principal reported offering remote learning programs to all students; and only 9% schools where live virtual lessons or digital materials were available. The key barriers identified by principals to providing remote learning were student access to digital devices or to the internet. The most common strategies principals reported were engaging the broader community and communication between staff and students.
A common finding is that support tended to be inequitably distributed. For instance, in terms of family inputs, students in Burundi, Côte d’Ivoire, Kenya, Senegal and Zambia who received more support tended to be more proficient in reading and mathematics compared to those who received less support. As for school inputs, the most commonly reported method of support to students was tips about self-study and follow-up to check students had completed schoolwork – but, even in this case, students in Côte d’Ivoire, Senegal and Zambia received more support if they were more proficient in reading and mathematics.
- Develop more affordable assessment tools as global public goods
The MILO project generated a set of tools under rigorous technical standards that countries and assessment programmes can use to measure and report learning outcomes against SDG 4.1.1b in the future. The toolkit, which includes the AMPL itself, also provides documentation to support the administration of the proficiency level test, technical standards, the assessment blueprint, contextual framework, field operations guidelines and a description of the analysis methods used in the study.
The AMPL can be implemented as a standalone assessment specifically targeted at the MPL to report against SDG 4.1.1b. Or, should a country, region or system wish to, it could be used to help measure and describe the broad range of abilities children at the end of primary acquire by being administered alongside existing national or regional assessments, as was done in the MILO project. Although, currently, the AMPL covers the end of primary schooling outcomes (AMPL–b), the same methods could be developed to measure learning outcomes in early grades (AMPL-a for indicator 4.1.1a) or at the end of lower secondary to address (AMPL-c for indicator 4.1.1c).
- Contribute to the UIS Global Item Bank
Among the constraints countries face to assess learning is access to high quality calibrated items that target SDG indicator 4.1.1 and align with the Global Proficiency Framework. The UIS aims to fill this gap with the Global Item Bank. The objective is to share as widely as possible assessment items of good quality created by countries and assessment programs for use in their assessments.
As part of the MILO project, the Australian Council for Educational Research (ACER) – through the Global Education Monitoring Centre – and the Conférence des Ministres de l’Éducation des États et Gouvernements de la Francophonie (CONFEMEN) added items to the UIS’s bank, expanding the pool of tools available to countries.
Quality assurance guidelines were also developed to enhance future contributions to the UIS Global Item Bank and a digital platform under development will provide access in 2022 in a user-friendly format.
The new AMPL toolkit offers an example of how the Global Item Bank can be used to efficiently develop assessments using high-quality material that also enable global reporting. The ultimate goal is to develop the capacity and the culture of monitoring at the country level in a cost-effective way by making available materials that can be considered as public goods.
