By Silvia Montoya, Director of the UNESCO Institute for Statistics (UIS); Hilaire Hounkpodoté, PASEC Coordinator; and Martin Gustafsson, Researcher, Research on Socio-Economic Policy (ReSEP), University of Stellenbosch

The Sustainable Development Goal 4 (SDG 4) indicators dealing with learning proficiency have received significant attention in recent years given the centrality of improving learning outcomes for tackling social inequalities and poverty.

The COVID-19 pandemic has had a devastating effect on learning around the world. Recent research suggests that as of October 2020 a year’s worth of learning was lost at the primary level, on average, if one takes into account the psychological and forgetting effects of school closures. In sub-Saharan Africa, somewhat less severe declines are expected in the percentage of children who are proficient. However, this is largely due to the fact that before the pandemic learning gains from one grade to the next were especially low in the region.

Comparing trends over time and across countries

While simply measuring learning is not the solution to higher proficiency, it can inform policy discussions around whether or not an intervention has been successful. If improvement is occurring, then something is working, and identifying the causal factors, while difficult, should receive attention. On the other hand, if no improvement is occurring, then it is likely that even ‘good-looking’ interventions are not working and need to be changed or replaced.

Given that even small improvements over time can be a significant sign of progress, assessment programmes aiming to detect trends need to establish – reliably – whether or not small changes are occurring. Achieving this is not easy, and any new assessment programme should accept that greater measurement accuracy is only reached gradually, as lessons are learned and technical enhancements made.

An African programme which is well-documented, and is moving towards better comparability over time, is Ghana’s National Education Assessment. Preceding the 2016 round of testing, item banks of questions were developed to better monitor trends. These enhancements put Ghana in a better position to report on progress against the SDG 4 learning proficiency indicators, and to have data-informed discussions on strategy.

At the regional level, the Southern and Eastern Africa Consortium for Monitoring Educational Quality (SEACMEQ) conducted testing across fifteen countries in 2000, 2007 and 2013, and plans to run assessments in 2021. This work has produced valuable trends over time, and comparisons across countries.

The CONFEMEN programme, PASEC has seen nineteen Francophone African countries participate in it since 1990. Between 1991 and 2012, PASEC carried out national evaluations in almost all the French-speaking countries of sub-Saharan Africa as well as in Lebanon and in three South-East Asian countries (Cambodia, Laos and Vietnam), using a value-added methodology (survey at the beginning and end of the school year). Additionally, thematic analyses, particularly on the status of teachers, repetition and cohort follow-ups also were undertaken.

Since 2014, attention has been given to improving comparability over time, specifically between the 2014 and 2019 evaluation. This was achieved through the introduction of anchor items – essentially test questions which are repeated in every assessment.

The recently published 2014 to 2019 PASEC trends point to two countries, Benin and Niger, displaying clear improvement trends, with statistically significant gains for both reading and mathematics in both Grades 2 and 6. These countries saw annual gains of just over 0.1 of a standard deviation. Such gains are very large compared to progress reported historically.

While programmes such as PASEC are not designed to identify precisely what factors are causing the changes in the average scores, they nevertheless provide a wealth of background information which can support efforts to determine what the causal factors are. Analytical methods are emerging which can assist in narrowing down the possibilities. Obviously, understanding what leads to better learning proficiency is vital. The official PASEC report is just a first step. Going forward, it is important that researchers, in particular African researchers inside and outside government, use the microdata and explore the patterns further.    

Together, SEACMEQ and PASEC cover 59% of sub-Saharan Africa’s school age children, with SEACMEQ’s fifteen countries representing 28% of the region’s children, and PASEC’s nineteen countries representing 31%. Mauritius is the only country participating in both.

Challenges for both programmes include strengthening the comparability of test results over time, and making the microdata and technical documentation more easily available. Along with building trust in the programmes, these issues are essential for developing local African expertise in psychometric techniques.

Non-psychometric and psychometric methods of cross-country comparison

National education planners are also interested in comparing their country to other countries, not just looking at national trends over time. Cross-country comparability is also important for global monitoring purposes. To this end, attention must be paid to strengthening the capacity of countries to develop evaluations that meet international standards.

The UNESCO Institute for Statistics (UIS) has employed non-psychometric methods to understand, for instance, how a level of proficiency represented by a score of X in PASEC is represented on SEACMEQ’s scale. Such methods rely strongly on bridge countries – those participating in more than one testing programme – to convert programme-specific scores to a universal scale. Mauritius is an example of a bridge country. While this approach is convenient and low-cost, it has many drawbacks. One clear drawback, for instance, is that even in bridge countries the tested learners tend to be from different grades and different points in time.

The UIS has also worked on more rigorous psychometric methods for linking different programmes. The Rosetta Stone initiative, launched in 2017, involves having the same learners take tests from two different programmes. In Africa, this dual testing was conducted recently in three countries, using PASEC Grade 6 questions, and questions from two IEA programmes, TIMSS and PIRLS, aimed at Grade 4 learners. This will enable global comparisons, allowing, for instance, Senegal, to gauge how well learners in that country perform in reading relative to learners in countries where PIRLS has been run, for instance South Africa, Morocco and Indonesia.    

The next frontier: vertical linking

While comparing learning across countries – for instance, reading proficiency of Grade 6 learners in Country X in PASEC and Grade 4 learners in Country Y in TIMSS – provides valuable information, it becomes more meaningful if Country X is also in a position to monitor how much learning is gained between Grade 4 and Grade 6. Vertical linking can provide reliable results. This is a technique in which more difficult test items in Grade 4 tests are repeated in Grade 6 tests, where they are likely to be relatively easy.

Even where there is no cross-country comparison, knowing how much learning is gained in each grade is of enormous value when planning interventions. For example, if Grade 6 results are weak, this could be because grade-on-grade gains are especially weak in the first three years – Grades 1 to 3 – or in the next three grades – Grades 4 to 6. There is scope for vertical linking in both PASEC, which covers Grades 2 and 6, and in many national assessment programmes in Africa.  

Measuring learning outcomes – even small, incremental changes over time – and being able to compare results across countries, is a key part of achieving the SDG 4 objective of improving the quality of education for children around the world. Implementing innovative methods of learning proficiency assessment between the countries in the world, especially in Africa, will thus enable a better understanding of how the continent, and different nations within it, are progressing towards the SDG 4 targets.