As the first in the new series of GEM Reports, the 2016 edition has looked in depth at the monitoring issues related to each individual SDG 4 target. This is timely: while the basic parameters of the global and thematic monitoring framework have been agreed, in many cases important details are still being decided. These issues need to be presented openly to the international community as a contribution to technical and political debate.

The scope of the monitoring agenda is wide, and the issues involved are often complex. This section aims to provide the bigger picture by drawing together conclusions, summarizing key messages and indicating common themes in challenges across targets.

A new mechanism is needed to help countries collect and compare information about policies successfully addressing disadvantages in education Click to Tweet

Target 4.5: In recent years, the World Inequality Database on Education has helped bring disparity in education opportunities between and within countries to the attention of the wider public. The launch of the Inter-Agency Group on Education Inequality Indicators will now advance this agenda by making effective use of a very large number of data sets.

Three major challenges remain. First, the choice of a suitable inequality measure is still open. The parity index, proposed as the global indicator, is easy to communicate but has notable weaknesses. Second, despite progress in global coordination, many education ministries do not yet monitor disparity. Third, global comparisons are currently possible only by sex, location and wealth. The search for measures of other markers – notably disability, language, migration and displacement – needs to continue.

Leaving no one behind, the rallying cry of the new agenda, will not be answered solely by a proliferation of disparity measures. It also calls for a concerted effort to monitor the policies pursued by countries to address education disadvantage, including policies outside education. A mechanism is needed to allow countries to collect and compare this qualitative information.

Target 4.6: One example of the gaps in monitoring adult education opportunities is the continuing absence of information on participation in adult literacy programmes. After years of advocacy, the new agenda has embraced a shift to a nuanced direct assessment of a range of literacy and numeracy proficiency levels instead of relying on dubious self-reported skills. However, as with assessments of learning outcomes in basic education, it is essential to help build national systems to monitor these skills. In addition, diversity in literate environment contexts and variations in national capacity for collecting and analysing data must be accounted for.

Target 4.7: In response to explicit reference in the target to knowledge and skills related to sustainable development and global citizenship, the international community has prioritized progress assessments based on the content of education. This is positive, as it will encourage countries to reflect on what is taught in classrooms. However, it has not been clarified how such information is to be collected and communicated at the global level.

UNESCO member states’ reports on implementation of the 1974 Recommendation concerning Education for International Understanding, Co-operation and Peace and Education relating to Human Rights and Fundamental Freedoms are proposed as the mechanism to monitor progress towards the target. But low response rates and submission quality mean the process is weak and needs to be complemented by a more systematic and rigorous approach.

The GEM Report has proposed an approach that would require a systematic listing of national curriculum frameworks and a coding protocol to analyse curricular materials. Such a mechanism would also require close collaboration between education ministries and regional or international organizations to ensure that the quality of the information is good and that the process is country-led. The mechanism could also cover other aspects of national policies, including teacher education programmes, learning assessments and textbooks.

Target 4.a: Interest in the concept of child-friendly schools drove the formulation of this target. The most meaningful measures of such environments are observation-based, making this a particularly difficult target for global monitoring. One potential entry point is the growing interest in measures of school violence, though further progress will require collaboration to ensure that definitions in student surveys, currently fragmented, are aligned. School infrastructure indicators may be the easiest to measure but also the least likely to capture the spirit of an effective learning environment.

Target 4.b: There is a surprising gap in information on scholarships. Providers need to collaborate to develop an entirely new global mechanism for reporting on government and non-government scholarships. Collected information must cover the basic characteristics not only of scholarships, but also of their recipients, notably their origin, destination and field of study. This process needs to start as soon as possible, since the target is to be achieved by 2020 and the currently proposed global indicator, based on scholarships funded from aid programmes, provides a very incomplete picture.

Target 4.c: The formulation of the teacher target also put the spotlight on a weak evidence base. Current measures of qualified and trained teachers are hard to compare because standards are not comparable. The fact that personnel databases are often not linked to overall education management information systems makes it difficult to monitor equity in staff distribution across schools, teachers’ working conditions and the rate at which they to leave the profession. Not enough use is made of labour force surveys that could show how teacher remuneration compares with other professions.

Considerable insights could be gained from a tool that collects information directly from teachers, along the lines of the Organisation for Economic Co-operation and Development (OECD) Teaching and Learning International Survey, whose contribution has been acknowledged by all stakeholders. Given the dual potential for allowing cross-country comparison with high quality data but also for informing policy, it is worth considering the expansion of such a tool to cover low and middle income countries.

Finance: The coverage of basic data on government expenditure is remarkably low and there is a considerable time lag. But the main challenge is providing a comprehensive picture of all financing sources. Therefore the priority is to support the institutionalization of national education accounts, similar to national health accounts. This key step would bring all financing sources into a common measure and clarify who benefits from public education services.

Civil society organizations have advocated the use of normative instruments for monitoring whether governments guarantee free education. While understanding whether education is free is important, the GEM Report argues that this is less likely to come from official policy documents than from monitoring the share of total education expenditure that households bear. Civil society organizations should therefore rally behind national education accounts and ensure that household income and expenditure survey data are available to facilitate analysis.

Systems: A strong case has been made for education system indicators to be monitored. This is not to suggest that governments should be held to account for their institutional arrangements. Rather, the purpose is to facilitate dialogue and encourage governments to learn from each other.

This idea has two implications. First, it would require agencies involved in collecting such information to collaborate to reduce overlaps in their diagnostic tools and make more effective use of resources. Second, regional organizations, which may include education as one of their areas of cooperation, would have to play a larger role in developing their vision vis-à-vis the Education 2030 agenda and use this as a first step to collect in-depth information on comparable aspects of education systems.

Education in the other SDGs: The scope for monitoring education in goals other than SDG 4 cannot be limited to the small number of global indicators in which education is explicitly mentioned. It also needs to consider education’s role in achieving particular development outcomes, building national capacity for SDG implementation and supporting adults in contributing to the broader transformation needed in economic and political structures and in relation to the environment, all of which are key to the sustainable development agenda.

RECOMMENDATIONS ON MONITORING THE EDUCATION AGENDA

The above review of the monitoring of each target points to several priorities for action at the national, regional and global levels, the most important being to produce sufficient comparable information that allows global dialogue on progress towards SDG 4. This aim does not contradict the message that every country should be empowered to monitor its own progress towards SDG 4, focusing on its own national context and needs.

The international education community has agreed on the monitoring framework, which consists of two complementary sets of indicators. First, there are 11 global indicators that were developed as part of the Inter-agency and Expert Group on Sustainable Development Goal Indicators process; countries are expected to sign these during the United Nations General Assembly in September 2016. These indicators will be mandatory for monitoring SDG 4.

Second, there are 43 thematic indicators, of which 11 constitute the set of global indicators. The need for this expanded list arose because the global indicators could not capture the full range of global education priorities. Countries are not required to report on the additional thematic indicators, unlike the global indicators. Instead, the additional indicators are meant to provide guidance as countries consider more in-depth ways to track their progress.

The original list of thematic indicators was based on an initial expert proposal. UNESCO and the UNESCO Institute for Statistics (UIS) then set up the Technical Cooperation Group (TCG) on the Indicators for SDG4- Education 2030 to support their further development and implementation. The TCG includes 28 countries as members and 14 as observers, in addition to 5 international bodies (including the GEM Report) and 2 international representatives of civil society. It is meant to be an umbrella group, incorporating bodies dedicated to specialized areas of the agenda, such as the Global Alliance for Monitoring Learning and the Inter-Agency Group on Education Inequality Indicators. The UIS serves as its secretariat.

For every thematic indicator in the framework, the secretariat has provided detailed information on the definition and sources and a preliminary classification according to the availability of methodology and actual data. One of the TCG’s first tasks is to address 8 of the 43 thematic indicators that have already been identified as poorly aligned with the target or difficult to implement. Initial decisions are expected when the TCG next meets in October 2016.

The establishment of a permanent group for technical cooperation, representing a large number of countries, is a considerable advance in the international dialogue on education monitoring, and fills a notable gap experienced during the Education for All period. At least two challenges lie ahead. First, countries must be assured an opportunity to contribute to discussions in an informed and meaningful way. Their active role in the TCG is critical. Indeed, one objective of the monitoring part of the 2016 GEM report is to serve as a reference document for such discussions.

Second, a mechanism is needed for future decisionmaking within the TCG, to help reach consensus and strengthen the group’s legitimacy.

Countries must be given a chance to contribute to discussions in the Technical Cooperation Group in an informed and meaningful way Click to Tweet

AT THE NATIONAL LEVEL: BUILD CAPACITY IN SIX KEY AREAS

Where do countries stand in relation to the emerging monitoring challenges? It is essential for education ministries and national statistical agencies not to lose track of the big picture with so many competing requests for their time and resources. This section focuses on six key steps countries need to take to strengthen national monitoring of education in the next three to five years while concurrently contributing to the development of a global monitoring framework.

Work is needed to address equity. Education management information systems produce data usually based on school censuses that fail to shed light on basic disparity – in access, participation, completion and learning – by student characteristics. But the rest of the national statistical system can often produce highly relevant information on education inequality through household or labour force surveys. Unfortunately, these two systems are disconnected in many countries so education ministries lack the capacity to recognize the complementary and critical nature of the evidence produced by national statistical agencies. A classic example is that education ministries are not sufficiently involved in the design of national household surveys and hence education questions are poorly formulated, undermining analysis and cross-country comparisons.

This situation needs to change. Dialogue and cooperation are needed between education ministries and national statistical agencies. Target 4.5 requires reporting on disparity through a series of education indicators. Countries need to agree on a new mechanism for this reporting. The work of the Inter-Agency Group on Education Inequality Indicators can help initiate a process for continued dialogue.

Definitions of learning outcomes need to be broadened. Countries need to ensure the establishment of samplebased national learning assessments that can be used to monitor progress on a range of learning outcomes over time. Such assessment frameworks need to be of good quality and meet standards that the international community can help define, regarding not only technical aspects of reliability and validity but also openness and transparency in the publication of results.

Countries will need good guidance to build assessment frameworks to ensure they produce information to improve teaching and learning processes. Reporting for a global indicator would be a helpful by-product, though not the main concern. Countries also need to take responsibility for monitoring the skills of those who have never been to school or who left school early. A first step would be to assess the skills of youth and adults.

A new strategy for assessing learning, and communicating its purpose, is essential to shift the nature of national education debates. In many countries, learning outcomes are still often mistakenly equated with pass rates in national high stakes examinations, such as those at the end of the basic education cycle or for transition into higher education. In contexts with high stakes testing, change is likely to be slow.

In terms of quality, learning outcomes are not the only key element. A focus on learning is essential to generate debate that has often been lacking. But such a focus can also create high expectations that learning outcomes will improve rapidly; experience over the last two decades suggests this is not likely (Clarke, 2016). If there is too much focus on learning outcomes, other urgent questions on education quality risk being neglected.

For example, target 4.7, on sustainable development and global citizenship, is a linchpin of the new global education agenda. To ensure commitment to its objectives, countries need to closely monitor policies, curricula, textbooks and teacher education programmes, as well as learning outcomes.

Another important aspect of quality is to identify the topics and concepts which should be conveyed through education, then define the desired results of education systems in relation to them. One way to start would be mapping how these concepts are promoted at different levels of the education system in policies, textbooks and classroom practices. It would also be useful for countries to engage in frank dialogue with each other on how effectively they address issues of tolerance, respect for diversity, group identity, collaboration, human rights and sustainability in their education systems. Such openness is not easy and requires political courage.

It is essential not to lose track of lifelong learning and its monitoring and expansion. Achievement of the SDGs calls for massive social and economic transformation. Schooling alone cannot deliver all the desired outcomes. Even if the entire cohort of 15- to 19-year-olds were to complete secondary education by 2030, this would be insufficient; the vast majority of the adults who will be called upon to make critical decisions relating to sustainable development will have already completed their schooling and not been exposed to the relevant content.

In most of countries, the education needs, opportunities and accomplishments of the adult population are not being monitored. Countries need mechanisms that track adult education and learning opportunities, including those geared to the sustainable development agenda. This report has provided the example of the systematic approach followed in Europe, yet even that was only from the perspective of what is relevant for work. The world needs to move beyond this to capture a fuller range of lifelong learning opportunities that is fit for the purpose of the SDGs.

Analysing education systems requires finding the right forums and prioritizing the right issues. It is relatively easy for countries to engage in debate on key education challenges through their membership in organizations of regional cooperation, which can offer the best channels for exchanging information on education structures and system characteristics. Countries need to champion such collaboration. This is not a call for increased homogenization of education systems, but rather for openness in comparing how countries deal with similar problems.

Equity and inclusion are examples of issues to prioritize. Much can be gained by comparing how countries approach them. The GEM Report advocates for countries to answer a set of basic questions on how they try to offset disadvantage among students and schools, so as to better understand which countries have the most effective results. Finally, countries are encouraged to adopt the national education accounts approach to improve monitoring of finance. Education finance has often focused on government spending or on how much poorer countries receive in aid. What is needed is to shift the debate to better understand who contributes what resources to what activities.

This outlook provides a completely different picture, showing how education expenditure is shared between governments and households. What emerges is the realization that while there may be a policy of free education, households often bear a very large share of the total cost. This undermines equity, a fact that remains underappreciated by policy-makers.

Shifting to the new approach will be tricky. The international community must encourage countries to adopt national education accounts, but not overburden them with procedures that pose excessively technical demands.

These recommendations in six key areas are relevant to all countries, reflecting the universal character of the SDG agenda, even if low and middle income countries need to do more work. The recommendations assume that countries take seriously their responsibility for monitoring the SDGs and are prepared to do what it takes, receiving technical and financial support where needed.

Countries have already engaged at the regional level with the UIS in important mapping exercises assessing where they stand vis-à-vis the global and thematic indicators (UIS, 2016). This rapid appraisal exercise served an important purpose of familiarizing countries with the monitoring framework as well as giving an early indication of data availability. But this exchange may also have overwhelmed countries with details and technical issues.

The recommendations above try to address the big picture. Numerous key changes are needed, which are a condition for reforms in education monitoring systems that will allow them to cope with the challenges of the Education 2030 agenda. Getting these changes right first will make it easier for countries to then take ownership of their monitoring agenda and make decisions on where to allocate resources and from where to seek technical and financial support.

Regional organizations should help make the global education agenda more specific and relevant to the contexts of their members Click to Tweet

AT THE REGIONAL LEVEL: SUPPORT PEER LEARNING

While the Education 2030 agenda is global, progress in many areas is more likely to happen at the regional level, particularly where qualitative, system-related information is needed. Countries are increasingly collecting better and more systematic information through quantitative indicators. This is insufficient, however; comparative information is needed to understand how countries in similar situations have responded and to help guide governments in the next steps.

The role of regional organizations is critical. Where such organizations include education among their concerns, this can help make the global education agenda more specific to members’ contexts. The GEM Report highlights the experience of bodies such as the European Union and the Organization of Ibero-American States. The European Union’s education strategy is supported by a network that regularly provides policy information on key issues to member states, which exchange information voluntarily, and take part in forums to learn from their peers.

Countries in a given region tend to have common education contexts. Their comparative reviews often reflect shared values, objectives and challenges. Members of regional entities are therefore more likely to express deeper political commitment to, and national ownership of, undertaking peer reviews. And the results of regional monitoring are much more likely to be used in policy-making and sustained over time, not least because governments have an interest in neighbouring countries’ performance.

AT THE GLOBAL LEVEL: FOSTER CONSENSUS AND COORDINATION

With the basic parameters of the monitoring framework and a global consultation mechanism in place, the GEM Report has three recommendations on how to improve the global coordination of monitoring approaches.

First, an international household survey programme, dedicated to education, is needed to cover many of the information gaps in the new agenda. There have been attempts to establish such a programme. Beginning in the early 2000s, alongside the Demographic and Health Survey (DHS) programme, USAID funded educationspecific EdData surveys in a few countries. However, this initiative was limited to only a few countries and did not become generalized. Since then, funders have shown a lack of willingness to finance an education-specific survey. Few questions are dedicated to education in the questionnaires of major cross-country multipurpose surveys such as the DHS, the UNICEF Multiple Indicator Cluster Surveys and the World Bank Living Standards Measurement Study. And these surveys are reluctant to add more questions on education, given the many demands on their resources.

In light of the expanded scope of the Education 2030 monitoring agenda and its many gaps, the question of an education-specific survey needs to be revisited. Such a tool could address issues such as participation in early childhood, technical-vocational, tertiary and adult education; the use of language at home and in school; the collection of detailed information on school attendance; and the direct assessment of literacy and numeracy skills. Potential donors need to discuss the cost-effectiveness of a new tool of this kind.

The second recommendation is for a consistent approach to support the monitoring of learning outcomes. Countries need support to build national assessment systems that have solid foundations and are in their best interests. Some countries face a multitude of options from development partners that stretch their capacity.
A code of conduct among donors and a common pool of resources are needed, with priorities including the building of national capacity, the provision of long-term support and the avoidance of overlap. Coordinated support would also help countries gain access to shared resources and knowledge networks.

Third, analysis of several measurement challenges in the GEM Report highlights the fact that, with the expanded scope of the Education 2030 agenda, many targets have not yet been measured on a global scale. Indicators, especially those related to learning outcomes, have not yet been fully developed. A common challenge is that large differences in culture and other contextual factors hamper clear, comparable definitions of such concepts as early childhood development, relevant learning outcomes in basic education, digital literacy and skills for global citizenship.

Therefore, this report recommends that institutions like the UIS, with the support of the TCG, can set a research agenda related to the challenges of comparative measurement in education. To be valid, proposed measures need to be tested on the ground and at scale. In key areas of global education, research on measurement is fragmented. While there is much national expertise, there is little pooling of resources to address questions in a cross-country, cross-cultural context. In the absence of a coordinating institution, it is strongly recommended that a research hub be established to focus on issues related to the major gaps in the global monitoring of education. Education could draw lessons from partnerships or networks in health or agriculture that have pooled research resources to similar effect.

The GEM Report believes that the concept of a ‘data revolution’ associated with technology and ‘big data’ is likely to be irrelevant in education, if not misleading Click to Tweet

Framing the debate: what data revolution for education?

Much of the discussion about global monitoring of the SDGs has been cast in terms of a ‘data revolution’. The term can be defined in many ways, but one of the most widely used descriptions is found in the final document of the United Nations Secretary-General’s Independent Expert Advisory Group on the Data Revolution for Sustainable Development: ‘New technologies are leading to an exponential increase in the volume and types of data available, creating unprecedented possibilities for informing and transforming society and protecting the environment. Governments, companies, researchers and citizen groups are in a ferment of experimentation, innovation and adaptation to the new world of data, a world in which data are bigger, faster and more detailed than ever before’ (United Nations, 2014c).

The GEM Report takes the view that the concept of a ‘data revolution’ associated with technology and ‘big data’ is likely to be irrelevant in education, if not misleading. Most countries are still grappling with compiling basic education data and understanding their purposes and uses. And most of the issues confronting countries are about basic concepts, such as what is meant by literacy proficiency skills or early childhood development. In addition, there is a need to invest in robust monitoring systems. In both these areas, the international community is called upon to provide assistance. Rather than overhauling ways of collecting data and bringing about a ‘revolution’ through technological means, better coordination between agencies and more resources to implement plans would bring about the changes needed to monitor Education 2030.

Moreover, accessibility, openness and accountability of data in education remain constrained. Many countries carry out household, school or learning achievement surveys but do not make the results or the data public. Countries need to do much more to promote the availability of data and encourage their use. This must be a key part of any ‘revolution’ if it is to be in the service of all.