What PISA measures and what it does not

Every three years, the liturgy is repeated. The OECD publishes the PISA results on a December morning and, before lunch, half the country has given its opinion. Headlines featuring the words "collapse" or "rebound," talk shows searching for culprits, comparisons with Finland or Singapore depending on the season. And in the staff rooms, a mixture of annoyance and fatalism: the feeling that an exam taken thousands of miles away, graded with statistical models that almost no one has read, has become the supreme judge of their work. Before next December, it is worth spending a thousand words to understand exactly what this instrument is: what it measures well, what it measures passably, and what it does not measure at all.

PISA — Programme for International Student Assessment — has been evaluating fifteen-year-old students in reading, mathematics, and science since the year 2000, with a rotating main domain per cycle: in 2022 it was mathematics; the 2025 cycle focuses on science and incorporates foreign language and learning in the digital world tests, in addition to inaugurating a four-year cycle; reading will take center stage again in 2029. It is not a curriculum exam: it does not ask about what the syllabi of any country say, but rather about the ability to apply knowledge to real-life situations. As a comparative thermometer of basic instrumental skills, administered to hundreds of thousands of students across dozens of educational systems, it is a serious instrument, probably the best of its kind. That is the part that should be said without hesitation.

The Letter of the Eighty Academics

The other part also has a history. In May 2014, The Guardian published an open letter addressed to the head of PISA at the OECD, coordinated by Heinz-Dieter Meyer, a professor at the State University of New York at Albany, and Katie Zahedi, principal of a New York public high school, and signed by dozens of academics from various countries. Their objections remain the best summary of the critical perspective. First: the three-year cycle of results pushes governments toward short-term fixes, when almost everything important in education takes decades to mature. Second: by measuring only a narrow band of what is measurable, PISA shifts public attention — and money, and reforms — toward that band, impoverishing the conversation about why we educate. Third: an economic cooperation organization, without a democratic mandate in educational matters, had acquired enormous power over school policies across half the world without the corresponding debate.

The OECD responded in writing and with data, and an honest counterpoint requires including it. It denied that there is evidence that PISA narrows curricula — "there is nothing to suggest" such a shift, its head replied —, recalled that only about a third of the items are multiple-choice, contrary to the caricature of a check-box test, and cited the cases of Poland and Germany as countries where the PISA diagnosis catalyzed reforms widely considered positive. Technical criticisms also joined the public ones: Danish statistician Svend Kreiner showed that rankings are sensitive to the assumptions of the psychometric model used — with other defensible assumptions, some countries shift several positions —, and in Spain, works like those of Antonio Fernández-Cano have pointed out sampling opacities and interpretive abuses. The OECD has been refining its methods and publishing more technical documentation. The dispute, like almost all serious ones, is not closed: it is domesticated.

The Thermometer and the Judge

What, then, does a principal or a faculty do with PISA? Perhaps the most useful formula is this: PISA is a reasonable thermometer and a terrible judge. As a thermometer, taken in aggregate and over a long-term trend, it informs: if the reading comprehension of fifteen-year-old students in a country steadily erodes, something is happening and it is worth knowing. As a judge — of a specific school, of an autonomous community, of a law, of school ownership —, it is an indefensible instrument, because it was not designed for that, and its own fine print warns as much: differences between systems are largely explained by socioeconomic factors that the exam can only partially adjust for. Using PISA to point fingers is cheating with the appearance of rigor, and this notebook will not play that game in any direction.

But the most important thing for a school with its own project is not how to read PISA, but to remember what is left out of it. PISA does not measure whether a student leaves school with the judgment to distinguish what is valuable from what is flashy. It does not measure character formation, nor the ability to converse with someone who thinks differently, nor the appreciation for beauty, nor the search for meaning, nor the habit of reading when no one is grading. In short, it measures almost nothing of what a mission statement promises to families, and almost nothing of what an adult thanks their school for thirty years later. This is not a reproach to the instrument: its developers never claimed to measure the entirety of education. The reproach, if anything, is for a public conversation that confuses what can be counted with what counts.

This is where a school stakes its ground. Centering an educational project on chasing an indicator is letting the thermometer dictate the treatment; the alternative requires having a clear — and faculty-aligned — canon of what truly matters. At Nomba Education, we work precisely in that area that no international exam will ever visit: the cultural and inner depth of teachers, which is where everything PISA cannot see is decided. When next December and its liturgy arrive, a well-prepared faculty will be able to read the headlines with interest and without trembling: knowing what the thermometer is telling them and knowing, above all, that their work is measured on a longer and deeper scale than any ranking.

Cover image generated with artificial intelligence.