Is it time for Aotearoa New Zealand to opt out of PISA?

PPTA Te Wehengarua has carried out an analysis of the research into the Programme for International Student Assessment (PISA). PISA is a three-yearly international assessment of 15 year olds’ reading, maths and science abilities.

What is PISA?

The Programme for International Student Assessment (PISA) is an initiative of the Organisation for Economic Co-operation and Development (OECD), whose mission is to shape policies that foster prosperity, equality, opportunity, and well-being for all.[1] It was first implemented in 2000 and it provides comparable data, every three years, that countries can use to improve their education policies and outcomes.[2] The assessment is voluntary; however, while PISA is sponsored, governed, and coordinated by the OECD, it is paid for by participating countries.[3] Nations pay to participate in PISA and fund the data collection in their own countries.[4]

Increasing numbers of countries are participating, wanting to see how their students compare with international standards. Lockheed, Prokic-Breuer, and  Shadrova (2015) point out that wealthier countries have participated more often than less affluent countries and that middle-income countries, particularly lower-middle-income countries, meet financial, technical, and contextual challenges when participating in PISA.[5] There are criticisms of the burden for developing countries, paying the PISA participation fee and significant implementation costs and wasting valuable resources that could be better used elsewhere.[6]

PISA measures the 15-year-old age bracket as these students are either close to or completing their compulsory education and are therefore expected to have enough knowledge to accurately reflect the state of their country’s education system.[7] The assessment measures the ability of 15-year-olds to use their reading, mathematics, and science knowledge and skills to meet real-life challenges.[8] PISA measures problem solving and cognition and is not an exam students can study for as its focus is not the memorisation of information, but rather how learning is applied to real-life situations.[9]

Mitchell (2024) states that the policymakers of participating countries consider PISA to be a valid and reliable tool to gauge how their system is performing, an international performance benchmark, and the changes over time. He cautions that “as countries focus on PISA's key domains (Maths, Reading, Science), there is a narrowing of the curriculum with less emphasis on humanities, arts, and social sciences”.[10] PISA thereby reduces what we consider education is and ought to be about. A tangible example of this in New Zealand is Prime Minister Christopher Luxon declaring he is prepared to see schools defer arts and music curricula to raise achievement in maths and reading.[11]

Officially, we are told that PISA allows the education sector to know how well-prepared students are to meet real-life opportunities and challenges after they finish school and monitors progress towards its education goals of raising achievement and reducing inequity.[12] Opinions are divided on how usefully PISA testing does this and how relevant it is to our education system.

PISA is considered by many to be an outdated tool for measuring overall education system performance. Internationally, critics argue that the tests could be doing more harm than good as they focus on a narrow range of educational outcomes, which reflect the “biases of Western educationalists”.[13] [14][15]

International assessments like PISA do not compare ‘apples with apples.’

The politics of the data

One of the major criticisms of PISA is, indeed, the use of the generated data. Increased participation in PISA testing has meant the generation of extensive databases claimed to be invaluable for education research,[16] with key findings widely shared in the research community.[17]  Critics point out that the data is often used selectively, with many a government, New Zealand’s included, concentrating on superficial features of PISA surveys such as country rankings and not the more detailed analyses.[18] For example, the National Party claims that  success of its education plan will be proven when, by 2033, New Zealand is in the “top 10 in the world on PISA rankings for maths, reading and science.”[19]

There are claims that international benchmarks can be a healthy driver for reform efforts worldwide,[20] yet, critics consider that inappropriate uses of assessment data are made to fit with government reform agendas.[21] PISA surveys are described as neutral, a third-party analysis of an education system; however, the PISA data and surveys can be selectively used to promote specific alignment on educational issues.[22] The ‘laser focus’ being applied to structured numeracy and literacy in New Zealand has been validated by our declining performance in the large-scale international testing rankings.[23]

Commentators suggest that PISA tests and the drumbeat for always being number one have played a big role in the bashing of teachers and the pointless rush to the top.[24] Hipkins (2019) highlights that critics of PISA consider that “it helps generate the very inequalities it then reports - teachers and school leaders are unfairly blamed for disappointing PISA results”, and there is potential for PISA data to be misused to 'label’ groups of students who have not done well.”[25] This is certainly a concern for New Zealand teachers, as the latest PISA report showed Māori and Pacific student performance falling faster than the New Zealand average, a message amplified in the media.[26]

Sjøberg (2019) states that politicians have tended to use PISA results to chase their agendas – to attack state schooling and argue for more privatisation or increased testing.[27] Academics consider that PISA has contributed to an escalation in testing and a noticeably increased reliance on quantitative measures.[28] The New Zealand coalition government introduced progression monitoring of reading, writing, and maths for children in Years 3 through 8 in 2024, claiming the twice-yearly testing would inform teachers about the next steps needed for a child’s learning.[29] Milne (2024) considers that in reading and maths, the narrower focus on teaching to the test may well bear fruit – but questions whether our children will emerge well-rounded from school.[30]

Economic success and a high ranking on PISA correlate,[31] and governments around the world are chasing this. There are questions to consider: Should PISA be so powerful in shaping[32] educational policy and practice?[33] Can it be used to judge the ‘quality’ of one country’s education system over another?[34] Critics believe that the OECD has assumed the power to shape education policy without debate about whether this is necessary or whether the OECD's goals are or should be limited.[35]

Sampling of students unfair  

In 2022, Singapore emerged at the top of the global PISA race, having been second to China in 2019.[36] The global trend shows Asian school systems getting the best results, with Estonia and Finland consistently being high achievers.[37]  Western countries have not recently been at the top of the board, seemingly confirming concerns over declining educational standards. However, the PISA scores reflect the students tested, and it is from the sample that inferences are drawn. If we look more closely at the sampling of the ‘apples with oranges’, PISA is certainly not fair.

PISA tests a sample of students, and the results are then adjusted to reflect a whole population of 15-year-old students. If a country isn’t equipped to conduct the test country-wide, then regions of that country can participate instead.[38] Roy (2017) highlights increasing criticisms that PISA’s sampling of several countries, such as China, Argentina, and Malaysia, might not be as random as intended, and there have been claims of result rigging with the ‘selection’ being students from their most educated regions, leading to higher scores.[39]

In 2009, Shanghai’s students were at the top of the PISA world, resulting in its educational system being held up as a model for the rest of the world. For countries wanting to replicate its success, Carnoy (2015) points out that Shanghai’s sample was not representative even of Shanghai’s 15-year-old population, nor China.[40] Shanghai schools systematically exclude migrant youth.[41] As a result, Bruyckere (2015) suggests that Shanghai scores should not have been included in any OECD comparison group, and its inclusion raises serious questions about the OECD as an international testing agency.”[42]

The OECD website explains that China has expanded its participation with Beijing, Jiangsu, and Guangdong in PISA 2015 and with Beijing, Jiangsu, and Zhejiang in PISA 2018 and PISA 2022, in addition to Shanghai.[43] The four regions combined scored significantly higher in some areas than other countries. Candido (2020) points out, however, that PISA participants are from China’s wealthiest areas rather than the entire country, with the OECD claiming that which areas are included is “individual countries’ decision”. [44] Loveless (2014) argues for greater transparency; data collected in other Chinese provinces has never been released. [45]

Roy (2017) considers PISA results a “huge propaganda victory for the educational systems of Asians and the Chinese, while covering up the huge disparities in education”, pointing out that “almost two-thirds of all Chinese children for example, live in rural areas, where school attendance rates can be as low as 40%.” [46] China is not alone in its lack of inclusion. In Canada, federally funded Indigenous schools are not considered in PISA data and would likely impact provincial scores.[47]

Here in New Zealand, the latest 2022 PISA results identified our student sample to be biased, with slightly more high-achieving students participating.[48]

Is it valid to compare ‘apples and oranges’?

The validity of PISA has been challenged by experts—does it assess what it says it assesses?[49] It is also sensible to ask how valid the comparison is between the sample of students in Finland, Estonia or Singapore with the students in a New Zealand classroom?

Clark (2020) sees cherry-picking and ignoring the details as a problem in the use of PISA results. Finland, for example, is seen as a high-performing PISA country, with particular aspects to its success – it is small, homogeneous, without substantial class divisions, and has high levels of vocational education and no streaming or private schools.[50] Donnelly (2015) points out that Singapore, a city state, geographically small with homogenous language and culture, employs educational practices that are not favoured in many nations (streaming, competitive testing, and examinations).[51] Are these countries comparable to NZ?

Estonia is a PISA highflier, yet its context as a nation is a key factor of its success in education. Jeffreys (2019) identifies that for Estonia, preserving a sense of national identity has been key through many occupations and has been achieved by a strong education system. She claims that a national desire has been fostered “to see every individual perform to their best, with education a key means towards this end.”[52] It would be interesting for our government to note that free in Estonian education means free. Estonians don't pay for textbooks, school lunches, or school transportation.[53]

PISA ranking, critics argue, lacks validity as the samples of students in different countries have different levels of family academic resources.[54] Despite claims that the tests are universally applicable, Thompson (2020) considers they are not, with the real-world challenges faced by students in poorer countries probably not being assessed by PISA.[55] Bloch (2014) says PISA tests lack cultural sensitivity in their avoidance of a country’s infrastructure, both physical and human.[56] Mitchell (2024) believes Western models of education shape the tests without regard for cultural diversity or approaches to knowledge in countries outside of these models.[57]

Students take the test in their native language, but Hipkins (2019, p.50) cautions that “even with strict protocols to follow, studies have shown translating the same question into different languages can change what is being asked in subtle ways that make the question comparatively easier or harder to answer in different languages”. Hipkins (2019. P.50) advises that “the need to use contexts that will make sense to students in many different nations severely constrains the actual contexts that can be used in the assessment items, and it is likely that some contexts will still give a comparative advantage to students in some nations.”[58]

Bruyckere (2015) warns that while comparative education is interesting, it is a difficult field of research and suggests that comparing education in different countries is almost impossible.[59] It is problematic to compare ‘apples and oranges’, as PISA does.

OECD policy advice challenged

Academics have been critical of PISA testing for some years ^[60] with several main criticisms based on academic research.^[61]  Acido & Caballes (2024)suggest that many education systems now emphasise the importance of tests and quantifying knowledge and warn of the arts and social studies being second class, a “teach to the test" approach, restricted concentration on test results and that other important aspects of learning and growth are ignored.^[62] Beghetto (2019) says that large-scale assessments often fail to capture valued educational outcomes such as creativity or social and emotional well-being.[63]

The OECD advises what nations should do next based on PISA outcomes, but PISA numbers are limited in what they can explain and the conclusions they can support. Hipkins (2019, p.51) suggests that PISA encourages a focus on a three-year improvement cycle rather than longer-term development and highlights critics' concern that “PISA assessments and policy advice will make everyone’s education systems the same, regardless of differences in national values.”[64]

There is increasing unease with the growing privatisation of public education, and critics point to the worrying partnership PISA has with Pearson, a global educational business enterprise. This is a conflict of interest, where PISA assesses and ranks school systems and Pearson, a provider of global and online charter schools, tests, and education consulting, provides the ‘help’, contributing to increased privatisation of education.[65]

Academics argue PISA/OECD has no mandate in countries, it has just ‘imposed’ itself on them. They highlight, for example, that ‘work readiness’ is not the main aim of education in some countries, yet this is what the PISA tests are designed to assess.[66] The critics point to the  OECD as an organisation of economic development and suggest a bias in favour of the economic role of public schools, pointing out that many educationalists do not consider preparing young people for ‘work’ to be the main goal of public education.[67]

The price of Eastern educational ‘success’

Yang & Fan (2019) state that “the outstanding performance of Chinese students in PISA mathematics, reading, and science has induced countries around the world to shift the focus of their education policy reform toward the East” (p.298).[68] However, it would be useful for the New Zealand government to reflect on the education systems of East Asia in some depth.

Yang & Fan (2019) emphasise knowledge transfer as a feature of the Chinese system, with teachers paying particular attention to knowledge and rote learning in classroom teaching.[69] They advise that Chinese teachers retain strict guidance and control over teaching and the practice and mastery of knowledge.[70] Other commentators propose that the knowledge-based learning preparation for exams is at the expense of students’ creative thinking and problem-solving abilities. [71]

Ho (2009, p. 333) identifies that in China, the role of the teacher is very different, and from an early age, students are taught” to respect, obey, listen, and follow their instruction and not to challenge their teachers.”[72] Respect for teachers extends to parents, who also follow what teachers ask.[73] Burge (2015) says that parental expectations in China are similar across different socio-economic groups.[74]

Donnelly (2014) points to the strength of Confucian values and ethics, seen throughout East Asia; respect for authority, the belief that success is possible with motivation, concentration and hard work, and the idea that education is central if one is to achieve a better life as cultural values that contribute to strong performance in assessments like PISA.[75]

Commentators claim that school streaming in China, where most schools have entrance requirements alongside the Government’s rules, leads to students in a school having similar intellectual ability and financial resources. [76] Hu et al. (2023, p.10) state that China’s families treat education as a necessity and spend 7.9% of their total annual household expenditures on education, surpassing any other country. They allocate an outsized portion of their budgets to prepare their children for the fierce competition in the education market.[77]

Yang & Fan (2019) found that in 2015, “Chinese students had the longest learning week among all countries participating in PISA, averaging 33 hr per week—significantly higher than the OECD average” and claims that Chinese students are overburdened with high rates of extracurricular tuition due to demands for academic achievement and parental expectations.[78] Tuition is important in other East Asian countries. Wise (2016) details that in Singapore, eight out of ten primary school-aged students receive private tuition.[79]

Cui et al. (2010) highlight that the stress of the assessments has been a factor in exam-induced suicide reported in places such as Korea, Singapore, Hong Kong, and China.[80] Wise (2018) reports that test anxiety and fear of failure are issues for the students of Singapore despite their performing very well academically, claiming that “70% of Singaporeans express fear of failure compared to 50% globally.”[81] In Japan, critics want a more progressive education system and highlight the mental distress of students experiencing mundane, rote learning, bullying, harassment, and/or excessive disciplinary action.[82]

Zhao (2014) considers that East Asian education systems produce ‘great test takers’, so they have not ‘risen’, as claimed by PISA. He states that Singapore, Korea, Japan, and Hong Kong scored extremely well on international tests such as TIMSS before the birth of PISA.[83] Other scholars suggest there is no room in these systems for “creative and unorthodox individuals to pursue their passion, question authority, and develop their strengths”.[84]

Jerrim (2014) says that East Asian families highly value their children’s education and send them to the best possible schools.[85]  Ho (2009, p.3) shares that “it is commonly perceived that having high academic achievement is a means to honour one’s parents and ancestors.” [86] Many of the key reasons East Asian children excel in PISA are cultural and not in the control of schools.

Do NZ parents value education this highly?

The long decline - is PISA a reliable measure?

New Zealand is far from alone in declining PISA results. Gomendio (2023) points out that after more than 20 years, PISA’s data shows that student outcomes have not improved overall in OECD nations or most other participating countries.[87] Gomendio (2023) suggests that other factors such as a global recession and the rise of social media, may have impacted efforts to improve schooling results but disagrees with the OECD’s explanation for the flatness of student outcomes over PISA’s lifetime, with the assertion that “its policy advice had not been followed” rather the greater problem being that “most policy recommendations are strongly context dependent.” [88]

The New Zealand government, however, may not be listening to the PISA policy advice! There is a correlation between poverty and lower test results, with the OECD noting that “up to 46 per cent of the differences in PISA mathematics scores among OECD countries can be explained by socio-economic disadvantage.”[89] Gomendio (2023 highlights that “several comparative analyses show a correlation between the degree of economic inequality and the extent of disparities in student outcomes.[90] The New Zealand government should address the economic imperatives for improving the educational outcomes of our young people.

The actual effort students make in completing the PISA assessment has been called into question. In many New Zealand schools, this is readily apparent to many senior leaders, begging and bribing students to participate in PISA, that students do not care about this assessment. Students are voting with their feet, which is also evident in other countries, with several failing to meet the required number of student participants in PISA 2022.[91]  

Australian researchers found that “nearly three-quarters of Australian students didn’t fully try on the PISA 2018 tests, with 91% saying they would put in high effort if the tests counted towards their school marks”.[92] This would be echoed in many New Zealand schools, where students want to put their efforts into assessments that give them credits.  Cobbold (2021), therefore, suggests that PISA results could be as much a measure of student effort as a measure of student learning.[93]

Cobbold (2021) highlights increasing student dissatisfaction at school across OECD countries saying there is no data on whether changes in student effort have contributed to the overall decline of several OECD countries; PISA results for OECD countries have fallen since 2000 while the proportion of students who feel they don’t belong at school has increased threefold from 7% to 22%.[94]

Many factors influence student achievement, and researchers consider that PISA is not the fully reliable assessment it is assumed to be.

Looking beyond the PISA ranking

“NZ records worst ever PISA international test results, amid global decline.” Gerritsen (2024)[95]

The attention-grabbing headline above highlights one of the major concerns with PISA testing: the focus on ranking. PISA has a savvy strategy in releasing results – its brand is built on impact and the idea that ‘we’ are part of a global educational race. The rankings tell us who the winners and losers are. Results are amplified through media, a ‘crisis’ story can be built to justify policies not necessarily supported by the data. This is an issue in New Zealand.

Sellar (2017) believes that the usefulness of PISA depends on looking beyond the ranking; it should be used as a conversation starter, not a report card. He claims that PISA cannot tell us about the differences between countries and what students learn but can make cross-national comparisons of how well students are prepared to perform on a particular PISA test.[96]

Hipkins (2019, p.51) advises that we should focus on PISA data that “throw up valid challenges within our education system” and in New Zealand these challenges include” the widest spread of achievement of all the nations in the programme, which also applies within individual schools”.[97] Hipkins also highlights that this is not new knowledge, with similar results seen in the National Certificates of Educational Achievement, the National Monitoring Study of Student Achievement, and most recently, our literacy and numeracy co-requisite results.[98]

Schleicher (2011, p.57) asserts that education reform requires teachers to contribute as the architects of change, not just its implementers. He notes that “some of the most powerful reforms are those supported by strong unions rather than those that keep the unions’ role weak”.[99] If the New Zealand government is sincere in its wish[100] to deliver a ‘world-leading’ education system it is advisable to ensure that the voice of teachers is heard.

If PISA participation is to be of any use in New Zealand’s educational reform, the government must address what PISA and other data consistently tells us matters; socio – economic disparity matters, parents matter and their interest in their child’s education contributes to educational success, individual teachers matter, and teacher collaboration is a positive. Behaviour matters and negative behaviours occur at a much higher rate in New Zealand classrooms than the OECD average and are strongly negatively related to achievement. School life matters to young people – they tell us that parents then life at school are the two most important things.[101]

PISA’s value questionable

PPTA remains skeptical that PISA provides meaningful value to improving the educational outcomes of our rangatahi. Clark (2020) summarises critics' concerns that PISA is flawed because of the “huge differences in nations being tested, which are diverse in demographics, socio-economic ranges, and linguistics”. [102] PISA has bias; there are immigration skews, issues comparing city-states against entire nations, and the impact of external tuition is ignored, which lead to serious doubts as to the validity and global influence of PISA.[103]

PISA testing may continue for the 15-year-olds in New Zealand, but we must be honest that this assessment compares ‘apples with oranges’, and how useful is that?

“I hope the government reads the writing on the PISA wall and realises that if it is serious about improving educational achievement, the most important thing it can do is address the widening gap between the haves and have nots in Aotearoa New Zealand.” Chris Abercrombie PPTA President, 2024