Op-ed: Did the Education Ministry influence our PISA 2015 results?

The OECD released the PISA 2015 Technical Report chapter on data adjudication at the end of last month, and the report says that Malaysia’s replacement schools were higher-performing than the non-responding initially selected schools that they replaced.

I wrote a piece about that, and about Malaysia’s participation in PISA 2015 more generally, for The Malaysian Insight—and I promise that it has less jargon than the preceding sentence. Read it here. Alternatively, here’s a version with the citation links:

Did the Education Ministry influence our PISA 2015 results?

by Hwa Yue-Yi

Like many other Malaysians, I want our country to have a good education system. So when the results of the Programme for International Student Assessment (PISA) 2015 were released last December, I was curious to see whether our 15-year-olds had developed more skills in reading, maths, and science than their peers in previous PISA rounds.

But, like many other Malaysians, I was disappointed. Not because of our average PISA scores – which had gone up – but because we weren’t included among other countries in the main database of PISA results. Apparently, we had been excluded because only half of the schools that had originally been chosen to take part in PISA 2015 had actually taken the test.

Within each PISA country, schools are randomly chosen to participate in the assessment, after considering certain school characteristics, such as the size of each school and whether it is urban or rural. The goal is to get a balanced, accurate picture of student learning across the whole country. Each selected school is also paired with a backup school. If one of the originally selected schools doesn’t want to take part, the backup school will be asked to participate instead. But when many originally selected schools drop out of PISA, it’s hard to tell if the results represent the country accurately.

In Malaysia, after including the backup schools, our weighted PISA 2015 response rate increased from 51% to 98%. However, at the end of last month, the OECD released an official report stating that Malaysia’s backup schools “had a significantly better result, on a national examination, than the non-responding schools in the original sample”.

So did the Ministry of Education strategically ask higher-performing backup schools to take the test, so that our PISA 2015 results would look better?

Looking at the evidence

To answer that question, let’s start with what the Ministry has said. Shortly after the results came out, the Ministry promised to release a report with full details on why we were excluded from the main PISA database.

Eight months later, no report has been released. However, in a March 2017 parliamentary reply to Tony Pua, the Ministry said that the 51% initial response rate was due to PISA 2015 being conducted using computers (unlike previous PISA rounds, which used paper-and-pencil tests). Because of this, some students were unfamiliar with computer-based tests and didn’t record their answers properly, and there were technical issues with data loss.

But the evidence suggests that computers probably weren’t to blame. And it also doesn’t seem likely that our low initial response rate was just a coincidence.

  1. The previous two times we participated in PISA, our weighted initial response rates were above 99%. This is also true of all five times we have participated in the TIMSS international assessments – including TIMSS 2015, which took place just six months before PISA 2015. Moreover, principals and teachers in Malaysia, as civil servants, usually obey government directives. It would be very surprising if half of the Malaysian school principals who were told to administer PISA simply refused.
  1. In PISA 2015, the Netherlands also had an initial response rate in the “unacceptable” range – although their 63% was higher than our 51%. But they were included in the main PISA database because national exam data showed that their PISA results probably weren’t biased. In contrast, the data submitted by our government showed that our PISA 2015 backup schools had higher national exam scores than expected, so our results may have been biased.
  1. On average, it’s likely – though unfair – that schools with better exam results also have better computer equipment, and that students in these schools are more familiar with computer-based tests. But it’s a lot less likely that (a) almost half of our initially selected schools faced computer-related difficulties in conducting PISA; but (b) such computer-related problems affected almost none of the backup schools, which had been selected using the same randomised process as the initially selected schools.
  1. The Ministry invested a lot of time and money to prepare for PISA – certainly enough to detect and solve computer issues. The PISA test was held in April 2015, but the Ministry had formed a committee on TIMSS and PISA by December 2013. Mock PISA tests were held as early as May 2014. In March 2015, the Ministry reported that students had been given PISA-style exercises to familiarise themselves with the test, and that teachers had been trained to conduct the computer-based test. In the weeks leading up to the test, students attended PISA training camps in hotels. Also, the OECD provided each country with a diagnostic programme several months in advance, so that PISA administrators could check if each participating school had adequate computers.
  1. PISA 2015 did not require fancy new computers – Windows XP was enough. Also, the test was delivered using USB sticks, so it did not need an internet connection.

All of this casts some doubt on what the Ministry has said about the problems with our PISA 2015 participation.

Measuring up

But why might the Education Ministry want to influence our PISA results?

Since our subpar TIMSS 2011 and PISA 2012 results were revealed, the government has been under tremendous pressure to improve our performance in international student assessments. As a result, the Malaysia Education Blueprint 2013–2025 uses these exams as the benchmark of educational quality: the aspiration is to be among the top third of countries participating in PISA and TIMSS.

But this might not be the best benchmark for Malaysian schools. For one thing, our ranking in PISA and TIMSS depends on which other countries decide to participate that year. If, for whatever reason, a lot of high-performing countries decide to drop out of TIMSS 2019, our relative ranking could rise, even if our average score doesn’t change.

If we want to benchmark our education system against international assessments, it would make more sense to use PISA proficiency levels, which are consistent from year to year. For example, we could say that, in PISA 2021, we want 80% of Malaysian 15-year-olds to reach at least at Proficiency Level 3 for science, which means that they can identify evidence supporting a scientific claim and construct explanations in complex situations.

Moving beyond exam obsessions

Most Malaysians would agree that our education system is too exam-oriented. To its credit, the Education Ministry has been trying to address this, by introducing coursework elements to the PT3 and the STPM, and by initiating consultations on whether the UPSR should be abolished.

But in emphasising our PISA and TIMSS rankings, we are choosing to worry about even more exams.

This is especially sad because PISA and TIMSS are not meant to be that type of exam. They are not the sort of test where you must do as well as you can because it will affect your future. PISA and TIMSS give national-level results, not results for individual students or schools. And it’s hard to think of any ways in which our national future would be directly affected by these results. It’s unlikely that multinational companies would use PISA rankings to decide whether to invest in Malaysia – they might look at graduate skill levels instead. Some Malaysians may migrate to other countries so that their children can attend better schools, but our TIMSS and PISA scores are just one data point among many indications that our education system is struggling.

Instead, PISA and TIMSS are more like the weekly quizzes that you might have taken in school, so that both you and your teacher have an accurate picture of what you know, and what you still need to work on. Similarly, these international student assessments are meant to help education systems understand what their students are good at (analysing literary texts? applying maths principles to everyday problems?) and what they can do to improve. Increasingly, it looks like one way our Education Ministry can improve is by embodying the integrity and responsibility that the national curriculum espouses.

Learning lessons, old and new

To practice integrity and responsibility myself, I’d like to highlight some evidence proving that the Ministry is innocent of a suspicion that had been raised earlier about our PISA results. When the PISA data were released last December, I noticed that 30% of students who took the PISA test were from fully residential schools (Sekolah Berasrama Penuh). This looked suspicious because less than 3% of students are enrolled in these asramas, which usually admit students based on their UPSR or PMR/PT3 results – so asrama students would probably get higher PISA scores than average.

However, when I later emailed the OECD to ask about this, they confirmed that the oversampling of asrama students was intentional, and that it was balanced out in all PISA calculations so that it did not bias our average results. It’s likely that the Education Ministry requested this oversampling to get a higher-resolution picture of asrama students’ performance – which is a legitimate reason, especially when the Education Blueprint discusses plans to internationally benchmark Malaysia’s education programmes for gifted children.

So in this process of looking at our PISA 2015 results, I have re-learned the lesson that I must be unbiased in seeking an accurate picture of Malaysian education. As a citizen and researcher, I must hold the government accountable for how it uses our national resources to prepare our children for the future – but I must also give credit where it is due. And as a former SMK teacher, I know that there are many hardworking teachers and Ministry officials who work sacrificially for our children’s futures, and that it can be dispiriting when everyone seems pessimistic about our schools.

I hope that the Ministry reveals more evidence about our participation in PISA 2015, as they promised in December. And perhaps this evidence will show that the bias from the backup schools was unintentional and unavoidable. But more than that, I hope that the Ministry will learn all that they can from this PISA 2015 process, for the sake of the millions of children whose education they are entrusted with.

Translation: Lant Pritchett membincangkan cabaran dalam pembaharuan sistem pendidikan

Terjemahan daripada Lant Pritchett, December 2015, “Creating Education Systems Coherent for Learning Outcomes: Making the Transition from Schooling to Learning”, RISE Working Paper, m/s 7–8. Dalam petikan ini, Pritchett membincangkan cabaran dalam pembaharuan sistem pendidikan, dan betapa pentingnya kesepaduan sistem. Tema lain dalam kertas kerjanya ini termasuk beza antara persekolahan (schooling) dan pembelajaran (learning). Cadangan dan pembetulan bahasa dialu-alukan.

Inilah soalan yang sukar dijawab: “Terdapat sebilangan negara yang telah berusaha selama 50–60 tahun untuk meluaskan pendidikan sekolah rendah kepada semua, tetapi keberhasilan pendidikannya terlalu rendah. Bagaimanakah keadaan menjadi begitu teruk?”

Lebih mencabar lagi, semua jawapan yang mudah kepada soalan sukar “Bagaimanakah keadaan menjadi begitu teruk?” ini tidak dapat menjadi penjelasan umum. Hampir semua teori tentang unsur utama dalam penambahbaikan sekolah telahpun disangkal oleh bukti tentang pembelajaran di tempat lain. (Sudah tentu, kebanyakan teori ini juga terbukti dampaknya di tempat-tempat tertentu, dan teori-teori ini juga mewakili ciri-ciri sistem pendidikan yang berprestasi tinggi.) Yakni, jika si pemerhati mendapati bahawa pembelajaran di sesuatu tempat terlalu teruk, dan pelajar di situ tiada buku teks, jawapan yang mudah ialah: “Pembelajaran tidak berjaya kerana murid tiada buku teks.” Namun, sekiranya kajian menunjukkan bahawa membekalkan buku teks di suatu tempat yang lain tidak dapat meningkatkan tahap pembelajaran yang teruk di sana, maka jawapan mudah ini ternyatalah bukan jawapan yang dapat dipakai umum. Penaakulan yang sama, berdasarkan kajian pro dan kajian kontra, juga menyangkal teori-teori lain, sama ada tentang meningkatan gaji guru, mengurangkan bilangan pelajar dalam kelas, mengetatkan syarat untuk pentauliahan guru, menambahkan geran pembiayaan pendidikan, dan sebagainya. Faktor-faktor ini (saiz kelas, buku teks, gaji guru, dsb) memang terbukti kepentingannya dalam sesetengah sistem lain, terutamanya sistem yang sudah mantap. Meskipun begitu, ini tidak bermaksud bahawa kekurangan faktor-faktor tersebut dapat memberi penjelasan lengkap tentang punca prestasi rendah di sesebuah negara.

Biar kita ambil kereta sebagai perbandingan. Berdasarkan bukti empirik yang kukuh, kita semua memahami bahawa sebuah kereta dapat bergerak lebih jauh apabila diisi dengan lebih banyak minyak. Ini memang benar untuk kereta yang berfungsi. Namun begitu, jika sistem gear kereta saya telah rosak, mengisi minyak tidak akan menambahkan jarak yang dilalui—walaupun motornya mungkin berjalan lebih lama. Sekiranya unsur-unsur sesuatu sistem tidak membentuk satu keseluruhan yang bersepadu dan berfungsi, adalah mustahil untuk meramal kesan daripada penambahan salah satu unsurnya.

Secara tentatif, jawapan (panjang) kepada soalan sukar tadi, “Bagaimanakah keadaan menjadi begitu teruk?” ialah: “Sistem pendidikan yang dibina di banyak negara, biasanya melalui kerajaan, tidak direka sebagai (atau tidak berkembang menjadi) sistem yang bersepadu ke arah keberhasilan pembelajaran yang tinggi untuk semua.” Sistem-sistem ini mempunyai sasaran utama yang lain, seperti perluasan akses kepada pendidkan. Kerap kali, sebahagian daripada mereka yang menganggotai sistem tersebut memang mengharapkan keberhasilan pembelajaran yang tinggi, tetapi sistem secara keseluruhan tidak pernah koheren untuk pembelajaran.

[original text]

The hard question is: “How is it that some countries are 50-60 years into pursuing universal primary education as a goal, yet have learning outcomes that are so awful?”

Worse, all the easy answers to the hard question of “How can it be this bad?” are ruled out as general explanations. Pretty much everything everyone believes is the key element of better schools has, by now, been rigorously disproved to have an impact on student learning somewhere. Of course, many of these same notions have also been rigorously proven to have an impact on student learning somewhere else, and are characteristics of well-performing education systems. That is, if one observed that learning outcomes of students were awful and that they lacked textbooks, then the easy answer suggests itself: “Learning is bad because kids lack textbooks.” But if studies show that, in places where learning outcomes are bad, adding textbooks doesn’t make things better, this obviously cannot be the answer as to why things are so bad. The same logic applies to better teacher pay, small class sizes, teachers with more formal qualifications, larger block grants, etc. Even if it is the case that these same factors (class size, textbooks, teacher pay, and so on) are proven to matter in some, often well-functioning, systems, this doesn’t mean the causal explanation of poor performance in a given country or region is a lack of these proximate determinants or simple policy elements.

An analogy is a car. We all have the empirically well-honed intuition that a car can go further with more gas than with less gas. This is because for cars that work, it is true. But if I have a car whose transmission has failed then adding more gas will not add to miles travelled—even if it allows the motor to run for longer.

If the system does not add up to a functional whole, the causal impact of augmenting individual elements is completely unpredictable.

The tentative (long) answer to the hard question “How can it be so bad?” is: “Systems of  education were built up in many countries, primarily within governments, that were never actually designed (or emerged) as systems coherent to the purpose of producing uniformly high learning outcomes.” These systems had other, often desirable, objectives, like expansion of access. They often had learning as one objective of at least some actors in the system, but the system was never coherent for learning.

—Lant Pritchett, December 2015, Creating Education Systems Coherent for Learning Outcomes: Making the Transition from Schooling to Learning”, RISE Working Paper, p. 7–8. See also the many other working papers from RISE (Research on Improving Systems of Education).

Column: Cilisos piece on Malaysia’s PISA 2015 results

Continuing with the PISA 2015 theme, I wrote a Cilisos piece here giving an introduction to PISA and explaining the issues with Malaysia’s PISA 2015 data.

Cilisos is like Cracked.com for Malaysian current issues, though somewhat politer. They also have a Malay-language counterpart, Soscili, as well as a wonderful team who put up with my rambling and nitpicking about statistics. (Thanks, Uihua!)

Radio interview: on Malaysia’s exclusion from official PISA 2015 results

Last week, I spoke on BFM, a business-oriented Malaysian radio station, in a ten-minute segment about the issues with Malaysia’s PISA 2015 results. The podcast is available here.

Background: The results from PISA 2015 (i.e. the most recent cycle of the triennial Programme for International Student Assessment) were released on the 6th of December.

When the results came out, many Malaysians were surprised to find that we weren’t included in the main results list, because sampling issues meant that our data weren’t comparable to other countries, or to previous PISA cycles. (For one local commentary, see this press statement by Ong Kian Ming, an opposition MP.)

A personal note: as far as research goes, it’s pretty frustrating that Malaysia’s 2015 PISA data have these quality issues. As far as I know, PISA and TIMSS (whatever their shortcomings and challenges) are the only accessible individual-level data on Malaysian education, because the Ministry doesn’t release national student and exam data.


_front page 1a
outside the library of the school I taught in, Nov 2015

(This is a pinned “About” post. Scroll down for blog updates.)

I am a Malaysian PhD student in Education at the University of Cambridge. My thesis research looks at teacher accountability policy and sociocultural context across countries.

Prior to starting the PhD in 2016, I spent two years teaching English in a high-need secondary school. I have also worked on a few research projects, including a guide to help Malaysian education researchers work with TIMSS and PISA microdata. For details of past and current research projects, click here.

Some of my non-academic writing is listed on this page, and my CV is available here. If you’d like to get in touch, you can email me.

If you’re wondering how to pronounce my given name, Yue-Yi, it’s 悦义. If, like me, you can’t read Mandarin, you can approximate my name by saying the acronym for the United Arab Emirates – U.A.E.

If anything on this blog is useful or interesting to you, I’d love to hear from you, whether in a comment or via email.


On respect and trust for teachers

“The teaching profession needs two things in order to thrive—respect and trust. The two go together. You can say nice words and be grateful to teachers, but if you do not trust them as professionals, you are not showing them respect. Trust means giving teachers (appropriate) autonomy in their classrooms, but it also means giving them influence over policy—real influence, not a few token teachers on some committee—and it means giving them control over their own professional growth. We need to stop fixing teachers and create environments in which teachers themselves fix their own profession. We need to trust them to do so.

“Will some teachers abuse that trust? Of course. That happens in every profession. We can deal with it. Far more will not, however, and on balance education will be greatly improved for everyone, and most especially for the students.”

John Ewing, president of Math for America

Danielle Allen on education and equality

A friend recently passed me the text of political theorist Danielle Allen‘s excellent 2014 Tanner Lectures on education and equality.

In the first lecture, “Two concepts of education“, Allen argues that current U.S. debates about education lack clarity about the purpose of education, and that this ambiguity skews policy approaches to equality in education. Drawing on John Rawls’ “Two concepts of rules” (1955), she posits that this murkiness comes partly from confusion between the aims of state education systems, and the aims of particular instances of teaching. While the former, social concept of education may vary with institutional goals—for example, the U.S. Common Core standards frequently mention “college and career readiness—the aim of individual instances of education must always be the development the student’s capabilities.

Allen goes on to unpack what it means to develop individual human capabilities. Here, she uses Hannah Arendt’s The human condition  (1958), which identifies three core human activities: labour, activities undertaken to enable us to feed ourselves; work, creative activities that shape our human spaces and relationships; and action, political activities that determine our shared freedoms. Allen agrees with Arendt’s assertion that every person should participate in all three core activities, rather than different social strata undertaking different core roles. Hence, education should prepare people for: (1) “bread-winning”; (2) “civic and political engagement”; (3a) “creative self-expression and world-making”; and (3b) “rewarding relationships in spaces of intimacy and leisure”. Thus, this humanistic approach to education harmonises the individual concept of education (i.e. education to enable each student’s flourishing in labour (1), work (3a and 3b), and action (2)), and the utilitarian concept of education (i.e. education for economic activity (1) and civic participation (2)). She then discusses some implications of these dual lenses—humanistic individual and utilitarian social—for accountability policies in education.

In the second lecture, “Participatory readiness“, Allen observes that public discourse (at least in the U.S.) about education usually neglects preparation for civic engagement, despite the fact that the civic is part of both the social and individual justifications for education. Her concept of “participatory readiness”, however, encompasses preparation for effective engagement in all social spheres, whether politics, the workplace, or groups of friends. Specifically, participatory readiness comprises: “verbal empowerment [i.e. interpretive and expressive skills], democratic knowledge [i.e relational bonding and bridging skills], and a rich understanding of the strategies and tactics that undergird efficiency”.

Allen then argues that “the most effective way for us to direct our education system toward egalitarian ends could well be to focus on participatory readiness”; because this would equip people not only to flourish across all spheres of human experience, but also to exert influence in the political institutions that determine how equitably the rewards of education are distributed. (In contrast, if the education system aims primarily to enable all graduates to get high-paying jobs, there will be little impact on equality if those in power alter the sociopolitical landscape to exclude certain people from such jobs, regardless of their qualifications.) This leads to a defense of the humanities in education, because:

Education’s most fundamental egalitarian value is in its development of us as language-using creatures. Our linguistic capacities are what, fundamentally, education taps, and it is their great unfolding that empowers students. … As we cultivate verbal empowerment in our students, we build the foundation for a politically competitive social and political system.

Although Allen doesn’t seem like a fan of economistic justifications for education, her arguments here dovetail with some the work I’ve been reading for my current research project, which argue that a central component of economic competitiveness is verbal and social competence—asking good questions, presenting your views persuasively, collaborating productively—because technological advances and automations can’t replace these skills. I also see resonances of her framework in Malaysian schools and civil society, where different mother tongues and media of instruction  feed into separate sociopolitical spheres—impoverished “bridging skills”, so to speak—which have all kinds of implications for equality and corporate flourishing.

A book containing the lectures, along with commentaries on the lectures by other scholars, will be published next month.

Danielle Allen membincangkan pendidikan dan kesaksamaan

Beberapa minggu lepas, saya diberi salinan teks daripada Kuliah Tanner yang disampaikan oleh ahli teori politik Danielle Allen di Stanford University pada tahun 2014.

Dalam kuliah pertama, “Two concepts of education (Dua konsep pendidikan)”, Allen merujuk kepada dua pemikir terkemuka untuk membincangkan matlamat yang patut dikejar dalam pendidikan. Daripada “Two concepts of rules (Two concepts of rules)” karya John Rawls (1955), Allen membezakan antara alasan sosial untuk sistem dan institusi pendidikan, yang biasanya berdasarkan manfaat umum seperti kemajuan ekonomi; dan alasan individu untuk setiap kali pendidikan berlaku, yang sepatutnya berdasarkan perkembangan kebolehan setiap pelajar yang terlibat.

Kemudian, Allen merujuk kepada The human condition (Keadaan manusia) oleh Hannah Arendt (1958). Dalam buku ini, Arendt berkata bahawa kehidupan yang bermakna dan bersepadu merangkumi tiga jenis perbuatan teras: kerja buruh (labour) untuk mengisi perut; kerja ciptaan (work) untuk membina ruang kemanusiaan secara fizikal dan sosial; dan tindakan (action), iaitu usaha bersama untuk menentukan hala tuju politik masyarakat dan negara. Dengan fahaman sebegini tentang kehidupan yang terbaik, Allen berpendapat bahawa matlamat pendidikan daripada segi individu adalah untuk menyediakan pelajar untuk: (1) mencari makan; (2) mengambil bahagian dalam dunia politik dan sivik; (3a) menggunakan daya cipta dalam wacana, seni, dan masyarakat; dan (3b) menjalin hubungan yang bermakna. Menurut Allen, pendikan patut mengejar kesaksamaan dalam semua kesediaan inidividu yang menyeluruh ini, dan bukan sahaja dalam kesediaan (1) dan (2) yang memberi manfaat dari sudut pandangan umum.

Dalam kuliah kedua, “Participatory readiness (Kesediaan mengambil bahagian”, Allen menegaskan bahawa cara yang paling berkesan untuk meningkatkan kesaksamaan melalui pendidikan adalah dengan memberi tumpuan kepada kesediaan pelajar untuk mengambil bahagian dalam semua dunia sosial. Dalam takrifannya, kesediaan mengambil bahagian ini terdiri daripada: kemahiran mentafsir dan menyampaikan pendapat, kemahiran menjalin hubungan, dan kecekapan dalam tentang strategi dan pendekatan dalam perhubungan. Dengan kesediaan ini, kita mampu bersuara dan mengakibatkan perubahan dalam arena awam, demi kesaksamaan.

Sebuah buku yang mengandungi teks kuliah ini bersama ulasan daripada cendekiawan lain akan diterbitkan bulan depan.