Academics put heat on half-baked reactions
The political and media stir caused by professor John Hattie's research on student achievement has prompted a group of academics to look closely at his work.
The authors were particularly concerned that politicians might use Hattie's work to justify ill-informed policy decisions.
Massey university academics Ivan Snook, John Clark, Richard Harker, Anne-Marie O'Neill and John O'Neill banded together to produce Invisible Learnings: A commentary on John Hattie's "Visible Learning: A synthesis of over 800 meta-analyses relating to achievement".
Commentary a result of politicised and simplistic discussion
Massey University Emeritus professor of education Ivan Snook said it was government and media interest in the research that prompted the commentary.
"We thought it was getting a bit too politicised in that the minister (of education) was expressing a great interest in it.
"The media also seemed to be taking quite a simplified view - the Times Education Supplement in England said a New Zealand professor had found the 'holy grail of education', which we thought was taking things a bit far," he said.
According to the introduction of Invisible Learnings? the announcement of Hattie's book had already led to a good deal of discussion in New Zealand and overseas and seemed to have captured the attention of policy makers.
Courtesy of critical scrutiny rather than 'half-baked reactions'
"It is therefore important that the members of the educational research community pay John Hattie the courtesy of subjecting his conclusions to critical scrutiny in a spirit of mutual truth - seeking to ensure that discussions are based on a careful reading of the book, rather than on half-baked 'reactions' in the popular media…(and) the findings are not 'appropriated' by political and ideological interests and used in ways which the data does not substantiate," it says.
Reducing class size enables 'doing things differently'
The commentary looks at Hattie's findings on class size and expresses concern at the validity of these.
"Hattie has been cited as 'finding' that class size is not important and this has excited the attention of those concerned about financing of schools, who conclude they can economise on class size," it reads. These results however fail to look at contributing factors such as professional development and the ability for teachers to spend more time with students.
"Reducing class size may have only a small effect when considered in isolation, but that's not the issue. What matters is that reducing class size permits the teacher (and children) to do things differently," it says.
No evidence to support performance pay
The report also voices concern that Hattie's conclusions on the importance of what teachers do have led some to advocate performance pay.
"We have seen no evidence at all to support the claim that performance pay improves teaching or learning and there is nothing in Hattie's massive research which even remotely suggests that it does. On the contrary, much of what he says suggests the very opposite… (he says) what is needed for school improvement is a caring supportive staffroom, a tolerance of errors and ... a peer culture among teachers of engagement, trust, shared passion and so on'. Such a co-operative, trusting and self-critical school atmosphere is the very kind of atmosphere which regimes of performance pay destroy."
Determining effect size and cut off points
The commentary looks closely at the methods Hattie used in his research. All the findings in his book derive from a synthesis of 800 "meta-analyses" of more than 50,000 studies of variables affecting the achievement of students.
The main aim of the exercise was to determine effect sizes of certain factors on student achievement and Hattie does this by reducing these issues to decimal points.
"There are debates about where a small effect size ends and a moderate or large effect size begins. Hattie adopts 0.4 as the cut off point, basically ignoring effect sizes lower than 0.4. Thus, for example, class size is interpreted as a small effect size since it is 0.2 (In public debate this tends to turn into 'class size has no effect at all'.) Selecting a cut-off point is a very hazardous exercise, as it means potentially important effects may be overlooked," the commentary says.
Classrooms are complex
Professor Snook did not believe the figures Hattie presented amounted to a "holy grail" for education. In fact he was rather dubious about the benefits of reducing a complex area like a classroom to mere decimal points.
"A classroom is a complicated sort of place."
Student background and social context are important
The commentary raises a number of concerns, including the fact that social effects and background context are ruled out.
"(This) is not a book about what cannot be influenced in schools - thus critical discussions about class, poverty, resource in families, health in families and nutrition are not included - but this is NOT because they are unimportant, indeed they may be more important than many of the issues discussed in this book. It is just that I have not included these topics in my orbit," Hattie says.
The commentators however are very concerned about this attitude.
"Hattie acknowledges the important role of socio-economic status and home background… but chooses to ignore it. That is his choice: but it is easy for those seeking to make policy decisions to forget this significant qualification," they say.
Professor Snook hoped the commentary would prompt John Hattie to discuss the issues it raised.
"I would like to see a good debate on these issues," he said.
(Published in the PPTA News April 2009)
Further information:
How to engage in pseudoscience with real data: A criticism of John Hattie’s arguments in Visible Learning from the perspective of a statistician
McGill Journal of Education, Vol 52, No 1 (2017), Pierre-Jérôme Bergeron, University of Ottawa Canada
The work of John Hattie on education contains, seemingly, the most comprehensive synthesis of existing research in the field. Many consider his book, Visible Learning (Hattie, 2008), to be a Bible or a Holy Grail: “When this work was published, certain commentators described it as the Holy Grail of education, which is without a doubt not too much of a hyperbole” (Baillargeon, 2014, para. 13). Editorial: Education can’t be reduced to simple measures
December 2, 2014, Jonathan Teghtmeyer, ATA News Editor-in-Chief (Alberta Teachers' Association)
...Visible Learning has made big waves worldwide, and Hattie even came to Alberta in the fall to speak to educators. His findings on class size, in particular, have been widely touted by politicians and others looking to discredit a push for smaller class sizes.
The shocking information making the rounds last month was an admission from Hattie that his work contained significant mathematical errors (see http://bit.ly/1xtmtDg).