A $50 million-a-year scheme to help NSW students learn to read and write through recognising words by sight doesn't work, according to a new study by the state Education Department.
A departmental evaluation of Reading Recovery – a 40-year-old programthat originated in New Zealand and is widely used in Australia, the United States, the United Kingdom and Canada – found it is only effective for a small proportion of students.
The report's findings will give more ammunition to critics of Reading Recovery, who say the program does not properly use phonics and explicit instruction to teach struggling students how to read – techniques that research has shown are the best ways to deal with reading difficulties in the early years of primary school.
IFERI committee member, Professor Kevin Wheldall had this to say:
Children's reading specialist Kevin Wheldall, now an emeritus professor at Macquarie University, was asked by the NSW Education Department to evaluate Reading Recovery in the early 1990s. He and his team found that it was effective for one in three children at most, and those whom it did help had the least reading problems. But the report given to the department was never released.
"It [Reading Recovery] has powerful advocates within the [NSW Education] department which has prevented any dramatic action to deal with it," he said.
Professor Wheldall said the Reading Recovery did not include phonics in "any structured, systematic, explicit way".
Reading Recovery was developed in the 1970s by New Zealander Marie Clay, who successfully promoted the program in other major English-speaking countries and in 1987 was made a Dame Commander of the Order of the British Empire for her services to reading.
However, Lorraine Hammond, a senior lecturer at Edith Cowan University and vice president of Learning Difficulties Australia, said Reading Recovery was an old program that had been "set in stone".
"The whole world of reading has moved beyond the 10-step plan she put forward," Dr Hammond said.
Kevin Wheldall is flagging this report up on Twitter with the comment that he has been telling the authorities in NSW for 23 years that RR does not work!
And I am asking via Twitter how many more reports do we need before RR is replaced with high-quality, first time, evidence-informed teaching.
Plus, I am pointing out that it is impossible to hold promoters and funders of RR to account.
Reading Recovery is entrenched internationally - but it is particularly embedded in Australia and New Zealand - and very influential for first-time teaching and intervention.
In other words, the multi-cueing guessing strategies prevail.
You can read about Professor Kevin Wheldall's work here:
IFERI committee member, Professor James Chapman commented privately via the DDOLL network about this NSW report. I asked his permission to copy his comments for this particular thread in the absence of a formal review of the report. James has given me permission to share his informal initial response here:
Yes, indeed!! It truly is a pretzel report with astonishing explanations for the data. No matter how they try to explain the data and emphasize the apparent better “success” for the lowest performing RR students at the end of K, the results are still poor. They are way short of the promised claims for RR persistently presented by Clay: bringing the lowest performing students after one year in school up to the level of their normally achieving peers.
The headline statement that the results show “some evidence that RR is effective at improving short-term reading outcomes at the end of Year 1” (p.21) is a nonsense statement. Of course the program will be of some benefit to some students. The last sentence in the same paragraph is simply not true: “These finding suggest that RR is an effective intervention for improving short-term reading outcomes among the poorest performing readers, which is the primary intention of the intervention [my emphasis].” Yes, there were some modest improvements that were greater than for those students who had higher scores at the end of K. However, the primary intention of RR is to bring students up to the average of their peers. This did not happen. RR was never designed just to improve reading performance. It was designed as an intensive intervention, with specially trained teachers, to significantly improve reading performance of the lowest achieving students—to the level of their peers.
Only 32.7% (p.14) of RR students met expected standards at the end of Year 1, according to classroom teachers’ assessments. Hardly the hallmark of a successful program.
This finding demonstrates that the acceleration goals of RR were not achieved by large numbers (two-thirds) of RR students.
Year 3 NAPLAN results showed “no evidence that RR yields any positive effects on students’ NAPLAN reading performance…” (p.21). Indeed! This finding is “excused” with the following comment: “The duration of the program is only 12-20 weeks so it is equally possible that RR students do not receive the level of support they need to sustain any short-term RR effects beyond Year 1” (p. 21). This excuse about levels of support has been used in New Zealand to try to explain the general failure to attain sustained improvements for those students who are discontinued from the program. Not the fault of the program, which is inherently sound, but of regular classroom teachers who fail to teach these ex-RR students properly.
The instructional approach and emphases of the program are seldom questioned in NZ and I’m surprised that the NSW report would buy into the same explanation without seriously considering the demonstrably flawed instructional approach. There is reference to the importance of phonics and word-level instruction (p.22) but it is effectively dismissed with the comment about the need for more research, and with the paragraphs that follow.
Much of the Discussion section is pretzel prose. I hope someone will take on a systematic critique of the report!
James Chapman
You can read about Professor James Chapman's work here:
Developmental Disorders of Language and Literacy Network
The DDOLL network was established in 2003 with funding from the Australian Research Council. The group consists of scientists, clinicians, teachers and parents interested in discussing, and disseminating information about, the investigation and treatment of developmental disorders of language and literacy through sound scientific methodology and evidence-based research. In order to maintain this focus, membership of the network is through invitation only. If you share the aims of the network and would like to join, please send max.coltheart@mq.edu.au a few sentences about your background and your reasons for being interested in the network. We would especially like to encourage current or prospective PhD students to become part of the network.
Sir Jim Rose's interest in the Reading Recovery NSW report is very welcome. Sir Jim contacted Professor Greg Brooks for his initial response to the report with the message below shared initially with the DDOLL network (permission received for this quote):
Sir Jim Rose wrote:
As you know, RR is entrenched as an intervention over here. I have laboured in vain to highlight its demerits for years but it keeps popping up under other labels and has continued to win unjustified funding. I thought the ‘Learning Curve’ report might be useful as evidence of flaws in RR so I sent it to Professor Greg Brooks for comment . Greg was a senior researcher at the NFER and has continued to critique research, for example , on intervention programmes and reading. He authors a report called ‘What Works’ which you may have seen.
Professor Greg Brooks responded to Sir Jim's request and gave his permission to share the following comment with the DDOLL network and subsequently via this IFERI forum.
Professor Greg Brooks wrote:
I will mention this study in the new edition .. (of What Works). However, in my opinion it isn't strong enough to overturn the findings from the US RCTs. It isn't an RCT; it isn't even a carefully (enough) matched-groups quasi-experiment. The comparison (not control) groups were (1) children at similar levels not receiving RR; (2) all children of the same age in the state. The 2nd group is clearly not a valid comparator. The 1st group, as far as can be judged from this document, was really just a convenience sample - all the other children at similar levels on whom there were data. There may of course be better detail in the fuller report behind this one, but so far this study seems to me to be less strong than the 2 London studies, where good attempts were made to recruit closely matched samples.
I also have serious doubts about the statistical analyses reported. Odds ratios are perfectly respectable measures, but we need to know what they were for all subjects, not just by starting level, where regression towards the mean may have influenced the results of the lowest-performing subset. And effect sizes would be more informative than odds ratios anyway.
The biggest missed opportunity of the statistical approach reported, however, is that the authors could (and probably should) have used propensity score matching (PSM). This is a technique specifically devised for use in circumstances where the intervention group is pre-determined and there is therefore no possibility of running an RCT. PSM requires the use of copious information about the participants available before the study begins - as here - and then searches among the available members of the comparison cohort for the best match for each intervention-group participant until a well-matched comparison group is assembled. This is much more rigorous than just using all members of a conveniently available comparison sample.
So you will understand that I have doubts about the reliability of the findings from this study.
Sir Jim Rose is an IFERI committee member and you can read about his work here:
