Critique of the i3 Scale Up Study of Reading Recovery (Chapman and Tunmer) is now available online

[Debbie_Hepplewhite]

I've been informed that a critique of the i3 Scale Up Study, by James Chapman and William Tunmer, to be published in the Journal Reading Psychology, is now available online.

See details below:

Is Reading Recovery an Effective Intervention for Students with Reading Difficulties? A Critique of the i3 Scale-Up Study

DOI: 10.1080/02702711.2016.1157538

James W. Chapman & William E. Tunmer

Published online: 24 Mar 2016

Abstract

The recently reported i3 Scale-Up of Reading Recovery (May et al., 2015) found an effect size of + 0.69 in favor of Reading Recovery compared to the control group. We discuss four issues: (a) many of the lowest achieving students were excluded from participation in Reading Recovery; (b) the control group received a range of different experiences; (c) the successful completion rate of students in the program was modest; and (d) no data supported the claim that Reading Recovery leads to sustained literacy learning gains. We question the value of this study as the basis for widespread implementation of Reading Recovery.

See: http://www.tandfonline.com/author/Chapman%2C+James+W

[Debbie_Hepplewhite]

In 2015, IFERI featured a new paper by Professor James Chapman and Professor William Tunmer reviewing the effectiveness of Reading Recovery.

See here:

http://www.iferi.org/a-new-paper-by-pro ... -recovery/

The Literacy Performance of ex-Reading Recovery Students Between Two and Four Years Following Participation on the Program: Is this Intervention Effective for Students with Early Reading Difficulties?

James W. Chapman and William E. Tunmer Institute of Education, Massey University Palmerston North, New Zealand

[Debbie_Hepplewhite]

This is a thread started in May 2015 featuring the questions raised about the efficacy of Reading Recovery and its continued promotion and funding by various educational and political establishments across the world:

http://www.iferi.org/iferi_forum/viewtopic.php?f=2&t=22

[Debbie_Hepplewhite]

I'm linking this thread to events in Australia where Professor Kevin Wheldall has written an open letter to Education Minister Adrian Piccoli about the continued official use of Reading Recovery in New South Wales:

http://www.iferi.org/iferi_forum/viewto ... ?f=2&t=532

It is looking like the questions raised about Reading Recovery are not being sufficiently addressed, or addressed at all, by the RR organisation itself and by others who continue to use and promote RR - including various government bodies and educational establishments around the world. Clearly this is an ongoing scenario of international concern, which has lasted many years, and it is arguably not an acceptable state of affairs.

Kevin relayed his experiences via a private forum and I asked his permission to share these on this thread. Kevin describes the chronology and reality of events thus:

As I have said before, RR is a great looking car with tremendous dealerships and owners associations but it has a really seriously underpowered and extremely unreliable engine. For many years I urged that they should keep the brand and the networks, not to mention their powerful links to government and funding, and simply replace the existing dodgy engine with a state of the art synthetic phonics engine. My blandishments fell on deaf ears.

(As my old English teacher once wrote in the margin of my essay, “This metaphor is now overworked!” and so I’ll stop.)

But it was only in the face of the RR community’s continual refusal to update their program to incorporate the research finding from the past thirty odd years, that led to our decision to devise a small group program alternative, MiniLit. I get very cross when folk say that we damn RR because we are pushing our own MiniLit barrow. I have been criticising RR for over 20 years now, since our evaluation of RR (subsequently published in RRQ in 1995) was completed; way before MiniLit was even a twinkle in my eye.

[Debbie_Hepplewhite]

Evaluating the four-year scale-up of Reading Recovery

Robert Pondiscio

March 23, 2016

http://edexcellence.net/articles/evalua ... g-recovery

...Big study, big results. However it’s worth noting that the study’s control group received no consistent, validated intervention from trained teachers. As psychologist Steven Dykstra waggishly observed, it's like comparing a new cancer treatment to aromatherapy, then claiming the results prove that your treatment is valid. Is the secret sauce Reading Recovery? Or would any well-structured, intensive program of one-on-one intervention work equally well? Answering that would require a massive study that compares Reading Recovery not just to the dull hum of workaday literacy instruction in struggling schools, but to competing programs like Core Knowledge Language Arts, Success For All, Teacher’s College Reader’s Workshop, and any number of “big box” programs from commercial publishers. Once we can line those initiatives up and say, “good, better, best,” we’ll start to seriously arrest and address the desultory reading achievement that chronically plagues American children, particularly low-income black and brown children in underperforming schools.

[Dick Schutz]

It doesn't require a "massive study" costing millions and taking several years to compare instructional programs, only to find that they don't reliably teach children to read. That result has been replicated ad infinitum throughout the English speaking over the last 100 years, but the finding has been attributed to faults of kids and their parents or to teachers lack of "program fidelity."

The research methodology to "solve the reading problem" is straightforward: and it need not entail fancy statistics, large incremental costs, or intrusive assignment of experimental participants to artificial groups. A Natural Experiment is in play in ongoing schools every year, with replication in each successive year. That is, each year parents send schools the best kids they have and expect schools to teach the littlies how to read. A fresh Cohort comes every year.

What you see is what you get, and what the kids get is an instructional Application--a "reading program" in educationese.

The differential "reading programs” schools are providing constitutes the Independent Variable of interest. School personnel will tell anyone who asks what this program is, constituting the Treatment. (You may not like what you are told, but what you see is what you get.)

The Treatment Effect can be determined using the Alphabetic Code screening check currently being administered to all primary school children in England. (Provocatively, in more than 600 schools in England, all children with few exceptions are currently passing this screen, indicating reliably teaching kids how to handle the complexity of English text in written communication as in spoken communication is indeed feasible.) The Check requires no more than 5-10 minutes and can be given and scored by the classroom teacher--akin to the "eye chart" used in driver license screening.

The reading instructional results will vary--we KNOW that. What hasn't been examined is the characteristics of the instructional Apps that are producing the differential Effect. The Population of schools and classes is sufficiently large to make it feasible to draw random samples, near infinitum, to cross check the reliability of the differential program results.

Simple as that. But academicians and researchers are resistant to change in the methodological status quo, and there are many powerful interests within and without academia that have become economically dependent on the "reading problem." It's problematic if-and-when "less research" will be implemented.

[Debbie_Hepplewhite]

I'm cross-referencing this thread with a thread featuring another report to be published about the Reading Recovery i3 Scale Up Study of Reading Recovery - a 'must' read with damning conclusions:

http://www.iferi.org/iferi_forum/viewto ... 1606#p1606

[Debbie_Hepplewhite]

Further commentary via the Ripe Tomato blog:

The $55M i3 trial of “Reading Recovery”

MARCH 4, 2018

https://ripe-tomato.org/2018/03/04/the- ... -recovery/

Results were biased by measuring outcomes when the intervention group hit their success target.

Reading Recovery (RR), a method for helping poor readers, designed on the basis of research in the 1970s by New Zealand psychologist Marie Clay, is a worldwide movement. It is controversial, partly because it is a semi-secret commercial programme for which schools or parents have to pay, and partly because it emphasises “whole language” rather than phonics based methods.

In 2015 RR was evaluated in a large and expensive ($55M) randomised controlled trial in the United States. The report of the first year of the randomised element (click here ) is behind a paywall, but a copy is available here i3 reading recovery trial. The full report of all four randomisation years, as well as the non-randomised scale-up phase is free (click here) or reading_recovery_final_report.

The trial has been criticised by some educationalists (click here) who appear to be more politically opposed to RR, than to have identified genuine flaws in the trial’s methods. They cite four “problems”. 1. Many low achievers were excluded. Answer: They were excluded from intervention and control groups equally, so did not bias the results. 2. The control group received a range of different experiences. Answer: Yes. They got “usual care”, common practice in a pragmatic trial. 3. The successful completion rate of students in the program was modest. Answer: Yes, but it still appeared to work. 4. No data supported the claim that Reading Recovery leads to sustained literacy learning gains. Answer: A valid criticism. Long term effects could not be measured in the randomised groups, because controls got RR at the end of the intervention period anyway. Other critics have repeated the claims that the effect of RR was small but not sustained (click here) or focused on lack of definition of what the control group got (click here).

The trial also has defenders (e.g. click here). The What Works Clearing House, an independent outfit evaluating evidence-based educational interventions not only issued a special evaluation (click here) or wwc_may_102814, but gave the trial its strongest possible endorsement; “The research described in this report meets WWC group design standards without reservations.”

I’m a medical doctor with no vested interest – I only learned of the existence of RR a few weeks ago – but I do know a bit about randomised trials. So far as I’m aware, the more fundamental flaws identified in the rest of this post, have not previously been described.

[Debbie_Hepplewhite]

Here is another important paper written by James Chapman and William Tunmer published in the UK journal, Review of Education - which I've cross-referenced with this thread:

Original Article

Reading Recovery's unrecovered learners: Characteristics and issues

James W. Chapman William E. Tunmer

First published: 10 July 2018

http://www.iferi.org/iferi_forum/viewto ... 2067#p2067

[Debbie_Hepplewhite]

https://js.sagamorepub.com/ldmj/article/view/8391

The Reading Wars and Reading Recovery: What Educators, Families, and Taxpayers Should Know

Pamela Cook, Deborah R. Rodes, Kay L. Lipsitz

Abstract

Reading Recovery, a meaning-based reading program designed for young children at risk of reading failure, is widely implemented across the United States. We discuss the recent Reading Recovery $45 million four-year i3-funded scaleup study that was designed to “cover the expansion of Reading Recovery around the U.S.” (May, Sirinides, Gray, & Goldsworthy, 2016, p. 1). While one of the two goals of the study was to determine the long-term impact of Reading Recovery, this study, described by its authors as “highly successful” (p. 4), found a “not significant” long-term effect on students’ reading skills. Subsequent Reading Recovery publications have failed to mention this “not significant” effect. With the exception of year one of the study, there are no publicly available test score data for the students when they were in Grades 2 or 3. Further, it appears that the actual lowest achieving students (special education students, students retained in first grade, and others) were systematically excluded from Reading Recovery instruction. Overall, there is very limited evidence of Reading Recovery’s efficacy as an effective long-term reading intervention. We discuss the limitations of the Reading Recovery approach, how Reading Recovery can be improved, and strongly recommend that schools do not adopt this program unless it incorporates all components of evidence-based reading instruction.