Eng: 2015 NFER Phonics Screening Check Evaluation - final report

[Debbie_Hepplewhite]

https://www.gov.uk/government/uploads/s ... _brief.pdf

Conclusions

This three-year evaluation has tracked developments in schools from the first national introduction of the PSC in 2012 to the current, 2014, round. Over this period, teachers’ responses suggest that most of them now see the standard of the check as appropriate. Teachers have integrated information from the check with their other records of children’s progress in phonics. Its introduction has required administrative effort in schools and gives rise to some, relatively low, costs in terms of time or resources. Little training is now required for teachers and many are familiar with the procedures for the check.

The three years have also seen a range of changes in schools which were, according to teacher reports, made in response to the check; the evidence suggests that a majority of schools have made some adjustments. These changes consist of improvements to the teaching of phonics, such as faster pace, longer time, more frequent, more systematic, and better ongoing assessment. Children are also introduced to the pseudo words that form part of the check. Most schools, however, continue to teach other strategies for word reading alongside a strong commitment to phonics. Nevertheless, according to these teacher reports, the introduction of the check has had impacts on teaching.

To assess whether its introduction also had impacts on pupils’ learning is more difficult, as the national introduction of the check made it impossible to have a control group. A further complexity concerns the date of implementation of the PSC. It was introduced for the first time nationally in 2012, but was piloted in 2011. Awareness of the introduction of the check may have given rise to a heightened emphasis on phonics in schools prior to its national introduction. The process evaluation of the pilot (Coldwell et al, 2011) found that the sample of schools in the pilot were already making some changes to their phonics practice.

While keeping these complexities and methodological limitations in mind, the national results show an improvement in performance in phonics, as measured by the check, which would be consistent with the adjustments to teaching methods reported above. Analyses of pupils’ literacy (reading and writing) scores in the national datasets over four years were not conclusive: there were no improvements in attainment or in progress that could be clearly attributed to the introduction of the check; attainment and progress improved in the years both before and after its introduction. As far as it is possible to report, given the methodological limitations of the study, therefore, the evidence suggests that the introduction of the check has had an impact on pupils’ attainment in phonics, but not an identifiable impact (or not yet) on their attainment in literacy. It will be of continuing interest to review the results at key stage 1 in future years and also the results at key stage 2 as the pupils who took the check progress through their later years of schooling.

Please note that it is looking like teachers persist in continuing with multi-cueing reading strategies against Government guidance and to disregard the warnings in research-findings (or not to know or understand the warnings) although their phonics teaching has been given more attention according to this description:

The three years have also seen a range of changes in schools which were, according to teacher reports, made in response to the check; the evidence suggests that a majority of schools have made some adjustments. These changes consist of improvements to the teaching of phonics, such as faster pace, longer time, more frequent, more systematic, and better ongoing assessment. Children are also introduced to the pseudo words that form part of the check. Most schools, however, continue to teach other strategies for word reading alongside a strong commitment to phonics. Nevertheless, according to these teacher reports, the introduction of the check has had impacts on teaching.

[/quote]

These issues are touched upon in my RRF 2015 conference talk - including reference to the 2014 NFER phonics evaluation findings (I hope viewers will find this a very pertinent talk not only regarding teaching in England but also in other English-speaking countries and wherever English is taught as an additional language):

https://www.youtube.com/watch?v=e5QT8kE ... e=youtu.be

And here is my graphic analysing types of phonics provision in England - but you may recognise the different types of provision in different countries and contexts: The Simple View of Schools' Phonics Provision

http://www.phonicsinternational.com/Sim ... chools.pdf

[Dick Schutz]

The Final Report raises more questions about the Report itself than it sheds light on the Screening Check. The second Interim Report sketched what the third year would do:

This chapter concludes by outlining the next steps in the evaluation, including the
completion of endpoint surveys and case-studies, analysis of NPD data and the final
value for money assessment.
Endpoint surveys and case-studies
The case-studies and surveys of literacy coordinators and Year 1 teachers will be
repeated with the same respondents in summer 2014. Additional schools will be recruited
to top-up the samples as required.
Analysis of National Pupil Database data
The primary objective of analysing NPD data is to explore the impact of the introduction
of the check on key stage 1 reading scores. This analysis will be undertaken in autumn
2014, modelling for each school the progress to key stage 1 for successive cohorts of
pupils up to those who complete key stage 1 in 2013/14. If the check has had a positive
impact, a step-change improvement in progress for the final two years of key stage 1
outcomes, i.e. for 2012/13 and 2013/14, is expected.
Value for money analysis
Year 3 of the study will see further development of the assessment of Value for Money
(VfM). Analysis of VfM will draw on cost-effectiveness analysis and cost-benefit analysis.
Through questions included in the teacher surveys, relative cost and time implications of
different approaches adopted by schools and the numbers of pupils engaged in
additional support will be explored. In combination with NPD analysis of relative impact,
this will enable cost effectiveness comparisons to be made.

The "Endpoint surveys and case studies" don't add to the interim results. The "impact on KS 1" inquiry was abandoned for the spurious reason,"absence of control group." No "value for money" assessment was conducted.

Who evaluates the evaluators?

[Debbie_Hepplewhite]

Here's a thought - if all schools in Scotland, Wales and Ireland took the Year One Phonics Screening Check at the same point as children in England, we would have had an extensive control group of sorts!

IFERI is urging teachers in other schools in other countries to utilise England's Year One Phonics Screening Check when it becomes freely available shortly after its official use in England's schools.

Dick - thank you for your posting - please do add what you would have done differently - and what could still be achieved using the Year One Phonics Screening Check - as I know that you are very knowledgeable about testing and statistics.

[Dick Schutz]

Actually, there are ample "controls" within the data for England, and there are lots of ways to go about the investigation. All that needs to be done is to ask schools and teachers what reading programmes they are using--as was done in the pilot study and as the Framework for the Check said would be done subsequently. Teachers and schools know what they are doing and are not reticent in telling anyone who asks. Whatever they say goes. We (anyone who has been paying attention) know that most will say "mixed methods" and most of those will be using "Letters and Sounds." but it doesn't matter for purposes of the investigation. What you get are natural experimental groups whose performance on the PSC can be compared. It doesn't take fancy statistics, and it doesn't take a large sample. The population of teachers and schools is so large, other samples can be drawn if there is any doubt about the information--yielding the the same control that "randomized controls" provide.

The NFER researchers had a good start on this with the 4-fold distinction they made in their 2nd interim report. They just didn't follow through.

It has also been reported that over 600 schools got "near perfect" passes on the PSC. Nobody has yet asked them about the instruction they provided.

Another example: Some view "training" as important. We know that most schools believe they don't need it--just as they believe they "already knew" the info provided by the PSC. The obvious counter to those beliefs is "Why then are they not getting "near perfect" performance on the PSC? Not too many schools bought into the "match funding" for training, but some did. Those who did, constitute a natural experimental group to compare with the "control group" of those who didn't receive training.

Similar opportunities for investigation are in what is going on in Yr 2 with kids who didn't pass in Yr. We know the general answer is "not much." However, it is almost certain that there are schools and classes who are doing "a lot" and have the performance data to show for it. It's just that no one has yet looked at where they are or what they are doing--providing the "evidence" that everyone gives honorific lip-service to.

Re "cost benefit analysis." It's almost a no-brainer. A "test" that is unobtrusive instructionally, requires only a few pages of paper for a full class, and that is scored by the test administrator "on the fly" is a time/cost bargain compared to any psychometric alternative. It's not even necessary to "run the numbers" to reach that conclusion. Opponents to the PSC have been successfully peddling self-serving poppycock. But absent replicable information for governmental officials and citizenry, the poppycock will continue to prevail.

[Debbie_Hepplewhite]

Great suggestions, Dick - many thanks.

We'll have to see what we can do re these suggestions.

There's no doubt about it, schools have a very wide range of attitudes and 'understanding' when it comes to phonics provision - and what it means to be 'rigorous' - or not.

Forgive me if you have seen this already, but I recently drew up a graphic to illustrate the wide range of 'types' of schools and why this is likely to produce very different phonics results in the Year One Phonics Screening Check:

http://www.phonicsinternational.com/Sim ... chools.pdf

Clearly it is not acceptable that some schools' phonics provision is so much weaker and less effective than others, and that many schools still persist with multi-cueing reading strategies despite official guidance and despite the findings of research.

There is definitely a need for more in-depth professional development - not only for teachers but also, in my view, for Ofsted inspectors!

Only a day or so ago, an influential former HMI replied to me on Twitter that 'there is no one way' with reference to phonics/reading instruction.

He is the chap who represented Ofsted when we took Ofsted to task for providing video footage with very weak phonics lessons in apparently 'outstanding' schools.

[Dick Schutz]

The thing is, some children will intuit the Alphabetic Code of English without any (apparent) formal reading instruction. Other children will learn how to handle the Code, irrespective of mal-instruction. Others will be able to cope using "work-arounds" that are not effective, but are not so ineffective they are handicapping other than that the children "find reading difficult."

However. . . It is the substance and structure of the Alphabetic Code that permits written English and the history of the language that accounts for the substance and structure. One Alphabetic Code fits all. About these matters, there is no rational argument; only ignorance, and that ignorance is widespread.

"How many ways there are to teach phonics" is an empirical question that to date has received scant empirical or analytic attention. Diane McGuinness' scale, running from "junk phonics" to "full Alphabetic Code," is a starting analytic point, and the PSC provides a "good enough" dependent variable for natural experimental investigation. The proof is in the pudding, and the proofing is feasible, but it has yet to be done.

People bristle when told there is "one way" to do anything. There is always some latitude/alternatives/"wiggle room" in doing anything. But there are always more ways to "go wrong" than to "go right." The standard way to untangle differences of opinion is "try it out and see what happens." There is more than one way to skin a cat, but one needs a cat, and there are lot more ways of doing the job wrong than doing it satisfactorily.

[Debbie_Hepplewhite]

And if all the teachers managed to teach virtually all of their children to read - and to spell - and to write (in a similar, realistic time-scale) - well, then it would be 'no worries' about how they taught their pupils.

The trouble is that when people refer to the 'no one way' argument in this particular field, it is invariably a suspect cliche.

Of course one would have to have a conversation to ask, 'What do you mean that there is no one way?' to get to the bottom of their thinking.

Experience, and perhaps a presumption too, suggests that the 'no one way' is likely to mean a lack of full belief in teaching the alphabetic code well - or perhaps a belief in something like 'different learning styles' and/or 'a range of reading strategies' (which usually means the multi-cueing guess from this, that or the other!).

And of course this field is SO important and there has been, and is, SO much weak literacy or illiteracy, that we are in need of drilling down to what is the most successful way of teaching literacy - technical knowledge and skills and higher-order knowledge and skills.

I don't even think we are off the starting block when it comes to properly exploiting the alphabetic code for teaching spelling over a more sustained period.

I still despair when I see material or hear guidance along the lines of 'Does the word look right?' and 'Think of words within words', and 'Write the word out in multiple colours' - and so on.

I totally agree that different children can intuit the alphabetic code in their own way in their own time with their own experience - but of course that includes the reality that many children can't, or don't do this well enough - to help them thrive in education and in life.

Therefore, as teachers, we simply must be concerned about 'how best' to teach ALL the children and to teach ALL the children well.

I get particularly concerned, indeed shocked, that any teacher (advisor, union leader, politician, inspector, parent, children's authors) cannot see the huge value in a quick, national snapshot of decoding knowledge and skills to give us a steer on how we are effectively we are teaching as individuals and as a nation in something so fundamentally essential to children moving forwards with their wider learning - and their self-esteem.

[Debbie_Hepplewhite]

I'll add here, too, that this issue is about far more than teaching English-speaking children in English-speaking countries to read and write in English.

In every English-speaking country, there are many children for whom English is a new or additional language - and therefore all teachers should be trained and knowledgeable about the value of teaching the English alphabetic code from a perspective far more complex than just picking up reading and writing in the mother tongue.

Then, we have a duty to hone our programmes, and our teaching, because much of the rest of the world chooses to teach English as an additional language.

Many teachers and pupils are placed into situations of great hardship in taking on such a huge task with the most complex alphabetic code in the world.

We simply must, as a duty, and a wider duty, find out how well our reading instruction (and spelling and writing instruction) is - and whether there is a BETTER way, even if there is NO ONE WAY.

[Dick Schutz]

I agree with all this, Debbie, but we're preaching to the choir. Although the NFER "Final Report" is flimsy, what there is of it is positive and the "main idea" is accurate. Despite opposition to the Check from most quarters, the Check and "phonics" has been incorporated successfully into primary schooling in England. Opponents to the Check can find nothing in the Report to support their negative claims. That won't stop the yapping, but it's now reduced to "barking at phonics"-- with no bite. Further, the National Curriculum has been revised with "mixed methods/searchlights" abandoned. Those are solid advances to build on.

Seems to me it's important to avoid the common error of "fighting last year's war." The battle at the National level in England has been won. It's also been won in some schools, but there is no intelligence, other than anecdotal, where these schools and classes are or what they are doing instructionally. The intelligence "data" has largely been collected but has not been pursued.

While it's tempting to envision "changing teacher preparation" and "training teachers and HT's," it's wishful thinking. Further, it's unnecessary. Parents with much less formal education than teachers and HTs and teachers in countries other than English-speaking countries are delivering "effective reading instruction" consistent with IFERI intent. This is in no way to discount "teacher prep and training"--just to point out the realities involved.

As the benefits of Alphabetic Code-based instruction become more widely realized, former critics will have easy defense mechanisms: "Of course, this is what we were trying to tell you all along. Everyone has always recognized the importance of phonics. We've just been trying to remind people of what Sir Jim Rose said--It's important that phonics instruction be embedded in a rich curriculum (blah, blah) that does not lose sight of 'meaning.'" By that time, there will be new "worries," but kids will be reliably reading--largely on electronic devices, but on whatever media they care to.

[Debbie_Hepplewhite]

Great post, Dick.

We know of parents who are more knowledgeable about phonics and reading instruction than their children's teachers but, as you say, we have achieved a great deal to get successive Governments more or less 'on board' in England with Systematic Synthetic Phonics and that is a huge achievement in itself.

Hopefully, this in itself will help to draw attention to phonics and increase the likelihood of improved phonics contents in curriculums elsewhere.