Centre for Multilevel Modelling

menu

The teaching of reading in 45 inner London primary schools: a critical examination of OFSTED research

Summary

Major policy conclusions have been drawn from this report by OFSTED. A close examination of the evidence reveals that a series of methodological errors and limitations makes these conclusions largely invalid.

Introduction

Learning to read is a key to a child's education. It is not surprising, therefore, that questions about reading standards and about the teaching of reading excite so much interest in so many people. A recent publication from the National Foundation for Educational Research (NFER)has shown that, although there has been no widespread national decline, there has been an increase in the proportion of pupils in urban schools in the lowest scoring groups (Gorman and Fernandes, 1995). As the report of the National Commission (1993) so clearly recognised, the world of the future is likely to require higher skills in a greater proportion of its population and any increase in reading failure is unacceptable.

The Teaching of Reading in 45 London Primary Schools was published in May 1996 (OFSTED, 1996). It is based on the results of an investigation in which inspectors from Islington, Southwark and Tower Hamlets, three inner London Local Education Authorities (LEAs), worked with a team from OFSTED in 45 school inspections.

The OFSTED investigation provided the opportunity for a major contribution to our understanding of how reading is taught in inner urban schools. It followed the 1993 report on Access and Achievement in Urban Education which had drawn attention to 'inadequate and disturbing' standards (OFSTED, 1993: 7) but which recognised the reality of the challenge, the need for a systematic programme of improvements, and the influence on education of other factors: 'Beyond the school gate are underlying social issues such as poverty, unemployment, poor housing, inadequate health care and the frequent break-up of families. Education by itself can only do so much to enable individuals to reach beyond the limiting contours of their personal and social circumstances and succeed' (OFSTED, 1993: 45).

Given that the new OFSTED work is likely to form the basis for crucial policy decisions, it is imperative that any evidence generated is valid and reliable. The combination of independent research data and inspectorial judgements present an opportunity to investigate in detail the processes of schooling but - as with any innovatory technique - early conclusions need to be cautious rather than definitive. It is also essential that such evidence should be open to public scrutiny. Accordingly, we have sought in a later section to examine the content of the OFSTED report in some detail.

Reaction of the LEAs to the publication of the report

Three LEAs readily agreed to take part in a cooperative investigation of reading in urban areas. In September 1995, the Prime Minister, speaking at a Birmingham conference, announced that three authorities had been targeted 'for punitive inspections' (TES,1996a). According to the Times Educational Supplement (TES), assurances were provided by OFSTED that this was not the case.

Once the analyses had been completed, meetings were held between the two groups of inspectors. The findings were seen by the LEA inspectors as worrying but, on the whole, fair. A draft report was circulated for comment and broadly agreed. Just before the publication date, however, the LEA inspectors were invited to visit OFSTED and given half an hour to read a further amended version of the report. They were not allowed to take away copies. Not surprisingly, their spokesmen and women expressed considerable anger over the changes to the previously agreed version and to the manner in which they were informed of them. The Chair of Education of one LEA stated that he thought 'the slanting of the report is clearly intended to pander to the prejudices of Mr Woodhead's political masters' (TES, 1996b). One Chief Education Officer commented, in a TES article, that neither the issue of progress nor that of resources had been seriously discussed (TES, 1996c).

Reaction of the press

The publication of the report was also controversial. All the major newspapers covered its launch. The Chief Inspector argued that 'it deserves to be studied with the utmost care by all involved in education. Its messages are too important to be ignored.' (Woodhead, 1996a). The Times, in a leading article argued for 'greater powers for the Chief Inspector to step up his campaign against the mediocre' (Times, 1996).

Other papers reported that the draft version had been altered by OFSTED just before publication. According to these accounts a number of emphases had been changed and the conclusions had been 'given a negative spin,' so that they appeared much less positive about inner city teachers and the three LEAs (TES, 1996b). The Chief Inspector commented that he had 'merely edited the report because it was too long' (Woodhead, 1996b). As a result of the publication of the report and the ensuing press discussion, the Secretary of State announced that she was - amongst other things - 'introducing performance league tables for teacher training colleges' and strengthening OFSTED powers 'to monitor pupils' progress in reading' (Daily Telegraph, 8 May 1996).

The publication of this report, despite being based on only one day's work in just 45 schools located in three boroughs of a single city, has clearly been seen as having considerable national significance. Ostensibly, it has focused attention on the importance of reading, dealt with issues of social disadvantage, addressed the relationship of OFSTED to LEAs and caused the Secretary of State to take action on initial teacher education. However, in the furore which attended the publication, the limited and atypical nature of the sample has been forgotten. Much of the comment appeared to assume that the findings could be generalised to all London schools, all urban schools and even all maintained schools.

Reaction of government

The report has been publicly cited by the Secretary of State on a number of occasions. The DfEE Press Release of 7 May listed a series of actions that flowed from 'a highly critical report by OFSTED… which clearly identifies poor leadership and poor teaching as the main reasons why hundreds of children in these London Boroughs are being so badly let down.' The actions included the consideration of the granting of new powers to OFSTED 'to inspect any LEA - with or without the authority's co-operation' (DfEE, 1996).

Details of the OFSTED report

In the 45 inspections the team observed 358 lessons in years 2 and 6 in 15 schools in each of the three LEAs. These data were supplemented by the results of questionnaires completed by class and headteachers and by local authority officials. A reading test was administered by the NFER to 1,767 pupils in the same 45 schools (885 pupils in year 2 and 882 pupils in year 6). The test results were analysed by OFSTED researchers.

Aims of the OFSTED investigation

The aims of the investigation were to follow up in greater depth the findings which had emerged from earlier OFSTED work as well as from most academic studies of urban schooling (Mortimore and Blackstone, 1982; Pilling, 1990; OECD, 1995; and Smith and Noble, 1995) and to explore the possible reasons for underachievement. The letter inviting the three chief education officers to participate in the investigation also stressed the collaborative aims of the exercise:.'it would also give the authority a significant voice in what is said about its own schools' (OFSTED, 1995).

Sample

Using data from the annual London Reading Test of Year 6 pupils, OFSTED selected a sample to form a cross section of schools in the three boroughs. Those schools which were grant maintained, in the process of inspection or otherwise engaged in major change, together with any separate infant or junior schools, were excluded. OFSTED stated that a subsequent check of the sample, using local reading data, showed that 'in each LEA the overall reading performance of pupils in the 15 schools selected was almost exactly the same as pupils in the schools not selected' (p.66).

Methodology

During the autumn of 1995 inspection teams visited each school for one day. The 358 lessons involving reading were judged according to the usual inspection criteria published in the Handbook for Inspection of Schools (OFSTED, 1994). Testers from the NFER visited the schools to test individuals and to collect background information on pupils. The Neale Analysis of Reading Ability, including both subscales (Reading Ability and Reading Comprehension), was used. It is worth pointing out that the use of such a test is fraught with difficulty. First, the last reported revision of this test was in the late 1980s (prior to the implementation of the National Curriculum) so that its relevance to reading in the mid 1990s is debatable. Second, even if it were considered an appropriate test, given that the sample was selected to represent children from highly disadvantaged backgrounds, it can be no surprise that reading levels in the sample are lower than a national average (see the study undertaken for OFSTED by researchers at the Institute of Education: Sammons et al, 1994). Third, serious doubts recently have been voiced about the validity of the standardisation (TES, 1996d).1

The background variables that were collected included gender, date of birth, number of years of nursery, infant and primary education, time spent in education outside the UK, Special Educational Needs, ethnic background, stage of learning English (based on the former Inner London Education Authority scale of fluency), free school meals and the postal codes of the pupils included in the sample.

Analyses

A variety of analyses were carried out on the data. These included the calculation of means, medians and standard deviation scores for different groups and the sample as a whole, as well as more complex modelling which sought to take into account the effects of the background variables and the variation amongst the respective intakes of schools. Unfortunately, no standard statistical tests are quoted for any comparisons.

Data from the reading tests

As the report makes clear, there are a number of different ways of dealing with such statistical information. The report draws on the use of measures of central tendency (mean and median) and of dispersion (standard deviations and percentages of pupils one year above or below their chronological ages).

The scores for Year 2 (pupils of six plus) were:

The report argues that, for this age group, 'reading performance overall.... is well below the national average, judged against test norms'. Seventeen per cent of pupils (151) failed to gain any score on the test, indicating that they had not yet really started to learn to read. Eight per cent of the pupils were reading at about the age of 8 - well beyond their chronological age. The average score for reading comprehension was similar to that for reading accuracy (5 years 10months).

The scores for Year 6(pupils of 10 plus)were:

About 53 per cent of pupils were more than one year below their chronological age and 12 per cent were four years below their chronological age. Thirty nine per cent of the pupils had a reading age at or above their chronological age and 28 per cent were more than one year above it. The reading comprehension score for year 6 pupils was again very similar to that for reading accuracy (9 years 5 months).

Gender differences

In year 2, girls out-performed boys in both reading accuracy and comprehension but, in year 6, few differences were found.

Ethnic group differences

The sample was divided into four groups: White (787 pupils or 54 per cent of those for whom this information was available); Black Caribbean (170 pupils or 12 per cent); Black African (174 pupils or 12 per cent); and Bangladeshi (317 pupils or 22 per cent). 2At both ages, Black African pupils had the highest mean scores and the Bangladeshi pupils, the lowest. There was little difference between the White and the Black Caribbean groups. The Bangladeshi group, in particular, performed poorly in year 2 and was the equivalent of about six months in reading age behind the other groups. Remarkably, in year 6, this large gap in reading accuracy between the Bangladeshi group and the rest had been made up (though a difference remained in reading comprehension).

Stages of english fluency

Less than half (49.3 %) of the year 2 and just over half (54.5%) of the year 6 pupils had English as a first language3. Of those whose families spoke another language, 77 per cent of year 2 and 70 per cent of year 6 had been categorised as non-fluent (or not known). In one school, however, 'over 98 per cent of pupils' had English as a second language. Inexplicably, no detailed data are provided for the reading levels of children categorised by the Fluency of English Scale although it must have been available because comment is made that 'the "stage of English" scale sharply distinguishes reading levels at both year 2 and year 6' (para 142).

Social factors

Reading scores are given for the sample, according to whether or not pupils were in receipt of free meals. In year 2, those receiving free meals (at least 418) had reading ages about six months behind their peers and in year 6 (at least 412 pupils) this gap was nearer to 12 months. Overall, those having free meals made up 57 per cent of the sample (of those for whom this information was available)4. In a number of schools the figure was much higher: 'in a further three of the schools, over 80 per cent of the pupils were receiving free school meals' (para 6). The national average is 18.7 per cent (DfEE, 1996b).

Estimation of progress between year 2 and year 6

The report quite fairly states that an investigation of this nature is not able to provide an accurate measure of progress over time. This would require studying the same pupils over a four year period. OFSTED wished, however, to gain some estimate of progress. To do this the investigators had to make an assumption that the intakes to the schools had not varied very much over the last four years and that the two groups of pupils were likely to be fairly similar. This assumption, which buttresses the main conclusions of the report is, in fact, highly dubious as we shall argue later.

In terms of the test's standardised results, there was a one year gap between the average reading ages (in reading accuracy) and the chronological age of the year 2 pupils. The equivalent figure for the year 6 pupils was 1 year 3 months, implying a further falling behind. However, the report also points out that, looked at in another way, in terms of the proportion of pupils reading at or above their chronological ages, the data tell a rather positive story. This proportion was 21 per cent at year 2, but 39 per cent at year 6. 'Overall it would be reasonable to conclude that either the schools or some other factors are helping the pupils overall to make up some of the ground between the year 2 and year 6 against the national norms'. This seeming discrepancy in the data could be explained by the good progress of those who had been in the school for some years being offset by the lack of progress of an increasing proportion of new entrants but there are no data to test this.

What the inspection data showed

It is difficult for readers of the report to appreciate, in full, the details of the inspection data gathered in this investigation. A summary, drawing on the 358 lesson observations, is provided in the form of 13 separate points. A further 31 or so pages are devoted to a discussion of reading. This section, however, goes well beyond the reasonable inferences which could be drawn from this brief investigation based as it was on observations - on only one occasion - of a limited number of classes and presents a general OFSTED view of many matters to do with reading. The summary points make reference to the roles of the LEAs, the schools, the headteachers as well as to the classroom approach of teachers and, finally, to teachers' perceptions of the value of their own initial training for the teaching of reading.

The LEAs

The three references to the role of the LEA deal with the specialist support teachers and teaching assistants, including those funded under Section 11 ('well used in the majority of schools'); in-service training ('relatively few teachers had experienced systematic in-service training since appointment'); and the level of resources available to the schools - although this might also be considered a central government issue - ('a quarter of the schools were not sufficiently resourced to teach the national curriculum requirements for reading effectively').

The schools

According to the report, in half of the schools surveyed the reading curriculum did not match the requirements of the National Curriculum. The inspectors also found 'unevenness in the quality and amount of teaching of reading' within some of the schools. In two out of three schools the 'written policies and associated guidelines were weak'. Finally, however, positive comment is made about the encouragement by schools of parents to support reading and the use of parents and volunteer helpers to listen to pupils' reading.

The headteachers

Leadership by the head - as it influenced the teaching of reading - was considered to be 'ineffective in one in three of the schools'.

The teachers

The inspectors report that weaknesses in the teaching of reading were apparent 'in one in three of the lessons of year 2 and nearly half of the lessons in year 6'. The inspectors single out the teaching of phonic knowledge and skills as being insufficient and of poor quality. They also comment that the teaching of higher order and information skills in year 6 was, in the main, weak. In their view, however, 'the amount of time given to teaching reading was over-generous in some classes and was not well managed'. The inspectors considered that listening to individuals, in many cases, 'had become an unproductive routine exercise'. Interestingly, in view of the wide range of reading skills already identified, the inspectors comment that 'effective teaching of pupils in groups and especially as a whole class, was uncommon'.

Initial teacher education

The inspectors comment that 'many of the teachers were strongly critical of their initial training to teach reading'. In addition to the section on the factual comments from the inspection, a much longer section of the report deals with broader issues to do with reading. However, it is not clear how much of this discussion emerges from the investigation and how much from the general approach towards reading adopted by OFSTED.

The OFSTED interpretation of the data

OFSTED interprets the data to mean:

  1. 'The wide gulf in pupils' reading performance is serious and unacceptable' (para 10).
  2. 'Such practices could profitably be replaced by more effective group and whole class teaching' (para 14).
  3. 'When it was taught well, phonics contributed to the accuracy and fluency of reading by children of all abilities' (para 17).
  4. '…a limited number of schools emerged as more or less effective than others' (para 151).
  5. 'These results suggest that some schools were significantly more effective than others when differences in intake were, as far as possible, taken into account' (para 153).

In the next section we show that none of these interpretations can be clearly supported by the evidence that has been made available.

Shortcomings of the report

Analyses

The analyses that have been carried out, whilst more adventurous than those frequently used by OFSTED, are still not adequate. The difficulty about collecting purely cross-sectional data is that this technique allows no causal conclusions to be reached. It is now well established that a necessary condition for passing any kind of judgement concerning the relative effectiveness of schools requires longitudinal follow-up information on the same group of pupils. Instead, OFSTED have relied upon school-level measures (such as the year 2 average test results) and a limited range of pupil-level information (such as gender and eligibility for free school meals). The report persists in referring to its analyses as identifying effective schools but it cannot be emphasised too strongly that the analyses carried out are incapable of supporting any reasonable conclusions about school effectiveness. The use of multilevel modelling without appropriate intake data is irrelevant and, despite its claims, OFSTED did not use intake data. The conclusions 1 - 5 above are simply not supported by the evidence. It is highly likely that a number of the schools are achieving considerably greater progress than other schools in far less disadvantaged circumstances but the limitations of the research design prevent any clear conclusions.

Sample

The sample chosen from three of the poorest boroughs in the country is quite atypical of urban education as a whole. The fact that over half of the children were receiving free meals, that over half were not first speakers of English, that only 45 per cent were white and that pupil and staff mobility was high, cannot be ignored. To use this sample to draw generalisable lessons about the teaching of reading simply does not stand up to critical examination. Of course disadvantaged groups need to be studied. Indeed, much of our own work takes place in schools with such pupils. But in any social science investigation there has to be concern about the generalisability of the findings.

Clarity and transparency

The report does not provide a clear way to evaluate its validity. Despite numerous introductions, summaries and lists of points it provides only partial data from which - with considerable trouble - it is possible to calculate some of the relevant numbers. There is far too little detail of the systematic linkage between the inspection information and the reading data. This is surprising, given that much of the impact of the report was due to the ostensible combination of these two sets of information.

The report notes that checks were carried out on the relationship of the two sets of data. Few details are given, however, and the wording of the report is unhelpful. We have already pointed out that the data used in this report do not allow judgements to be made about the relative effectiveness of schools. The fact that the inspection judgements are consistent with the test scores, therefore, does not support the assertion that inspection judgements are fair estimates of such effectiveness. Thus, this report provides no evidence to validate inspectors' judgements in this respect.

Details of the sample

A further shortcoming is the absence of full data on the proportions of pupils from different ethnic groups speaking English as a second language or receiving free meals. This conceals the true picture of the population. Whilst it is possible to calculate the numbers and proportions of some of these groups (as has been done in this article) it would have been helpful to have provided a proper array of relevant data. Where inferences are being made about the efficacy of teaching techniques with certain groups of pupils, it is crucially important that heads and teachers are aware of the bases of such claims.

Information on pupil and staff turnover

An even more misleading omission is the failure to provide figures for the turnover of pupils and staff in the schools. This information was obviously available to the investigators because the report refers to 'a third to over a half of schools having higher pupil turnover than the average for their respective authorities' and one school having a staff turnover of 50 per cent in the previous year (para. 8). Given that inner London has much higher rates of turnover than the rest of the country it can be assumed that many of the schools had extremely high turnover figures. This information needs to be taken into account when judging the efficacy of reading. The year 2 and year 6 populations of these schools may well have had different social and other characteristics as a result of differential mobility in and out of these LEAs. This possibility makes any attempt to derive progress measures from a comparison of the two years pointless.

The study of urban schooling

Perhaps the most serious omission is the failure to address the critically important issues of urban schooling. If the investigation had seriously addressed the special challenges of teaching in inner urban areas, this sample would have been appropriate, although it was always going to be an extreme case. As it was, the investigation simply failed to consider the detailed circumstances of life in such schools. Given that OFSTED was working with local inspectors who knew the schools and the teachers involved, this represents a missed opportunity to make a significant contribution to our knowledge of how reading can be taught well, even in the most difficult of circumstances. The survey shows that some schools were highly successful, with numbers of pupils exceeding their chronological age with their reading capability. The investigation could have shed light on how this was being achieved. It could have detailed the reading schemes that were used. It could have given us rich data about how teachers coped with classes where half to three quarters of the pupils had joined during the year or where those with statements of special needs and/or language difficulties formed the majority.

Instead the report gives only generalised OFSTED remarks about the use of phonics and whole class methods. It represents a wasted opportunity. As an exercise, it fell between inspection and research and carried out neither task satisfactorily. No figures are given for the cost of the investigation but it could not have been cheap and it must represent extremely poor value for money. We are surprised that this could be considered 'the most important document that OFSTED has produced' (Woodhead, 1996a).

The tone of the report

Interestingly, some sections of the report do imply a sympathy for the conditions faced by the teachers and pupils and a respect for their achievements. There is a sensitivity to the limitations of the data and the reality of life in schools in some of the most disadvantaged areas in the country. In paragraph 76, for example, the report comments on the problems associated with 'the state of the buildings' which preoccupied some headteachers . But these sections are outweighed by the tone of the rest of the document which presses strongly held views about aspects of the teaching of reading (and of teacher training), regardless of the context in which the teaching is taking place or the nature of the data collected by the investigated team. A comprehensive analysis of factors associated with learning must surely take into account the wider context within which teaching and learning take place, and it is the failure to do this which makes this report such a poor example of its kind.

Given the high proportion of pupils whose families are disadvantaged, their reading outcomes cannot be surprising. We agree wholeheartedly with the OFSTED conclusion that reading is supremely important and that these pupils - even more than others - need fluent reading skills in order to improve their life chances. Using the data underpinning this report to blame teachers; to generalise to the teaching of reading nationally; to stress the need for whole class teaching and group work and a greater use of phonics (no matter how desirable this might be on other grounds) is quite unjustified.

In view of the national concern about reading standards, we note the comment in the report that 'a quarter of the schools were not sufficiently resourced to teach the National Curriculum requirements for reading effectively'(p.5). Furthermore, it is regrettable that central government has withdrawn support from the Reading Recovery Programme (noted in the report as one of the places where the effective teaching of phonics took place), after the detailed and highly positive evaluation it received.

Implications of the investigation for the use of research in policy formulation

There are a number of important lessons to be learned from this OFSTED exercise. We hope that our analysis, which has drawn attention to the weakness of the methodology and the inappropriateness of the interpretation, will help to clarify them. The lessons extend beyond the subject of reading to the roles of those working within the education system. First, a strong, independent OFSTED should concentrate on its fundamental task of inspecting schools. Second, those who fund and support research (including OFSTED) need to understand that good research has to be rigorous and that short cuts can lead to flawed information.

It is manifestly foolish for important policy decisions to be made on the basis of unsatisfactory evidence. Whilst 'cherry-picking' particular findings and ignoring the quality of the methodology which has led to them, may appeal to some policy makers, it is not a sensible manner in which to manage an education system. In contrast, using sound evidence produced by well-conducted research to help formulate policies is clearly prudent.

The best research is generally conducted by those having neither managerial nor political roles. It should be methodologically sound, long-term where appropriate, and independent.

Notes

  1. A paper presented at the 1996 Annual Conference of the British Educational Research Association by a part of the OFSTED team (Smith et al. 1996) also draws attention to the questions raised about the standardisation.
  2. Smith et al. (1996), in their paper provide a more detailed breakdown of ethnic groups. In their list, only 47.7 per cent of year 2 and 43.8 per cent of year 6 pupils are classified as white.
  3. Full data are not given but it is likely that those for whom this information was unavailable were not English first speakers.
  4. These data have been gleaned from a table combining ethnic groups and pupils on free meals. It is not known whether this represents those eligible for or actually receiving free meals. Missing data could be for either free meals or ethnic category information.

References

by Peter Mortimore and Harvey Goldstein

Feedback

Edit this page