Centre for Multilevel Modelling

menu

A Fresh Start – comments on a DfEE report on adult literacy

(British Educational Research Journal, 26, 2000, pp 547-550)

This collection is a series of discussions about the effect of OFSTED inspections on education. It arises from a widespread concern with inspections and their place within a continuing programme of politically driven educational change - a concern which also led to the setting up of a Parliamentary Select Committee enquiry whose report was published in Summer 1999 (see below).

The 12 chapters are contributions from researchers interested in the inspection process and deal mainly with schools in England, with one contribution from Wales, one from higher education initial training and one from the further education sector. In addition to discussing these contributions I will look at a critique of the volume by HMCI Chris Woodhead which appeared in The Guardian newspaper (5 October, 1999).

While this volume is ostensibly about OFSTED it comes at a time when there is a considerable debate in education, and within the Social Sciences more generally, about the acquisition and use of research evidence to inform policy. This context is important because the volume claims to provide and interpret research evidence in an important area of policy, so that in the wider debate about evidence it may come to be seen as a model to follow. Whether it should provide such a model is an underlying theme of this review, which I address both implicitly and explicitly.

In his introduction, Cullingford stresses the difficulty of evaluating OFSTED's work and the importance of well-constructed empirical research. Other contributors echo this, but sadly, with one or two exceptions, fall short of this aspiration. The first chapter by Kogan and Maden briefly describes much of current school improvement research and also provides an account of the inspection system. They summarise and discuss the results of a questionnaire survey, supplemented by interviews, and describe how OFSTED inspections affected normal school activities and existing management structures as well as people's perceptions of the effects of inspections on 'standards'. The authors go into some detail about true inspection costs. They raise several issues also made by other contributors, for example on the need for the views of those being inspected to be incorporated into reports and argue in favour of school 'self evaluation'. Nevertheless, there are difficulties with this contribution. To claim, as they do, that "OFSTED has made strenuous efforts to monitor its own reliability", when in fact it has only (to date) carried out one small, poorly controlled, study (see select committee report for a description and discussion) hardly inspires confidence. More seriously, the authors provide no details of how their study was carried out, for example how the interviews were conducted, what the survey response rate was, etc. Results are presented as 'facts' with no indication of their reliability or representativeness and no reservations expressed about the validity of their data. In fact the full report of the study (CEPPP, 1999) does give some details: the response rate to the main postal questionnaire was 22% for governors and parents and 33% for heads. These rates are extremely low and would generally be felt to cast considerable doubt upon the validity of any findings. Moreover, although some basic data about the sample are given in the full report, there appears to have been no serious attempt to check whether the sample can be considered to be representative. When authors omit such caveats from any serious discussion of results it weakens their whole argument.

In the following chapter Winkley characterises a classroom lesson as a 'performance', for example of a play, with pupils as audience and OFSTED inspectors as critics reporting on it. Using this analogy Winkley makes several observations about the importance of the teaching 'context', the lack of opportunity to contest the 'critic's' judgement and the (partly) subjective nature of critical judgement. Pursuing the analogy he makes assertions about the level of disagreement (reliability) to be expected among inspectors. All of this, however, is speculative and rather a long way from having a secure evidential basis. Winkley goes on to discuss the results of a survey of recently inspected primary schools and their views about the inspections. He notes, for example, failures to take account of particular kinds of achievement in addition to Key Stage test results and discusses the stress induced by the inspection process. Like Kogan and Maden he argues for the school's views of inspection to become incorporated into reports. He attempts to set OFSTED within the wider contemporary political scene and sets out his views about how a better inspection system might be constructed, using a school improvement model.

Interesting as much of this chapter is, it is let down by weak logic and poor research methodology. Thus, for example, Winkley makes statements about the lack of evidence for beneficial effects of inspection with a clear invitation to the reader to interpret this as an argument against the existence of such benefits. In another example he writes about 'schools with serious weaknesses' needing special treatments – yet gives no indication of how he defines 'serious weaknesses': yet the labelling of such schools by OFSTED is contentious, as many other contributors point out. Like Kogan and Maden he makes reference to the results of a survey, but again with an absence of detailed information that would allow the reader to evaluate his evidence – and also with no indication where the reader might go to see such details.

The merit of the chapter by Cullingford and Daniels is that it does attempt to obtain empirical evidence about the relationship between OFSTED inspections and changes in educational 'standards'. The authors describe a survey of exam results, which they relate to inspections, when and where these occurred. Details of their survey are set out. Their analyses show that the percentage of A* to C grades at GCSE for a school decreases the closer is the inspection to the time of the examination. The authors recognise some of the pitfalls in interpreting such data, for example that 'weaker' schools were chosen closer to the exam period, and go on to dismiss these possibilities, but they provide no empirical data that would justify such dismissal. More importantly, the use of GCSE results to measure 'standards' is naïve. At the very least a 'value added' definition is required, adjusting for prior achievement, but nowhere is this recognised. This study, therefore, provides poor ammunition against OFSTED.

Grubb provides a US view of OFSTED, based upon observations in a sample of schools and conversations with heads and inspectors. He discusses ways in which the establishment of OFSTED led to the bureaucratisation of inspection, with the introduction of criteria and formal databases. The chapter is characterised by attempts to make generalisations based upon his own experiences and this makes it much more an exercise in educational journalism than serious research. As such, however, it does provide some insights; the waste of information involved in summarisation; the failure to match inspectors' judgements to those of teachers and the conflicts endured by the inspectors themselves. He contrasts the inspection system in further education (FE) with that in schools. He comments favourably on the former's emphasis on self evaluation and avoidance of some of the stress associated with OFSTED, and makes some suggestions for how others might learn from the English experience.

FitzGibbon and Stephenson-Forster consider the nature of any claim by OFSTED to objectivity and report on a survey designed to monitor OFSTED's impact. They look at the inspection process in terms of whether the information provided is 'representative' of what happens in schools, whether it can achieve reasonable reliability, and whether it approaches acceptable standards of validity. In their survey they found strong support for the principle of inspection, they also found a general reluctance among heads to believe in the validity of OFSTED judgements and also to see problems with reliability. The authors present a limited analysis of the relationship between a crude measure of poverty (based on free school meals) amount spent on pre-inspection preparation and the inspection outcome as perceived by headteachers . Limited as this study is, it is well documented and is a careful attempt to provide relevant empirical evidence about OFSTED's perceived impact.

Alexander's chapter is based upon his evidence to the Parliamentary select committee. He sets out to question OFSTED's claim to independence and integrity and argues that it should be much more concerned with critiquing government policy, based on professional judgements, rather than acting largely to implement that policy. The main difficulty with this chapter is that it consists largely of brief statements of personal views and presents little new empirical evidence beyond some anecdotes about individual OFSTED judgements. Alexander is very clear in his attitude to Woodhead, accusing him of abusing his position, not by speaking out, but by doing so irresponsibly and with more regard for ideology than evidence.

Thomas traces the history of the creation of OFSTED and looks at a survey of inspection in Wales that has its own inspection system. Schools were asked about the factors they felt affected their performance and inspection appeared as less important than, for example, INSET or within-school monitoring. Schools, however, had quite different views about the factors affecting performance of other schools than their own. Without any more objective evaluation, and coupled with a relatively low (64%) response rate to the survey, it seems difficult to draw clear conclusions from these findings.

Law and Glover look at about 30 schools inspected within one Shire County. They point to a relationship between inspection judgements and social background factors and discuss the difficulty of properly taking account of these when making judgements about schools. They look at changes in a small number of their schools that were reinspected and found that those with more social problems made less progress between inspections. They argue that such schools will tend to be those that more often fail and have less chance of being judged as 'improving'. Despite being based upon a limited sample of schools and not using a proper value added analysis, this chapter does offer some potentially useful insights into the important issue of relationships between OFSTED judgements and social background.

In the next two chapters Hustler looks at the lay (non-educational-professional) inspector within inspection teams and suggests that these inspectors are increasingly becoming 'socialised' into the system and Cuckle and Broadhead describe a survey of headteachers ' views about OFSTED inspections. Hustler's study consists of a series of interviews, but we are given no details of how representative the sample is and the presentation consists of vague statements with little quantification. Cuckle and Broadhead conclude that heads value inspections, especially when schools are able to respect the quality of the inspection team. Again, no sample details such as response rates are reported for the survey so that it is also difficult to judge the quality of this contribution.

Griffith and Jacklin discuss the OFSTED inspection of the University of Sussex primary PGCE course, which eventually led to the withdrawal of the University from Initial Teacher Education (ITE). They present this as a case study and invite other PGCE institutions to share their experiences. They make several points:

  1. That changing requirements and inspection criteria for ITE have made it extremely difficult for institutions to respond in the time required.
  2. Inspectors are often inexperienced and confused.
  3. There is often a serious lack of communication between OFSTED and the Teacher Training Agency (TTA).

They offer suggestions for improvement, especially changing from a 'punitive' to a 'supportive' culture, and in this echo much of the debate about OFSTED inspections in schools.

In a concluding chapter Cullingford describes 'A modest proposal' for a new inspection system based around the notion of an 'undercover state inspector' in every school monitoring everything that happens. The satire is spoilt, unfortunately, by stopping (or does it?) part way through, to be replaced by bold assertions such as "OFSTED lowers standards" and "we also know that standards of achievement are not rising".

Chris Woodhead is the key figure in contemporary debates about OFSTED, although Grubb's view that he has 'single-handedly done considerable damage to the process (of inspection)' seems something of an overstatement. Nevertheless, it is important to try to understand Woodhead's role. There is in this volume, unfortunately, no serious attempt to do this, although Alexander does make some observations about it. Nevertheless, Woodhead's critique of this volume (Guardian, 5 October 1999) is quite revealing.

In his critique Woodhead selects particular evidence or misquotes contributors in order to defend OFSTED. For example, he accuses Grubb of 'dismissing out of hand' the possibility that poor teaching can be a result of personal incompetence. In fact Grubb does nothing of the sort as a careful reading shows. Has Woodhead really absorbed Grubbs chapter and then deliberately misquoted him, or has he simply superficially skimmed it and not understood what is being said? Woodhead is often contemptuous of research; in the critique he uses the phrase 'the treacle of academic reference'. On several occasions in the recent past he has also demonstrated his ignorance of educational research. Thus, for example, as Alexander points out, Woodhead's introduction to Tooley and Darby's (1998) review of four education research journals draws quite unwarranted conclusions from that same report. Furthermore, Woodhead shows little inclination to learn about research and its methods. It is such attitudes, rather than any particular opinions he may hold, which in the longer term may have more serious consequences, since they make it difficult for policy to be mediated by evidence. Even though all within OFSTED do not share such attitudes those other voices are generally not heard.

There are certainly many deficiencies in OFSTED and in its leadership, and these need well founded critiques that the present volume, on the whole, fails to provide. For those readers who wish to become well informed about OFSTED a much better source is the Select Committee report, and the DfEE and OFSTED responses to it are also interesting.

References

Harvey Goldstein. Institute of Education, January 2000

Feedback

Edit this page