Review of ‘Education and Social Justice'

• Back to Harvey Goldstein's commentaries >>

242pp Cardiff, Univ. of Wales Press, 2000. ISBN 0-7983-1619-0

(British Journal of Educational Studies, 49, 2001, pp 354-357)

The main aim of this book, based partly upon the author's own research, is to question the received wisdom of the 1990s that educational participation and performance have become increasingly polarised as a result of market reforms pursued during that period; the author seeks to disprove and even reverse some of these views.

In the first chapter the author sets his scene by quoting 'evidence' from various sources about how English and Welsh students perform poorly on international comparisons and how performance has deteriorated over time. He then claims that existing interpretations of data are flawed on the basis of three alternative interpretations. The first involves essentially a presentation of numerical comparisons of percentages on a multiplicative rather than arithmetic basis, what he calls 'proportionate analysis'. The second point he makes is that comparisons, especially those over time, must take into account any underlying population changes; sensible but hardly new. The third is that raw comparisons of performance are misleading and that 'value added' analyses are needed. Again, this is hardly new. The rest of the book applies these interpretations to data, and I shall consider each one in turn and whether Gorard's reanalysis does takes the debate forward.

Social segregation

Chapters 2 and 3 address the issue of whether social segregation between schools has increased. Gorard rightly points out the limitations of generalising from small-scale unrepresentative studies and introduces data that include information on social background such as free school meals entitlement. Interestingly Gorard never seems to address the much more important segregation issue with which many have been concerned, namely the issue of increasing achievement segregation. Irrespective of social background, the ability of schools to choose children can be expected to lead to increasing differences in achievement, given the emphasis on producing 'good' league table results. The evidence from school effectiveness research is that initial achievement is a far better predictor of final performance than social background, so that a 'rational' school would wish to select on that basis, even resisting efforts to enter 'desirable' schools by the socially advantaged who cannot also demonstrate high achievement. If such mechanisms are at work they may in fact lead to less social segregation, but that would not necessarily make them desirable. What, then, of his analyses?

The data used in these analyses come from administrative school level records and provide information such as the proportion eligible for free meals (a crude measure of social disadvantage), ethnic minorities and special educational needs, supplemented by area level social data. The real problem with this analysis, however, is a technical one. Gorard defines a 'segregation index' as follows.

Within a given area, such as an LEA, if there is no underlying difference between schools in the proportion of disadvantaged children, then the expected number of disadvantaged pupils in a school is simply obtained by applying the overall proportion for the area to the number of children in the school. The difference between the expected and observed numbers in each school is then calculated, the absolute value taken and then these are added together across the area and divided by (twice) the total number of disadvantaged pupils in the area. This index is used to compare LEAs and, crucially, to make comparisons over time; Gorard finds that across England between 1989 and 1997 there is actually a decrease in the value of this index and this is used to argue against the thesis of increasing social segregation. Unfortunately, the value of this index is a function of the size of school, and without controlling for this, differences in the index cannot be interpreted as measuring changes in segregation. This dependence on size becomes negligible as the underlying segregation increases, but in the case where the true segregation is small, as a result of sampling fluctuations this index will be (approximately) inversely proportional to the square root of the size of school. Even where there is no real underlying variation among schools, sampling variations generally will ensure that the index will be greater than zero (contrary to Gorard's claim) by an amount depending on the school sizes (for a set of primary schools of size 200, the expected value of the index for an average of 20% eligible for free meals and where there is no real underlying difference, is about 6%). Thus, for example, consider an LEA with no underlying differences between schools, apart from random fluctuations, measured at two time points. If the number of pupils remains constant but some schools close so that the average size of schools increases by 25%, then the segregation index will decrease by about 10%. In addition, even where the sizes of schools remains unchanged and there is no real difference among schools, an increase in the percentage of those eligible for free school meals will also tend to decrease the index, again approximately inversely proportional to the square root of the percentage. For at least some of the LEAs considered by Gorard, the actual observed values of the index may in fact be consistent (on the basis of a significance test) with no underlying differences among schools.

Over the period he considers, the average size of secondary schools increased by 18% which would tend to reduce the segregation index. Furthermore, amalgamations would also tend to reduce the index; Gorard does mention this possibility but dismisses it without empirical justification. This is not to say, of course, that in some sense segregation has not changed, simply that Gorard's failure to recognise and account for these measurement problems means that his results must remain an unsound basis for drawing conclusions.

International comparisons

Chapter 4 looks at comparative international studies of student achievement. Gorard makes some useful criticisms of some of the wilder claims made on behalf of these studies. He points out the problems of translation and curriculum coverage, although he fails to comment on the fact that these studies are almost all cross sectional which makes it extremely difficult to make any kind of causal inference about educational systems. He also fails to point out that in Mathematics and Science the overall scores mask some quite different sub test comparisons and that reliance on overall scores means that one accepts the actual balance of items (e.g. predominantly arithmetic in Maths) making up the total. Thus, although England does not appear to do well overall it does do very well in problem solving. Gorard's criticisms of school effectiveness research suffer from the same superficiality and apparent unfamiliarity with much of the research literature. Nevertheless, he does repeat some useful criticisms of attempts to make absolute comparisons over time.

In Chapter 5 Gorard compares examination results in England and Wales. He uses data for LEAs and adjusts for LEA level social factors including free school meals. He concludes that there is no evidence for Wales performing worse than England. Unfortunately, again there are technical problems with this analysis. First of all it uses aggregate level data and thus is open to what is known as the 'ecological fallacy' whereby quite different relationships can occur for data fitted to aggregate level units and data fitted to individual pupils. Secondly, the school effectiveness literature has demonstrated quite clearly that the most important variable to adjust for when comparing 'output' performance is intake achievement, social factors simply do not make an adequate adjustment. Gorard seems not to recognise these difficulties so that once more his conclusions are unsound.

Chapter 5 reverts to the 'proportionate analysis' comparisons over time and seeks to reverse existing interpretations, claiming that in fact there is convergence between schools in terms of examination results. For the reasons explained earlier all of this is debateable and it is difficult to rely upon such analyses. An appropriate analysis of 'convergence' would require, at the very least, longitudinal 'value added' data on cohorts of pupils in schools.

Chapter 7 suffers from the same drawbacks as previous chapters in using the segregation ratio, failing to take account of prior achievement when comparing schools and carrying out an aggregate level analysis using schools as units.

Chapter 8 uses 'proportionate analysis' to revise conclusions of others. Here, as elsewhere, however, Gorard does make some useful criticisms of other people's simplistic interpretations. In Chapter 9 Gorard makes the important point in comparing boys and girls over time, that entry changes need to be taken into account. He again uses a 'proportionate analysis' and simply claims that this is superior to other methods of presentation and on this basis comes to different conclusions from other researchers.

In his final Chapter Gorard makes several claims on the basis of his analyses; that social segregation is decreasing, that international differences are exaggerated, that there are no clear differences between types or sectors of schooling and that differences between different types of pupils are decreasing over time. Unfortunately, none of these conclusions can be relied upon to follow from Gorard's analyses.

Interestingly, he makes an attack on 'qualitative' researchers who are dismissive of quantitative research, and I have some sympathy with that. But he then goes on to dismiss 'complex methods of analysis' in favour of his own 'middle way'. The trouble is that Gorard's analyses are over-simple, use insecure statistical techniques and actually demonstrate the real dangers of over-simplification. The processes being studied are genuinely complex and only by developing techniques that seek to match that level of complexity can we hope to advance quantitative contributions to knowledge (Goldstein, 1998). Far from providing a step in the right direction this book provides a sad example of how not to approach the analysis of complex educational data.

Reference

Goldstein, H. (1998). Models for reality (PDF, 78kB). London, Institute of Education.