Value added data for schools
A commentary on a paper from SCAA
The consultative paper published July 14, 1997 from SCAA on 'value added indicators' provides another opportunity to debate the issues surrounding the publication of school league tables. This is reinforced by the Government White Paper on Education with its commitment to the continuing publication of 'raw' league tables.
In response to mounting criticism of the unfairness associated with the publication of 'raw' league tables of GCSE and A level exam results, the Government in 1994 asked SCAA to look at possibilities of properly contextualising these so that, at the very least, children's prior achievements could be allowed for when comparisons were made between schools. This was a welcome recognition of the complexities involved in comparing schools using children's' performance data. Thus the SCAA paper is especially timely in view of the present Government's commitment to the publication of simple 'raw' league tables.
The consultative paper consists of two virtually identical sections, one for primary and one for secondary, and these set out the background of the associated research project, a brief attempt to describe the value added notion itself, a summary of the project findings and a set of recommendations and topics for debate. The authors of the paper have gone to some lengths tosummarise a complex and sometimes quite technical debate, and to suggest howvalue added data might be used by schools for their own improvement purposes. SCAA hopes that the paper will stimulate discussion in schools and sensibly calls for more training of teachers in the use and interpretation of performance data. All this is highly commendable, but, sadly, the consultative paper has several important weaknesses which seriously undermine its usefulness. In the limited space available here I shall restrict my comments to the use of value added data for school improvement and comparison purposes, although the report also raises the issue of using them for screening individual children.
The main problem with this paper is that it fails to set out the full range of issues surrounding the construction and use of value added measures. It gives little hint that there is a considerable debate going on about these issues and in many places it oversimplifies matters and by so doing closes important areas of discussion. In particular it claims that, once prior attainment has been taken into account when comparing schools, other factors, such as social background have only a small additional effect and can be ignored. In fact, such factors, including ethnic origin, eligibility for free school meals, gender and type of school, should not be ignored since they can interact with prior attainment in important ways. One of the key findings of school effectiveness research is that schools are not uniformly 'effective' or 'ineffective' but differentially so for different groups of children, such as those from particular ethnic groups. More importantly, this research shows that many schools are differentially effective for initially low and initially high achieving children. In a recent value added analysis of A level exam results carried out by Sally Thomas and myself in conjunction with the DfEE, we showed that it was possible, for each school, to calculate separate value added scores for the top 10% of attainers at GCSE and the bottom10% of attainers at GCSE. When this was done there was only a moderate correlation (0.6) between these scores. In other words, schools were being differentially 'effective' according to GCSE attainment. Comparable results have been found in other data for both primary and secondary children. Such detailed information is very important for school improvement: it may be very misleading to report a single, overall, value added score which is effectively averaged over all the kinds of children attending a school. To be fair, the SCAA paper does recognise the need to report separately on different school subjects rather than in terms of an overall school 'effect', and a forthcoming book by Pam Sammons, Sally Thomas and Peter Mortimore presents new evidence on the importance of this. Of course, the same issues about differential effectiveness will apply to each subject.
Another failing of the paper, and the research project upon which it is based, is the use of weak statistical procedures. The diagram used to illustrate how value added scores are derived not only uses a limited and inefficient technique (simple regression analysis as opposed to multilevel modelling), but also assumes that there is a simple straight line relationship between key stage2 and key stage 1 test levels. In general such relationships are not straight lines. This is more than a mere technical quibble since the particular effect of this oversimplification will be to overestimate the predicted key stage 2 levels for some students and to` underestimate the levels for others (the above mentioned DfEE A level value added report discusses the implications of this in detail). Furthermore, the experience of several researchers working with LEAs and schools is that it is perfectly possible to explain complex relationships and procedures in ways that are understandable to teachers and parents.
The other issue given scant attention is that of the reliability of value added scores. The paper mentions that most schools cannot be separated in any ranking because of the 'sampling error' resulting from relatively small numbers of pupils. It recommends that value added scores be aggregated over three years to minimise this, but fails to point out that a large number of schools still cannot be ranked. The position is much worse when we look at how school performance changes over time, since time trends are estimated with far less reliability than single year scores. The report also fails to point out that value added data is out of date by the time it can be computed. Thus, for example, using GCSE scores obtained in 1998, adjusting for key stage 2 prior attainment for the same children in 1993 means that the results of the analysis are available for a cohort of children starting at the schools five years previously: schools change and it is a moot point whether such data can or should be used to judge a school's current performance. This difficulty is exacerbated if scores are averaged over a period of three or more years. This point, of course, and the problem of sampling error apply equally to 'raw' league tables as currently published.
Finally, there are a number of broader implications of this debate. The paper sensibly recommends that issues such as reliability and student mobility (which school should mobile children be assigned to?) should be sorted out before value-added league tables are published. To this list should be added the issues of improving the statistical procedures and taking account of the historical nature of performance data. Yet, in the case of unreliability, mobility and out-of-date ness the same caveats apply to 'raw' league tables - in addition tot he fact that they are completely uncontextualised. If the Government accepts SCAA's reservations about value added tables, then it ought logically to do so for the current tables. The recent white paper, however, not only endorses these without any serious reservation, it also proposes to introduce more in the form of so called 'improvement' tables whereby the trends in schools' performance over time will be compared. Unfortunately, in terms of reliability and out-of-dateness the evidence suggests that such tables will be even more suspect than current ones, in addition to the fact that they have no value added element.
An informed debate on all these issues is long overdue. It is clear that there are potential uses for value added analysis and several LEAs, in conjunction with schools, are beginning to implement value-added systems with a range of outcome measures as a means of reflecting upon performance. The SCAA paper is to be welcomed in providing a starting point for a debate to occur, but it will only be useful if the broader issues are also discussed. If Government is to realise its aim of raising standards then it needs to come to terms with some difficult and complex issues and should be careful to avoid any tendency to oversimplify the complex realities of children's performance.
Harvey Goldstein, 17 July 1997