Value added pilot tables
December 1998
The 1998 school performance tables contain results of a 'value added' pilot study using 30,000 pupils in 200 volunteer schools. For each school value added scores are presented together with a corresponding 'band' (A-E). The prior achievement score is an average over all KS3 scores for each pupil and the outcome is the average GCSE/GNVQ score. An overall value added result is presented together with separate ones for boys and girls and for low, middle and high achieving KS3 pupils.
The use of value added indicators for school improvement is a welcome development and schemes such as those in Hampshire and Lancashire and the ALIS scheme which retain the confidentiality of the results, so that they can be used by schools without the problems associated with public rankings, have much to commend them. The present attempt by government to move into this area, however, is highly problematic, for the following reasons.
First, this claims to be a pilot, yet there is full publication of results for individual schools with an invitation to make comparisons. As I will argue,the procedures used are flawed - something that a pilot project should be designed to detect, prior to publication. It seems irresponsible to publish these results before they have properly been evaluated.
Secondly, it is now accepted practice, when publishing value added scores (or bands) to include 'confidence intervals' so that instead of a single number a range of plausible values is given. The government is fully aware of this issue yet chooses not to even mention it. In so doing it is withholding crucial information from readers to the point of actually misleading people.
Thirdly, while it acknowledges that pupil mobility is an issue, it still publishes results for some very high mobilities and does not discuss seriously the biases that mobility can lead to.
Fourthly, the statistical procedures used in these calculations are inefficient, being based upon an over-simple 'regression' model rather than a more accurate 'multilevel' model; this may seem a technicality but does underline the naivete with which the exercise has been approached.
Finally, the government invites responses to this exercise, yet makes it clear that whether or not to publish value added tables is not negotiable. Yet publication is the key issue; it has important consequences for how schools react, and given the uncertainties surrounding any value added measures there must be considerable doubt about the usefulness of comparisons based upon published tables.
Harvey Goldstein