Performance related pay - assumptions and problems

• Back to Harvey Goldstein's commentaries >>

Introduction

In September 2000 new arrangements for monitoring the performance of teachers come into force. Following consultation and considerable controversy and opposition from teachers’ representatives, the DfEE issued a full description of how it is planned to operate this system.

The following notes are not intended to be a general critique of the new arrangements; rather they look at the way in which the various documents describe how teachers are meant to use measures of pupil progress to support annual judgments of their performance – the performance management framework, or for an application to meet the threshold standards for promotion to a higher pay scale.

The government documents place a considerable emphasis upon pupil progress as a means of judging the performance of teachers. While the documents deny that the new proposals are a ‘return to the Victorian system of payment by results’ and that pupil progress is only one of the performance ‘objectives’, the documents themselves continually stress this particular objective, or related ones to do with attendance or behaviour. The documents also explicitly link teachers’ achievements of progress targets with the targets that have already been set at school level, for example in terms of percentages of students reaching key stage test levels at various ages. There appears to be some confusion in this respect since while the Performance management documents state that it really is progress that is to be assessed, taking into account pupils’ prior achievement, the school level targets generally have not been set on that basis. This confusion is also evident in the documents themselves; in examples of performance management objectives ( Ref: DFEE 0051/2000, Performance management framework) in the 5examples given reference is made to pupil progress in each case, but in only one of these is there even a hint that prior achievement is to be taken into account – the others refer to judging the ‘absolute’achievements of a class or group.

On the other hand, the documents on Threshold Assessment are clearer and do explicitly refer to ‘value added’assessment, although as I will argue below, betray a lack of understanding about what such assessments can be used for.

Pupil progress measurement – a brief review.

A more detailed description of the possibilities and limitations of using test scores to make judgments about schools or classes (PDF, 21kB)

There are two key issues. The first is the need to make careful adjustments for prior achievement – providing ‘value added’scores, and the second is to take account of the fact that when any score fora school or class or teacher is quoted an ‘uncertainty interval’ is included. These uncertainty intervals are typically very wide and this means that most schools or classes cannot be separated from the average or from each other. A research report commissioned by OFSTED), also demonstrates that commonly used ‘proxy’ adjustments such as ‘free school meal entitlement’ are inadequate substitutes for a full value added analysis.

Proposals for the use of pupil progress measures

a) Annual performance review

I have already mentioned the lack of clarity about the definition of pupil progress. Nevertheless, the documentation (Annex C on setting objectives) does say that ‘what is important is that the planning discussions are based on an understanding of pupils’ prior attainment’.Nowhere, however, is there a discussion of just how this is to be done, or its severe limitations. In particular, for any one teacher the prior attainment most relevant to pupil achievement at the end of a school year is that at the start of the same year. The problem is that there are few data available for measuring progress over a single year – most of the value added studies are over longer periods between key stage testing times. Without LEA or national norms it is difficult to see how any sensible judgements can be made – leaving aside the problem of the very wide uncertainty intervals that one would expect for such yearly comparisons.

If sound value added judgments cannot be made then presumably teachers will be assessed on ‘absolute’ achievement levels –and it is therefore perhaps no surprise that the examples given are mainly of this nature. The unfairness of this is now well understood and it would be especially unfortunate if the current requirements for performance reviews were to reward teachers, at least partly, on the basis of the selective nature of the classes they teach.

Finally, throughout the documents there is the implicit assumption that teachers and schools alone are what influence the achievements of their pupils. Yet, not only is prior attainment important, so is the education at any previous schools attended, mobility, special needs etc. Moreover, especially in secondary schools, it is very difficult to ascribe the progress of any one pupil in a given subject to the teacher of that subject. Pupil progress will be affected by other teachers and features of schooling– perhaps to an even greater extent – and the attempt to associate progress for a pupil with a single teacher is not only divisive, it is also likely to be misleading.

b) Threshold applications

In applying for threshold promotion, teachers ‘should show that as a result of their teaching their pupils achieve well in relation to their prior attainment, making progress as good or better than (sic!)similar pupils nationally’. Teachers are expected to use national test data‘where appropriate’. When this is not the case, and as pointed out above this will commonly be the case, it is stated that alternative valid methods for showing progress are ‘examination marks, general coursework marks etc. Quite how these are supposed to show progress, when they generally will not take into account prior achievement, is not stated.

In discussing the evaluation of progress several sources of evidence are referred to. One of these is the PANDA reports, in particular where these attempt to make adjustments on the basis of free school meal entitlement. I have already referred to the inadequacy of this kind of adjustment. Another reference is to the various schemes, such as ALIS, QUASE etc which make comparisons between schools and departments after adjusting for prior achievement. As already pointed out, these will typically not refer to the relevant period, such as a school year, are only available for a minority of schools, and it is difficult to see how they could be used to make a judgement about any individual teacher. There is reference to optional QCA tests that produce scores every year, at least for primary children, that could be used. These potentially would be the most useful information, but there is no recognition of the limitations of such information. The documents refer frequently to the ‘expected acceptable range’ of progress but nowhere define this. As I have already pointed out, the uncertainty surrounding any measures, especially when based upon one class for one teacher is very large and a proper contextualisation of any measure would need to involve all the other influences that a child is subject to, from the school as a whole, other teachers and external factors.

In short, in terms of pupil progress, it seems that teachers are being asked to provide evidence that, in many if not most cases,they simply will be unable to do in any objective fashion. A teacher may indeed have provided conditions in which pupils have made outstanding progress, but through no fault of their own be quite unable to demonstrate that fact.

More generally

The regulations for performance management and threshold assessment betray a sad lack of understanding about the appropriateness of using pupil test score data to make judgments about teachers. While there is,in places, a recognition of some of the limitations of such test score data,the overall impression is one of a belief in the ‘objectivity and reliability’ of the measures proposed. For a government which aspires to promoting evidence based policy, these regulations demonstrate a remarkable lapse of judgement and an unawareness of what the evidence actually shows.

Finally, however, a welcome comment from Michael Barber, head of the Standards and Effectiveness Unit at the DfEE:

'If the government continues to ignore the profession,as it seems intent on doing in relation to the pay structure and performance related pay and to continue its attacks on initial teacher education, in spite of the outrage they have generated …'

(NUT Education Review, 7,2, pages 4-7, 1993)