Centre for Multilevel Modelling

menu

How should we measure school performance and hold schools accountable? A study of competing statistical methods and how they compare to Progress 8

ESRC standard grant

  • Grant Code: ES/R010285/1
  • Duration: 01/10/2018 - 30/09/2021
  • Amount Awarded: £467,000

People

Summary

In 2016, the Department for Education radically overhauled their secondary school accountability system and introduced 'Progress 8', arguing it to be the simplest and fairest school performance measure to date. Progress 8 aims to quantify and communicate the average academic value each school adds to their pupils' learning. Specifically, Progress 8 measures how much higher each school's pupils score in their age 16 GCSE examinations than expected given their age 11 KS2 test cores when they started secondary schooling. Progress 8 scores are used to hold schools to account, with the lowest scoring schools judged 'underperforming' and 'coasting'; classifications that trigger intense scrutiny and intervention from the school inspectorate, Ofsted. Given the high-stakes involved, research is urgently needed to first evaluate the strengths and weaknesses of the statistical method underlying Progress 8, and second, to explore the potential benefits of alternative methods for measuring school performance. Our proposed research will address these needs and in doing so will advance scientific understanding about school performance measurement.
 
See the ESRC page ES/R010285/1 for more details.

Videos

The following video is of the Northern Powerhouse Partnership giving oral evidence to MPs on the Education Select Committee about our recent Adjusted Progress 8 research on 05/06/2019. We previously gave evidence to this committee on this same topic at a private meeting with them in March 2019. Video.
 
The following video is a research presentation of the working paper below given to mixed audience of academics and non-academics at UCL Institute of Education 0n 14/11/2018:

Journal articles

  • French, R., Sariaslan, A., Larsson, H., Kneale, D. & Leckie, G. Estimating the importance of families in modeling educational achievement using linked Swedish administrative data. Journal of Research on Educational Effectiveness. Forthcoming. Article. Pre-print.
  • Gallo Cordoba, B., Leckie, G., Browne, W. J. Decomposing ethnic achievement gaps across multiple levels of analysis and for multiple ethnic groups: A mediation approach. Sociological Methodology? Forthcoming. Article. Pre-print.
  • Goldstein, H., Haynes, M., Leckie, G., Tran, P. (2020). Estimating reliability statistics and measurement error variances using instrumental variables with longitudinal data. Longitudinal and Life Course Studies, 11, 289-306. Article.
  • Leckie, G., Browne, W., Goldstein, H., Merlo, J., Austin, P. (2019). Variance partitioning in multilevel models for count data. Psychological Methods, 25(6), 787–801. Article. Pre-print. Slides.
  • Leckie, G. and Goldstein, H. (2019). The importance of adjusting for pupil background in school value-added models: A study of Progress 8 and school accountability in England. British Educational Research Journal. Forthcoming. Article. Video. Slides.
  • Leckie, G. and Prior, L. A Comparison of Value-Added Models for School Accountability. School Effectiveness and School Improvement. School Effectiveness and School Improvement. Article. Pre-print.
  • Prior, L., Goldstein, H., & Leckie, G. (2021). School value-added models for multivariate academic and non-academic outcomes: A more rounded approach to using student data to inform school accountability. School Effectiveness and Improvement. Article.
  • Prior, L. and Leckie, G. (2022). Student mobility: extent, impacts, and predictors of a range of movement types for secondary school students in England. British Educational Research Journal. Article
  • Prior, L., Jerrim, J., Thomson, D., & Leckie, G. (2021). A review and evaluation of secondary school accountability in England: Statistical strengths, weaknesses, and challenges for 'Progress 8' raised by COVID-19. Review of Education. Article.‌

Working papers now published as journal articles (see above)

  • Leckie, G., Browne, W., Goldstein, H., Merlo, J., Austin, P. (2019). Variance partitioning in multilevel models for count data. arXiv:1911.06888. Article.
  • Leckie, G., Goldstein, H. (2018). Should we adjust for pupil background in school value-added models? A study of Progress 8 and school accountability in England. arXiv:1811.09240. Article.
  • Leckie, G., and Prior, L. (2021). A Comparison of Value-Added Models for School Accountability: arXiv: 2107.09410 [stat AP]. Article.
  • Leckie, G., and Prior, L. (2021). A Comparison of Value-Added Models for School Accountability. Bristol Working Paper in Education Series. Working Paper, 01/2022. Article.
  • Prior, L., Goldstein, H., & Leckie, G. (2020). School value-added models for multivariate academic and non-academic outcomes: A more rounded approach to using student data to inform school accountability. arXiv:2001.01996 [stat AP]. Article.
  • Prior, L., Jerrim, J., Thomson, D., Leckie, G. (2021). A review and evaluation of secondary school accountability in England: Statistical strengths, weaknesses, and challenges for 'Progress 8'. arXiv:2104.06299 [stat.AP]. Article.
  • Prior, L., Jerrim, J., Thomson, D., Leckie, G. (2021). A review and evaluation of secondary school accountability in England: Statistical strengths, weaknesses, and challenges for 'Progress 8'. Bristol Working Papers in Education Series. Working Paper, 01/21. Article.
  • Prior, L., Jerrim, J., Thomson, D., Leckie, G. (2021). A review and evaluation of secondary school accountability in England: Statistical strengths, weaknesses, and challenges for 'Progress 8'. (CEPEO Working Paper No. 21-04). Centre for Education Policy and Equalising Opportunities, UCL. https://EconPapers.repec.org/RePEc:ucl:cepeow:21-04. Article.
  • Prior, L., Jerrim, J., Thomson, D., Leckie, G. (2021). A review and evaluation of secondary school accountability in England: Statistical strengths, weaknesses, and challenges for 'Progress 8'. UCL IOE Quantitative Social Science Working Paper No. 21-12, April 2021. Article.

Working papers

  • Prior, L., Evans, C., Merlo, J., Leckie, G. (2022). Sociodemographic inequalities in student achievement: An intersectional multilevel analysis of individual heterogeneity and discriminatory accuracy (MAIHDA) with application to students in London, England. Bristol Working Papers in Education Series. Working Paper, 06/2022. Article.
  • Prior, L., Evans, C., Merlo, J., Leckie, G. (2022). Sociodemographic inequalities in student achievement: An intersectional multilevel analysis of individual heterogeneity and discriminatory accuracy (MAIHDA) with application to students in London, England. arXiv: 2211.06321 [stat.AP]. Article.
  • Leckie, G., Parker, R., Goldstein, H., Tilling, K. (2021). Multilevel models with random residual variances for joint modelling school value-added effects on the mean and variance of student achievement. arXiv: 2110.02079 [stat AP]. Article.
  • Goldstein, H., Leckie, G., Prior, L. (2020). Providing educational accountability for Local Authorities based upon sampling pupils within schools: moving away from simplistic school league tables. arXiv:2002.09897 [stat AP]. Article.
  • Leckie, G., Prior, L., Goldstein, H. (2019). The implications of Labour’s plan to scrap Key Stage 2 tests for Progress 8 and secondary school accountability in England. arXiv: 1911.06884 [stat AP]. Article.

Work in progress

  • Prior, Evans and Leckie (2022). Sociodemographic inequalities in student achievement: An intersectional multilevel analysis of individual heterogeneity and discriminatory accuracy (MAIHDA). Sociological Methods and Research, submitted.

Representative articles

The following articles are representative of the style of work we are aiming to do:
  • Leckie, G. (2018). Avoiding bias when estimating the consistency and stability of value-added school effects using multilevel models. Journal of Educational and Behavioral Statistics. Forthcoming. DOI: 10.3102/1076998618755351. Article. Slides.
  • Leckie, G., & Goldstein, H. (2017). The evolution of school league tables in England 1992-2016: ‘contextual value-added’, ‘expected progress’ and ‘progress 8’. British Educational Research Journal, 43, 193-212. DOI: 10.1002/berj.3264. Article. Video. Slides.
  • Leckie, G., & Goldstein, H. (2011). Understanding uncertainty in school league tables. Fiscal Studies, 32, 207-224. DOI: 10.1111/j.1475-5890.2011.00133.x. Article. Visualization.
  • Leckie, G., & Goldstein, H. (2009). The limitations of using school league tables to inform school choice. Journal of the Royal Statistical Society: Series A (Statistics in Society), 172, 835-851. DOI: 10.1111/j.1467-985X.2009.00597.x. Article.

Keynote conference presentations:

  • April 2022. International Multilevel Modelling conference, Utrecht.
  • June 2020. Santiago. Cancelled due to COVID-19.
  • August 2018. '25 years of school leagues tables, accountability and choice: Lessons from England'. Educational Effectiveness EARLI SIG 18 & 23 conference. Groningen. Flyer. Slides.

Other presentations:

  • September 2022. 'School value-added models for multivariate acadmemic and non-academic outcomes in school accountability systems' and 'Student mobility: extent, impacts and predictors of a range of movement types'. EARLI SIG 18, Freiburg.
  • April 2022. 'Incorporating student mobility in studies of school improvement: Comparing two cross-classified multiple membership models', Heck, R., Reid, T., and Leckie, G., 2022 AERA Annual Meeting, San Diego.
  • September 2021. 'Pupil mobility: the extent, impacts and predictors of moving school'. Society for Longitudinal and Lifecourse Studies (SLLS). Online.
  • September 2021. 'School value-added models for multivariate academic and non-academic outcomes: Exploring implications for performance monitoring and accountability'. British Educational Researcch Association (BERA). Online.
  • June 2021. Review of Education article summary.
  • March 2021. Northern Powerhouse Partnership. Zoom.
  • June 2020. Lewisham. Cancelled due to COVID-19.
  • March 2020. Manchester. Cancelled due to COVID-19.
  • February 2020. 'Secondary school performance measures and league tables in England: An overview of the official measures and their statistical modelling, data, policy, and interpretation issues'. York.
  • January 2020. 'School value-added models of academic and non-academic student outcomes: A more rounded approach to using student data to inform school accountability with application to attainment, absences and exclusions in England'. NPD User Group Meeting, London. 
  • January 2020. 'Secondary school performance measures and league tables in England: An overview of the official measures and their statistical modelling, data, policy, and interpretation issues'. Wolverhampton.
  • September 2019. 'School accountability in crisis - holding the judges to account'. British Educational Research Association (BERA) . Manchester.
  • April 2019. ‘Progress 8’ National Education Union annual conference, Liverpool.
  • April 2019. 'Calculating intraclass correlation coefficients in multilevel models for count responses'. 12th International Amsterdam Multilevel Conference, Utrecht.
  • November 2018. ‘Accountability and school differential effects’. Department of Learning and Leadership, IOE UCL, London. Flyer. Article. Slides. Video.
  • September 2018. Bristol Cathedral School, Bristol. Article. Slides.
  • September 2018. 'Avoiding bias when estimating the consistency and stability of value-added school effects'. RSS Annual Conference, University of Cardiff. Article. Slides.
  • May 2018. 'Avoiding bias when estimating the consistency and stability of value-added school effects'. Q-step Seminar Series, University of Exeter. Article. Slides.
  • April 2018. 'Avoiding bias when estimating the consistency and stability of value-added school effects'. 2018 AERA Annual Meeting, New York. Article. Slides.
  • July 2017. ‘Modelling heterogeneous variance-covariance components in two-level models'. JEBS: Methods for Eductional and Behavioral Applications session at Joint Statistical Meetings (JSM), Baltimore. Article. Slides.
  • June 2017. Data Intensive Research Workshop, Cardiff University. Slides.

Book chapters and Encyclopedia entries

  • Leckie, G. (2021). Cross-classified multilevel models, Encyclopedia of Quality of Life and Well-Being Research. Second edition.
  • Leckie, G. (2019). Multilevel models for continuous outcomes, SAGE Research Methods Foundations series.
  • Leckie, G. (2021). Multiple membership multilevel models, Encyclopedia of Quality of Life and Well-Being Research. Second edition.

Teaching presentations:

  • 'The importance of adjusting for pupil background in school value-added models: A study of Progress 8 and school accountability in England'. M-level: Statistics in Education. University of Bristol. December 2018.
  • 'Introductory overview of quantitative methods and basic analysis approaches'. EdD: Conducting Educational Research. University of Bristol. June 2018. 

Multilevel modelling short courses:

The following short courses use our school league table research as teaching examples:
  • December 2018, 2019, 2020, 2022, Introduction to Multilevel Modelling Using MLwiN, Royal Statistical Society, London
  • Jan and July 2019, 2020, Jan 2021, July 2021 Introduction to Multilevel Modelling Using MLwiN, Bristol.
  • October 2018, Introduction to Multilevel Modelling, Bamberg.
  • August 2018, Multilevel Modelling: Three-level, Cross-Classified and Multiple Membership Models, EARLI Special Interest Group 18 Educational Effectiveness, Groningen, Netherlands (Course Instructor). Flyer.

Education Select Commitee

  • June 2020. Submitted written evidence to Education Select Committee on “Left behind white pupils from disadvantaged backgrounds"
  • March 2019. Submitted written evidence to Education Select Committee on “Progress 8 and the need to adjust for pupil background”

Response to Ofqual consultation

  • April 2020. Submitted statement to Ofqual consultation on ‘GCSE and A level grading proposals for 2020’

Response to EPI consultation

  • February 2022. Submitted statement to Education Policy Institute (EPI) consultation on 'The Features of Effective School Groups: Measuring pupil inclusion at school-group level'

Policy briefings

  • Leckie, G., Goldstein, H. (2019). The Government’s Progress 8 school performance measure needs to account for pupil background. PolicyBristol. Policy Briefing 66: 01/2019.
  • Prior, L., Jerrim, J., Thomson, D., and Leckie, G. (2021). Is it worth calculating Progress 8 anymore? PolicyBristol. Policy Briefing 109:11/2021.

Impact case study

Blogs

  • Prior, L., Jerrim, J, Thomson, D., Leckie, G. (2021). Is it still worth calculating Progress 8? BERA blog
  • Leckie, G., Goldstein, H., Prior, L. (2019). School accountability by Progress 8: A critique of Conservative anf Labour proposals. BERA blog. 

Radio coverage

  • January 2019: Heart radio
  • January 2019: Love sport radio

Newspaper and other print coverage

  • March 2019: Times Educational Supplement.
  • January 2019: Bristol Post, Daily Express, The Guardian, The Independent, Leigh Journal, The Star, The Times, Times Educational Supplement, Warrington Guardian, Wigan Today, Yorkshire Post
  • June 2018. 'Testing children gives a poor measure of ability'. Letter in the Guarduan in response to Simon Jenkins’s piece on how ‘the cult of tests is ruining our schools’. Article.

Other online media coverage

  • March 2019: Schools Week
  • January 2019:  Ekklesia, Greensheets, Humanists UK, Local Gov, My Science, Reclaiming Schools, Schools Improvement, Schools Week, World News Live 4 You

Knowledge exchange meetings:

  • July2021: Fischer Family Trust: Education Datalab. Discussed their related research work.
  • January 2021: Fischer Family Trust: Education Datalab. Discussed their related research work.
  • April 2020: Round table discussion on Progress 8 with 15 others. Via Zoom.
  • August 2019: Fischer Family Trust: Education Datalab. Discussed their related research work.
  • April 2019: Department for Education, London. Discussed Adjusted Progress 8 work.
  • February 2019: Ofsted, Bristol.  Discussed Adjusted Progress 8 work.
  • December 2018: Fischer Family Trust.  Discussed our Adjusted Progress 8 work.
  • September 2018: Ofsted, Bristol. Discussed our Adjusted Progress 8 work.
  • November 2017: Department for Education, London. Discussed grant proposal.
  • November 2017. HEFCE, Bristol. Discussed grant proposal.

Other meetings 

  • January 2022. Round table discussion on educational equity hosted by Loic Menzies and Cambridge Assessment Network
  • August 2019. Discussion with Education Campaigns Manager at Humanists UK about Progress 8
  • May 2019. Discussion with Director and Chief Economist of Northern Powerhouse Partnership
  • April 2019. Presentation of Progress 8 research to Emma Hardy MP (Education Select Committee member) at Portcullis house
  • March 2019. Presentation of Progress 8 research to MPs at private meeting with UK Parliament Education Select Committee
  • February 2019. Discussion with Head of Co-Op Academies about Progress 8

Related advisory work:

  • June 2019. Discussion with Israel Academy of Sciences about school value added models.
  • 2018-present: Discussions with Trinidad and Tobago Department for Education about school value added models.
  • 2018-present. Discussions with Sweden Department for Education about school value added models.
  • 2018-present. Office for Student (OfS) Expert Advisory Group on benchmarking methodologies.
  • March 2018. Discussions with Bulgarian Minister of Education and heads of directorates as expert member of World Bank team.
  • 2017-present. Higher Education Funding Council England (HEFCE) Learning Gain Expert Group.

Related conference organisation:

  • Spring 2021. Co-organiser (with Dave Thomson) of NPD User Group meeting, Bristol.
  • January 2020. Co-organiser (with Dave Thomson) of NPD User Group meeting, London.
  • September 2018. Co-organiser (with Becky Allen) of NPD User Group meeting, Bristol.

Northern Powerhouse Partnership

In joint work we produced a website where schools could look up their Adjusted Progress 8 scores.

The following reports were produced for the Northern Powerhouse Partnership (NPP) on Progress 8 and Adjusted Progress 8:

The following datasets provide the school Progress 8 and Adjusted Progress 8 scores used to compile the reports:

Previous Grant: ESRC Future Research Leaders Grant on school performance statistics.

Feedback

Edit this page