Universities should review support, assessment and contextual offers for widening participation students
Value-added measures of educational effectiveness provide indicators of the extra value that is added by schools to student attainment over and above the progress or improvement that might normally be expected.
About the research
These techniques could be used by Higher Education (HE) institutions to inform aspirations and facilitate greater HE participation by disadvantaged students. While rarely used to date, these approaches might also be helpful in supporting the development of evidence-based policy and practice in widening participation, an area of critical importance to HE institutions (and associated funding criteria).
Value-added measures seek to establish whether students in different schools/departments make relatively greater or less progress over a specified period of time. These measures can also be used to examine the progress of specific groups of students such as those from low-income families, facilitating the targeting of additional support for disadvantaged students where progress may be less than expected.
The methodology involves studying the effect of the school/department experience on individual student outcomes (what students achieve) and the extent to which student intake characteristics (such as their prior attainment, gender, ethnicity and social class) affect student outcomes.
So far, contextual admission and other Widening Participation (WP) policies have been underpinned through fairly crude statistical testing. Typically it has been found that students are more likely to get a 2.1 or higher degree if they attend a state rather than a private school; have high socioeconomic status (SES); are white.
This study is innovative in that it has been able to measure progress from intake to final degree. It has examined in much more detail (using multilevel modelling) the relationship between student degree outcomes, and their background characteristics. To do this we used a largescale matched dataset from Department for Education (DFE)/Higher Education Statistics Authority (HESA) for all UK Universities including almost 950,000 students across five consecutive cohorts, starting their 3 or 4-year degrees 2007-11 and taking GCSEs 2005-09. A further aim was to examine the variation in students’ value-added performance across different UK universities and academic subject areas for different student groups and the time trends across 5 cohorts.
Some disadvantaged students may perform below their potential at A-level due to poor schooling quality or lack of other types of support.
Consequently, they may not be offered a university place in spite of having the ability to achieve very well at HE.
In contrast, after starting university, some disadvantaged students may underperform at HE due to lack of adequate support or assessment. This report shows how value added-measures can help to identify these students in order to provide additional support.
Key findings
The overall patterns in English undergraduate performance across all UK universities need to be seen in the context that the proportions obtaining degree classes differ greatly by institution and also by HE cohort year. The analysis did allow for these differences and showed that:
• The probability of students obtaining a 2.1 or above degree outcome was predicted by HESA tariff score (A-level or equivalent), total GCSE score and Key Stage 2 attainment (prior attainment variables), as well as other student background characteristics controlled for in the statistical models. This analysis could explain 56% of the degree outcome differences between universities and similarly 21% of differences between subject areas, after adjusting for student prior attainment and background variables.
• Students typically found to perform less well than other students included those who were male, entitled to free school meals (FSM), aged 17-18 years on entry, with lowest socioeconomic status, parents not university educated, disabled, Black and Minority ethnicity (BME) or privately educated. However, this was not the case for some combinations of student factors such as the most able privately educated students and least able disabled students.
• This evidence suggests A-level and other prior attainment measures may not adequately reflect the potential future achievement for certain student groups, especially lower A-level attaining state school students (see plot 1) or mature students (see plot 2).
• Having taken account of student prior attainment and background variables, we also found for the most able A-level students the apparent gap in value added performance decreased to almost zero between the following student groups: males/females; BME/non-BME; aged 17-18 years/ 22-25 years on entry. This evidence possibly suggests an alternative explanation of lower degree performance that requires further investigation; that the experience of disadvantaged student groups while at university may unfairly restrict their performance in undergraduate degree outcomes, especially lower A-level attaining FSM or BME students.
Key findings - Value-added performance varies between universities
Having controlled for student prior attainment and background factors, there remain substantial and statistically significant differences between UK universities as well as across HEFCE academic subject areas in the probability of obtaining a 2.1 or higher degree, as estimated by our models (see plot 3).
Whilst this might be expected, our aim has been to show how differences are related to A-level scores. We found that after controlling for student intake and background factors, 4% of the remaining variance in degree outcomes was attributable to differences between universities, 5% was attributable to differences between subject areas and 9% was attributable to differences between universities and subject areas.
In further analyses (not shown here) we also found these differences vary significantly over time, suggesting using a single indicator of value-added performance for one cohort may be misleading.
Policy implications
Prior attainment (A-level tariff, GCSE scores, Key Stage 2 attainment) and other relevant student background factors such as age, gender, Free School Meals (FSM), Black and Minority Ethnicity (BME) and school attended, should be controlled for in Higher Education value-added analyses.
• Universities should use specific indicators when making contextual offers. These include noting that mature students and state-schooled students with average (AAB) or below A-level attainment, typically perform better than expected having controlled for intake factors.
• Universities should consider what possible explanations underlie the apparent lower performance of some student groups, across all UK universities as well as specifically in their own institution. Universities should review whether more assistance or alternative assessments are required for particular disadvantaged groups - such as BME or FSM students - to fairly support their progress while at university.
• Universities should also consider what possible explanations underlie relatively higher or lower value-added performance in some universities and/or disciplinary subject areas in comparison to others. Given the increasing competition between universities, it would be beneficial for HE stakeholders to reflect on whether differences in curriculum or assessment approaches or overall quality standards might be expected, or whether they should be reviewed to create better equivalences between degree content and outcomes.
• Universities should work with educational researchers and data analysts to identify the optimal approach to developing HE value-added methodology for all UK universities and how key findings could be best presented to non-statistical audiences. Evidence of this kind is crucial for HE professionals and policymakers to raise pertinent questions about the value-added performance of different student groups, universities, cohorts and subject disciplines. Moreover, other types of evaluation evidence, such as student satisfaction, are needed to provide a comprehensive evaluation of educational quality.
In terms of application, the HE institutional value-added measures and other findings reported here are estimates produced by the statistical analysis.
Therefore, similar to other quantitative evaluation measures, there are some limitations to HE value-added methodology and approaches, such as measurement error, which need to be recognised and understood when interpreting the findings (Goldstein, 1997, Domingue et al, 2014).
It is also important to note that we do not envisage the use of these measures to publicly rank universities with all the problems that are almost certainly likely to arise as they have with schools ‘gaming’ the system, but rather used sensitively by institutions as part of their self-evaluation.
Policy Report: March 2019
Universities should review support for widening participation students (PDF, 571kB)
Further information
Research funded by University of Bristol Widening Participation Research cluster see http://www.bristol.ac.uk/sraa/home-recruitment-and-conversion/widening-participation-research/
References
Crawford, C. (2014) The link between secondary school characteristics and university participation and outcomes CAYT Research Report. DfE June 2014. Goldstein, H. (1997) Methods in school effectiveness research. School effectiveness and school improvement. 8: 369-95
HEFCE (2014) Differences in degree outcomes: Key findings. March 2014/03. Issues paper. Higher Education Funding Council for England.
Thomas, S. (2001) Dimensions of Secondary School Effectiveness: Comparative Analyses Across Regions, School Effectiveness & School Improvement. 12 (4):285-322.
Thomas, S., Zhang, L. & Goldstein, H. (2018). Widening Participation and Progress in Higher Education Paper presented at European Educational Research Conference, Bolzano, Sept 2018.
Authors
Professor Sally Thomas, Dr Lei Zhang and Professor Harvey Goldstein, University of Bristol
Related briefings
Universities should be accessible to everyone, with lifelong learning at the heart of their work
The Government’s Progress 8 school performance measure needs to account for pupil background
