UCAS has a large and increasing data science capability, which it has put to use in a number of ways, for public benefit. One such need is the demand for an evidence base in relation to widening participation in higher education.
An enormous amount of investment takes place within the sector relating to widening participation, and access to higher education. This support is typically offered through outreach activities, and aimed at potential applicants to higher education who find themselves at a disadvantage due to their background or individual circumstances. Despite the scale of this investment, and the transformative impact it has on young people’s lives, little activity exists to evaluate the effectiveness of these activities.
The Analysis and Insights business unit at UCAS consists of over forty data scientists, with a variety of remits, products, and services. An example is STROBE – a service designed to evaluate widening participation activity and fill the gap in the sector. In 2017, STROBE underwent a transformation from a simple tracking service, to a highly sophisticated evaluation service.
The STROBE model is a simple one. UCAS is not permitted to share individual level data with third party organisations for this purpose, and therefore has built the functionality to receive data on named individuals from these third parties (with their permission), match it to the UCAS database, and return statistics relating to the application journey of these individuals.
At its most basic level, STROBE will provide information on the proportion of the named individuals that apply to university, the proportion that receive offers and d the proportion that are accepted to an institution. STROBE will also break this information down by individual universities or aggregated university groups.
This level of detail alone is valuable to the sector, but with approximately £200 million spent on these activities each year, it would be remiss for organisations not to demand a more robust evaluation, which is what STROBE aims to achieve.
STROBE creates control groups from both the UCAS admissions, and potential admissions, databases. These control groups are designed to be statistically similar in make-up, across multiple demographic and academic dimensions. The use of the potential admissions database controls for behavioural dynamics in the customer’s data – a predisposition towards ambition, or even specifically to higher education is likely to already exist in what is known as an intention to treat bias. These control groups will of course be subject to random variation, but the enormity of UCAS data asset allows the creation of multiple control groups which normalises for this effect.
Once this process has taken place, it becomes relatively easy to calculate assessments of significant difference in the customer’s data. A successful customer would be able to state that their intervention meant students were significantly more likely to apply or be accepted to university, or a specific type of university. This assertion would hold a significance it didn’t have before.
STROBE reports on the admissions process, from application to acceptance. Looking at the individual stages of the process can fine tune interventions. For example, a customer seeing success regarding university applications, but not acceptances, may devote more effort to their students receiving offers or meeting conditional offers.
STROBE will also report on the academic tariff level of the applications and acceptances the customer’s students are making. It’s not uncommon for the headline data to show an intervention hasn’t yet had a significant impact at headline level, but they can further interrogate the data to show whether there has been change in the type of provider a student may apply to (e.g. from low to high tariff institutions). Therefore, the intervention could have potentially affected the student’s aspiration.
Customers of STROBE have included charities and higher education providers, as well as other not-for-profit and commercial entities. Feedback has been positive, with one customer remarking:
‘Working with UCAS has given our evaluation work a level of rigour that we could not have achieved independently. The methodology underpinning the project was extremely well thought through.’
UCAS is uniquely placed in the higher education sector, with the data, capability, and connections to deliver many other initiatives for public benefit, and always welcomes the opportunity to discuss these. For more information on STROBE, including how to become a customer, please contact the team at strobe@ucas.ac.uk.