Research and analysis

Research to support the independent TEF review: surveys of HE applicants

Published 21 January 2021

Applies to England

Research overview

This research consisted of 3 distinct elements which were brought together to provide an understanding of applicants to higher education (HE) use of and views about the Teaching Excellence and Student Outcomes Framework (TEF). The 3 research elements were:

• UCAS annual applicant survey with UCAS TEF questions
• UCAS omnibus survey with TEF questions designed by the TEF review team and DfE research team
• TEF focus groups with applicants, pre-applicants, students and careers advisors

UCAS annual survey with UCAS Teaching Excellence Framework (TEF) questions

Headline TEF insights from existing UCAS annual survey with new applicants to HE:

• TEF questions were designed by UCAS
• the survey was undertaken September 2018 to June 2019 at the point of application
• results from the 2019 new applicants’ survey are based on responses at 11 March 2019
• there were approximately 80,000 applicants from 2019 entry cycle at 11 March 2019

Questions for 2019 HE (higher education) applicants:

• Are applicants aware of TEF and TEF ratings?
• When making decisions about HE, how important is TEF to applicants compared to other factors?

Omnibus survey with bespoke TEF questions

Specific TEF insights based on bespoke TEF questions included in UCAS applicant omnibus survey:

• 10 TEF questions designed by TEF review team and DfE HE social research team were added to the omnibus survey
• there were approximately 2,500 applicants from the 2019 entry cycle
• the survey was undertaken in March 2019

Questions for 2019 HE applicants:

• Did applicants use TEF awards to make decisions?
• When making decisions about HE, how important are subject-level and provider-level TEF to applicants compared to other factors?
• What do applicants think about subject-level TEF?

Focus groups

Four TEF focus groups with pre-applicants, 2019 applicants, students, postgraduates and career advisors:

• 2 held at schools (boys grammar school, mixed comprehensive)
• 1 UCAS Student Advisory Group
• 1 UCAS Secondary Education Advisory Group – careers advisors discussion
• the focus groups were undertaken February 2019

Questions for all focus groups:

• How do they interpret TEF ratings?
• What do they think of the TEF rating system?
• How would they value and use the subject-level TEF?

Methods

The research was facilitated by UCAS’ survey data services, and advisory and focus groups.

Survey data was weighted based on response rates, to account for some characteristic groups being more likely to respond than others. This ensures that the aggregated results are representative of the UCAS applicant population for the 2019 application cycle.

We have stated where statistical significance testing was undertaken.

Knowledge of TEF

Applicants’ knowledge of provider-level TEF was compared between 2018 and 2019 using TEF questions from the annual UCAS applicants’ survey.

Applicants’ knowledge of provider-level TEF has increased between the 2018 and 2019 application cycles.

Knowledge of TEF before applying to HE	2018	2019
Knew what TEF was	17.1%	22.9%
Heard of TEF but didn’t know what it was	17.7%	22.5%
Had not heard of TEF	65.2%	54.6%

Knowledge of TEF ratings of universities applied to (among those who knew what TEF was)	2018	2019
Yes, I knew all their ratings	42.3%	45.2%
I knew some of their ratings	50.9%	48.7%
No, I didn’t know any of their ratings	6.8%	6.2%

Insights

• English applicants were most likely to know about TEF: 27% of respondents in England, compared to 24% Wales, 13% NI and 12% Scotland, 17% International and 15% EU.
• Younger applicants were most likely to know about TEF: 25% aged 18, compared to 23% aged 19, 20% aged 20-24 and 18% aged 25 and over.
• International applicants were more likely to know the TEF ratings for all their HE choices: 55% International compared to 45% EU, 47% Wales,44% England, 41% NI and 38% Scotland.
• Older applicants were most likely not to know the ratings of any of their HE choices: 14% aged 25 and over, compared to 7% aged 20-24, 6% aged 19 and 5% aged 18.

Source ^{[footnote 1]}

Focus group comments

I haven’t heard it mentioned in my school.

I’ve seen universities promoting it, it means their lectures are good.

Importance of TEF

Importance ratings of TEF have increased from 2018 to 2019

Importance of TEF in deciding where to apply, among applicants who knew about TEF when applying

Year	Extremely important	Important	Slightly important	Not very important	Not at all important	Total
2019	22%	53%	18%	5%	2%	100%
2018	19%	39%	27%	10%	5%	100%

Predicted importance of TEF in deciding where to apply, had they known about TEF, among applicants who did not know about TEF when applying

Year	Extremely important	Important	Slightly important	Not very important	Not at all important	Total
2019	16%	55%	21%	5%	3%	100%
2018	16%	44%	28%	8%	4%	100%

Insights

TEF was rated important/extremely important:

• by 74% of those who knew about TEF in 2019, compared to 58% in 2018
• by 71% who did not know about TEF when applying in 2019 (but were subsequently informed via the survey), compared to 60% in 2018

International applicants who knew about TEF were more likely to rate it important:

• International 82%
• UK 74%
• EU 73%

International applicants who did not know about TEF were more likely to rate it important:

• International 78%
• EU 73%
• UK 70%

Source ^{[footnote 1]}

Use of TEF

The omnibus survey was used to update the headline figures of TEF use from the DfE TEF evaluation undertaken in 2018.

Applicants’ use of TEF has increased from 2018 to 2019.

An estimated 22% of all 2019 applicants actively used TEF to make decisions about where to study, an increase from 15% for 2018 applicants.

Of those who knew about TEF, 41% of applicants actively used TEF to make decisions about where to study, an increase from 34% for 2018 applicants.

Of those who knew what TEF was:

Question	Yes	No	Don’t know
Did you use the TEF awards to help you make your decision about where to study?	41.4%	50.3%	8.3%

Applicant group	% using TEF to make decisions (average 41%)
Men	45%
Women	39%
Polar Quintile 1 – most disadvantaged	36%
Polar Quintile 2	46%
Polar Quintile 3	42%
Polar Quintile 4	44%
Polar Quintile 5 – most advantaged	44%
International	47%
Wales	46%
England	42%
EU	38%
Scotland	27%
NI	24%
18 year olds	43%
19 year olds	37%
20-24 year olds	39%
25 year olds and over	37%
Asian	44%
Black	30%
Mixed	55%
White	40%
Other	54%

Source ^{[footnote 2]}

Focus group comments

I looked at it, but it wasn’t a deal breaker.

I think it’s important; I found it useful.

Value of TEF

Applicants’ views about the value of TEF are broadly positive.

Statement	Agree	Neutral	Disagree	Not sure	Total
TEF gives useful information about teaching and education experience	70%	12%	2%	16%	100%
Single rating like TEF makes it easier to use information	67%	14%	7%	12%	100%
I value the independence and standardisation of assessment process	63%	19%	3%	15%	100%
TEF makes it more confusing when trying to decide where to apply	17%	23%	45%	15%	100%
TEF duplicated information available and is not needed	17%	29%	34%	20%	100%

Insights

For demographic subgroups, there were few consistent or substantial differences from the average:

• Over all statements, applicants from Northern Ireland and Scotland had the highest levels of “Don’t know” responses (9 to 20% higher than the average) and correspondingly the lowest levels of agreement and disagreement with the statements, potentially reflecting current UK coverage of the TEF awards.
• International applicants from outside the EU tended to agree more with statements 1, 2 and 3 than average (74%, 73% and 70% respectively).
• Applicants from the EU agreed less strongly (12%) and disagreed more strongly (43%) that TEF duplicated information available elsewhere (statement 5).

Source ^{[footnote 3]}

Importance of TEF compared with other decision-making factors

When considered alongside other decision-making factors TEF rating of HE provider is least important of 15 decision-making factors:

Decision-making factor	% saying important or extremely important
Quality of staff/teaching	94%
Quality of teaching facilities	89%
The range of modules	87%
Graduate prospects	86%
Good student reviews	83%
Challenging course content	76%
Academic entry requirements	74%
Earning potential of graduates	73%
Work experience as part of course	70%
Incentives on offer	68%
Institution pre-application communication	68%
Cost of tuition fees	64%
Position in league tables	56%
TEF rating of subject at institution	51%
TEF rating of institution	40%

Insights

Subject-level TEF was ranked 14th in importance, more important for decision making than provider-level TEF, ranked 15th.

There was no apparent change between applicants 2018 and 2019 in importance of top and bottom decision making factors:

• 1st – quality of staff/teaching: 94% rated as important (95%, 2018).
• 15th – provider-level TEF rating: 40% rated as important (40%, 2018)

For different demographic groups, there were a few exceptions to ranking TEF importance 14th and 15th.

Subject-level TEF was ranked 13th by:

• those age 20 to 24 (league tables and subject-level TEF joint 13th)
• those age 25 and over (ahead of league tables)
• polar quintile 4 (ahead of cost of tuition fees)

Polar quintile 3 rated subject-level TEF 12th (ahead of both league tables and tuition fees).

Whilst TEF was ranked lower compared to other factors, many of the factors that are rated more highly are composite elements within the TEF rating.

Source ^{[footnote 2]} and source ^{[footnote 4]}

Focus group comments

Ticks a box but I wouldn’t base a decision on it

Gold offered a bit of reassurance

Importance of TEF – factors predicting applicants’ rating of the importance of TEF

Two regression analyses were undertaken to identify which applicant characteristics were the strongest in influencing and predicting applicants’ importance ratings of:

1. provider-level TEF
2. subject-level TEF

Findings from the regressions identify applicant characteristics that either increased or decreased importance ratings.

Reported findings include:

• The estimate of the amount of change in the TEF rating due to the factor: the higher the percentage reported, the more it will change the importance rating.
• The confidence interval (95% CI) predicts a range of values within which the true amount of change will fall. The range is usually wider for smaller populations and a narrower range means the effect is more robust.

All findings presented are statistically significant (p<.05).

Overall predictors for provider-level and subject-level TEF

The most consistent applicant factors influencing increases in both their subject-level and provider-level TEF ratings of importance were:

• having or gaining some knowledge of TEF
• having a clear idea about job or career

Demographic factors of being of White ethnicity and male predicted lower ratings of TEF.

Source ^{[footnote 4]}

Factors predicting applicants’ rating of the importance of provider-level TEF

• Those who had heard of the TEF and knew a lot or a fair amount about the TEF were respectively 83% and 84% more likely to have a higher importance rating than those who had heard of the TEF, but knew nothing about it (CI, 70 to 91%; 78 to 88%).
• Those with a very clear idea or some idea about their job or career were respectively 51% and 35% more likely to have a higher importance rating than those who had no idea about their occupation (CI, 33 to 63%; 15 to 51%).
• Those with a White ethnicity were 63% more likely to have a lower importance rating than those with an Asian ethnicity (CI, 24 to 111%), whereas those with an ‘Other’ ethnicity (ie not of Asian, Black, Mixed or White ethnicity) were 55% more likely to have a higher importance rating than those with an Asian ethnicity (CI, 14 to 76%).
• Women were 26% more likely to have higher importance rating of provider-level TEF than men (CI, 7 to 47%).

Factors predicting applicants’ rating of the importance of subject-level TEF

• Those who had heard of all 3 items of TEF before the survey (TEF name, Gold/Silver/Bronze awards, and Provisional awards) had a 51% increased likelihood of higher importance ratings than those who had not heard of TEF before the survey (CI, 9 to 74%).
• Those who had heard of the TEF and knew a little, a fair amount or a lot about it were respectively 28%, 77% and 79% more likely to have a higher importance rating than those who had heard about TEF, but knew nothing about it (CI, 6 to 45%; 69 to 83%; 62 to 89%).
• Those who had a very clear idea or some idea about their job or career were respectively 40% and 28% more likely to have to have a higher importance rating than those who had no idea about their occupation (CI, 20 to 56%: 5 to 45%).
• Those with a White ethnicity were 69% more likely to have a lower importance rating than those with an Asian ethnicity (CI, 29 to 121%).
• Women were 25% more likely to have higher importance rating of provider-level TEF than men (CI, 7 to 46%).

Clarity of subject and career path, and the use of TEF

• 75% of applicants had a very clear idea about which subject they wanted to study at university or college.
• 42% of applicants had a very clear idea about their future job or career path.

Applicants’ clarity about subject area they would like to study and their future career path

Subject/career	Very clear idea	Some idea	No idea	Total
Subject	75%	24%	1%	100%
Career	42%	49%	9%	100%

Insights

Subject clarity:

• Applicants in Wales (80%), Scotland (78%) and England (78%) were more likely to have a very clear idea of subject to study than applicants from NI (68%) and International (67%) and EU applicants (65%).
• Applicants aged over 20 (82%) were more likely to have a very clear idea of subject to study than applicants aged 18 (74%) or 19 (70%).

Career clarity:

• Applicants in Scotland (50%), Wales (47%), England (43%), NI (40%) and International (39%) were more likely to have a very clear idea about their job or career than applicants from the EU (27%).
• Applicants from polar quintile 1 (43%: most disadvantaged) were more likely to have a very clear idea about their job or career than applicants from polar quintile 5 (33%: most advantaged).
• Applicants aged over 25 (66%) and aged 20 to 24 (51%) were more likely to have a very clear idea about their job or career than applicants aged 19 (38%) or 18 (37%).

Source ^{[footnote 3]}

Combined clarity of subject and career path, and use of TEF

This section of the research combined the 3 levels of clarity of subject and 3 levels of clarity of career to establish 9 subgroups. These 9 subgroups reflect the degree of readiness of applicants, in terms of subject and career, to make their HE decisions. The level of knowledge and use of TEF was established for each of the groups.

When prompted about specific elements of the TEF, 53% of all applicants recalled they had heard of TEF before taking part in the omnibus survey.

When asked if they used TEF to make their decisions, 22% of all applicants indicated they had used TEF to help make decisions about where to study.

TEF use varied, depending on how clear applicants were about the subject they wanted to study and their job or career path.

Group	Subject	Career	Proportion of all applicants in each group	Proportion of all applicants who had heard of TEF	Proportion of all applicants who used TEF to make decisions
1	Very clear idea	Very clear idea	39%	21%	9%
2	Very clear idea	Some idea	32%	18%	7%
3	Very clear idea	No idea	4%	2%	1%
4	Some idea	Very clear idea	2%	1%	<0.5%
5	Some idea	Some idea	17%	8%	3%
6	Some idea	No idea	4%	2%	1%
7	No idea	Very clear idea	0%	n/a	n/a
8	No idea	Some idea	<0.5%	<0.5%	<0.5%
9	No idea	No idea	1%	0.6%	<0.5%
-	-	All applicants	100%	53%	22%

Insights

Applicants with a very clear idea about the subject they wish to study and either a very clear idea or some idea of a job or career path together represented the largest proportion of the applicant sample at 71%. As the largest group of applicants, they also form the largest proportion of applicants who had heard of TEF (39%), and had used TEF to make decisions (16%).

Anticipated use of subject-level and provider-level TEF ratings

The majority of applicants favoured using both subject-level and provider-level TEF ratings.

If both provider-level and subject-level ratings were available, which do you think you would use?

Which would you use?	Response
Both provider-level and subject-level ratings	82%
Only provider-level ratings	3%
Only subject-level ratings	11%
Neither provider-level or subject-level ratings	4%

What impact would subject-level TEF ratings have on likelihood of applying?

Subject-level rating	Much more likely to apply	A little more like to apply	It wouldn’t impact my decision	A little less likely to apply	Much less likely to apply
Gold	54%	29%	16%	0%	1%
Silver	14%	53%	29%	3%	1%
Bronze	4%	23%	43%	22%	8%
Provisional	3%	10%	47%	26%	14%
No award	2%	3%	37%	26%	32%

Insights

Most applicants say they would use both provider-level and subject-level TEF but subject-level is slightly more popular:

• 11% say they would use only subject-level
• 3% would only use provider-level

Relative importance of subject-level TEF:

• subject-level TEF rated more important than provider-level TEF
• Provider-level:
  • 11% say extremely important
  • 29% say important
• Subject-level:
  • 14% say extremely important
  • 37% say important

Subject-level TEF was still only rated 14th overall (out of 15 factors).

Source ^{[footnote 3]}

Focus group comments

Subject-level ratings would definitely be more helpful. I’m not really interested in what goes on with other courses.

Bronze is seen as bad, but it depends on how the other courses I opt for are graded

Views and evidence about TEF from applicant, student and career advisor focus groups

Focus groups sought feedback in the following areas:

• What did participants think of the TEF and the TEF rating system?
• How could TEF ratings better inform applicants’ choices? Were Gold, Silver and Bronze useful rating names? What else would be useful? Are 3 levels of ratings sufficient?
• How do they interpret TEF ratings for different institution types? What information would help make sense of these differences?
• How useful would subject-level ratings be? How would subject-level ratings and provider-level ratings be used?

Respondents appeared unsure about the added value of the TEF

It’s just another form of league table

Dubious about gold/silver ratings as they seem to be everywhere

It’s based on existing data anyway

We use Unistats, league tables, Which? University – what is this adding?

There’s a danger of information overload

They questioned whether the TEF had sufficient emphasis on teaching

TEF should be solely around teaching and not other factors such as student satisfaction

What’s the correlation between this data and being good at teaching?

The name is misleading – it doesn’t look at the teaching

Maybe it should be rebranded if the decision is to continue with factors that aren’t just teaching related

There were some questions about what TEF was and what Gold, Silver and Bronze represented

I don’t understand the criteria

I don’t understand why widening participation is part of it

Can you get anything less than a bronze?

It sounds like you can scrape through and still get a bronze

Respondents felt the TEF rating was a useful metric - more granularity was needed to differentiate between providers and more focus should be made on the TEF being independent

How gold is this gold rating I am looking at? Basically, a good silver plus a bit more or a solid gold?

Not gold silver or bronze, but maybe a more granular scale, eg 1 to 5

It is important to be clear this is an impartial rating system unlike some of the rankings

Respondents mostly thought that subject-level TEF would be useful, more useful than provider-level TEF, so long as there was a good level of understanding about what TEF covered

Subject-level ratings would definitely be more helpful.

I’m not really interested in what goes on with other courses. Prefer subject ratings over provider ratings

Subject ratings would be much more important and render the provider-level ratings irrelevant

1. Source: UCAS TEF report from annual applicants survey, 2018; 2019 (at March 2019, survey open to June 2019) ↩ ↩²

2. Source: 2019 – UCAS omnibus survey with DfE questions; 2018 – Same questions, IFF/DfE Provider-level TEF Evaluation (2018) ↩ ↩²

3. Source: 2019 UCAS omnibus survey with DfE questions ↩ ↩² ↩³

4. Source: 2019 UCAS omnibus survey with combined UCAS/DfE questions ↩ ↩²