SUBU Responds to National Independent Review of the TEF

No ratings yet. Log in to rate.

What is this about?

The TEF is the Teaching Excellence and Student Outcomes Framework which was introduced by the Government in 2016 as a way of measuring excellence in teaching and student outcomes at different Universities. Universities can be rated: Bronze, Silver and Gold. These ratings are dependent on the level of quality of: teaching; the learning environment; and graduate outcomes.

Two of the key things TEF aims to do are:

  • to better inform students' choices about what and where to study 
  • to recognise and reward excellent teaching

TEF has gone through a number of changes since it was first introduced so the independent review aims to see how well it is currently working i.e. is it achieving what it sets out to do?

You can read more about TEF here.

Why has SUBU sent a response?

The review was won by NUS and some Students' Unions after they lobbied the Government back in 2017 on planned changes to Higher Education. By sending a response to this independent review, SUBU is taking part in these national conversations, using the experiences of students studying at BU. Any changes from this will impact past, present and future students. 

Teaching is a really important part of the student experience; we receive and act on thousands of comments from you about it and have referenced some of your feedback in our response! 

Our answers were informed by NUS and BU and you will notice us referring to and endorsing their responses.

You can read BU's response here.

If you have any questions on SUBU's response, please message suresearch@bournemouth.ac.uk 

SUBU's Response

Do you support the aim of assessing the quality of teaching excellence and student outcomes across providers of higher education?

Yes

Please explain why:

We believe it is important that there is a framework for measuring teaching quality and the quality of the student learning experience, which has a big impact on the student experience, and that the framework is designed to work well for students and that it clearly identifies excellent teaching.

The Student Outcomes aspect is also important as the value of a student attending University is projected into their life prospects and the breadth of opportunities available to them as a result; however the very simplistic measurement of Student Outcomes and the focus on graduate salaries does not foster a healthy approach for provider enhancement. Strategies to support employability such as alumni mentoring and specialist programmes for WP students to address progression enhance student outcomes for providers and recognise an important aspect for students. 

Transition for students into HE is a public conversation; prospective students will not know the full breadth of what University has to offer, especially with the conversation over value for money where students will reflect on things like contact hours and overlook opportunities to develop themselves personally and professionally, which is unique to each provider and what they can offer students.

TEF is an excellent example of a good idea in theory which has not yet been executed effectively. This would involve rejigging the metrics so the data is current and useful to measure the criteria.

These purposes fall into two main areas: providing information, and enhancing the provision of higher education.

(a) Which of these is the most important (select one option only)?

Enhancing Provision

(b) Please outline below the reasons for your answers:

Students require clear, easy to use information to help them choose a course and HEI. It is worth noting that some students have ‘more’ of a choice than others, for e.g. those restricted to HEIs more local to them because of costs, family commitments or ongoing health needs, particularly mental health provision with 1 in 8 applicants entering HE with a mental health condition (‘Reality Check’ - HEPI & Unite Students 2017).

There are currently a number of metrics, including NSS, that students can access to tell them how ‘good’ a course and HEI might be – these do not tell them other information which might be important to them around what an HEI feels like to study at (for e.g. the information students search for in places like Student Room). TEF in its current design does not add value to the information already available to students (for example on Unistats) and it also does not recognise that students need additional data to make an informed choice. SUBU endorses Bournemouth University’s response on this point.

TEF does not inform students about teaching quality with the current metrics being measured.

An HEI should focus on using the data it has available to enhance learning provision for students – in particular teaching quality. At our HEI the university uses the collective student voice at course, department, faculty and institution level to inform how well they are doing in terms of the overall quality of learning opportunities and the student experience.

Should there be any other purposes for TEF?

SUBU endorses Bournemouth University’s comments that the purpose of TEF should be to highlight excellence and provide inspiration, best practice and encouragement for the sector.

In terms of TEF providing information to students, SUBU also endorses Bournemouth University’s comment that students, as potential users of TEF information, should be consulted on what would be useful information and how it could be provided.

If, as currently, TEF is used to place providers in a simplistic 3-tiered system, we would argue that this does not add anything useful to the information that is already available to students to inform their choices. An institution-level ‘badge’ might not mean much to a student as they are looking at individual courses and will be more interested in the quality of the experience they could hope to have on a course level. However we believe that subject level TEF will not add valuable information as it is not at course level.

Are the criteria used in TEF appropriate? (Teaching Quality, Learning Environment, Student Outcomes & Learning Gain)

No

If not, what criteria would be more appropriate?

SUBU endorses NUS’ comments that the current metrics in TEF do not serve to enhance and assess teaching quality, therefore changes to the metrics need to be made, which would require alterations to the criteria.

TEF should recognise student engagement with learning from opportunities for co-production and co-creation, which as NUS has said, will allow students to be meaningful members of an academic community. This could be incorporated under Learning Environment as well as Teaching Quality.

Enhancing provision of teaching is an important part of improving the student experience. To demonstrate this; in the last 18 months our student representative system has collected 15,088 comments about the academic and overall student experience with 59% of this total number of comments relating to teaching excellence:

  • ‘Teaching on my course’ 33%
  • ‘Assessment and Feedback’ 18%
  • ‘Academic support on m course’ 8%

Results from SUBU’s annual evaluation survey shows that employability is also an important aspect of the University experience to students. We ask students an open question on what the 3 most important things would be to determine a successful University experience and the answers have followed the same patter over the last 5 years:

  • Employability prospects
  • Friends made
  • Good degree grade

This supports the continued inclusion of Student Outcomes within the TEF framework however it should move away from the focus on graduate salaries. There should be criteria that measure the extent students feel they have developed their employability prospects i.e. gains in social capital, skills development etc. since entering HE.

There is no direct measurement of teaching quality currently available. As a result, the TEF uses existing data as indirect measures of teaching quality. These measures are known as “proxies”.

(a) Are the metrics used in TEF the best proxies for measuring the TEF criteria?

No

(b) If you answered no, what metrics would be more suitable proxies?

We consider TEF has only been ‘tinkered with’ – the current metrics available are not ideal and no amount of alterations will make them so (for example the changes in the weighting of NSS - although we recognise the limits of this as a metric as it is only captures the views of final year students).

In particular, we have reservations about use of following metrics:

  • Learning gain – we endorse Bournemouth University’s comments on this as there is no agreed definition or metrics for learning gain
  • Graduate salaries -particularly when not contextualised as this really has no bearing on teaching excellence, please see our comments in the previous question and the question below on 'Are there any other aspects of the process that you wish to comment on?' on this
  • Contact time/ teaching intensity – we endorse Bournemouth University’s point on this that the metrics are not robust or effective measurements

We share NUS’ concerns that institutions can work with the current metrics to show their institution in the best light, gaming their way to excellence rather than actually enhancing provisions for students.

We would favour the following metrics:

  • Those around co-production and co-creation (like initiatives at our HEI – funding and prizes for co-creational and co-production activities)
  • Those around engagement in curriculum design (like at our HEI – students are equally paid, full panel members on review/ development panels)
  • Use of the collective student voice to measure teaching excellence, which could be facilitated through the students’ union, association or guild (our HEI uses this at all levels to measure the quality of the learning and overall student experience)

The TEF metrics are benchmarked to account for factors such as the subject of study, prior attainment; ethnicity and educational disadvantage of the provider’s student intake (see that ‘What is TEF?’ section for detail).

(a) Should the metrics be benchmarked to allow for difference in a provider’s student population?

Yes

(b) Does TEF benchmark for the right factors?

We consider that the TEF should definitely use student demographics (including WP markers) and also:

  • Regional aspects
  • Geographical aspects

The TEF process uses both quantitative evidence (for example, the core metrics) and qualitative evidence (for example, the written submission).

(a) What are your views about the balance of quantitative and qualitative evidence considered in arriving at ratings?

We approve of the opportunity for an HEI to include a qualitative submission and we would recommend that there is a requirement to provide evidence of the collective student voice from the students’ union, association or guild in such a submission. As with the Subject Level TEF pilots, we would recommend that there is a requirement to have a Lead Student Representative from the students’ union (unless they have constructively disengaged) to sign off the submission and confirm that they have had an opportunity to engage with the process.

We would also strongly recommend that students’ union could report back to OfS regarding their HEI’s performance, independently of their HEI’s submission. This would allow for independent commentary on student engagement and teaching quality, evidenced by the collective student voice. This would give unions an opportunity to highlight where there is excellent partnership work between the HEI and students.

(b) Are there any other aspects of the process that you wish to comment on?

As stated above, from a student perspective we are not certain that a single award per institution gives students any additional valuable information. There will be differentiation between HEIs in all three categories and it does nothing to provide additional information to students at a course level, which they already have access to.

We have concerns that the subject level TEF involves artificial groupings of courses, some of which are not similar, having the effect of shoe-horning them together in a way that will not provide meaningful information to students.

We endorse Bournemouth University’s point about it not being necessary for courses that require PSRB accreditation to be included in a subject level TEF, as we also believe that a TEF award will not add value for a student or employer in these cases.

We would also like to see a formal recommendation that the link between the TEF and tuition fees is discontinued.

We support the continued inclusion of Student Outcomes within TEF due to our student feedback, mentioned in our answer to 'Are the criteria used in TEF appropriate?', indicating that employability prospects are an important element of the value of a University experience. However, the metrics on student outcomes need to move away from the focus on graduate salaries as this is not a true depiction of student outcomes. There should be criteria that measure the extent students feel they have developed their employability prospects i.e. gains in social capital, skills development etc. which help to level the playing field, especially for students from WP backgrounds.

Are the purpose(s) of TEF met by:

(a) awarding a single rating?

No

Please explain your answer:

We do not believe that single awards provide useful information to potential students, students, employers or other stakeholders. The TEF does not measure some of the things that form part of a student’s choice of university (their personal circumstances, course content, learning environment) and a single award does not reflect this complexity (we are endorsing Bournemouth University’s point on this).

(b) with three levels of differentiation, plus a fourth rating for those unable to be assessed?

No

Please explain your answer:

We are concerned about the lack of differentiation within the current three categories and would recommend that using just three is reconsidered, to make sure that any changes do not involve a framework being shoe-horned to fit into three pre-existing categories.

We also endorse BU’s point on this regarding the perception of Bronze as ‘poor’ and agree with BU that that public perception of the Gold/Silver/Bronze categories is that they equate to good/fair/poor. This is misleading for students, as Bronze institutions have achieved an accepted level of quality in their provision and are better than ‘poor’. Whilst this association might not have been the intention, it is what has happened and does not provide students with useful information about what institution would suit them best.

(c) ratings named Gold, Silver, Bronze and Provisional?

No

Please explain your answer:

As we describe above, we believe the Gold/Silver/Bronze awards are not helpful in aiding students’ decisions about what institution might suit them best. Significant changes need to be made to the structure and labels in order to change the perception that an institution in the ‘bottom’ category is ‘poor’.

Has the introduction of TEF positively changed the educational experience of students (e.g. teaching and learning)?

We endorse the point from NUS here about TEF not delivering so far on providing information for students – UCAS research from June 2018 shows that 19% of applicants was aware of TEF (https://www.ucas.com/corporate/news-and-key-documents/news/ucas-analysis-shows-growing-awareness-tef-among-applicants-higher-education ).

Has the introduction of TEF negatively changed the educational experience of students (e.g. teaching and learning)?

Yes

If yes, how?

We are concerned that the TEF does not encourage collaboration or sharing of best practice across the sector and rather encourages competition between institutions; in particular we believe that institutions risk focussing efforts on how best to perform in the metrics, rather than on enhancing the quality of teaching and learning opportunities for students.

Does TEF help you as a student/student union/provider/employer/other? Please explain the reasons for your answer.

No

Please explain the reasons for your answer:

We consider that a framework to assess teaching excellence could work very well for students, provided that there is a clear focus on enhancement and that the framework includes a requirement for HEIs to provide evidence on how the collective student voice has been used to inform their enhancement processes.

We believe that TEF does not currently support the consistent use of the collective student voice within the HEI as it doesn’t require this as part of the qualitative submission. As described earlier in our answer to 'What are your views about the balance of quantitative and qualitative evidence considered in arriving at ratings?', we believe that a re-worked framework would have the potential to raise the impact of the collective student voice, by requiring that  institutions provide evidence that they use it to inform and drive their teaching excellence enhancement work.

As already noted, we would highly recommend that the institution’s qualitative submission is supplemented by a student written submission from the union/association/guild in order to guarantee that the collective student voice is at the heart of the framework. As mentioned in our answer to 'Are the criteria used in TEF appropriate?', a very high proportion of the student feedback we collect through the student representative system relates to teaching excellence: 59% of 15,088 comments collected in the last 18 months.

Explaining your reasoning, what are the most significant benefits of:

(a) Provider-level TEF?

As described in responses elsewhere in this consultation, we do not consider that there are significant benefits of a provider-level TEF and it does not provide prospective students with meaningful information to help them choose the most appropriate institution and course for them.

(b)Subject-level TEF?

As with our response above

Are there particular types of students, provision or providers that are disadvantaged by the current design of TEF, in a disproportionate way?

Yes

If so, what changes could be made to address this?

We endorse NUS’s point in reference to research from students’ unions that shows there are certain groups of students who are more likely to hold the opinion that Gold-rated institutions are ‘not for them’; for example 11% of Black students would have reconsidered applying to their HEIs if they had been Gold-rated, compared to 5% of white students (https://studentsunionresearch.com/2017/11/09/48/ pg4).

This type of phenomenon is something that the sector is trying to remedy in its widening participation work and it is unfortunate that the current labels seem to reinforce assumptions and stereotypes in terms of what types of institutions might ‘suit’ some students more than others.

We also endorse NUS’ concern about reliance on LEO data disadvantaging students whose backgrounds might mean that they are less likely to do well regarding employment after university; HEIs might be less likely to recruit from these groups as a way to ‘manage’ their TEF score.

In terms of using salary data as a proxy for teaching excellence, we endorse the views from both NUS and BU. Taking this approach might make HEIs focus on enhancing careers advice/ recruitment information rather than enhancing teaching and the student experience. To help avoid this, there should be a greater focus on co-production/ co-creation and learning gain as a proxy for teaching excellence.

Comments

No comments have been made. Please log in to comment.