Consultation outcome

Consultation decisions: Terms of Reference for EPRS in Summer 2021

Updated 23 July 2021

Introduction

Between 25 June and 2 July 2020 we ran a consultation on proposed changes to the Terms of Reference for the Exam Procedures Review Service (EPRS) which would affect qualifications awarded this summer, on the basis of Teacher Assessed Grades (TAGs).

We received 7 responses to our public consultation. In this document we set out our analysis of the responses we received as well as our final decisions.

We have published final Terms of Reference for EPRS in summer 2021 alongside this document, and will publish further guidance on EPRS for students before results are issued.

Summary of decisions

We have decided to set Terms of Reference which reflect those we proposed. We have made 2 small changes, unconnected to our proposals, which are to confirm that the EPRS will accept applications for Project qualifications at all levels, not just Level 3, and in respect of any in-scope qualification which was awarded to a learner in the UK and not, as the draft suggested, only learners in England.

Details

We described the changes we proposed to make to the Terms of Reference for EPRS for qualifications awarded in summer 2021 using 4 headings:

• AEA qualifications,
• Technical Qualifications forming part of T levels
• the role of the EPRS for summer 2021 qualifications
• who may make an application

We have used similar headings to explain our analysis of the responses we received and to set out our final decisions.

AEA Qualifications

The remit of the EPRS does not usually include Advanced Extension Award (AEA) qualifications because there are important differences between the review and appeal arrangements for those qualifications and those which must be in place for GCSE, AS, A level and Project qualifications (which are in scope for EPRS).

We explained in the consultation that AEA qualifications will this summer be awarded under the General Qualifications Alternative Awarding (GQAA) framework which also specifies how GCSE, AS, A level and Project qualifications must be awarded.

We proposed that the remit of the EPRS for summer 2021 should include all of the qualifications awarded under the GQAA framework.

All respondents to the consultation either agreed with, or expressed no view on, our proposal that the EPRS should consider applications in respect of AEA qualifications this summer, as well as GCSE, AS, A level and Project qualifications. We have decided to include AEA qualifications as we proposed.

Technical Qualifications

We decided in 2019 that the remit of the EPRS would be extended to include Technical Qualifications which form part of T levels (TQs) as those qualifications became available.

Although some TQs will be awarded this summer, we proposed that EPRS should not consider applications in respect of TQs this year. This is because TQs this summer will be awarded under the Vocational Contingency Regulatory Framework which means that the reviews and appeals process which is in place for those qualifications this summer may not be the same as that which is in place for GCSE, AS, A level and Project qualifications. In a normal year, the reviews and appeals process for TQs must be the same as that for A level qualifications.

Six respondents to the consultation agreed or strongly agreed with our proposal in respect of TQs. One respondent disagreed, but only to the extent that it suggested it should be possible to apply to the EPRS in relation to every Ofqual-regulated qualification.

We do not consider this is the right time to make any broad change to the remit of the EPRS. The EPRS has historically considered applications only in respect of GCSE, AS, A level and Project qualifications, following from the work of the Exams Appeal Board. These qualifications all share arrangements for reviews and appeals which are specified in conditions and requirements. The changes we are making for 2021 will align the EPRS process this year with previous years, by including all of those qualifications for which the review and appeals process is specified in the GQAA framework.

We have decided the remit of the EPRS should include only those qualifications which must be awarded under the GQAA framework, as we proposed. We will delay implementation of the decision we made in 2019 in respect of TQs until those qualifications are first awarded under the Qualification Level Conditions and Requirements for Technical Qualifications.

The role of EPRS

An application to the EPRS is an application for a procedural review. In other years, the EPRS considers whether the awarding organisation followed the processes it says it will follow to determine a result, or consider an appeal, and whether, in following those processes, the awarding organisation secured the outcomes required by the applicable conditions and requirements. The EPRS cannot change a learner’s grade.

The exceptional regulatory arrangements we have put in place, following government’s decision that exams should not take place this summer, specify the way in which results must be determined this summer and set an approach to reviews and appeals which differs substantially from that in place in a normal year.

We set out in the consultation the specific aspects of the process which will be in place this year which we considered the EPRS should review. We explained those proposals using 4 headings:

• the determination of results, and awarding organisation errors
• appeals on procedural grounds
• unreasonable exercise of academic judgement
• correcting results

For each area, we proposed that the EPRS should be able to consider whether the awarding organisation followed its own procedures properly and consistently and whether it secured the outcomes required by the GQAA conditions.

Four awarding organisation respondents made representations which are relevant to each of the 4 areas we considered in this section. Those representations concern:

• whether the EPRS might consider information or argument which was not raised at the review or appeal stage
• whether the EPRS would consider an application on behalf of a centre
• whether it was appropriate for the EPRS to consider if a process was capable of securing the outcomes required by the GQAA conditions
• whether the EPRS would consider broader questions of fairness
• correcting results

New Information

Two awarding organisations commented that it would be undesirable for the EPRS to consider applications made on the basis of information or argument which could have been raised as part of the centre review, or the appeal to the awarding organisation, but which was not raised at that time.

They observed that for the EPRS to investigate (or require the awarding organisation to investigate) entirely new grounds of appeal would be inefficient and would be inconsistent with a procedural review, which is how the EPRS process is described.

One awarding organisation suggested that it may be preferable, if fresh information comes to light within a reasonable period after an appeal, for the awarding organisation to accept a second application for appeal, on an exceptional basis, so the new information could be considered through its appeals processes.

We agree that the main role of the EPRS is to consider how an awarding organisation managed an appeal, and in particular whether it followed its process properly and consistently in relation to that appeal, as well as whether in that case the process secured the outcomes required by the conditions. In the majority of cases it will not be necessary to consider (or to ask the awarding organisation to consider) whether a different appeal decision might have been reached had different information been available, because the EPRS process will evaluate the decisions taken by the awarding organisation and not those which it might have taken in different circumstances.

However, it may be appropriate, by exception, to consider whether particular information might have affected the outcome of an appeal. For example, this might be appropriate where the premise of the application to the EPRS is that an awarding organisation could and should have considered a particular piece of information, but failed to do so.

We will keep in mind the possibility that it might be more appropriate in some cases to refer an applicant back to the awarding organisation, rather than consider fresh information through EPRS, and will welcome effective engagement with awarding organisations should such cases arise.

Applications by centres

Four awarding organisations questioned whether a centre would be able to make an application to the EPRS on its own behalf where it disagreed with an awarding organisation’s decision following a learner’s appeal. Three awarding organisations suggested that it would be useful to clarify the position, and one suggested a centre should be able to apply to the EPRS where it disagreed with the awarding organisation’s decision, following a learner’s appeal, to change the grade awarded to that learner.

In practice, the opportunity for a centre to make such an application would be relevant only where the learner did not wish to make an application for an EPRS review. This would represent a significant extension to the remit of the EPRS, which will usually consider only applications made on behalf of a learner, with their agreement.

We do not think we should extend the remit of the EPRS in this way. In particular, this would allow an application to be made without the learner’s knowledge where the centre considers an awarding organisation’s decision to increase that learner’s grade, following an appeal, was incorrect. We do not think this would be an appropriate use of the EPRS process. A decision by an awarding organisation that a centre did not follow its procedure properly or consistently, or that it reached an academic judgement which was unreasonable, is not pejorative - in the same way a decision by an awarding organisation that a Marking Error occurred in a normal year is not pejorative. In these circumstances, we do not think there is sufficient public interest in allowing centres to challenge those decisions through the EPRS process when the cost of doing so would be to create uncertainty for learners as to whether a grade which has been corrected by an awarding organisation following an appeal will stand.

The draft Terms of Reference specified that applications could be made by, or on behalf of, learners. We consider this is sufficient to make clear that any application which is not supported by the affected learner will be rejected.

Compliance with the conditions

Three awarding organisations questioned whether the EPRS should consider if an awarding organisation had secured the outcomes required by the conditions, in relation to those aspects of the process (to issue results and manage appeals) which EPRS would review.

The awarding organisations noted that Ofqual was aware of the processes they had in place to secure compliance with the GQAA framework, and suggested that any concerns about whether those processes were capable of securing compliance could (and should) be raised now, and not considered for the first time through an application to the EPRS.

We agree it is likely in most cases this summer the answer to the question whether an awarding organisation secured the outcomes required by the conditions will be the same as the answer to the question whether the awarding organisation followed its process properly and consistently. This reflects the nature of the awarding process this year and the requirements specified in the GQAA conditions. We also agree it is unlikely, given the engagement with us which has taken place, that any awarding organisation will have in place a process which cannot secure compliance with the GQAA conditions, although we note that it is possible for flaws in a process to be discovered only when that process is tested through live, and unpredictable, casework.

Moreover, we recognise that it is not the role of the EPRS to evaluate the merits of an awarding organisation’s decision where that decision is essentially a matter of judgement for the awarding organisation – for example whether a result was or was not an unreasonable exercise of academic judgement, or whether an incorrect result should or should not be corrected.

However, it remains possible, as in a normal year, that in a particular case an awarding organisation might follow its policy in a manner which is not consistent with the conditions. For example, an awarding organisation might follow its policy to consider an appeal on procedural grounds but the decision-maker might overlook a clear procedural error by the centre. In such a case the EPRS might consider that, although the process was followed faithfully, the process was not effective in that instance, and the awarding organisation did not secure the outcomes required by Condition GQAA4.1(b)(i). Such a decision would not mean the process was itself flawed, but that the execution of the process in the particular case had been ineffective.

Against this background, and noting that the EPRS (like the Exams Appeal Board which preceded it) has always considered both procedural grounds and compliance grounds, we do not consider that the EPRS should be prevented this year from determining that an awarding organisation did not secure the outcomes required by the GQAA conditions in a particular case.

One awarding organisation further argued that it was inappropriate for EPRS to consider issues of compliance, because those matters were more properly considered through Ofqual’s enforcement process. We consider this argument, which if it were relevant now would be as relevant in a normal year, to be out of scope for this consultation.

Broader questions of fairness

Two awarding organisations reflected that an application to the EPRS should not be an opportunity for a learner to argue, or for the EPRS to evaluate, whether a Teacher Assessed Grade (or by extension a replacement grade issued following an appeal) was or was not ‘fair’.

We agree. The EPRS might evaluate how an awarding organisation has reached any judgements it makes when it considers an appeal, but it will not seek to evaluate the judgements themselves nor to substitute its own judgement. Questions of academic judgement, as well as whether an incorrect result should be corrected, are matters of discretion and judgement which awarding organisations are uniquely placed to make.

Correcting results

One awarding organisation raised a concern about the possibility that the EPRS will anticipate an awarding organisation creating and retaining records to show that the appeal decision-maker had regard to our Guidance on Correcting Incorrect Results (the Guidance) before determining that a result should be corrected.

In practice, they suggest, decision-makers will not routinely have access to the information which would be necessary properly to consider all of the factors specified in the Guidance. In the majority of cases, they argue, particularly where the grade change reflects a decision that the original grade was unreasonable, the Guidance will not be relevant because the learner will have given consent to the appeal – knowing the grade could go down as well as up.

Awarding organisations must comply with the conditions, which specify that they must have regard to the Guidance as part of deciding whether to correct a result which is incorrect. A consultation on the Terms of Reference for EPRS cannot affect the meaning of the conditions.

However, the Guidance sets out a number of factors which an awarding organisation should consider when deciding whether it is appropriate to correct a result. It is for the awarding organisation to decide what weight to give to those factors in each case. Moreover, the Guidance does not alter the nature of the final decision on whether it is appropriate to correct a result, which is essentially a matter of judgement for the awarding organisation. In an appropriate case, an awarding organisation might decide a particular decision is indicated by a factor which is not contemplated in the Guidance but to which it nonetheless considers it is appropriate to attach significant weight.

We recognise that an awarding organisation might legitimately take the view that the combination of informed consent from the learner seeking the appeal, and an expert view that the grade was an unreasonable exercise of academic judgment, would generally be given so much weight that it would outweigh any argument against correcting the result which might arise from specific consideration of the Guidance. In cases of this nature it is likely to be sufficient for the process to require the decision-maker to consider whether there was anything obviously exceptional about the particular case, and to record their decision.

However, we think the weight to be given to consent and expert opinion will lessen as time passes between the application for appeal – and the learner giving consent – and the determination of that appeal, particularly where a result may change to a learner’s disadvantage.

As such, where the decision to change a grade reflects an unreasonable academic judgement, the learner affected by that decision consented to the appeal and where the appeal is determined promptly, we consider it will be sufficient for the EPRS to see records only that the decision-maker considered whether there was anything obviously exceptional about the case, which might indicate that a departure from the default position was appropriate.

We agree with the awarding organisation that different considerations will apply where the outcome of one learner’s appeal affects other learners, who have not consented and who were not aware of the appeal. We also think the factors in the Guidance might have greater weight in an appeal on procedural grounds, particularly where correcting a procedural error leads to a small adjustment which, by reason of the thresholds used by the centre, would have a detrimental impact on the learner’s result. In cases such as these, we anticipate an awarding organisation would be able to show that specific consideration was given to the Guidance.

Finally, an awarding organisation noted that it is unclear how the EPRS might approach a case in which, having identified an error, an awarding organisation allowed an original grade to stand because there was insufficient evidence to form a reliable view as to an alternative grade. We think for present purposes it is sufficient that the EPRS will be aware that such circumstances might arise.

Conclusion – the role of the EPRS

For the reasons given in this section, we have decided that the Terms of Reference for summer 2021 should describe the role of the EPRS as proposed.

Who may apply

In a normal year, applications to the EPRS must be made by a centre on behalf of one or more learners. The only exception is for private candidates, who may apply directly. This mirrors the arrangements awarding organisations have in place for the reviews and appeals process in a normal year.

This summer, although centres will submit appeals to awarding organisations, it is the learner and not the centre who has the final decision whether to appeal. A centre must submit an appeal to the awarding organisation where the learner requests this.

We proposed to allow learners to apply directly to the EPRS this summer, as this was the most effective way to reflect the arrangements for submitting appeals. This is because (unlike awarding organisations) we cannot impose requirements on centres.

Five respondents agreed with our proposal and one did not respond to this question. One respondent argued that we should permit applications from learners only where they were not known to the centre and in all other cases the centre should make the application.

We remain of the view that it is appropriate, for qualifications in scope of the EPRS, that learners should have equivalent access to the EPRS as they will to the awarding organisations’ appeals process. We remain of the view that, because we cannot require centres to submit EPRS applications where a learner wants this, allowing learners to have direct access to the EPRS is the most appropriate way to achieve this aim. We recognise that in some cases centres may wish to support learners, and we will make sure the EPRS process allows this.

Finally, in this connection, we recognise the need to make sure the EPRS process is accessible to learners. We are reviewing our systems to this end, and will publish a student guide to the EPRS before results day.

We have decided to allow learners to apply directly to the EPRS, as proposed. We have amended the Terms of Reference to make clear that, instead of allowing a Centre to make an application on their behalf, or making the application themselves, a Learner may ask a parent, carer or other representative to make an application to the EPRS on their behalf.

Equality Impact Assessment

Respondents commented on the need for clear communication about the role - and limitations of - the EPRS, as well as clear explanation of how to make an application. No respondent identified any impact on persons who share protected characteristics arising from our proposals.

Regulatory Impact Assessment

Respondents reflected that the regulatory impact associated with the EPRS depends on the way in which individual cases are managed. This in turn depends on the nature of the applications which are made. We agree that it will be useful this year for officials involved in EPRS casework to understand in advance the processes the awarding organisations have in place for appeals and, so far as possible, to agree standard materials which will be provided in each case.