Pearson’s Undertaking and Action Plan regarding reviews of marking arrangements (June 2020)

Question 1

Pearson’s Undertaking and Action Plan regarding Reviews of Marking arrangements (June 2020)

Accepted Answer

Failure to comply with Conditions

1. Pearson admits that in respect of its GCSE and GCE qualifications in 2016, 2017, 2018 and 2019, it failed to ensure that its arrangements for reviews of marking complied with GCE / GCSE Condition 17.6(b)^{[footnote 1]}, which provides that:

…an Assessor who was previously involved in the marking of a task in an assessment in respect of a Learner must not be involved in a review of marking in respect of that task…

2. This failure to comply with the Conditions affected a total of 46,797 reviews of marking.

3. Pearson charges Centres a fee for its review of marking service, if the review does not result in a grade change. The number of affected reviews where a fee was charged was 36,807. The fees associated with those reviews totalled £320,510.

Undertaking

4. Pearson hereby undertakes to:

(a) compensate all Centres with learners whose reviews of marking were not undertaken by a fresh examiner in 2016, 2017, 2018 and 2019, where a fee was charged. This compensation will be calculated based on the fees applicable at the time for the particular services requested and performed by Pearson and will be proportionate to the extent of non-compliance.
(b) pay this compensation to affected Centres in the form of credit notes, on a date to be agreed with Ofqual. The credit notes will be redeemable against the full range of Pearson’s services, with choices being entirely in the hands of Centres. Pearson will ensure that the accompanying communications do not convey any promotional content or seek to advantage Pearson in any way.
(c) perform the actions set out in Annex 1^{[footnote 2]} within the timeframe set out therein in order to ensure compliance with the Conditions for the next exam series and beyond.

Declaration

5. In giving this Undertaking, I acknowledge on behalf of Pearson that:

(a) Ofqual will publish this Undertaking (on a date to be agreed with Pearson); and
(b) Ofqual may take regulatory action in respect of any failure to comply with the terms of this Undertaking which may include the issue of a Direction, the imposition of a Monetary Penalty, Costs, or any other such action as it deems appropriate; and
(c) Ofqual may take other regulatory action in respect of the breaches set out in this Undertaking and any other breaches that may be determined by Ofqual’s Enforcement Committee in due course. Any regulatory action will be taken in accordance with Ofqual’s ‘Taking Regulatory Action’ policy (2012).

Timing

6. Ofqual and Pearson have agreed that:

(a) Pearson’s payment of compensation to Centres (as set out in paragraph 4(a) of this Undertaking); and
(b) Ofqual’s consideration of any regulatory action in respect of the breaches set out in this Undertaking (as set out at paragraph 5(c) of this Undertaking)

will be postponed until further notice to enable both parties to prioritise critical work associated with the Coronavirus pandemic.

Name: Derek Richardson

Responsible Officer, Pearson

Date: 8 June 2020

Question 2

Pearson Action Plan for PRS compliance

Accepted Answer

Action plan

We outline here our high level action plan, with associated estimated timelines, to achieve compliance with the condition, GCSE/GCE Condition 17.6(b), in relation to the allocation of reviewing examiners to review of marking cases. The action plan is split into two linked sections:

the high level action plan for achieving compliance by the next examination series
a longer term action plan for series following the next examination series

Proposed short term solution

The following section outlines our action plan for securing compliance with GCSE/GCE Condition 17.6(b) by the time of the next post results period following the autumn examination series.

The action plan provides details of the solution that is proposed and the IT development and changes in our way of working that will be required to achieve it. Timelines are given later on in this document.

Summary of the short term solution

There are 2 strands to the work to achieve compliance for the next examination series:

availability of examiner resource to complete reviews of marking, and increasing the likelihood that sufficient compliant reviewers can be recruited from the pool of examiners used in the summer, and;
updates to systems in order to ensure that any potential non-compliant allocation of reviews of marking are not allocated, and creating an exception system whereby scripts which cannot be allocated to an examiner who has not marked any part of it before are allocated to two, or more, different examiners who then review the parts of it that they have not marked before.

Making better use of the examining workforce

Alteration of the process for reviewing the auto generated forecast

Prior to each examination series, a forecast of examiner demand is created automatically. The forecast is based on an analysis of the number of expected entries for each component, the number of discrete items within the assessment for each component, the type of examiner that is needed to mark the items within the assessment, and a reasonable workload for each examiner in the period available for marking. In this way, an estimate is generated of the number of examiners required and the number of items each of the examiners will mark.

When the forecast is generated, subject teams are required to review it to ensure that there are enough examiners to complete the work, that examiner workload is reasonable, and that the work can be completed in the time available. The forecast is updated, if necessary, following this review.

Subject teams will now be required, in addition to the above, to analyse the forecast to ensure there is a sufficient pool of review examiners which are available during the Post Results Services window because of the condition that reviewers must not have had involvement with the original assessment of the candidate work, even when they had only marked a proportion of the candidate work originally. This work had already been completed for the Sumner 2020 examination series prior to its cancellation. A similar process will be followed for the next GQ examination series.

Criteria for reviewing the forecast through the PRS lens

Subject teams will be briefed, ahead of the next available examination series, to review the forecast of examiner demand to ensure that as far as possible the pool of reviewers is sufficient to ensure that item marked scripts can be reviewed in a compliant manner.

Subject teams will give particular attention to the following areas when reviewing and updating the forecast. Additional information, such as the volume of items which are forecast to be marked by senior AA s, and the volumes of PRS requests in previous years, will also be considered.

Chief and PE roles are covered by the same person - The Chief Examiner role is usually filled from the ranks of Principal Examiners. As such, for at least one component in each qualification, the Chief and Principal Examiner will be the same person. In such cases, the PRS forecast will be amended so that the PE from another paper can review PRS cases.

Sole marked - the PE will not be eligible to conduct PRS cases; teams will recruit the Chief or Chairs of Examiners to complete ROMs and/or recruit another examiner and update the re- forecast. Consideration will have to be given to how additional examiners or other senior examiners are standardised when no standardisation meeting has been scheduled.

Smaller entry, but not sole marked - there is a chance that for some ROM cases, neither the PE or the examiner(s) on the paper are eligible to conduct them as between them all, they marked some of the original script. Consideration will be given to recruiting additional AAs to mitigate this risk and/or reduce the allocation of the PE so that the chance of them being able to complete ROM cases increases. When taking this action, teams must balance the risk to PRS with the need for PEs, for example, to complete enough first line marking that they can meaningfully participate in awarding.

Small entry where the PE/TL s would usually mark the graduate items - the PE, in this case, will not be able to mark any ROM cases because they marked all of the graduate items. Recruit a graduate marker or increase the number of examiners so that they can share the graduate items between them.

Larger entry papers - there is a decreased risk that ROM cases are not able to be allocated to the TLs, PAs and PEs and above, but the risk remains, particularly if there are large scale mark reviews such as borderlining. Reduce the allocation size of the senior examiners in order to increase the chance that they are able to conduct the majority of ROMs and also ensure there are sufficient graduate markers contracted to cover the total graduate allocation. When thinking about a reduced allocation size, balance the risk to PRS with the need for PEs, for example, to complete enough first line marking that they can meaningfully participate in awarding. When considering whether to conduct borderlining, decide whether the benefit of borderlining is greater than the risk to delivery of ROM cases.

Recruitment of additional reviewers

In some cases, it will be necessary, in addition to the steps outlined above, to increase the size of the examining pool so that sufficient experienced reviews are available for the purposes of PRS reviews of marking.

The forecast demand will be adjusted and additional reviews recruited through business as usual processes. In shortage areas, such as lesser taught languages in Modern Foreign Languages, where there is a scarcity of personnel with sufficient skills and experience, a combination of the measures outlined in this action plan will have to be enacted.

Alteration to the PRS forecast

As per normal business processes, when an examination series is nearing completion, a further forecast is generated which automatically estimates the demand for the reviewers needed to complete review of marking cases. The forecast, at present, is based on an analysis of the percentage of entry of the volume of reviews of marking from the previous equivalent exam, which is applied to the present series to generate the estimated demand for reviewers.

The PRS forecast represents another opportunity in which to ensure that the pool of reviewers is sufficient to ensure that reviews are conducted compliantly. The forecast data will therefore be amended to display the total proportion of items/scripts which the current pool of senior examiners and other nominated reviewers had marked originally so that, where needed, the pool can be added to.

Contingency exception process

There will remain a need to ensure there is a contingency arrangement in place should the actions taken above fail to completely mitigate the risk of there being insufficient examiners available in order to complete reviews of marking in a compliant and timely manner. As such, an exception process is being developed and implemented whereby reviews of marking can be reviewed by item. This exception process, and the adaptations to technology that are required for it, is detailed below.

Recruitment: Summary of actions to be taken against the conditions to ensure compliance

The table below summarises the actions that will be taken to improve recruitment practices to ensure that a sufficient pool of reviews will be available during the post results window following the autumn series.

Condition of Recognition

GCSE (9‐ 1) 17.6; and GCE (reform) 17.6 The arrangements must provide that –

(a) all reviews of marking of Marked Assessment Material will be carried out by Assessors who have appropriate competence and who have no personal interest in the outcome of the review being carried out,

Pearson actions	Progress to date
We will ensure that sufficient reviewers with appropriate competence and no personal interest in the outcome of the review being carried out are recruited by training staff to review the exam series AA demand forecast to ensure that sufficient AAs are recruited for ROMs	This was completed for the cancelled summer 2020 exam series. This will now be planned for to be repeated in November of 2020 for the autumn exam series
We will ensure that sufficient reviewers with appropriate competence and no personal interest in the outcome of the review being carried out are recruited by training staff to review the PRS forecast so that in the case of small or sole marked papers, additional reviewers are recruited, or drawn from the same team of senior examiners	This will be put in plan for November so that staff are trained to review the forecast and recruit additional AAs
We will ensure that sufficient reviewers with appropriate competence and no personal interest in the outcome of the review being carried out are recruited by recruiting sufficient examiners for larger papers	Teams will be trained to recruit sufficient AAs to complete reviews of marking ahead of the autumn series
We will ensure that sufficient reviewers with appropriate competence and no personal interest in the outcome of the review being carried out are recruited by ensuring that examiners have no personal interest in the review through current recruitment practices	ROM reviewers will be recruited through the current process which ensures they have no personal interest
We will ensure that sufficient reviewers with appropriate competence and no personal interest in the outcome of the review being carried out are recruited by conducting compulsory training and standardisation for reviewers prior to completing ROMs	Reviewers will be trained and standardised in November. Reviewers will not be able to conduct ROMs until standardisation has been satisfactorily been completed

Condition of Recognition

GCSE (9‐ 1) 17.6; and GCE (reform) 17.6 The arrangements must provide that –

(a) an Assessor who was previously involved in the marking of a task in an assessment in respect of a Learner must not be involved in a review of marking of the Learner’s Marked Assessment Material in respect of that task

Pearson actions	Progress to date
We will ensure that reviews of marking are conducted by assessors who were not involved in the original marking by recruiting additional examiners so that the pool is sufficient to ensure assessors do not conduct ROMs on all or parts of scripts they had assessed before	As above, forecasts will be reviewed and additional AAs recruited
We will ensure that reviews of marking are conducted by assessors who were not involved in the original marking by ensuring there is management oversight of reviewer recruitment plans and forecast	Additional reporting will be established for the autumn series onward so that resource plans are reviewed at regular Closure and Completion meetings prior to and during the post results services window
We will ensure that reviews of marking are conducted by assessors who were not involved in the original marking by ensuring that ROM allocations are not made to original assessors	The actions in relation to this are detailed more thoroughly below. Reviewers will be trained so that they understand that they must only complete ROMs on scripts, or parts of scripts, they have not assessed before. Reviewers will be further trained on the processes and systems that ensure they cannot do this.

Allocating reviews when all examiners have marked some part of the original script

We expect that the majority of review of marking (ROM) cases will be allocated as normal. For the cases that cannot be allocated as normal as there are no review markers available who have not marked some or all of the script previously, an exception process will be deployed. The exception process, which is under development, will allow the review to be conducted by 2 or more reviewers in order that each of the review examiners is only making a judgement on the parts of the script they haven’t seen.

The purpose of handling the script differently is to allow the script to be first reviewed by an examiner who has marked the least number of items originally, and once the examiner has submitted marks for the ROM allocation, for our ROM systems to keep the case in quarantine, with the aim of getting the non-compliant items (items originally marked by the reviewing examiner) reviewed by another examiner. The outcome of the review of marking would then be reviewed, if necessary, and published to the centre after the review on the non-compliant items has been completed by another examiner.

Handling these exception scripts differently will involve making some significant changes to our ROM systems to allow for 2 allocation scenarios:

Scenario 1 - Business as usual. The ROM system attempts to find a suitable examiner who has not been assigned as a marker for all items on the script at the time of allocation, and finds a suitable examiner. The review is completed as normal.

Scenario 2 - Exception process. The ROM system fails to find a suitable examiner who has not been assigned as a marker for all items on the script at the time of allocation. The following steps will be executed.

the request is not auto allocated and need to be managed as an exception script for manual examiner allocation
processing teams will manually identify a suitable examiner to complete the review based on which of them has marked the least amount of items originally, taking into account the seniority level of the original examiner
processing teams will manually allocate the script to the identified examiner using the ROM system
the system will display a warning to the operator that they are assigning a non-compliant allocation
the request is allocated to a review examiner
the system will record that the allocation is a non-compliant allocation
the system will flag the allocation so that it knows to handle the script differently
The first review examiner acknowledges the allocation, completes the review and submits a new paper mark
the new mark submitted by the review examiner will not be published and the mark will be kept in a quarantine area for another review to be completed on the items which the first reviewer had marked originally.
once the additional review on the item(s) which the first reviewer had marked originally has been completed, assessment teams will review and publish the ROM mark from a quarantine area which will update the paper mark for the review cases directly
the new mark is graded and the outcome of the review will be issued for the request if all criteria to generate an outcome has been met

In this way, as described above, the items which the first reviewer had marked originally will be reviewed by a second reviewer who had no involvement in the original assessment of the items. The scores given by the second reviewer for these items will be aggregated with those of the first so that each item on the review script is reviewed by an examiner who had not seen the items before.

Timelines for short term solution - Workaround for Item Marking

Reviewing examiner availability and recruitment

Status	Task	Expected completion date in 2020*
Complete	Review and adjustment of Summer 2020 examination series forecast	Closed
Not due	Review and adjustment of the forecast for the next examination series	Sept 10
Not due	Recruitment of additional examiners for pinch points	Sept 10 - Oct 05
Not due	Review of the PRS forecast for the next examination series	Nov 21
Not due	Further recruitment activity for ‘pinch points’	Nov 21 - Dec 16

*At the time of writing the operational calendar for the autumn series was still being reviewed. The dates given here are therefore provisional and subject to change.

Systems and Process updates

System	Timeframe for scoping, development, testing and implementation	Expected Completion date in 2020*
iSeries	Work to take a total of 13 weeks to complete. Delivery of the updates to the iSeries will be aligned to complete at the same time as the other systems, because of the interdependencies across the systems.	Nov 30
EOL	Work to take a total of 24 ‘person weeks’ (total of 120 hours). Delivery of the updates to the EOL and Online Marking will be aligned to complete at the same time as the other systems, because of the interdependencies across the systems.	Nov 30
Online Marking	Work to take a total of 24 ‘person weeks’ (total of 120 hours). Delivery of the updates to the EOL and Online Marking will be aligned to complete at the same time as the other systems, because of the interdependencies across the systems.	Nov 30

*At the time of writing the teams involved in this work are also heavily involved in working on different solutions for summer grading and the autumn series due to Covid-19. The dates given here are therefore provisional and subject to change.

Conducting reviews of marking: summary of actions to be taken against the conditions to ensure compliance

In addition to the recruitment actions to ensure compliance that are identified, above, the following actions will be taken to ensure that reviews of marking are carried out in accordance with the conditions.

Condition of Recognition

GCSE (9‐ 1) 17.6; and GCE (reform) 17.6 The arrangements must provide that –

(b) an Assessor who was previously involved in the marking of a task in an assessment in respect of a Learner must not be involved in a review of marking of the Learner’s Marked Assessment Material in respect of that task,

Pearson actions	Progress to date
We will ensure that reviews of marking are allocated to reviewers that had no prior involvement in the marking of the task by updating our systems so that auto allocations to non- compliant reviewers are blocked	This work has been scoped and is in plan to commence in the next three weeks, with full delivery prior to the autumn series, by no later than Nov 30
We will ensure that reviews of marking are allocated to reviewers that had no prior involvement in the marking of the task by providing training to our processing staff about the action taken when scripts need to be allocation by exception to more than one reviewer	Processing teams have been fully briefed on the requirements of the conditions and will be trained prior to the next examination series
We will ensure that reviews of marking are allocated to reviewers that had no prior involvement in the marking of the task by updating our systems so that manual allocation to more than one reviewer is possible by exception	This work has been scoped and is in plan to commence in the next three weeks, with full delivery prior to the autumn series, by no later than Nov 30
We will ensure that reviews of marking are allocated to reviewers that had no prior involvement in the marking of the task by training our assessors to understand the exception ROM process and the systems involved with it	Prior to each series reviewing assessors are trained and must pass standardisation. The training will be adapted this autumn so that it makes clear to them the requirements of the conditions and the processes that must be followed.
We will ensure that reviews of marking are allocated to reviewers that had no prior involvement in the marking of the task by training assessment staff about the new process and the requirement for them to review and approve candidate outcomes	Assessment teams have been fully briefed on the requirements of the conditions and will be trained prior to the next examination series
We will ensure that reviews of marking are allocated to reviewers that had no prior involvement in the marking of the task by adapting systems so that non-compliant reviews that are made by mistake are recorded and escalated for remedial action	This work has been scoped and is in plan to commence in the next three weeks, with full delivery prior to the autumn series, by no later than Nov 30
We will ensure that reviews of marking are allocated to reviewers that had no prior involvement in the marking of the task by monitoring the allocation of reviews to ensure compliance and to escalate allocations which are not compliant	Additional reporting will be established for the autumn series onward so that resource plans are reviewed at regular Closure and Completion meetings prior to and during the post results services window
We will ensure that reviews of marking are allocated to reviewers that had no prior involvement in the marking of the task by monitoring the completion of reviews which are conducted by item to ensure that assessors review the parts of scripts they have not assessed before	Additional reporting will be established for the autumn series onward so that resource plans are reviewed at regular Closure and Completion meetings prior to and during the post results services window

Question 3

Long term solution

Accepted Answer

Summary of long term solution

The process described above to manage reviews of marking which cannot be allocated to a compliant reviewer will be labour and time intensive. As such we are scoping the development needed to the pre existing functionality within our ePen marking platform with a view to enabling and testing it for use in a future live series.

The pre existing system allows assessments which were previously marked onscreen to be reviewed by item. Following centre requests for reviews of marking though our post results services, each script will be broken down into its constituent parts and fed into the system.

Reviewing examiners who have been successfully standardised will access the item reviewing system and will be able to view the pool of items that are available to be reviewed for any component they have been contracted for. Reviewers will only be able to view and mark items for which they had not previously had any involvement in assessing.

When the review of each item is completed, the marks will be aggregated and the outcome communicated to the requesting centre.

This system is currently configured so that reviewing examiners can only award a score which is equal to or greater than the original score. Development work will be required to update the system so that all score points are available to the reviewing examiner. Furthermore, other systems will need to be adapted so that they are compatible.

Timelines

At present, we provisionally plan full implementation of item reviewing for summer 2021, though this will be kept under review to ensure systems and processes can be developed and tested in time to minimise the risk of a major change to the manner in which we operate reviews of marking. It is not possible, given the rapidly changing nature of the autumn examination series, to commit to an earlier series.

The solution we present above for the manual allocation of review cases in the next examination series will be kept live as a contingency for future series when the item review system is launched.

The item-review system, the adaptations to it and other systems, and the changes to processes are still in the process of being scoped and development work scheduled. At present, therefore, we do not have firm estimates for the timelines for development and implementation.

Long term systems updates: summary of actions to be taken against the conditions to ensure compliance

In addition to the actions described above in relation to the recruitment and completion of ROMs, the following systems updates will be made to ensure ongoing compliance with GCSE/GCE condition 17.6 in future series.

Condition of Recognition

GCSE (9‐ 1) 17.6; and GCE (reform) 17.6 The arrangements must provide that –

(a) all reviews of marking of Marked Assessment Material will be carried out by Assessors who have appropriate competence and who have no personal personal interest in the outcome of the review being carried out,

Pearson actions	Progress to date
We will ensure that sufficient reviewers with appropriate competence and no personal interest in the outcome of the review being carried out are recruited by updating our processes and systems for generating forecasts	Current forecasts for in-series item marking will be adapted so that item reviewing forecasts can be generated
We will ensure that sufficient reviewers with appropriate competence and no personal interest in the outcome of the review being carried out are recruited by briefing staff on the forecast and systems for item reviewing	All teams will be trained on the requirements of the item forecast for ROMs on the item review system
We will ensure that sufficient reviewers with appropriate competence and no personal interest in the outcome of the review being carried out are recruited by recruiting sufficient review assessors to conduct ROMs by item	Current recruitment practices as for an exam series will be adapted for recruitment of reviewers for item reviewing
We will ensure that sufficient reviewers with appropriate competence and no personal interest in the outcome of the review being carried out are recruited by briefing the reviewing assessors on the requirements for item reviewing prior to the series in which it is implemented	All reviewers must complete mandatory standardisation and training prior to completing ROMs. The training will be adapted to include training on the usage of the new system.

Condition of Recognition

GCSE (9‐ 1) 17.6; and GCE (reform) 17.6 The arrangements must provide that –

(a) - an Assessor who was previously involved in the marking of a task in an assessment in respect of a Learner must not be involved in a review of marking of the Learner’s Marked Assessment Material in respect of that task,

Pearson actions	Progress to date
We will ensure that reviews of marking are allocated to reviewers that had no prior involvement in the marking of the task by implementing item reviewing so that the original assessor will not be able to view or review items they had marked originally	Systems and processes are being scoped for implementation in a future exam series

These are Qualification Level Conditions and Requirements that were in force in 2019 for GCE and GCSE (9-1) qualifications. There were a variety of Conditions in force during 2016, 2017 and 2018, depending on whether the affected qualifications were legacy or reformed qualifications. For the purposes of this undertaking, there is no material difference between the relevant Conditions. ↩
Pearson Action Plan for PRS compliance, 28 April 2020 ↩

Question 4

Pearson Action Plan for PRS compliance (updated August 2021)

Accepted Answer

Action plan

We outline here our high level action plan, with associated estimated timelines, to achieve compliance with the condition, GCSE/GCE Condition 17.6(b), in relation to the allocation of reviewing examiners to review of marking cases. The action plan is split into two linked sections:

the high level action plan for achieving compliance by the next examination series
a longer term action plan for series following the next examination series

Proposed short term solution

The following section outlines our action plan for securing compliance with GCSE/GCE Condition 17.6(b) by the time of the next post results period following the autumn examination series.

The action plan provides details of the solution that is proposed and the IT development and changes in our way of working that will be required to achieve it. Timelines are given later on in this document.

Summary of the short term solution

There are 2 strands to the work to achieve compliance for the next examination series:

(a) availability of examiner resource to complete reviews of marking, and increasing the likelihood that sufficient compliant reviewers can be recruited from the pool of examiners used in the summer, and;
(b) updates to systems in order to ensure that any potential non-compliant allocation of reviews of marking are not allocated, and creating an exception system whereby scripts which cannot be allocated to an examiner who has not marked any part of it before are allocated to two, or more, different examiners who then review the parts of it that they have not marked before.

Making better use of the examining workforce

Alteration of the process for reviewing the auto generated forecast

Prior to each examination series, a forecast of examiner demand is created automatically. The forecast is based on an analysis of the number of expected entries for each component, the number of discrete items within the assessment for each component, the type of examiner that is needed to mark the items within the assessment, and a reasonable workload for each examinerin the period available for marking. In this way, an estimate is generated of the number of examiners required and the number of items each of the examiners will mark.

When the forecast is generated, subject teams are required to review it to ensure that there are enough examiners to complete the work, that examiner workload is reasonable, and that the work can be completed in the time available. The forecast is updated, if necessary, following this review.

Subject teams will now be required, in addition to the above, to analyse the forecast to ensure there is a sufficient pool of review examiners which are available during the Post Results Services window because of the condition that reviewers must not have had involvement with the original assessment of the candidate work, even when they had only marked a proportion of the candidate work originally. This work had already been completed for the Summer 2020 examination series prior to its cancellation. A similar process will be followed for the next GQ examination series.

Criteria for reviewing the forecast through the PRS lens

Subject teams will be briefed, ahead of the next available examination series, to review the forecast of examiner demand to ensure that as far as possible the pool of reviewers is sufficient to ensure that item marked scripts can be reviewed in a compliant manner.

Subject teams will give particular attention to the following areas when reviewing and updating the forecast. Additional information, such as the volume of items which are forecast to be marked by senior AAs, and the volumes of PRS requests in previous years, will also be considered.

Chief and PE roles are covered by the same person - The Chief Examiner role is usually filled from the ranks of Principal Examiners. As such, for at least one component in each qualification, the Chief and Principal Examiner will be the same person. In such cases, the PRS forecast will be amended so that the PE from another paper can review PRS cases.

Sole marked - the PE will not be eligible to conduct PRS cases; teams will recruit the Chief or Chairs of Examiners to complete ROMs and/or recruit another examiner and update the re- forecast. Consideration will have to be given to how additional examiners or other senior examiners are standardised when no standardisation meeting has been scheduled.

Smaller entry, but not sole marked - there is a chance that for some ROM cases, neither the PE or the examiner(s) on the paper are eligible to conduct them as between them all, they marked some of the original script. Consideration will be given to recruiting additional AAs to mitigate this risk and/or reduce the allocation of the PE so that the chance of them being able to complete ROM cases increases. When taking this action, teams must balance the risk to PRS with the need for PEs, for example, to complete enough first line marking that they can meaningfully participate in awarding.

Small entry where the PE/TLs would usually mark the graduate items - the PE, in this case, will not be able to mark any ROM cases because they marked all of the graduate items. Recruit a graduate marker or increase the number of examiners so that they can share the graduate items between them.

Larger entry papers - there is a decreased risk that ROM cases are not able to be allocated to the TLs, PAs and PEs and above, but the risk remains, particularly if there are large scale mark reviews such as borderlining. Reduce the allocation size of the senior examiners in order to increase the chance that they are able to conduct the majority of ROMs and also ensure there are sufficient graduate markers contracted to cover the total graduate allocation. When thinking about a reduced allocation size, balance the risk to PRS with the need for PEs, for example, to complete enough first line marking that they can meaningfully participate in awarding. When considering whether to conduct borderlining, decide whether the benefit of borderlining is greater than the risk to delivery of ROM cases.

Recruitment of additional reviewers

In some cases, it will be necessary, in addition to the steps outlined above, to increase the size of the examining pool so that sufficient experienced reviews are available for the purposes of PRS reviews of marking.

The forecast demand will be adjusted and additional reviews recruited through business as usual processes. In shortage areas, such as lesser taught languages in Modern Foreign Languages, where there is a scarcity of personnel with sufficient skills and experience, a combination of the measures outlined in this action plan will have to be enacted.

Alteration to the PRS forecast

As per normal business processes, when an examination series is nearing completion, a further forecast is generated which automatically estimates the demand for the reviewers needed to complete review of marking cases. The forecast, at present, is based on an analysis of the percentage of entry of the volume of reviews of marking from the previous equivalent exam, which is applied to the present series to generate the estimated demand for reviewers.

The PRS forecast represents another opportunity in which to ensure that the pool of reviewers is sufficient to ensure that reviews are conducted compliantly. The forecast data will therefore be amended to display the total proportion of items/scripts which the current pool of senior examiners and other nominated reviewers had marked originally so that, where needed, the pool can be added to.

Contingency exception process

There will remain a need to ensure there is a contingency arrangement in place should the actions taken above fail to completely mitigate the risk of there being insufficient examiners available in order to complete reviews of marking in a compliant and timely manner. As such, an exception process is being developed and implemented whereby reviews of marking can be reviewed by item. This exception process, and the adaptations to technology that are required for it, is detailed below.

Recruitment: Summary of actions to be taken against the conditions to ensure compliance

The table below summarises the actions that will be taken to improve recruitment practices to ensure that a sufficient pool of reviews will be available during the post results window following the autumn series.

Condition of recognition

GCSE (9‐ 1) 17.6; and GCE (reform) 17.6 The arrangements must provide that –

(a) all reviews of marking of Marked Assessment Material will be carried out by Assessors who have appropriate competence and who have no personal interest in the outcome of the review being carried out,

Pearson actions	Progress to date
We will ensure that sufficient reviewers with appropriate competence and no personal interest in the outcome of the review being carried out are recruited by training staff to review the exam series AA demand forecast to ensure that sufficient AAs are recruited for ROMs	This was implemented for the Autumn 2020 series and will be put in place again for the Autumn 2021 series.
We will ensure that sufficient reviewers with appropriate competence and no personal interest in the outcome of the review being carried out are recruited by training staff to review the PRS forecast so that in the case of small or sole marked papers, additional reviewers are recruited, or drawn from the same team of senior examiners	This was implemented for the Autumn 2020 series and will be put in place again for the Autumn 2021 series.
We will ensure that sufficient reviewers with appropriate competence and no personal interest in the outcome of the review being carried out are recruited by recruiting sufficient examiners for larger papers	A sufficient number of examiners were recruited for larger papers as required y the additional PRS forecast
We will ensure that sufficient reviewers with appropriate competence and no personal interest in the outcome of the review being carried out are recruited by ensuring that examiners have no personal interest in the review through current recruitment practices	ROM reviewers were recruited through standard practices to ensure no examiners allocated a review had a personal interest in the review
We will ensure that sufficient reviewers with appropriate competence and no personal interest in the outcome of the review being carried out are recruited by conducting compulsory training and standardisation for reviewers prior to completing ROMs

Condition of recognition

GCSE (9‐ 1) 17.6; and GCE (reform) 17.6 The arrangements must provide that –

(b)an Assessor who was previously involved in the marking of a task in an assessment in respect of a Learner must not be involved ina review of marking of the Learner’s Marked Assessment Material in respect of that task,

Pearson actions	Progress to date
We will ensure that reviews of marking are conducted by assessors who were not involved in the original marking by recruiting additional examiners so that the pool is sufficient to ensure assessors do not conduct ROMs on all or parts of scripts they had assessed before	As above, forecasts were reviewed and additional AAs recruited as required
We will ensure that reviews of marking are conducted by assessors who were not involved in the original marking by ensuring there is management oversight of reviewer recruitment plans and forecast	Additional reporting will be established for the autumn series onward so that resource plans are reviewed at regular Closure and Completion meetings prior to and during the post results services window
We will ensure that reviews of marking are conducted by assessors who were not involved in the original marking by ensuring that ROM allocations are not made to original assessors	The actions in relation to this are detailed more thoroughly below. Reviewers were trained for the Autumn 2020 exam series so that they understand that they must only complete ROMs on scripts, or parts of scripts, they have not assessed before. Reviewers also received further training on the processes and systems that ensure they cannot do this.

Allocating reviews when all examiners have marked some part of the original script

We expect that the majority of review of marking (ROM) cases will be allocated as normal. For the cases that cannot be allocated as normal as there are no review markers available who have not marked some or all of the script previously, an exception process will be deployed. The exception process, which is under development, will allow the review to be conducted by two or more reviewers in order that each of the review examiners is only making a judgement on the parts of the script they haven’t seen.

The purpose of handling the script differently is to allow the script to be first reviewed by an

examiner who has marked the least number of items originally, and once the examiner has submitted marks for the ROM allocation, for our ROM systems to keep the case in quarantine, with the aim of getting the non-compliant items (items originally marked by the reviewing examiner) reviewed by another examiner. The outcome of the review of marking would then be reviewed, if necessary, and published to the centre after the review on the non-compliant items has been completed by another examiner.

Handling these exception scripts differently will involve making some significant changes to our ROM systems to allow for two allocation scenarios.

Scenario 1 - Business as usual. The ROM system attempts to find a suitable examiner who has not been assigned as a marker for all items on the script at the time of allocation, and finds a suitable examiner. The review is completed as normal.

Scenario 2 - Exception process. The ROM system fails to find a suitable examiner who has not been assigned as a marker for all items on the script at the time of allocation. The following steps will be executed.

the request is not auto allocated and need to be managed as an exception script for manual examiner allocation
processing teams will manually identify a suitable examiner to complete the review based on which of them has marked the least amount of items originally, taking into account the seniority level of the original examiner
processing teams will manually allocate the script to the identified examiner using the ROM system
the system will display a warning to the operator that they are assigning a non-compliant allocation
the request is allocated to a review examiner
the system will record that the allocation is a non-compliant allocation
the system will flag the allocation so that it knows to handle the script differently
The first review examiner acknowledges the allocation, completes the review and submits a new paper mark
the new mark submitted by the review examiner will not be published and the mark will be kept in a quarantine area for another review to be completed on the items which the first reviewer had marked originally.
once the additional review on the item(s) which the first reviewer had marked originally has been completed, assessment teams will review and publish the ROM mark from a quarantine area which will update the paper mark for the review cases directly
the new mark is graded and the outcome of the review will be issued for the request if all criteria to generate an outcome has been met

In this way, as described above, the items which the first reviewer had marked originally will be reviewed by a second reviewer who had no involvement in the original assessment of the items. The scores given by the second reviewer for these items will be aggregated with those of the first so that each item on the review script is reviewed by an examiner who had not seen the items.

Timelines for short term solution - Workaround for Item Marking

Reviewing examiner availability and recruitment

Status	Task	Expected completion date in 2020*
Complete	Reviewing examiner availability and recruitment	Closed
Complete	Review and adjustment of the forecast for the next examination series	Closed
Complete	Recruitment of additional examiners for pinch points	Closed
Complete	Review of the PRS forecast for the next examination series	Closed
Complete	Further recruitment activity for ‘pinch points’	Closed

Systems and Process updates

System	Timeframe for scoping, development, testing and implementation	Expected Completion date in 2020*
iSeries	Work to take a total of 13 weeks to complete. Delivery of the updates to the iSeries will be aligned to complete at the same time as the other systems, because of the interdependencies across the systems.	Closed
EOL	Work to take a total of 24 ‘person weeks’ (total of 120 hours). Delivery of the updates to the EOL and Online Marking will be aligned to complete at the same time as the other systems, because of the interdependencies across the systems.	Closed
Online Marking	Work to take a total of 24 ‘person weeks’ (total of 120 hours). Delivery of the updates to the EOL and Online Marking will be aligned to complete at the same time as the other systems, because of the interdependencies across the systems.	Closed

Conducting reviews of marking: summary of actions to be taken against the conditions to ensure compliance

In addition to the recruitment actions to ensure compliance that are identified above, the following actions will be taken to ensure that reviews of marking are carried out in accordance with the conditions.

Condition of Recognition

GCSE (9‐ 1) 17.6; and GCE (reform) 17.6 The arrangements must provide that –

b.) an Assessor who was previously involved in the marking of a task in an assessment in respect of a Learner must not be involved in a review of marking of the Learner’s Marked Assessment Material in respect of that task,

Pearson actions	Progress to date
We will ensure that reviews of marking are allocated to reviewers that had no prior involvement in the marking of the task by updating our systems so that auto allocations to non- compliant reviewers are blocked	This work was completed in advance of the Autumn 2020 series
We will ensure that reviews of marking are allocated to reviewers that had no prior involvement in the marking of the task by providing training to our processing staff about the action taken when scripts need to be allocation by exception to more than one reviewer	Training was provided to all processing staff in advance of the Autumn 2020 series
We will ensure that reviews of marking are allocated to reviewers that had no prior involvement in the marking of the task by updating our systems so that manual allocation to more than one reviewer is possible by exception	This work was completed in advance of the Autumn 2020 series
We will ensure that reviews of marking are allocated to reviewers that had no prior involvement in the marking of the task by training our assessors to understand the exception ROM process and the systems involved with it	Training was provided to all assessors in advance of the Autumn 2020 series & only those that passed standardisation were allocated work. The training was adapted so that it makes clear tothem the requirements of the conditions and the processes that must be followed.
We will ensure that reviews of marking are allocated to reviewers that had no prior involvement in the marking of the task by training assessment staff about the new process and the requirement for them to review and approve candidate outcomes	Assessment teams have been fully briefed on the requirements of the conditions trained prior to the next examination series
We will ensure that reviews of marking are allocated to reviewers that had no prior involvement in the marking of the task by adapting systems so that non-compliant reviews that are made by mistake are recorded and escalated for remedial action	This work was completed in advance of the Autumn 2020 series
We will ensure that reviews of marking are allocated to reviewers that had no prior involvement in the marking of the task by monitoring the allocation of reviews to ensure compliance and to escalate allocations which are not compliant	Additional reporting was established for the autumn series onward so that resource plans are reviewed at regular Closure and Completion meetings prior to and during the post results services window
We will ensure that reviews of marking are allocated to reviewers that had no prior involvement in the marking of the task by monitoring the completion of reviews which are conducted by item to ensure that assessors review the parts of scripts they have not assessed before	Additional reporting was established for the autumn series onward so that resource plans are reviewed at regular Closure and Completion meetings prior to and during the post results services window

Cookies on GOV.UK

Applies to England