Transparency data

Chief Regulator's report: 21 September 2022

Updated 20 July 2023

Applies to England

This document has been subject to redactions

Date

21 September 2022

Title

Chief Regulator’s report (OPEN Paper)

Report by

Dr Jo Saxton, Chief Regulator

Paper for information and discussion

Preface

As a crown appointee, I would like to open this paper by recognising the life of service of Her late Majesty Queen Elizabeth II, whilst also welcoming our new monarch, King Charles III.

Recommendations

1. 1. The Board is asked to note the matters reported.

Overview

1. 2. Ofqual oversaw the delivery of the first summer exam series since 2019 and overall this was successful. Students told Ofqual they wanted the opportunity to prove themselves in exams and formal assessments and I’m delighted that they were able to do so.

2. 3. Many of the potential issues identified to the safe delivery of GCSEs, AS and A levels did not materialise. Marking was completed on time. Working with the exam boards, we successfully delivered the policy intent of results that were a staging post between those of 2021 and 2019, reflecting the so-called ‘midpoint’.

3. 4. I wrote to the Education Select Committee (ESC) on 1 July, to update them on the series, including to address some of the issues that had been reported in the media on advance information and breaches of paper security. On 12 July the ESC wrote to the Secretary of State, copying the Ofqual chair, raising concerns about exam board errors, including in relation to advance information, and requesting a response by the end of August. The Secretary of State replied to the ESC on 21 July, and I wrote in detail on 31 August. I have been called to appear before the Committee alongside OCR and Pearson – this was initially scheduled for 21 September but is now expected to take place sometime between the end of September and mid-October.

4. 5. As with every annual series, Ofqual will publish reports in December summarising how we monitored GCSE, AS and A levels and vocational and technical qualifications, covering each phase of the summer exam series from planning, through to exam delivery, marking, setting standards and post results. These reports include official statistics of reviews of marking, moderation and appeals, access arrangements, reasonable adjustments and malpractice cases. They will also provide an overview of the number of assessment material errors and other delivery issues in the series. In the autumn equalities analyses of this summer’s GQ and VTQ results will also be published.

5. 6. Proposals for arrangements in 2023 were discussed and agreed with the Board on 13 September. Decisions will be announced before the end of September and any relevant consultations will be launched, after the official period of mourning following the death of Her Majesty Queen Elizabeth II.

Summer series of exams and formal assessments

General Qualifications

1. 7. Overall delivery of GCSE, AS and A level results in summer 2022 was achieved according to plan, and with public confidence upheld. Marking was completed in time. Exam boards did, though, take all the available time to complete marking.

2. 8. A limited number of high-profile issues from this series were reported in the media (some of which were noted in the ESC’s letter to the Secretary of State). These included leaks of secure assessment material in GCSE Maths for Pearson and A level Chemistry for AQA, perceived and actual issues with advance information in some subjects and a labelling error in a GCSE Geography paper. We monitored the management of these via our established ‘event notification’ (EN) process.

3. 9. The actual number of confirmed errors where the advance information differed from the content of question papers was small. In each of these cases, exam boards took appropriate steps to address potential adverse effects on learners. In the greater number of cases, perceived issues with advance information appear to have resulted from misunderstandings about its purpose.

4. 10. Our requirements for awarding general qualifications were published in June and we monitored exam boards’ awarding throughout the summer. This included observing award sessions and reviewing emerging outcomes for each specification.

5. 11. On results days, enabled by our team’s analysis, we published headline information as well as a series of interactive data sources which were very well received.

6. 12. The National Reference Test showed no statistically significant change in English and a statistically significant drop in maths, compared to 2020. In governance terms, whether or not to require exam boards to adjust grade standards when setting GCSE grade boundaries following the NRT is a decision for the Chief Regulator. I confirmed recommendations not to make any adjustment to standards based on NRT evidence. An NRT annual statement was prepared and published, alongside NFER’s results digest, on GCSE results day, here.

7. 13. Post-results we are working on equalities analyses for GQ and VTQ 2022 results, with the intention of publishing this in the autumn.

8. 14. Appropriate data continues to be collected and Ofqual continues to meet with the exam boards as necessary, including through the governance arrangements we have in place to oversee delivery, to monitor and hold them to account for their delivery of qualifications this year. Our monitoring focus has now turned to post-results processes, including reviews of marking, moderation and appeals (RoMMA).

9. 15. Despite speculation about a significant increase in applications for reviews of marking this year, which exam boards factored into their resource planning, they are currently reporting a reduction in volume of between 20 – 40% compared to the same time in 2019. Although it is possible that this changes, as the deadline for submission of applications is not until later this month, it appears that volumes will be appreciably lower than the last time exams took place.

10. 16. Due to the cancellation of exams in 2020 and 2021, this year is the first in which the requirement Ofqual set for all exam boards to make marked and annotated copies of exam scripts available to centres in time to inform decisions about seeking a review of marking or moderation came into effect. Some previously chose not to do this until we required them to do so, while others offered it voluntarily. Some boards are reporting significant uptake of this service – one has seen a 600% increase in requests for access to copies of marked scripts at GCSE. Nevertheless, the reflection from exam boards who have offered this service previously is that these are unlikely to translate into review of marking requests (in line with the original policy intent of reducing the volume of baseless reviews).

11. 17. As part of the normal cycle of regulatory monitoring, we will undertake an evaluation of summer delivery issues experienced by the exam boards, to be discussed with the boards in post-summer meetings, to identify potential areas for improvement for summer 2023. As in any year, separate consideration will be given to whether there is a need for us to take regulatory action in relation to any non-compliance with our Conditions that we have seen this summer.

Vocational and Technical Qualifications

Performance table qualifications

1. 18. Intensive work was undertaken with awarding organisations (AOs) in the lead up to, and during, the summer series to agree approaches to awarding VTQs. While this year signalled a return to exams and formal assessments, the final grades for VTQs taken in schools and colleges were based on student outcomes from a range of sources, including one or more of:
   1. a. results from exams or formal assessments, some of which may have been adapted by the AO (as permitted by our Vocational Contingency Regulatory Framework)
   2. b. results based on teacher assessed grades that were awarded in summer 2021
   3. c. results based on centre assessment grades from summer 2020

2. 19. Because vocational qualifications are often modular, and assessment typically happens throughout the course of study, these may span 1, 2 or more years. We recognised this meant that gathering the evidence needed to support results this year would be more complex for students, schools, colleges and awarding organisations. In April work began to actively monitor this risk and routinely raise it with AOs.

3. 20. Timetabled exams and assessments for all PTQs concluded by 1 July.

Summer 2022 engagement and communications

1. 21. While exams and formal assessments were taking place in June, we toned down our proactive external communications to avoid distracting students.

2. 22. In late June and July, I undertook a number of speaking engagements. This provided an opportunity to remind the sector that 2022 results would likely be lower than in 2021, amongst other things.

3. 23. At the Association of Employment and Learning Providers’ national conference on 28 June, I spoke about adaptations, grading for vocational and technical qualifications, including Functional Skills. In early July, I spoke at the inaugural conference of the National Association of Exams Officers, thanking them for the work they had done to help deliver exams and assessments successfully.

4. 24. A great deal of communications activity took place in July. We produced the second podcast in our series with Laura McInerney, on the reliability of marking. We published a regular stream of content on our social media channels, including short videos of students explaining how they prepared for results days. We signposted to our student guide and reminded followers of the package of support in place for this year’s students. We published 3 blogs – 2 on what to expect with grades for general and vocational and technical qualifications, and a third which considered the reforms of functional skills qualifications.

5. 25. I prioritised regular stakeholder engagement over this period and in the run-up to results days – with teaching associations, equality organisations and the higher education sector. Other colleagues also spoke with education commentators and academics.

6. 26. UCAS were proactive in managing expectations about university places this year due to an increased demographic of 18-year-olds. With their Chief Executive, we published a joint letter just ahead of results, highlighting that in 2019 only 21% of placed students achieved their predicted grades, and reminding students of the choices available in clearing if they did not receive the grades they had hoped for. Our Chair took part in a UCAS Facebook live event with the then schools minister on the eve of A level results.

7. 27. During results weeks, several colleagues and I undertook media interviews. These spanned Sky News, ITV, Channel 4 News, TES, the BBC news and radio and local radio station, BBC CWR which covers Coventry and Warwickshire. Coverage seemed fairly balanced, drawing on regional disparities in particular for GCSEs. Attention swiftly turned to arrangements for 2023.

Stakeholder engagement

1. 28. A programme of extensive stakeholder engagement was undertaken. In particular, new energies were invested in reaching equalities stakeholders in advance of GCSE and A level results to increase understanding of this year’s arrangements and manage expectations about our planned equalities analyses this autumn. This was well received.

2. 29. Extensive one-to-one stakeholder engagement around GCSE and A level results was undertaken, in addition to our participation in the JCQ briefings to all relevant stakeholders on results days.

3. 30. During July and August, VTQ extended its reach to stakeholders through conference presentations. A presentation on the benefits of regulation was delivered at the Apprenticeships Conference, and at the Westminster Forum we spoke alongside IfATE representatives on the future of the level 3 qualifications system following the Department for Education’s review of post-16 qualifications.

Evaluation

1. 31. Work to evaluate the adaptations to exam arrangements for GCSE, AS and A level exams in 2022 is underway. This work focuses on advance information, formulae and equation sheets and content optionality. Survey elements include students, teachers, tutors and home-educators, undertaken prior to the exams as well as afterwards. Student focus groups were undertaken immediately after the exam series.

Qualification design and development

1. 32. Work continues on the development of 2 new GCSEs and 3 revised GCSEs:

   1. a. For the GCSE in British Sign Language (BSL) we have provided further detailed written and verbal feedback to DfE on their draft subject content. We expect to receive a further draft from DfE during the autumn. Once the draft has been finalised and we are confident the content can be regulated, we will bring a paper to the Board setting out our proposed assessment arrangements.

   2. b. For the GCSE in Natural History we continue to engage with the DfE and exam board representatives to support their work as they develop the draft subject content. While this work is in its early stages, it is progressing well.

   3. c. For the revised GCSEs in French, German and Spanish we are preparing for accreditation, which will begin with first submissions of proposed qualifications and sample assessment materials from exam boards in GCSE French in November 2022.

Functional Skills Qualifications

1. 33. Research conducted in 2019 on the expected level of difficulty of items in legacy and reformed FSQ papers was published, with an accompanying blog that sets out the next phase of our evaluation programme.

Recognition activity

1. 34. In total, 58 new End Point Assessment Organisations (EPAOs – those who offer Apprenticeship End Point Assessments) have now been recognised, as part of our programme of work to transition End Point Assessments into regulation.

Technical Education Reform – Post-16 Qualifications Review

1. 35. Work to strengthen the qualifications that will remain eligible for funding following the Post-16 Qualifications Review continues to progress. We have concluded both the policy and technical consultations on our proposals for the relevant Level 3 qualifications.

Equalities work

1. 36. The annual equalities report was published on 18 July 2022, setting out progress against Ofqual’s equalities objectives.

Research and Analysis

Research

1. 37. Our research about teacher and student perspectives on TAGs in 2021 was published in July, along with analysis of inter-subject comparability in 2021. We also published research on Functional Skills looking at the expected difficulty of items following reform.

2. 38. A number of other planned research projects are underway and on track. Our analysis of differential item functioning (DIF) in a sample of pre-pandemic qualifications is underway. This work explores the extent to which such analysis may support awarding organisations in identifying exam questions which may be unfair.

3. 39. Following the February 2022 publication of our landmark research report How CASLO Qualifications Work, we have moved into a new phase of research, which will focus on Quality and Value in CASLO Qualifications. The work is landmark in that, for the first time, a key group of qualifications which require assessors to Confirm the Acquisition of Specified Learning Outcomes (hence ‘CASLO’) is being codified and the acronym, coined by Ofqual’s research Chair, Professor Paul Newton, is beginning to gain circulation. (The CASLO approach to qualification design involves specifying unit content and standards in terms of learning outcomes and assessment criteria, with a mastery approach to aggregation.) An extensive literature review is underway and we are detailing the history of this approach. During the autumn and spring term we will work closely with awarding organisations to understand the pros and cons of this approach in more detail.

4. 40. Progress is being made on research projects on the topics of remote invigilation, public perceptions of on-screen assessment, the use of optionality in exam papers, the assessment of behaviours in End Point Assessments, and perceptions of extra time as a reasonable adjustment. These are at various stages of development and delivery but are all broadly on track. We continue to provide research support for our nascent programme of work about the use of technology in assessment, exploring, with the Department for Education, the opportunities and barriers for on-screen and online high stakes, sessional assessment.

5. 41. Ofqual’s research report on the Evaluation of 2020 Centre Assessment Grades (CAGs) published in 2021 has been shortlisted for the People’s Choice category of the ONS Research Excellence Awards alongside a few other quantitative projects in the field of social sciences.

Data & Analytics

1. 42. The data team has been focused on the preparation and delivery of analysis that was published on results days in August. This entailed close collaboration with internal colleagues as well as with AOs to ensure the punctual submission of the data and the redesign of the user-centric interactive analytics to be published. Between A level and GCSE results days, 9 interactive visualisations were refreshed with 2022 data, and in many cases this also entailed improvements to the design of the interactive visualisations to make sure they were insightful.

2. 43. Following up on the commitment made in spring to increase the statistical publications on VTQs, a new Performance Table Qualification Dashboard was developed to allow users to explore data on VTQ PTQs. This was published on 18 August with Level 3 outcomes, and refreshed on 25 August with Level 2 and below outcomes. On Level 3 results day an interactive visualisation on Technical Qualifications was also released. In addition to analytics on results, in June an interactive visualisation was also published to present data on the volume of certificates issued over the past 10 years.

3. 44. The team now turns its focus to the Official Statistics autumn season that will see the publication of a range of reports reporting on the awarding of GQs in summer 2022 and to plan for further VTQ publications.

Strategy Department

1. 45. Arrangements to assess risks to the financial viability of awarding organisations are now confirmed using the annual statement of compliance process. This process is important, given the implications of the pandemic on the market, and continued economic uncertainty. All awarding organisations will be required to answer specific questions regarding their viability, which will inform further where further analysis is required. All high and medium risk entities will be asked to provide detailed information on their financial position, at their year-end. This approach is designed to ensure that the burden on awarding organisations is reasonable, whilst protecting students in a volatile economic climate. We have also agreed with regulators in Wales and Northern Ireland that we will, for the first time, split the financial review workload across the 3 organisations. This reflects the positive joint working that has developed over the pandemic; ensures efficient use of public resource; and enables coherence for entities regulated by multiple regulators.

2. 46. The Ofqual approach to risk management continues to evolve, with ARAC briefed in September on a range of improvements underway and programmed for the coming years. This includes resetting our approach to risk appetite, which has been problematic given our regulatory role and the often-limited scope to reduce strategic risks due to wider systemic factors. This work will see a range of improvements to how risk is managed at directorate level, commensurate with the growing size and complexity of Ofqual and our work.

3. 47. I provided an op ed to Schools Week on Ofqual’s view on the future of assessment – making clear that our focus is on evidenced based deployment of technology where, and only where, this is clearly in the interests of students and good assessment.

4. 48. The first Ofqual all-staff event post-pandemic is now confirmed for 3rd November. This will be an important opportunity to meet together given how much the organisation has grown and changed since the last in-person event in 2020. Our commitment to regulate on behalf of students and apprentices will underpin the topics and focus for the day. All Board members are warmly invited to attend the day if they are available, which will be structured around a range of workshops.

Regulatory and Corporate Services

1. 49. The key focus for the directorate over the last quarter has been to support delivery of the summer period. All IT systems remained secure and stable. Colleagues working within our public facing services, which include the public enquiries helpline, mailbox, and complaints portal, were able to respond quickly and clearly to contacts over the results period without drawing on resource from outside the core team. The team also supported colleagues in other directorates in their response to issues arising by providing accurate and regular reporting of volumes and themes.

2. 50. Aside from the summer, the directorate continued to deliver its routine activities, corporate plan commitments, and investigations into and analyses of a range of awarding organisation activities.

3. 51. Recruitment campaigns continue for several senior posts within Ofqual including the Deputy Chief Regulator and Executive Director Finance and Operations. Both campaigns are being led by a Civil Service Commissioner, with Moloney Search undertaking the search element, and interviews due to take place for the Deputy Chief Regulator in September and the Executive Director Finance and Operations in October. Interim measures remain in place to manage these vacancies. Charles Scott joined Ofqual in September as the Interim ED Finance and Operations for a period of 6 months.

4. 52. The 2022-2025 People Strategy has been published internally. Following an external recruitment process Charlie Henning has been appointed as Director People and Culture as part of the capacity increase to the HR Team to support the delivery of the People Strategy that the Board contributed to and approved with the Corporate Plan.

5. 53. Our 2022-23 Pay Award has now been implemented. This is the earliest the pay award has been made in the past 5 years, in recognition of the cost of living pressures, and for the same reason it has been tapered to give a greater percentage increase to those in lower paid roles. Further work will be carried out in September and October to review and adjust any salary anomalies that are identified.

6. 54. As longer standing Board members are aware, work is underway to reform our Register of regulated qualifications. The Register, found on gov.uk, receives visits from around 25,000 users per month, reflecting the interest in qualifications. An expert has been recruited to lead this transformation, which is an opportunity to bring to life and integrate a range of our statutory objectives.

Publication of paper

Paper to be published: Yes

Publication date (if relevant): N/A

Exemptions to publication: Section 36, Development of government policy