Corporate report

Electronic evidence gathering review

Published 16 January 2025

Applies to England

Purpose of the review

The purpose of this review was to explore how robust and reliable Ofsted’s electronic evidence gathering (EEG) application is, including the guidance and processes for inspectors to follow if they encounter technical difficulties. The review covers the period from March 2019 to May 2024.

Four of Ofsted’s education remits use EEG. These are:

• schools
• further education and skills (FE and skills)
• teacher development – made up of initial teacher education (ITE) and lead providers of the early career framework (ECF) and national professional qualifications (NPQ)
• the online education accreditation scheme (OEAS)

We have considered all 4 remits as part of the review.

This report focuses on 3 areas: the benefits and challenges of recording inspection notes electronically; maintaining the integrity of inspections if there are technical difficulties; and inspector training and guidance. It sets out the review’s findings, followed by an annex outlining next steps.

Background

EEG is a digital application built using Microsoft technology.

Both employed inspectors (His Majesty’s Inspectors, HMI) and contracted inspectors (Ofsted Inspectors, OIs) use EEG:

• OIs make up approximately 70% of the inspector workforce. These inspectors provide their own laptops and are responsible for downloading and installing the Microsoft technology and EEG themselves. Most do not inspect every week.
• Ofsted provides HMI with a laptop. We install the Microsoft technology and EEG for them. HMI inspect most weeks during an academic year.

The introduction of EEG

Ofsted commissioned EEG in 2017. We used a commercial supplier to develop it.

Field tests and pilot inspections took place ahead of all inspectors receiving training on how to use the application in January 2019.

We started to use EEG on a small number of live inspections from March 2019. In September 2019, it launched for all school and FE and skills inspections, at the same time as the education inspection framework (EIF).

We added ITE inspections in May 2021. In November 2022, we further developed it for inspections of ECF and NPQ lead providers and visits to OEAS providers.

Between March 2019 and 1 May 2024, EEG was used by 6,016 unique inspectors to record their notes on 33,254 live inspections.

How inspectors record notes on inspection

Inspectors use EEG to record written notes during an inspection. They are trained not to record notes verbatim. Instead, we train them to spend time listening and responding to those they meet with, and to record the most important points from what they see and hear. Inspectors use their professional judgement to decide what is important to record during each inspection activity.

Often, an inspector will make notes during an inspection activity. At other times, it will be more appropriate for them to record some, or all, of their notes after an activity has finished. For example, they would make notes after observing social times in a school or speaking to parents and carers at the school gate.

At the end of an inspection, the inspection team uses the full range of what they have seen and heard to make the final judgements. They use their written notes to inform their discussions and decision making. They invite leaders from the provider to be present at end-of-day team meetings, when these discussions are taking place.

Following the on-site part of the inspection, the lead inspector uses the inspection team’s notes to write the draft inspection report. We then send this report to the provider so that they can check the content for factual accuracy.

The benefits of recording notes electronically

On inspection

Inspectors tell us that recording notes electronically is an improvement on using pen and paper. EEG enables inspectors to review and synthesise their notes more efficiently. Using EEG also means inspectors’ notes are more legible compared with the handwritten alternative.

EEG provides a secure place for inspectors to record their notes. This is a notable improvement on the previous system. Inspectors used to make handwritten notes and post carbon-copies to a central Ofsted depositary, risking damage and loss while in transit.

When there is internet connectivity, EEG provides lead inspectors with full oversight of their team inspector’s notes. This has 2 benefits.

First, the lead inspector can identify where enough evidence has been collected and where more is needed. This helps them to deploy their team efficiently.

Second, it helps the lead inspector to build a picture of the emerging findings ahead of sharing these with the provider. This helps the provider to understand what the inspection team has found so far and to identify any further evidence they might want to share.

Quality assurance

EEG has strengthened the quality assurance process significantly.

Once inspectors’ notes are uploaded to the cloud, a lead inspector can access them instantly. Lead inspectors can check the notes and give team inspectors ongoing feedback to make sure the inspection is carried out to a high standard.

Senior colleagues in regional teams can also view inspectors’ notes remotely in real-time. This increases the number of inspections that regions can quality assure.

Moreover, if lead inspectors contact the national duty desk for advice during an inspection, HMI can access the inspection notes to inform the guidance they give.

Research and analysis

Electronic evidence recording has enhanced our ability to analyse evidence bases for research purposes.

Analysts and researchers within Ofsted use the notes inspectors record in EEG to evaluate aspects such as consistency, the impact of training and how effectively inspectors consider certain elements on inspection.

The challenges of recording notes electronically

Technical issues

Since the introduction of EEG, there have been some technical issues with stability and reliability. Some of these issues link to the application itself, and some to the Microsoft technology it is built on.

We have made improvements to EEG’s stability and reliability over time. This has reduced the extent of technical issues that inspectors experience. However, on some inspections there are still issues with the application intermittently ‘freezing’ and closing unexpectedly. This does not automatically affect the recording or saving of an inspector’s notes, but it does create uncertainty and additional workload for inspectors.

Variables that affect performance

There are several environmental variables that can affect the performance of EEG. The most significant of these are the settings on a provider’s internet network and the availability and strength of connectivity.

Each aspect can make it difficult for inspectors to establish an internet connection and to stay connected as they move around a provider’s site during an inspection. Although EEG is designed to work offline, intermittent, low or no connectivity increases the likelihood of technical issues.

Another variable is the laptop an inspector uses. As stated above, HMI are provided with a laptop that meets the recommended technical specification to run EEG. OIs are contractors who are either serving or non-serving practitioners in the relevant sector. As such, they provide their own laptop. We give them the technical specification to inform what laptop they should use.

Actions we have taken so far

In autumn 2023, we commissioned an external technical review of EEG. This review examined the root causes of the issues relating to stability and reliability. We are in the process of acting on the recommendations from this review.

One of the most significant actions is moving EEG onto a new version of the Microsoft technology. This new version offers improved offline capability and is expected to perform well with intermittent and low connectivity. Inspectors tested and piloted the updated EEG between July and November 2024. We released the application to all inspectors at the end of December 2024.

At the start of September 2024, we issued an updated technical specification to all OIs. This came into effect from 31 October 2024.

Functionality

Inspectors receive training on saving their notes manually at regular intervals during an inspection. We also advise them to upload their notes to the cloud whenever they have strong enough internet connectivity.

However, there is some inconsistency in how often inspectors save and upload their notes. Sometimes, this is due to not being able to upload because of poor internet connectivity. At other times, inspectors choose to delay uploading until they have checked their notes are complete.

Although EEG has an auto-save function, any delay in uploading to the cloud increases the risk of information not being saved if there is a technical or human error.

Actions we have taken so far

In the short term, we have reminded inspectors that it is important to save and upload their notes as frequently as they can.

In the longer term, moving EEG onto Microsoft’s new technology means that inspectors’ notes will now upload automatically whenever their laptop is connected to the internet. Inspectors’ notes will also save more securely when they are working offline.

Feedback from inspectors

Since September 2019, we have routinely sought inspectors’ feedback about their experience of recording inspection notes electronically. Over time, feedback shows a consistently high level of confidence in how to use EEG in most remits. Feedback also shows a steady improvement in the stability and reliability of EEG.

Despite these positives, we recognise that inspectors’ priorities for improvement have remained unchanged since EEG was launched in September 2019. These priorities are to further improve:

• stability and reliability
• functionality to structure and demarcate text
• navigation within the application
• functionality to search and filter notes

We have begun to improve these aspects, and inspectors appreciate the work that has been done. However, the capacity to make functional changes has been limited in the past 18 months due to the need to re-prioritise finite funding and resource.

In addition, some aspects of EEG are interdependent. For example, we removed rich text formatting to make EEG more stable. However, this has made it more difficult for inspectors to structure and demarcate text. This has led to many inspectors choosing to record their notes in Microsoft Word or an equivalent application. Inspectors then transfer their notes into EEG during or after the on-site part of the inspection. This makes the recording process easier but creates additional work for inspectors. It also means that regional Senior HMI cannot access evidence bases remotely.

Actions we have taken so far

One of the main aims of moving to Microsoft’s new technology is to improve the stability and reliability of EEG, particularly when working with intermittent, low or no internet connectivity.

Once the application is stable, we plan to implement changes to improve useability, starting with the structuring and demarcation of text.

Support to resolve issues

Our approach to supporting inspectors when they encounter issues with EEG has evolved over time. Between March 2019 and February 2021, a dedicated support team of Ofsted staff handled all queries. Once EEG was an established product, first-line support moved to a third-party IT service desk. This is in line with other Ofsted systems.

Issues reported to the IT service desk can range in scale – from needing advice on how to reset a password to EEG not opening. However, inspectors tell us that the pace of inspection means they often do not have time to report issues or to wait while they are resolved. Therefore, we know that the IT service desk records only give a partial picture of the issues inspectors experience. Under-reporting of issues makes it difficult to monitor how the application is performing and to respond proactively where remedial action might be needed.

Actions we have taken so far

Inspectors are advised to record their notes in Word if it is not possible to use EEG. This mitigates the risk of any technical issues affecting the inspection by making sure that it can continue uninterrupted. Inspectors then transfer the notes they have recorded into EEG as soon as practicably possible and always within 5 working days. Written guidance to this effect has been in place since December 2021.

In addition, we are extending the reach of current background analytics to provide more information about the performance of EEG. This will reduce some of the reliance on inspectors reporting technical issues.

Maintaining the integrity of an inspection if there are technical difficulties

Quality assurance

Processes are in place to ensure the integrity of the evidence base on every inspection. Judgements are always based on the evaluation of findings from all members of an inspection team across the full range of inspection activities.

Primarily, it is the role of the lead inspector to make sure that there is enough evidence to make secure and reliable judgements.

Regional Senior HMI, under the direction of each regional director, also have responsibility for quality assuring inspections. They do this by sampling inspections for quality assurance, for example through a live visit, checking inspectors’ notes remotely, a post-inspection evidence base review or an enhanced check of an inspection report.

In addition, regional Senior HMI provide remote support for lead inspectors when they are on site. All inspections receive an additional layer of quality assurance if any of the key judgements are ‘inadequate’.

Inspectors’ notes not saving successfully

Occasionally, a system or human error can result in some of an inspector’s notes not saving in EEG. There is an internal policy to guide inspectors on the actions they should take if this occurs. This has been in place since March 2019. The policy states that inspectors should recapture the salient points from memory as soon as possible after they detect that any notes are missing. In most instances, only a small proportion of the evidence base is affected. Here, the inspector can recall and record their notes before the end of the inspection. This happens without any disruption to the inspection process.

Where the amount of unsaved notes is more extensive, inspectors should record a summary of what is lost as soon as they are able. This review has found that the extent to which inspectors summarise their notes varies. Inspectors should also notify their region if they experience significant or substantial loss of recorded information. The regional director will then decide whether any additional quality assurance is needed. In rare instances, this may lead to Ofsted returning to the provider to gather additional evidence. At the time this review was instigated, some inspectors were not clear about the need to report this type of incident to their region.

Between 1 February 2021 and 17 May 2024, EEG was used on 26,431 inspections.^{[footnote 1]} We have identified that in 191 of these inspections, an inspector on the team contacted the IT service desk because a proportion of their notes did not save. These particular notes were unrecoverable. In 4 of the 191 inspections, we returned to the provider to ‘gather additional evidence’ at the time. This was done to secure the judgements.

As part of this review, we have examined the evidence bases of the remaining 187 inspections. In all 187 events, the inspector whose notes did not save shared their findings verbally in the end-of-day team meetings. The inspection team considered these findings, alongside the recorded notes, when making their final evaluation and reaching judgements. This is in line with standard inspection practice.

In 182 of the 187 inspections, the affected inspectors also followed Ofsted’s internal policy and summarised their missing notes. In 5 inspections where this was not done, the other notes recorded by the inspector or colleagues in the inspection team, and/or the summary drawing together the team’s findings, supported the judgement. 

To conclude, we can confirm that in all 187 events, the full range of inspectors’ findings was considered when reaching final judgements. The written notes available in EEG are enough to support these judgements.

Actions we have taken so far

We have reminded all inspectors of the need to summarise any notes that do not save and to report these incidents to their region. In June 2024, we gave inspectors updated guidance on the process to follow. We further updated that in September and October 2024.

We have also tightened our monitoring of the rare incidents of information not saving, including the actions regions take to ensure the integrity of the evidence base.

Moving EEG onto Microsoft’s new technology will significantly reduce the risk of inspectors’ notes not saving.

Inspector training and guidance

Overall, inspectors are positive about the training they receive in how to use EEG. In May 2023, 83% of inspectors told us that they felt confident using EEG within 6 months of their initial training. Less than 7% said they lacked confidence after a year or more of using it. We recognise the need to provide additional support for these inspectors.

Each remit within Ofsted takes a slightly different approach to how they train inspectors to use EEG. For example, new HMI and OIs in schools and teacher development receive face-to-face training before they go on their first inspection. These inspectors also practise recording notes on at least one shadow inspection. In FE and skills, less time is spent on the face-to-face element of training and more emphasis is placed on learning how to use EEG as part of inspectors’ shadow experiences. In this remit, HMI and OIs are trained in different ways.

Each remit also takes a different approach to the way in which they ask inspectors to record their notes in EEG. In schools and teacher development, we have a suite of templates to help inspectors structure their evidence bases in a consistent way. Inspectors are positive about the support these templates provide. Most inspectors follow the agreed methodology. In FE and skills, there is more flexibility in how inspectors organise and populate their evidence bases. The way in which the application is built can make it time consuming for all inspectors to navigate and complete some essential tasks.

All inspectors have access to training materials and guidance on Ofsted’s internal ‘EEG learning hub’. This includes technical guidance. Written technical guides are also available for HMI on Ofsted’s self-service portal. However, the content of some of these documents does not align with what is available to OIs. The EEG learning hub is well used, particularly in the schools and teacher development remits where the training and guidance materials are more fully developed. The hub is less well used by FE and skills inspectors.

Actions we have taken so far

We recognise the need for more consistency in the training and guidance inspectors receive within and across remits. We have completed a full review of the training and guidance available to inspectors and have begun to update the information available on the EEG learning hub. Induction training for new inspectors will be consistent across all remits from January 2025.

Starting in the spring term 2025, we are introducing additional training for those who need it along with ongoing refresher modules to keep inspectors’ skills sharp. We will also be considering what more we can do to assess and train on generic digital skills to support inspectors’ use of EEG.

Finally, we have identified a need for policy teams and inspectors in remits other than schools to make a greater contribution to how the application is developed and improved. We are building workforce capacity to achieve this.

Conclusion

EEG has many benefits for individual inspectors, inspection teams, providers and the wider organisation. The integrity of inspections is our priority. Policies are in place to make sure that this is not adversely affected if a technical or human error occurs with EEG.

EEG is built on a Microsoft platform, which offers security, speed of development and future-proofing. However, there have been issues with the stability, reliability and useability of EEG over time. These have caused uncertainty and additional workload for inspectors.

We have taken actions to improve EEG since its introduction, and this work continues. Our short-term strategy for EEG prioritises improvements to the following aspects:

• stability, reliability and useability of the application
• reporting, monitoring and analysis of incidents
• consistency in the training and guidance that inspectors receive
• implementation of agreed policies and guidance

Annex A: actions to be taken

Stability, reliability and useability of the application

• Monitor the stability and reliability of EEG following the move to Microsoft’s new technology.
• Continue with existing plans to improve aspects of useability within EEG, including exploring functionality around structuring and demarcation of text.
• Explore the best approaches and technology for recording inspectors’ notes efficiently and consistently across all remits.
• Further develop partnership working with policy teams and inspectors so that all remits contribute to helping shape:
  • the way in which EEG is used on inspection
  • how EEG is further developed and improved.

Reporting, monitoring and analysis of incidents

• Implement changes to EEG so that we can monitor technical performance and respond more proactively to emerging issues.
• Explore how we can make it easier for inspectors to report issues and request technical support during an inspection.
• Further improve the reporting, monitoring and analysis of incidents where inspectors’ notes do not save to:
  • make sure that we have a full picture of what is happening and why
  • identify where we can make technical improvements to EEG.

Consistency of training and guidance

• Review and standardise the training offer in each remit to make sure that all inspectors have the necessary knowledge to use EEG confidently and effectively.
• Further improve inspector/operational guidance and ongoing training, and make sure that HMI and OIs have access to the same resources.
• Review and update the existing policy for ensuring the integrity of the evidence base if inspectors’ notes do not save, and help inspectors to follow this consistently.

Implementation of agreed policies

• Make sure that all inspectors are fully aware of, and understand, the expectations set out in policies and guidance relating to EEG.
• Monitor inspectors’ compliance with EEG policies and guidance more closely, and support them to apply guidance consistently.

1. Formal recording of reports to the IT service desk began in February 2021. A new process for recording, reporting and responding to incidents of inspectors’ notes not saving started in May 2024. ↩