Research and analysis

Remote invigilation within vocational and technical qualifications

Published 18 July 2023

Applies to England

Authors

• Diana Tonin
• Stuart Cadwallader

With thanks to

The authors would like to thank the awarding organisation staff and the candidates who gave their time to speak to us in depth and share their experiences and views.

Executive summary

Remote invigilation (RI), also known as remote proctoring, allows candidates to take assessments at a location of their choosing, while being supervised by an invigilator that ensures the assessment is completed under controlled conditions. The candidate may be in their own home, their place of work, or some other location as they take the assessment. RI is distinct from remote assessment in that the invigilator is not assessing the person sitting the assessment, their role is to ensure that the assessment is completed under the specified conditions. RI solutions provide similar services to those offered by invigilators at an examination school or college. They seek to prevent malpractice during the assessment, they validate each candidate’s identity, and, in some cases, they offer administrative support to candidates before, during and after the assessment.

RI has become increasingly prominent in the last decade, but its growth has accelerated considerably due to the coronavirus (COVID-19) pandemic. During the pandemic, Ofqual set out special regulatory requirements for awarding organisations (AOs) offering vocational and technical qualifications (VTQs). These conditions allowed AOs to use a range of approaches to facilitate the provision of results to candidates taking VTQs in circumstances where normal assessments were disrupted. AOs were permitted to adapt their assessments where they were able to ensure that the validity and reliability of the qualification would be sufficiently maintained. An adaptation that was commonly used was the implementation of onscreen assessment paired with RI where candidates sat their assessments on a computer device and were supervised by an invigilator present in a different location.

While Ofqual permitted the use of RI during the pandemic to mitigate unprecedented challenges, this research should not be interpreted as an endorsement of widespread use of this form of invigilation in high-stakes assessments. With this research, we seek to deepen our understanding of RI and how it is used and experienced, therefore informing our approach to regulation in the future.

It is worth noting that this research took place at a time when generative artificial intelligence tools such as ChatGPT were not widely available and so the challenges such tools present for RI have not been explored in this report. These will, however, become increasingly important as AI tools are more widely used in the education sector, including the impact they have on the way assessments are designed and delivered.

The study

This report describes a qualitative study which was undertaken to foster greater understanding of how RI was used, in practice, for a small sample of VTQs in 2020. AOs and candidates were interviewed between spring 2021 and autumn 2022 (the latter interviews were with a small group of candidates who had been remotely invigilated in 2021). Interviewees from AOs were asked to describe the RI systems they used, how they had implemented these systems, whether they encountered any challenges and what they perceived to be the strengths of RI. Candidates were asked about their experience of using RI during their assessment. The discussions were diverse and covered various RI systems.

Key findings

Data were analysed thematically and the findings are presented in relation to 5 broad areas:

1. selection of a RI system, namely between ‘live’ or ‘record and review’ system, with the decision often depending on available resources and the nature of the assessment

2. technical considerations around RI implementation, which included ensuring that the devices are well set up prior to the assessment, that the candidates have access to reliable internet connection, that the system allows for a room scan and a reliable authentication of the identity of the candidates. In addition, assessments should be designed in a way that allows for rest breaks without compromising the security of the assessment and ensures that students with special needs can be offered Reasonable Adjustments

3. identifying and managing malpractice, where various processes could be adopted to identify and investigate possible malpractice instances

4. the candidate’s experience, being reported as positive overall

5. long-term planning, with all interviewed AOs making efforts to accelerate their plans to adopt RI long term and introduce it as an option for more of their learners across a broader range of their assessments

The research, however, highlights various challenges and barriers to implementing RI and the array of decisions that AOs must make about the configuration of their systems, including the need for bespoke solutions. The AOs suggested that they would find the definition of a minimum acceptable standard for RI systems helpful.

Introduction

Invigilation, or proctoring, refers to the supervision of a candidate as they take an assessment task. This process helps to preserve the integrity of the assessment by ensuring that all candidates are offered an equal opportunity to show what they know and can do under the specified conditions of the assessment. Its main purpose is to prevent cheating and ensure that the work produced is the candidate’s own, thereby maintaining the validity and the fairness of the assessments.

Typically, invigilation is carried out by an invigilator (or proctor) who supervises candidates, live, as they are completing the assessment. However, due to the coronavirus (COVID-19) pandemic, live in-person invigilation was no longer possible. Therefore, Ofqual established regulatory requirements for Awarding Organisations (AOs) offering Vocational and Technical Qualifications (VTQs) in summer 2020. These requirements were set out in an Extraordinary Regulatory Framework (ERF) and later a revised framework, the Extended Extraordinary Regulatory Framework (Ofqual, 2021). Under these frameworks, a range of approaches were made available to AOs to facilitate the provision of results to candidates who were taking VTQs in circumstances where normal assessments were disrupted. AOs were permitted to adapt their assessments where they were able to ensure that the validity and reliability of the qualification would be sufficiently maintained. An adaptation that was commonly used was the implementation of onscreen assessment paired with RI, the latter of which is the focus of this report.

RI, or remote proctoring, refers to the process of invigilation when it takes place remotely; that is, when the candidate and the invigilator are not in the same room. RI can occur live as well as after the candidates have taken the assessment through video recording. The candidate may be in their own home, their place of work, or some other location as they take the assessment. RI is distinct from remote assessment in that the invigilator is not in any way assessing the performance of the examinee: their role is instead to ensure that the assessment is completed under the specified conditions.

RI is not new – it has been in use, in various forms, over a decade. It was already becoming increasingly prominent, but its growth has accelerated considerably due to the pandemic, particularly following its use by some AOs as an adaptation under the ERF. Some of the key benefits that tend to be identified around the use of RI, include that it can allow candidates flexibility in the locations and timings of the assessment (Alpha Plus, 2020). Moreover, it can be cheaper than delivering invigilation in traditional centres (that is, in schools or colleges where exams are administered), with the potential for allowing rapid growth in testing capacity without compromising the security of the assessment.

RI presents both opportunities and challenges to AOs, and there are also potential advantages and disadvantages for educators and candidates alike. This report draws on the available literature and interviews with staff from AOs, and examinees, to explore the capabilities of existing RI systems and how they are used and experienced in theory and in practice. The intention is to deepen our understanding of RI, therefore informing our approach to regulation. It is critical for Ofqual to evaluate the opportunities and risks that RI presents and compare these against the expectations from all forms of invigilation. It is also hoped that this report will be a helpful resource for AOs who are considering RI options for their own assessments.

The report begins by broadly describing RI systems and their capabilities, outlining some of the key decisions that AOs must make when introducing such systems. This is followed by an exploration of the research literature about RI, which provides insights into its impact upon assessment and the experience of candidates who are invigilated remotely. The report then describes the methodology for our interviews with AOs before discussing the themes that emerged from our analysis. Finally, these insights are drawn together and considered in context.

Remote invigilation systems

RI is usually used in tandem with some form of remote computer-based assessment (it is possible to remotely invigilate paper-based assessments, but this approach is not common). Any RI system must be integrated with the mode of assessment, which can present technical challenges for AOs, teachers and examinees. Therefore, AOs seeking to introduce RI must consider whether any changes to the assessment they wish to invigilate are also necessary and whether any such changes undermine the validity of that assessment.

Most RI solutions rely on the use of one or more cameras to monitor the candidate and their environment during the assessment. Some solutions may also offer features that ‘lockdown’ the candidate’s system, preventing them from accessing either unrelated programmes or the internet while the assessment is in progress (though these features are often part of the online assessment platform rather than the RI platform, if the 2 are not integrated).

Most of the focus on the use of RI is around the prevention of candidate malpractice, commonly described as ‘cheating’, but RI can also be used to provide live technical and administrative support to candidates in case of any issues, depending on the system. Therefore, in addition to deciding which RI technology works best with their qualification and assessments, AOs introducing an RI system must make several fundamental decisions about the features they want their RI system to include. In this section, we will first describe the main 2 types of RI systems and then we will draw out some of the key considerations for AOs. Please note that this is not intended to be an exhaustive list of these considerations, but a broad outline of some of the key decisions that need to be made.

Live versus recorded

In basic terms, there are 2 common systems of RI: ‘live’ and ‘record and review’. The former system involves an invigilator watching a contemporaneous video feed of the candidate as they take the assessment, often interacting with them at specified points during the session. This system is similar to a traditionally invigilated exam. Record and review systems rely on retrospective analysis of video footage which can be undertaken by a human reviewer, or it can be automated. In the latter case, potential issues are usually ‘flagged’ by the automated system for further review by a human invigilator.

As we will discuss later in the report, the decision about the overarching system must be followed by a series of further choices about the details of the invigilation approach. For example, the AO needs to decide, based on the specifics of their assessment, the exact skills and experience that they require of their invigilators, how many candidates they will permit to be invigilated simultaneously (if live), and how potential issues will be escalated and to whom. For live invigilation, the invigilator is responsible for supervising the candidate during the online assessment and for flagging any potential instances of malpractice or issues that are observed during the session. They also often check the identity of the candidate and can sometimes provide basic technical support. The invigilator is either employed by the AO directly or invigilation is sometimes outsourced such that the invigilators are employed by the RI platform provider.

RI systems vary in the extent to which they allow the candidate and the invigilator to interact. In some live RI systems, the invigilator may be a visible presence with whom the candidate may interact if they have any questions. The invigilator may be able to audibly inform or warn the candidate if they are contravening any rules (whether intentionally or not) and thus ensure the ongoing security of the assessment. Although unable to offer the same level of interaction, record and review RI is often cheaper to operate and may be better suited to circumstances where a candidate’s internet connection is weak or intermittent (Alpha Plus, 2021).

On-demand versus scheduled

Whether live or recorded, the test session may be scheduled or on-demand. A scheduled session requires a time slot to be arranged either by the AO or by the candidate on a specific date and time. If the provider offers on-demand assessment, the candidate can take the test when they are ready, providing higher flexibility to the candidate. When considering whether to provide on-demand assessments, AOs need to consider how their assessment process works with the RI system.

Prevention of malpractice

Prevention of malpractice is arguably the main purpose of RI, but there are a few specific subsystems to consider. One way in which RI systems vary is their use of hardware such as cameras to monitor the candidate. Alongside the typical screen mounted camera (the type used in video conferencing) it is also possible to use an additional camera, known as a secondary or environmental camera. This is usually set up either behind or beside the candidate and allows the invigilator to observe them in their wider environment. The intention of this environmental camera is to further increase the likelihood that malpractice would be detected through invigilation.

In addition, all forms of high-stakes assessment require that the identity of the individual taking the assessment is verified – a candidate sitting an examination in another candidate’s name would constitute malpractice or maladministration. While in many face-to-face examinations this process is conducted by invigilators onsite, an alternative approach is required for RI. This usually involves pre-registration followed by verification of the individual alongside a photo ID via a camera, but there are alternative approaches, including the use of facial recognition technology.

Many RI systems work in conjunction with a ‘lockdown’ system, which can block the use of prohibited software to prevent the candidate from accessing the internet during the test. Some systems can also monitor the use of connected external devices (for example, a mouse or keyboard), to ensure that they are being used appropriately and that it is the candidate who is undertaking the assessment (and not, for example, someone who is responding on their behalf from off camera). Some systems may not offer a lockdown feature, but may allow the invigilator to directly view and monitor the candidate’s screen (Foster & Layman, 2013). It is worth noting that such systems may technically be integrated into the onscreen assessment software, rather than the RI platform itself.

In summary, there is a multitude of ways in which RI can be implemented. There are many service providers who offer AOs ‘off the shelf’ solutions, while it is also possible for AOs to develop or adapt their own in-house systems and platforms. It is also possible to use non-specialised remote conferencing software to deliver RI. As we will see, most of the systems we discussed with the AOs we interviewed were in fact developed and tailored to the AOs’ needs, combining and adapting software in such a way that the invigilation works in tandem with the onscreen assessment system.

Literature review

In the previous section we broadly described RI systems and how they may be configured. In this section, we turn to the research literature to build a wider understanding of the main issues surrounding the use of RI in educational assessment contexts. The literature highlights the need for AOs to carefully consider the processes and infrastructure that underpin RI, including exactly how candidates are monitored, the robustness of the identification process, academic integrity, privacy and security (Bohmer, Feldmann, & Ibsen, 2018), the quality of candidates’ internet connections, the availability of appropriate IT equipment, and the provision of a practice exam (Davis et al., 2016). In addition, previous research provides evidence about how RI may impact upon the experience and performance of examinees. In the section below, we review this evidence in 2 parts. First, we discuss the impact of RI on test scores and the potential for it to reduce malpractice. Second, we discuss the concerns and experiences of the candidates.

The impact of RI on assessment scores

Research on the impact of RI has been carried out in the context of higher education, predominantly comparing candidates’ remote performances under RI with their remote performances without any invigilation (Alessio, Malay, Maurer, Bailer, & Rubin, 2017; Dendir & Maxwell, 2020; Hall et al., 2021; Hylton, Levy, & Dringus, 2016; Karim, Kaminsky, & Behrend, 2014; Steger, Schroeders, & Gnambs, 2020).

Evidence based on this literature is mixed and inconclusive. Some studies suggest that, compared to non-invigilated assessments, RI is associated with lower test performance (for example, Dendir & Maxwell, 2020; Steger et al., 2020) while other studies report mixed (for example, Hall et al., 2021) or no impact of RI on candidates’ scores (for example, Karim et al., 2014).

Findings based on a few studies that compared students’ test scores in contexts where the assessments were remotely invigilated with those in contexts of in-person invigilation were also mixed (Jones, Tong, Liu, Borglum & Primoli, 2022; Jaap et al., 2021; Weiner & Hurtz, 2017; Wythisatian, 2020). For example, Weiner and Hurtz (2017) implemented a quasi-experimental study with 14,623 examinees to evaluate the comparability of RI (live) exams and on-site in-person invigilated exams for a professional licensing qualification. The research showed that test results were similar in both conditions. These results are, however, in contrast to a study of 65 graduate economics students by Wuthisatian (2020), who found that students invigilated on-site performed better than students who were remotely invigilated.

Candidates’ performance under different invigilation conditions is often used as a proxy for the occurrence of malpractice (Dendir & Maxwell, 2020; Hall et al., 2021; Hylton, et al., 2016; Karim et al., 2014). It is argued that, in absence of invigilation altogether, students are more likely to cheat and hence receive inflated scores. The mixed and limited evidence so far available makes it difficult to draw confident conclusions about the merits of RI in reducing incidents of cheating solely based on candidate performance data.

There is some evidence that suggests that candidates under RI take less time to complete assessments compared to those who are not invigilated (Alessio et al., 2017; Hylton et al., 2016), which it has been argued may be because unmonitored students have more opportunity to engage in time-consuming misconduct (Hylton et al., 2016). Such a conclusion is, of course, speculative. Alternatively, it may be something about the RI process, or the experience of assessment under RI, which may be causing candidates to commit less time than they normally would.

Of course, there are many factors that can affect assessment performance, and the time candidates take to complete an assessment, and differences between remotely invigilated, in-person invigilated, and non-invigilated assessments need to be carefully interpreted. Although there may indeed be differing levels of malpractice, variations in performance and time to complete an assessment may instead (or also) reflect other differences between the cohorts, such as ability, conscientiousness and test anxiety (Sinha et al., 2020). It is also important to note that there are likely to be differences across qualification types and assessment cultures, meaning that generalisations are extremely difficult to make. For example, much of the existing research has taken place in the American higher education system and it is unclear whether the findings apply beyond this context.

Overall, and keeping these caveats in mind, the research does seem to suggest that, compared to having no invigilation, RI has the potential to increase the integrity, the security and hence the validity of online assessments by decreasing the opportunity for malpractice (Davis, Rand, & Seay, 2016; Dendir & Maxwell, 2020; Paredes, Peña, & de La Fuente Alcazar, 2021; Vazquez, Chiang, & Sarmiento-Barbieri, 2021). Context-specific research would, however, be required to understand the potential impact of using RI over traditional invigilation methods across the range of qualifications available in England.

Experiences and concerns around the use of RI

At present there is little evidence which reflects on the candidates’ experience of being assessed under RI. Milone and colleagues (2017) surveyed students about their experience of using particular RI software in their course evaluation form across 2 semesters. Approximately 60% of students in one semester and 90% in the following reported satisfaction with their experience of RI. Those who were unhappy explained that the set up was too time consuming, or that they had issues with the invigilator. About half of the students also indicated that their experience of the RI software would influence their decision about whether to take a course that used it in the future. Specifically, 34% of the students indicated that they would rather take a course that used RI (instead of in-person invigilation), while 23% of the students would choose not to take a course with RI. The remaining 44% of students indicated that it would not make a difference to them (Milone et al., 2017).

Woldeab & Brothen (2019) explored test anxiety and students’ performance in a sample of 631 psychology undergraduate students at an American university. Most of the students (587) took their final exam on computers in a centre and were invigilated traditionally (in-person), while 44 took the same examination using RI. The performance of these 2 groups did not differ overall, but students with higher anxiety traits taking the RI test felt that the proctor made them feel uncomfortable. Although based on small samples, these studies illustrate how the impact of RI may vary between individuals and that AOs deciding to use RI should consider how it may influence the experience of the candidates.

In a context of increasing demand and rapidly changing technology, it is critical that staff and students have the necessary skills to set up and interact with RI systems. Dawson (2021) suggests that students should be supported in practical and technological matters, such as setting up equipment, and should be provided with clear information about their rights and how their information will be used. On the other hand, AO staff should be supported in learning how to use the software and how to write assessments that are compatible with it. Teaching staff require training in how to administer exams and to interpret information to ensure that the exams are appropriately invigilated.

Similarly, to alleviate stress and anxiety linked to technical issues, Cramp and colleagues (2019) suggested that candidates should be informed about the rules and technical requirements, and equipped with the necessary technical skills, prior to sitting the assessment (Cramp, Medlin, Lake, & Sharp, 2019). In addition, they suggested that support should be available to candidates before and during exams, and that clear guidance should be provided to explain what candidates should do if they believed that something has gone wrong.

As previously discussed, high quality internet access and the availability of suitable devices is critical (Langenfeld, 2020). Moore and colleagues (2018) surveyed a sample of students taking a test as part of a national assessment in the USA and reported that a proportion of students taking online tests rated their internet service as “unpredictable” or “terrible”. Although almost all students had access to an electronic device, over half of the students reported that this was a smartphone and therefore inappropriate for the purposes of assessment. These findings indicate that, although the candidates had the opportunity to sit the test remotely, a proportion of students may have been at a disadvantage compared to their peers due to the lower specification of the technology they had at their disposal.

The evidence presented in this section is primarily from research studies of low-stake tests, university assessments or work-based certifications. There is limited research about the impact of RI on high-stakes assessments. Alpha Plus (2021) has, however, recently conducted research into the RI approaches implemented by AOs in the UK and the experiences of AOs and candidates using RI to navigate the pandemic. Their research suggests that, overall, the candidate experience of RI has been positive and that the AOs and their candidates’ satisfaction levels increased as they adjusted over time to RI delivery. The research emphasises the need for RI systems to provide support to candidates both prior to and during the assessment, and the importance of capacity planning with providers to ensure that candidates do not have to wait long to access their exams.

Indeed it is important to ensure that candidates are comfortable about taking an assessment using RI, particularly as the assessment often takes place in the candidate’s home. RI often collects information about a candidate, such as recordings of their voice and images of their face and home environment. Various studies raised issues around candidates’ sense of privacy. For instance, Parades et al. (2021) conducted interviews with examinees experiencing RI and reported that around 40% of the respondents said that they had experienced a sense of intrusion to their privacy.

Similarly, Karim and colleagues (2014) presented participants with a cognitive test online that was either invigilated or not invigilated. Participants in the invigilated condition were asked to film their own testing session and were led to believe that this would be viewed by a researcher to verify their identity and to ensure test integrity. Test takers reported negative emotional reactions, experiencing greater pressure and describing concern about their privacy.

More recently, Kharbat and Abu Daabes (2021) explored the attitudes and concerns of IT students using RI in their final university exams during the pandemic. While one third of the students felt that the place where they conducted their exams was suitable and free from distractions, 80% of students felt that their workspace was too small for them and some expressed concerns around their family being disruptive during the exam. In this study, around 75% of students across different qualifications and ages expressed concerns around invasion of privacy and anxiety about being continuously monitored.

While RI can be unsettling for some candidates, others have expressed that the presence of a live form of RI can be reassuring. Research has shown that participants who experienced this type of RI felt better supported and experienced less stress about the potential for something to go wrong (Lilley, Meere, and Barker, 2016).

Research questions

Building on the literature, this research study aims to engage with the following explorative questions, focussing on the perceptions of a sample of AOs that offered RI with assessments for at least some of their vocational and technical qualifications in 2021:

1. What are the typical capabilities of existing RI systems and how are they used in practice?
2. How was RI employed in 2021, and what were the challenges and benefits?
3. What was the perceived impact of RI on the validity and reliability of the assessment?

These questions, particularly the last 2, were also explored through interviews with candidates who had recently undertaken assessment which was remotely invigilated. The intention is to deepen our understanding of RI and how it is operated and experienced, therefore informing our approach to regulation.

Method

Interviews

Given the exploratory purpose of the research, an opportunistic approach was taken to recruiting AO participants who had recent experience of implementing RI. We identified a range of VTQ AOs who had used RI in summer 2020 and invited them to take part in the research. We then asked those AOs to invite their candidates to contact us if they had recently taken an assessment under RI and were willing to talk to us about their experience. Though this approach does not allow us to draw generalisable conclusions about the prominence of certain approaches to RI, it does generate insights into the factors and mechanisms which underpin how decisions about RI are made, surfacing helpful ideas and potential issues alike.

Interviews were conducted with 7 AOs and 7 candidates between January and March 2021 via video-call. The AO interviews usually involved 2 or more representatives from that AO. This was because those AOs who agreed to take part wanted to ensure they had a range of expertise available during the discussion (given the relatively technical nature of the topic). A total of 17 representatives were involved from across the 7 AOs.

Discussions were guided by a semi-structured interview schedule which aimed to elicit information and views about the process of implementing and using RI. The interview schedule provided a structure and focus to guide the interview, but neither the participants nor the interviewers were prevented from sharing information about anything outside of the interview schedule which they felt was pertinent. Each interview lasted approximately 1 hour for the AOs and 30 minutes for the candidates. All interviews were audio recorded and transcribed verbatim by an external supplier for transcription services.

Analysis

The content of the transcripts was analysed thematically by 2 researchers, using a flexible method for exploring data and identifying patterns and themes (Braun & Clarke, 2006). The analysis was organised and conducted using the NVIVO qualitative analysis software package. An initial coding scheme was prepared on the basis of the questions within the interview schedule and used by the 2 researchers to individually analyse one transcript. Following this initial process, the code was refined accordingly, based on the information and themes discussed in the interview. A final coding scheme was then developed and agreed between the researchers. Written transcripts of the interviews were coded by a researcher, with a subset of these double coded by the second researcher to ensure consistent interpretation. To ensure that themes and quotes were captured in context, passages of text were coded (rather than just individual sentences). Researchers had access to the original audio transcript of each interview, allowing them to account for the tone of discussion wherever there was any ambiguity.

Findings

The findings of the qualitative analysis are presented within 5 broad topic areas:

1. choosing an RI system
2. technical considerations around RI implementation
3. identifying and managing malpractice
4. learner experience
5. long-term planning

Before delving into these themes in detail, it is worth noting that we have chosen not to provide counts for how frequently each theme arose or how often particular views were expressed. Instead, we present an extensive range of quotes to represent the range of experiences of those interviewed in their own words.

This approach has been taken because findings from qualitative research of this type and scale are limited with regard to generalisability. In this study, we were interested in capturing the nuance of the experience of each AO, along with the experience of students. The approach does not allow us to ascertain how RI is operated and perceived across the entire sector, but it does allow us to develop a deeper understanding of the factors that are important when implementing RI, and the factors which are likely to affect the experience of AOs and candidates.

Most quotes in this report are from the AOs as the themes were discussed more in detail with them, however, we wanted to get the personal experience from the examinees. While the experience of the examinees is mainly presented under the ‘candidates’ experience’ theme, some quotes appear in other sections of the report.

Selecting a RI system

There are a wide range of RI platforms, all of which are customisable. AOs either used a third-party provider or developed a system ‘in-house’ for delivery of RI. Similarly, some AOs used a third-party provider for sourcing invigilators, while others decided to use their own staff to perform the invigilation. Decisions around which RI system to implement appeared to be largely driven by the specific needs of the AOs, how the RI system interacts with the remote assessment system, and the IT skills of the candidates intended to use it. Under this section we will discuss the different RI systems used by the AOs, how AOs introduced RI and the decision processes that informed its implementation.

Remote invigilation methods

The RI methods used by the AOs in our sample were ‘live invigilation’ and ‘record and review’, the 2 approaches we outlined in the introduction. Most of those we interviewed subscribed to a third-party system, while a minority of AOs implemented a system developed in-house. These systems used either AO staff as invigilators or relied on the recruitment of external invigilators (which was often managed by the RI platform provider).

The AO quoted here adopted a ‘record and review’ invigilation and both the RI system and the invigilators are provided by a third-party. Students can take their assessment on-demand .

We use a system, which is a third-party add-on to our existing online assessment platform. That effectively is a service that we buy in. The proctoring itself is not completed by [the AO] staff, it’s completed by people all around the country, but one of the things that they were able to offer us was effectively 24/7 assessment. And one of the considerations we had was if we were to do it ourselves then you’d only be able to have assessment at certain times. AO5

One of the advantages of the ‘record and review’ system is that assessments can be taken at any time. The candidate can book in for their assessment and log in when they are ready, without the need to wait for a session. This flexibility would be more challenging to achieve with live invigilation.

In terms of record and review, clearly that gives us a really great option because it allows us to offer the candidate the chance to take the exam at a time and a place of their choosing… we currently offer a 6-week window… And of course, you can’t do that with live invigilation, because live invigilation involves a booked appointment and a timed appointment for a specific person. […] I think from an administrative perspective and also from a learner perspective, live invigilation isn’t quite as flexible as record and review. AO3

Although ‘record and review’ offers more flexibility, this type of invigilation may not be suitable for all assessment types and candidates. We discuss this later in the report.

‘Live’ invigilation offers a unique benefit, namely that the invigilator can interact with the candidate, if necessary. This is usually through a chat function that is built into the system. The AOs we spoke to were keen to emphasise that the only help invigilators could provide related to the logistics of the assessment, and they could not help candidates to interpret or answer questions.

At any time, the invigilator can interact with the test taker, so they can use chat. Because they’ve got their mobile phone… they can also be called if the invigilator wants to validate anything. AO1

So while a candidate can’t say “what’s the answer to question 4”, they can reasonably say “I can’t remember how to move onto the next question”, or “I want to go back to question 2, how do I do that?”, and it’s not unreasonable that we should be able to answer those questions. AO2

Live support was considered to be particularly valuable to candidates at the beginning of the assessment, during the set up.

Well, obviously, you can’t support them if you’re not doing it live […] I think where I was going was to say that a lot of them don’t necessarily have the IT skills that you might expect of someone taking another qualification. We found that live proctoring was much better because we could give them that live support. And in the most extreme cases we’ve even ended up messaging the candidate and saying tell me your phone number… AO4

One of the AOs explained that in live invigilation, the invigilator can intervene if the candidate is doing something against the rules. This is not possible with record and review.

…the advantage of live invigilation is that if you see a learner doing something wrong or is about to do something wrong then the invigilator can intervene immediately and stop anything happening and certainly stop the situation where there might be any malpractice and therefore the learner could be disqualified. With record and review… [the assessment] is looked at and reviewed anywhere between 24 and 72 hours after the exam has taken place. AO3

The number of invigilators that AOs employ must be proportionate to the number of candidates taking the assessment, which means that high-entry assessments require more resource to mitigate any potential risk of malpractice. As one AO explained, although their preferred method was live invigilation the high volume of candidates meant that, for some qualifications, they were considering a record and review approach instead.

We’re exploring both live proctoring and record and review route, mainly because the volume of learners that we have go through [our] system […]. So if we were only to use the live proctoring route, we could quite quickly run into resource implications and issues for centres for ourselves for the software solution we’re using. Whereas, if we can also look at the record and review route, and then have the checks and balances in place to make sure we are managing the risks around any of those test issues and the malpractice etc., then we think that that is also a scalable option using that approach as well. AO1

All of the AOs we interviewed had been considering the use of RI as a potential optional solution for their assessments before the pandemic, and to varying degrees, had been testing and trialling software and systems. The need to make the assessment available for candidates during the pandemic had accelerated the implementation of RI. Some felt they would have preferred more time for piloting.

It was conversations we were having anyway. And then with the pandemic it sort of came more to the fore. I think if I’m being honest, we probably would have preferred to wait a bit longer, but with needing to get assessments up and running and try and ensure that learners weren’t disadvantaged by not being able to take those assessments. […] So that’s not to say that we hadn’t performed testing of systems, it was just more really around the guidance and the comms to providers and learners, that wasn’t quite in place to the extent that I think we would have liked, because we were reactive at that point… AO5

Most AOs explained that they would not have been able to provide online assessment with RI if they hadn’t already trialled RI and negotiated some of the challenges and complexities of using it.

I would say there’s no way we could have done it if we’d not been so far down the road towards it already, definitely. There’s so much to think about that we’d already done a lot of that thinking… We were already working with an online exam platform. And they were already integrating or working towards integrating… the overlay of the proctoring over the exam platform. And so there’s honestly no way we could have started it from scratch and had it up and running in 2 weeks. AO4

This section has provided an overview of the different RI systems and has described some of the factors that AOs considered when implementing them. In the next section we will discuss the factors influencing the choice of an RI system.

Factors influencing the choice of RI system

There are a wide range of RI platforms on the market, each with different services and costs. AOs took into consideration several factors in the process of deciding which RI system to implement. Overall, 4 main factors were considered:

• the integration with the AO’s remote assessment system
• the cost of the viable systems
• the security of the assessment
• the type of candidates taking the assessment

AOs had considered the logistics and costs of introducing a RI system. Those that already had a remote assessment system in place, preferred to implement software that could seamlessly integrate with their existing systems.

Our system, we’ve got our own e-assessment system, which we’ve had for years and years and years… But our system talks seamlessly with the [RI software] system and it’s as good as being in the room. AO6

Security was often also mentioned as a priority. For a paper-based exam, papers are delivered to the school or college physically and kept secure until the exam begins. With remote assessment, exam papers are delivered to the student digitally. It is critical for the AO to ensure that the integrity of their assessments is maintained. For example, if an assessment is on demand, AOs seek to mitigate the risk that a candidate may share the questions, either through taking a screenshot or writing the questions down. Therefore, most AOs sought a system where they could view the candidate’s screen and monitor if they were making any attempt to compromise the assessment material. They also sought a system that would prevent, for example, use of the copy-paste function or screen recording software.

…we’ve got an on-demand paper bank so the security of our papers is obviously at the utmost of importance for us, we didn’t want to go down the less secure route of [conferencing software] or anything like that. We were really keen to have a really secure service. AO3

The availability of an ‘environmental’ camera was often considered significant. Some AOs preferred a solution that offered a multiple camera view during the exam because they felt it better ensured that the candidate and their environment were sufficiently monitored.

…one of the key things is those multiple camera views and the screen view. I think that’s absolutely critical. Some of the third-party solutions on the market will just literally give you a webcam view, and we didn’t feel that was enough. It’s too much potential for malpractice going on out of that camera view, candidates accessing mobile phones or notes or all of that. That was an absolute solid requirement for us. We didn’t want to proceed without those multiple camera views. AO1

For those AOs implementing ‘live’ invigilation, the invigilator could intervene in real-time if there was an attempt to compromise the assessment materials. For those AOs that used record and review, any issues arising would already have affected the assessment itself because they could only be identified in a review of the footage after the assessment had taken place. We discuss malpractice further in a later section.

It is important to note that AOs balanced the need for security with the need to provide candidates with a good experience in different ways. For example, one AO decided not to use multiple cameras because of the additional technical challenges that this approach might introduce for candidates.

And the reason we don’t is it comes back to the type of candidate and what they can cope with. And it really would just be too much for a lot of them to cope with [an environmental camera], so we decided that we weren’t going to go with the third stream and that we would initially tentatively see how it went without that. And we’ve found that it works really well and I think it would make it a lot more stressful for a lot of them who just wouldn’t be able to cope with that. AO4

Indeed, the experience of the candidates was one of the most important drivers for influencing the AOs’ decisions. Not all candidates are comfortable with technology, and this can affect their ability to sit assessments under RI.

Because these are individuals who are not necessarily used to using computers in their daily life… I think some of our learners have really struggled and some of our providers have actually backed away from using it, because they felt it didn’t work for the cohort of learners… AO5

Most of the AOs interviewed stated that they used a third-party provider to deliver RI, while others preferred to develop a platform in-house that met their invigilation needs. The AOs stated that they considered various solutions, including using video-conference software rather than specialist RI software. But although viable in the short-term to help navigate the pandemic, they generally felt that non-specialist software did not offer the required level of security in the long term. Although the details varied between the AOs we interviewed, most explained that they had undertaken research and ran pilot studies before finding the provider that fitted their specific needs.

Independently of the RI system chosen, the invigilators play a crucial role in the delivery of remotely invigilated assessment. In the next section we explore the role of the invigilators.

The role of the invigilators

We have seen in the previous section that invigilators can be either AO staff or those employed by an external company. Invigilators are given guidance and training about the checks that they need to complete before, during and after assessments, such as checking the identity of the candidate and ensuring that assessment conditions are maintained. In the following sections we will explore the role of the invigilators in ‘live’ and ‘record and review’ invigilation.

Live invigilation

Live invigilation may take place on a one-to-one basis, with a single invigilator assigned to a single candidate. It is, however, more common for several candidates to be invigilated at one time. The candidate to invigilator ratio varied across AOs and was often dependent on the nature of the assessment and the level of experience of the individual invigilators. More experienced invigilators were often permitted to scrutinise a higher number of candidates at one time, while more junior invigilators would usually inspect fewer candidates.

So there’s 4 [learners per invigilator] at any time. Yeah, the maximum’s actually 13 [in the system], but the way we do it, when we bring a new invigilator on board, the new invigilator will only do 2 until the invigilation team are happy that they know exactly what they’re doing, and then they move up to 4, and 4 is the number that we’ve decided we’re happy with. AO6

Each AO carefully considered the ratio of candidates to invigilators to ensure the quality of the invigilation was to a high standard and that there was no negative impact on the candidate’s experience of taking the assessment.

And we’ve been pretty careful in not either disadvantaging the learner or reducing the quality of the invigilation […]. We don’t want to overstretch or overreach the invigilator […] It is about us being satisfied that the invigilation procedure and process is carried out thoroughly and correctly and so that it doesn’t have a negative impact on the learners and the experience of it as well. AO2

Depending on the RI system, each invigilator will see the video feed of the candidate(s) (their main camera), an audio feed, a ‘mirror’ image of their screen, and the secondary (environmental) camera. AOs using this system highlighted the benefits of having access to various pieces of information simultaneously and being able to cycle through the various feeds of information.

So, each invigilator will have a feed for each of their 4 learners, but that feed would have all of the 3 feeds together. So, you’ll be able to flick between screenshare, webcam and the separate device that shows you the room. AO6

All of [the invigilated] candidates will be on screen all the time, so they can always see all of them. So in the top left of their screen there’ll be a double display for one candidate. So that would be the candidate’s webcam feed and their screenshare. And then underneath, there’ll just be one display per candidate and the proctor can toggle between camera feed and screenshare and then at any point can pull any of them up to the prime position where they can see both at the same time, or… you can just hit a button and it will automatically just circulate those that it puts in the prime position and move around. You can also double click on them and it’ll give you a full screenshot of either a candidate’s camera feed or their screenshare if you just want to look closer. AO4

With live invigilation, some of the AOs were keen to ensure that invigilators in a given assessment session were connected using conferencing software, independently from the RI system. This allowed them to speak to each other and ask for advice from other invigilators, including more experienced invigilators. These more experienced invigilators often held a supervisory role.

[What] I think is really valuable is the fact that the proctors are all connected through [Microsoft] Teams . So when we’re doing the exam session the supervising proctor will call all the local proctors, make sure everybody’s there and then the candidates will start to appear. But we’re all in communication. […] And there’ll always be more than one invigilator, so we’ll always have a supervising invigilator and then at least one, but usually 2, sometimes more than 2, local proctors. The supervising proctor will be responsible for the session overall and will have access to the system through what’s called the back office, which means that they can focus in on any individual candidate from the session. It means they can also do things like check ID photographs and that sort of thing. […] It’s an extra pair of eyes. And then once everybody is in and doing the exams and running smoothly, the supervising proctor will just randomly be double checking on candidates that are all being watched by local proctors. AO4

Record and review

In our sample, where ‘record and review’ RI systems were used, invigilators tended to be employed through the third-party RI provider and AOs had no direct interaction with them. The RI provider was, however, required to ensure that invigilators met agreed standards and carried out work in line with a contract. For example, the RI provider needed to provide a certain level of training and to ensure certain checks and safeguards were in place.

For some AOs, ensuring the quality of this third-party invigilation was a process of trial-and-error during their testing phase. One of the AOs explained that after the first session, and following a thorough review, several issues and inaccuracies had been detected with the invigilation process. As a result, a rigorous training programme had been put in place so that new proctors were required to qualify as ‘tenured proctors’. Through this mechanism, this AO chose to only allow experienced invigilators to proctor their assessments.

Essentially [new invigilators] go through a three-step process and they have to have all of their work monitored for the first time they do it. …And then they don’t become what they call tenured proctors, so more experienced proctors, until they’ve done it for a certain length of time, or got through a certain quantity without error. …that’s been a specification that we’ve put in place following the August when we first trialled it… AO7

AOs often prepared guidance to identify the potential infringements that invigilators should be monitoring for during the review process. During our interviews, this guidance was often discussed in terms of ‘thresholds’, the point at which an issue or infringement was clear or frequent enough for it to be flagged for further review. For example, a candidate may occasionally be looking at something off camera, and the invigilator will make a judgement as to whether such behaviour is suspicious enough to warrant further scrutiny.

Flagged footage is then reviewed by someone within the AO, who acts as an arbitrator to decide whether malpractice has taken place. If the threshold for flagging an issue is too high, there is a risk that potential malpractice may go undetected. On the other hand, if the threshold is too low, there is a risk that too many flags are sent to the AO and review becomes a significant drain on resources. One AO explained that they had to refine their guidance for the invigilators with regard to the threshold for issues that should be flagged.

The first element is we put together a violations list that we sent to our partner and they just follow that list to the letter. So if we turn round and say if someone’s done this then you need to grade them amber, they will grade them amber and pass it to us for a final check. So they don’t make the decision on whether to disqualify an individual, that decision is made by us when we’ve had a chance to sit and review it a second time. And that list is an extensive list, there’s over 90 different violations that can possibly take place… And we review that list regularly. We’ve reviewed and updated it as a result of our pilot programme and we came in I think quite hard at the beginning, we were pretty ruthless about what we wanted to see. And based on experience we’ve been able to relax that slightly. […] And so there are things that we’ve been able to reduce in terms of the red or amber to green and say oh that’s OK, that’s fine. It’s a learning process. It’s been a learning process for us too. AO3

…I think that’s the hardest thing when you’re using these proctors, but you can’t rely on them to make a judgement call on your assessment, because that’s not what they’re paid for. They’re literally paid to follow your guidance. So maybe our guidance possibly needs refining, so that we get fewer flags added onto. […] That’s one of the challenges at the moment, because we don’t know how to effectively refine our guidance to maintain enough flags coming to us to look at things, so we’re not going to miss anything. How do you define that, that’s really hard. AO7

As noted in the quote above, invigilators are not expected to make a judgment as to whether an event is malpractice or not, they are instructed to follow the guidelines and flag any potential issues in the recording. This internal review process is undertaken for both record and review and live invigilation and it is further described in the Internal Review section. In the next section, we will discuss the technical challenges encountered by candidates, centres (schools and colleges) and AOs.

Technical considerations around RI implementation

One of the major themes of discussion was about the technical challenges around the implementation of RI. The need for a reliable internet connection and other potential technological limitations were considered carefully by all the AOs interviewed. They were keen to ensure equality of access to candidates. RI poses different challenges when compared to invigilation in an exam hall. Some of these will be presented in this section: they include the device set up, the verification of the identity of the candidates and ensuring that there are no materials (for example, books or notes) in the room to help the candidate.

Device set up

Undertaking exams under RI usually requires the candidate to have a certain level of IT skill. To ensure that the candidates are ready to undertake the assessment, the AOs explained that they invited those who had opted to take the assessment using RI to download the required software and test the functionality before the exam. This is critical for the candidates to familiarise themselves with the system and to ensure that their broadband meets the minimum requirements. Other checks include testing audio and video.

We always say 2 weeks out do the log on practice. Practice logging on to make sure you’re familiar with the system. There’s a system check through [the system] as well, so we make sure that everybody does that to ensure they meet the requirements basically and that’s done in advance. So hopefully all being well on the day they log into a system they have already downloaded, they’re already familiar with. AO7

The complexity of the IT requirements varied across AOs. While the set-up process may be straightforward for candidates that are comfortable with the technology, others may find it challenging. Taking into consideration the type of candidates that are likely to sit a particular exam and tailoring the guidance and (where possible) the system to their needs played a critical role in shaping the AOs’ RI implementation process.

Reliability of internet connection

The availability of a reliable internet connection is critical for allowing candidates to access RI exams without disruption. This is because the broadband often needs to be able to support screen sharing and one or more cameras alongside several software programmes that are running at the same time (that is, both the remote assessment system and the RI system, if the 2 are not integrated). A connection of insufficient bandwidth will have a negative impact on the candidate’s experience.

There is a minimum bandwidth required, because there’s 4 streams of information that need to be stable throughout this assessment. You’ve got webcam, tethered cam, digital audio, screenshare, that’s a lot – well, it’s not a lot, but in terms of poor internet connections it’s a lot. And if it doesn’t work, you know, you will be met with frustrating dropouts and error messages, and learners will attribute it to your [system]. AO6

To mitigate the risks associated with the reliability of the broadband, 2 AOs reported that their remote assessment system was designed to handle a short connection drop-out, allowing candidates to reconnect and resume their test if they were able to do so within a certain timeframe.

If the internet goes down briefly they are able to log back in again, they can put a QR code in and their keycode in again and as long as it’s done within 15 minutes they can actually resume the exam. And so they don’t lose out if they can get their internet back up and running. AO3

For many AOs, in the event of a connection failure, the invigilator or AO has no means to contact the candidate, let alone monitor the candidate’s environment during the affected timeframe. While AOs accepted that this may happen and allowed the candidate to continue with the exam as soon as the connection was restored (if this was a short dropout), other AOs felt that this would compromise the integrity of the test and that the test would need to be re-sat.

Room scan

Checking the candidate’s environment before starting the assessment is one of the processes that AOs used to minimise malpractice. This check (which is commonly referred to as a ‘room scan’ or ‘environmental check’) requires the candidate to use a camera to capture a video of the room, allowing the invigilator to check for prohibited notes or books, as well as to check that the candidate is alone.

There’s also a series of pre-flight checks. Just before the exam we ask them to show us the environment using their tethered phone. So all 4 corners of the room, under the desk, even in their ears to check for listening devices and Bluetooth headsets and what have you and just give us a good view of the room prior to the exam. AO6

A few AOs decided not to perform a room scan to avoid adding stress for their candidates, in particular those who are less comfortable with the technology.

Authentication

Authentication refers to the process of ensuring that the person sitting the exam is the person that signed up for the exam. In instances where there is a live invigilator, authentication usually involved the candidate showing an ID document to the invigilator via a camera, replicating the way a candidate’s identity would be verified at an exam centre before an exam.

Authentication of learners is quite an important element of that, and that’s certainly something that the proctoring system works very well with. They’d have to show identification for the proctor, so it’s things like a driver’s licence or something like that […] AO5

For ‘record and review’ invigilation, the process is similar – candidates are usually required to show ID to the camera, ensuring that the information is visible and clear. In instances where the information on the documents is found to be unreadable in the footage, the video is flagged and the AO subsequently contacts the exam centre to verify the identity of the candidate who has been filmed. Given that the invigilation is not live, the exam is not interrupted but the release of results can be delayed.

It’s part of the setup process for the exam. And they have to show to the camera in a very clear way a passport or driving licence and that is verified against their name. And indeed, quite a number of times a number of the violations are because we’re not able to verify the individual and at that point then we have to go back to the centre and say we now need to confirm who that individual was, because we either couldn’t read or they didn’t show it to the camera properly or something like that. We’re very careful on making sure that if we can’t read what they’re showing us we have to go back to the centre… [These violations] don’t stop the exam taking place, it just comes through as a red violation and therefore it takes just a little bit longer for us then to release the results at some stage once we’ve confirmed their ID. AO3

The candidates we interviewed mentioned the authentication process but did not report experiencing any difficulties. One of the candidates did, however, experience stress when the invigilator was unable to see the document on their screen due to poor lighting. The candidate explained that they managed to get through the authentication process eventually, but the initial failure had made them feel anxious before the start of the test.

I’d get a message pop up saying, we can’t see you, you need to move to a better lit room, and I’m like, I’m in a room with 6 lights on… that was a little bit stressful and unnecessary. Candidate 1

Dealing with unexpected technical challenges

Along with connectivity issues, AOs described some of the other unexpected technical problems that may occur during a live session. The quote below exemplifies one such issue, describing a situation where anti-virus software interfered with the RI systems functionality.

We’ve had 2 that spring to mind where there was a technical issue, basically, and again we can see from the recordings that an error message flashes on the test itself, and then the learner just can’t do anything because the test has just crashed basically. And they were upset by that. But they understood eventually with a bit of back and forth with the centre and with the learner they would have to do that test again. AO1

To minimise the stress around technical issues, one of the AOs explained the value of live technical support for candidates during the assessment.

…prior to the exam, if we see them struggling, our support staff will pick up the phone to them, you know, they’ll say I can see you’re struggling here, I can see you’re not quite getting your phone tethered or whatever, then we can provide that support there prior to the exam. AO6

Rest breaks

Assessment may vary in length and candidates may require comfort breaks. This poses a challenge for AOs, who need to ensure that exam integrity is not compromised but are unable to control or monitor the entirety of the candidate’s environment. If the assessment is short, the AOs may advise the candidate that they are not allowed to leave the room. For longer exams, or where the candidate has specific personal needs, breaks may be required. Most of the AOs we interviewed did not allow rest breaks during their assessments and made this clear in their guidance.

With regards to comfort breaks and toilet breaks, our rule is that the learners aren’t permitted to leave the exam environment. We make that very clear to them, they get an info video, which is only a minute long but it’s short, sharp and punchy and just explains that you can’t leave the exam environment. So, you know, we’re not saying that everyone is going to cheat, but what we’re saying is if you leave that environment we can’t offer any sort of supervision as to what’s going on, so therefore we can’t say definitively that our assessment is valid and therefore we have to invalidate. We have had a small number of learners that have said look I’ve got to go to the toilet, and we’ve got to take the hard route and say we’re terribly sorry but you’re going to have to do it again. AO6

In general terms though once you’re sitting down for that exam you’re not allowed to leave the room. …If you’re at home, I’m afraid there is no way that we can invigilate you if you leave the room. AO3

A few AOs said in the interviews that candidates were allowed a very short break and added that not many candidates needed it.

AOs offering RI need to consider the suitability of their assessment for RI, in particular for longer assessments, and are exploring alternative approaches to mitigate any risks associated with breaks, such as breaking the assessment into sections. The interviewee below reflected on how this might affect the nature of the assessment.

Normally a learner can look through the question paper, decide which questions to tackle first. They can come back to answers they’ve already completed, look them through and add to their response. They can do a final check of the whole paper before they finish the exam. If you put in break points to keep the exam secure, you would need to say right here’s your opportunity to take a 5-minute break. But you can’t see the content in the latter half of the paper until you come back. And once you’ve done that you won’t be able to go back and revisit your answers in the first half of the paper. So it really changes the way that learners are used to approaching that lengthy exam. AO1

Reasonable adjustments

Awarding bodies are required to make reasonable adjustments where a candidate, who is disabled (as defined by the Equality Act 2010), would be at a substantial disadvantage in comparison to someone who is not disabled. The most common adjustments discussed by those AOs we interviewed were extra time and modified papers. Invigilators were made aware of the adjustment ahead of the assessment.

Mostly for […] learners who are able to do online exams, it’s normally going to be extra time, type font and it might be changing the colour of the screen. So the invigilator would need to know so that they aren’t shocked where there’s a black screen and white writing, for instance, or a yellow screen and blue writing. …additional time would be set on the exam itself, so your invigilator would see that that exam is 50 minutes long instead of 35 or whatever. AO2

This research did not include candidates who required reasonable adjustments. Therefore, potential issues that relate to how such adjustments would interact with RI processes and systems cannot be discussed in depth within this report.

Overall, this section demonstrates the different technical considerations and challenges for AOs and candidates during the implementation of RI. Although some of the technical challenges can be mitigated, any unexpected issue during the assessment can have a negative impact on the candidate’s experience. The candidate experience needs to be carefully considered, particularly as RI is largely a novel area where new practice is emerging. The next section provides more detail on how RI systems mitigate malpractice.

Identifying and managing malpractice

The AOs we spoke to believed that RI was effective for identifying malpractice and could often provide effective evidence to support (or refute) any allegations. This confidence was largely based on the fact that the assessment was recorded and that they could access the recording for review.

Internal review

All the AOs had a process in place to internally review the videos that were flagged for potential malpractice. The internal review process is carried out by trained staff, who review each frame that has been flagged by the invigilators according to the list of infringements. In the case of those we interviewed, this process was carried out for both ‘live’ and ‘record and review’ invigilation.

The invigilator literally just flags and produces a report of their observation. We will then check whether we think something untoward has gone on or if we’re quite happy that it was just someone absentmindedly humming or something like that. We will make those decisions, not the invigilator. AO5

During the internal review process, the AO staff look at all flagged events in each video and use their judgement to consider each in context. This means that each violation is reviewed and assessed on a case-by-case basis.

[The invigilators] will flag everything, they’re very, very thorough and then it’ll come to my team. Now, if I’m reviewing and I see yourself and you’re doing an assessment now and I can see your environment and your toddler comes bursting through the door, […] then I’ll take a measured approach, I’ll do a little internal note and I’ll say I’m comfortable that whilst there was an interruption, there was no breach to the exam conditions based on the evidence that I have in front of me and therefore I’m happy for this one to be released. […] It’s not steadfast for everything. We do review and we do base it on the evidence available to us. AO6

As we have already noted, the review of the videos requires attention and time. While some videos may only have a small number of flags, others may capture more complex situations that require a thorough review.

We’ve got a team of 20 reviewers, first stage reviewers from centre staff who are trained to go through the process. The trouble is it takes a really long… we work on average about 3 and a half minutes per video to review internally. Now that could take 10 minutes if it’s one that’s got loads of stuff on it and it’s something you’ve really got to look at and you’ve got to look at the desktop to work out what questions they were on when something’s happening or what have you. Or it could just be a bit of a room scan and you can whizz through it and just we’ll send a warning letter to them and then that’s that. […] And spending 10-to-12-hour days reviewing videos, it’s one of the most difficult things I think I’ve ever done. AO7

Although reviewing the flags can be time consuming, AOs did not feel comfortable in sampling flagged videos (rather than reviewing them all) because the risk of missing instances of malpractice was considered to be too high.

We can’t [sample], because if you sample them and you miss something that is clearly malpractice, then we just can’t risk it as an AO. […] We’ve identified people having their phone up or covering the camera and then changing their answers on the screen and it’s flagged as a camera cover. […] So this is the trouble, you can’t sample it, because you just don’t know what you’re going to come up against. […] We have to go through every single flag, so it’s a hugely arduous process. I’m not disputing it’s important, it has to be done there for remote assessments, but as part of developing remote assessments it’s massive in terms of the resource implication and time commitment. AO7

Indeed, reviewing the videos was one of the main challenges cited by the AOs in terms of demands on time and resources.

One of the AOs explained that if malpractice is identified, they usually contact the exam centre and the candidate to open a discussion about what happened. The AO below describes a flexible and communicative approach, which in some instances resulted in allowances being made for candidates.

It’s always a conversation with a centre to explain what’s happened, to get feedback then from the learner as to why they did what they did. Before any final result is issued the learners know and understand and the centres know and understand what’s happened and what the concern we have is. We are just trying to introduce the human touch here to make sure that as far as possible there’s this sensible positive feedback, or negative feedback, about what’s happened so that you can have a conversation before any final decisions are made by the compliance team. And very often you can make allowances and turn around and say OK we understand why that’s happened and therefore we can pass this particular occasion with a warning… The intention is not to disqualify people, the intention is to try and pass people if we can. AO3

Effectiveness of RI in identifying malpractice

Those we interviewed provided some helpful examples of how incidents of potential malpractice were identified using RI and how such incidents could be scrutinised during the review stage.

I think when it’s set up properly, I think it’s really very difficult [to cheat]. We have a screen recording of your screen. So if you go onto Google and try and look up the answer we can see what you’re doing. And we can see your face of course through the camera in your computer at all times. So if you’re shifting, looking down all the time, it might be that you’ve got notes and things there. And so we’ve also got a side camera that’s also viewing your actions to see whether there’s anybody coming into the room or whether there’s a second person in the room. AO3

Particularly in those instances where unusual or unexpected events happen during the session, having access to a recording supports potential malpractice investigations. This, however, may require the staff looking at each video to make a judgement. The quote below describes the process in an instance where a candidate had left the room during the assessment session.

We try and establish what’s happening on the desktop before they leave, what question they’re looking at, whether they can gain any advantage by going outside to look at notes or what have you. What they do immediately when they come back into the exam room, so whether they literally got back onto that question, change answers, or move around or quickly fill it in. Or whether they come back in and sit there and think well actually no, I’m going to read this question, I’m actually going to do it now. AO7

Those AOs which employed RI systems that used multiple cameras were keen to point out the strengths of this approach, both in terms of identifying potential cases of malpractice and in managing the assessment.

If sometimes you get a failure in the recording, perhaps the phone stops recording, if the battery runs out and they stop recording, at that point you take a view. If you’ve got through 2 thirds of the assessment and it’s just a camera falling but you’ve still got 2 others and there’s nothing to suggest that they’re cheating in the other screens, then I think you can take a considered view on whether that’s a pass or not. AO3

More generally, some of our interviewees compared RI to face-to-face invigilation, describing how minor infringements may be handled differently in each context.

…in a real-life situation if somebody sees somebody trying to read somebody else’s script, or look outside the window for somebody to wave an answer, or whatever, they’d probably say stop doing that. They’re unlikely to report it. So I don’t know that we have any real knowledge that we could compare [in-person and remote invigilation] with. […] I think that’s really a very, very safe system; […] and it’s being recorded, so you’ve got a record of what is going on there as well. AO2

Exam centre versus candidate malpractice

One of the striking things that emerged from the interviews was that most RI systems essentially shift the focus of malpractice investigations from the exam centre (who are traditionally required to prevent malpractice through invigilation) to the candidates (who are directly invigilated). The exam papers are delivered by remote assessment to the candidate directly, meaning that the exam centre does not have access to the papers ahead of the assessment and malpractice at the level of the exam centre is less likely.

What [RI] doesn’t allow is the malpractice that we would previously have come across most frequently, which was centre malpractice. So either the trainer or invigilator giving the candidates the answers, or the centre changing a candidate’s answers after they’ve completed the paper and before it’s sent to us. There’s just no opportunity for them to do that at all. And what we’ve found historically is that most malpractice is actually at centre level rather than candidate level. So I think there’s been a flip, because now what we see is candidate level cheating. […] [Candidates] just either forget or they think they’re cleverer than the system, and they’re not. AO4

In this section we have explored how AOs implemented RI to ensure the validity of the assessment was maintained and to prevent malpractice. Although specific examples of malpractice were identified and discussed, it is worth noting that AOs felt that most candidates were keen to engage with the rules and guidance.

If people are given the opportunity to sit an assessment in the comfort of their own home then so be it, you know, and they’ll want to do it. And I’ll say another thing as well, I mean it’s easy for me to get lost in these investigative incidents, 99.9% of all of our learners want to do this legitimately. They want to do this, they want to pass their own exam, they want to earn their certificate. It’s just easy for me to get lost, because the only ones I do get involved in are the ones that have bent the rule. But if we’re doing nearly a thousand a week and only one passes my desk then that’s a great statistic. AO6

Candidates’ experience

This section discusses AOs’ and candidates’ views about the experience of being assessed under RI. It begins by discussing the guidance that is issued to candidates before reflecting on the experience of undertaking the assessment itself.

Candidate guidance

The guidance provided by the AO and/or the exam centre (school or college) seeks to provide information on how candidates can set up the RI system and resolve any potential issues so that they are comfortable when they sit the assessment. AO guidance has often been developed over time to try to meet the needs and expectations of candidates.

We developed it [the guidance] all ourselves, because we realised right from the beginning that the candidates would need a lot of screenshots in the instructions, for example, rather than just a list of sentences telling them what to do, they would need to be shown […] AO4

Depending on the AO and assessment, exam centres sometimes played a critical role in supporting candidates with RI, often being the first point of contact in case of any issues.

But we’ve also, we have a relationship with the centre. So any technical issues, again they go to the centre, the centre comes to us. So there’s still a relationship there where the centre has a responsibility to their learner to make sure they are set up, they are ready to go. They’re first line of support in this situation. And then any issues that are still ongoing obviously we can do our best to support. AO1

We’re providing the invigilation service and the technical requirements…. It’s [the centre’s] responsibility to brief their learners pre-learning and make sure they understand this and that they talk about exam conditions and violations and make sure they understand that they are on camera the whole time and to be aware of that. So it’s a whole process, an iterative process we go through to try and make sure the learners are prepared and understands everything that’s going to happen. AO3

AOs reported that candidates did not always follow the guidance closely, particularly in those instances where they misunderstood the software or hardware requirements. One of the candidates, however, said that the guidance had been shared on multiple occasions and was clear.

The company that I trained with sent me an email with that clearly explained, then the email from the company invigilating the test, that also was very, very clear, plus our instructors on numerous occasions throughout the thing when we were doing prep for the test kept saying, you will be watched, you will be watched, you will be watched, you must do this, you must do this, you must do this. So, if anybody says they didn’t know, I would doubt that very much indeed. Candidate 5

The guidance provided to the candidates usually detailed what would be considered as malpractice. The AOs can refer to this guidance if any potential infringement is detected during the assessment session.

The majority of [cases are] where learners just have no idea what’s going on and they just ask someone or they just try to open a Google page or they just get their phone out or they just leave the exam environment. And as soon as these happen we make it abundantly clear to these people, if you break any of these rules we’ve got to void your exam. We’re not saying that you’ve cheated; we’re saying that you broke the rules! AO6

As part of the guidance and for safeguarding, AOs and exam centres also advised candidates to remove any personal pictures or information from the room during the assessment.

…we’re filming in that learner’s environment, so the centre should be advising the candidates to make sure that it’s a quiet space, that it’s a room, there’s nothing really visible behind them that would give us, as external bodies, any information about themselves. AO2

From the interviews with the candidates, no specific concerns emerged around being invigilated in their own home.

Experience of using RI

Candidates appreciated that they could take the assessment in the comfort of their home, without the need to travel to an exam centre.

They liked the fact that they could do it in a comfortable environment that they felt safe and secure in at home. We had a lot of them commented on the fact they were very happy they were doing it in an environment they felt comfortable in. They liked the fact they could take the exam when they wanted, because we offer a 24/7 service, they can take their exam at any time: 19% of our learners have undertaken assessments outside normal working hours. AO3

One of the candidates commented that not having to travel to the exam centre made the assessment day less stressful and that they were more relaxed for their exam.

I wasn’t as anxious because I knew I hadn’t got the travel to begin with and the traffic and that makes you stressed anyway, so being more relaxed I think already was the best thing for doing a test online, because I don’t like exams. Candidate 3

AOs reported that most students had a positive experience of RI, however, there was a consensus that those experiencing any technical issues or difficulties during set up, felt more stressed and anxious.

Looking at some of the videos we’ve seen the learners don’t appear to be anxious, with the exception of those where you can see that they’ve obviously got frustrated because of the technical hitch that’s happened. But the ones who’ve received and done their tests, you can see from the videos they seem to be quite comfortable doing what they’re doing. […] I’m not aware of anything where there’s been particularly stressful scenarios or anything over and above what they would have had in a normal scenario. AO1

I think where the technology has been a problem then getting started can create a few nerves. And I’ve definitely seen that in one or 2 cases. But then generally once they start the test it goes away, then they’re doing the test. And I’m not sure that that would differ materially from somebody standing outside an examination room and all that that process entails. It’s a different kind of experience. AO2

Familiarisation with the system before the assessment played an important part in ensuring that the candidates were comfortable during the assessment.

But a learner will get […] nervous if they’ve never engaged with the systems before, but that’s all about engaging with the technology and the process of it all. If they’ve been able to engage with it and do a sample or a sample test, for instance, they won’t be as nervous when it comes to the test, because they’ve already got through this part once before… So of course their stress levels are not going to be the same as queueing up outside a room getting tense with everybody else that’s there. AO2

One candidate did describe some frustration with a technical limitation of the RI system. In the quote below, the candidate explains that the RI software would only work with a specific browser that they were not comfortable using.

The negatives I found was I had a problem with mine back at the very beginning trying to sort out the actual software for the examination and didn’t like the fact that we had to use [a specific browser]. I felt that that was a bit bad, forcing people to use [a browser] that they don’t believe in. I don’t trust [it] for security measures […] So, I had a few issues with it and I had to speak to the person in charge of the actual software, who actually gave me their direct number, because we couldn’t work out why my system wasn’t actually accepting it, but we managed to sort it out in the end. I think, because with me I’m quite calm when it comes to stuff like that, but I think for other people it would make them a bit anxious and a bit wound up. Candidate 4

Those AOs that offered ‘live’ invigilation described how candidates valued the help of the invigilator during the session.

I think the most feedback we probably see is in the proctor textbox that we have between the local proctor and the candidate and quite often at the end there’ll be a glowing, oh thank you so much for your help, I couldn’t have done it without you, thank you. And I think probably a little bit of relief that they’ve finished the exam, but a lot of them do actually put in the box how much they’ve appreciated the proctor’s help. And that means a lot to the proctor actually, quite a lot of them say that’s really good, they really appreciated the input that we did, yeah. AO4

A similar experience was reported by one of the candidates who used the chat box functionality and valued the help that was provided.

As I remember, there was a message saying, there was assistance if you had any trouble, like not sure if they are hearing you or internet problems, you had always a number or something to call, I think, and also a live chat, and I wasn’t sure if I submitted my test, if it really submitted and I asked in the chat box and they explained to me the steps that I had to take and if I take them that’s it, and it was good, it was really helpful. Candidate 2

There was, in one case, a concern about the invigilation system failing. The candidate quoted below filmed the whole session using an additional device so that they would be able to provide evidence of their compliance with the guidance if anything went wrong with the RI system.

And because, in a way I wasn’t sure if that surveillance that they put in the system is going to work or not, but I already put another phone of mine just to film me from the beginning to the end so I can prove myself I’ve done everything by myself, just in case the system is not going to work or something like that. But there wasn’t any problem, everything was good. Candidate 2

Candidates’ readiness

Overall, AOs did not notice differences between live invigilated assessment in the exam hall and assessment taken under RI in terms of pass rate. Some AOs, however, commented that because candidates are not travelling to an exam centre and they are working at home, they may be less prepared, or in some cases may not take the exam quite as seriously.

I think there possibly is a tendency for candidates to be less prepared because they’re not having to go to a training centre and sit in an exam room, because they’re just doing it from home; although, having said that, we haven’t seen a significant drop in achievement rates at all. To be perfectly honest, I haven’t looked at the data for a little while, but last time I did it was surprisingly close to the data that we had for face-to-face. I think given the times that we’re in at the moment a lot of them feel a massive relief to be able to still do something, so still do the training, get a qualification. AO4

Student safeguarding

It is important to note that all AOs had safeguarding policies in place to protect their candidates. Whether the assessment was internally invigilated by AO staff or outsourced to external invigilators, all those involved were required to have ‘Disclosure and Barring Service’ (DBS) checks (or an equivalent check, if they were based outside of the UK) and were instructed to report any concerns.

Going back to your safeguarding point, we’ve made the decision as an awarding organisation that anybody who reviews videos of candidates should have relevant DBS checks, just basic checks, just because they have access to minors on video. Not that they can do anything with it, but that’s what we’ve decided was best practice. Anybody reviewing those and the same for [the Service Provider], they’ve all got the right safeguarding. It’s not DBS because it’s not UK, but they have the relevant equivalent. AO7

All our reviewers are DBS checked. The videos are destroyed, and all data is destroyed after 6 months, automatically. And we restrict the number of people who can see those videos. They’re held securely in the system with password access. And yet we are conscious, that we’re sitting inside people’s homes, and you have to have the policies in place on the HR side to turn around and say you’re not allowed to discuss those sorts of things outside and the things you see and do. AO3

Long-term planning

All the AOs we interviewed considered implementing RI prior to the COVID-19 pandemic and had conducted research and pilot schemes. They all felt that the pandemic had accelerated their timescale for the implementation of RI.

I think if I’m being honest, we probably would have preferred to wait a bit longer, but with needing to get assessments up and running and try and ensure that learners weren’t disadvantaged by not being able to take those assessments, I think we went sooner than we ideally would have liked in terms of our own planning. So that’s not to say that we hadn’t performed testing of systems, it was just more really around the guidance and the comms to providers and learners, that wasn’t quite in place to the extent that I think we would have liked. AO5

Challenges and limitations

Although all AOs were confident in the RI systems they were using, some expressed concerns around ensuring that they had the resources to meet demand.

We’re pretty confident that the systems and processes we’ve put in place are safe, secure, fair, and we’ve put the resource in place internally to be able to meet future demand. And I guess the biggest challenge we have now is understanding and trying to understand how big that demand might be, because we’ve anticipated a spike when we go live because centres may well want to offer [assessment to] learners that have been waiting to go through the system, and so we’re planning for that. But in terms of general demand we do have the ability to be able to adjust our demand profile, as it were, as and when things settle down a bit. AO3

One of the biggest challenges that AOs are facing is that there is a significant number of candidates whose access to remote education and assessment is undermined by limited access to digital technology or inadequate IT skills.

In some respects, you could argue that a strength of RI is that it has exposed the digital poverty issue, because actually if you haven’t got IT equipment at home or you haven’t got internet connection or a very poor internet connection then you simply can’t do this at home. And for a number of our learners that’s the case, I’m afraid, and we can’t help them, we can’t supply computers to everybody. But the centres can and very often the centres have laptops they could lend a learner to do at home and some of the better centres will do that. AO3

From an assessment validity perspective, IT skills are normally not part of the assessment; however, without these skills, it was not considered possible for the candidate to complete the assessment.

Certainly, one of the things in terms of a consideration from our point of view was this does start stepping into the realms of secondary assessment and we’re not assessing their IT skills, but actually you can’t complete the assessment unless you’ve got a reasonable level of competency with IT. AO5

AOs were planning to offer RI at scale eventually. However, due to the challenges faced with the technology, rest breaks and reasonable adjustments, they made it clear that RI will be an additional option for candidates for the time being rather than the default mode of invigilation.

We’re aware [learners] may feel uncomfortable in this situation, therefore nobody is obliged to take assessment under this system at the moment. What we’re trying to do is make it an option for people. I don’t think it would ever be a situation where we said you categorically cannot do these assessments under the watchful eye of a face-to-face invigilator if that’s what you chose to do, because obviously then it’s things like people don’t like being recorded and having that video somewhere in the ether. AO5

I don’t think we ever see this as being the only route, but I personally think it will be where the candidate chooses to, it’s likely to be because it suits them better and then why would they get upset about it, why would they get stressed about it, they’ve chosen that option. AO2

Overall, the AOs we interviewed considered RI to be a useful tool to be made available for candidates that preferred it, and although it resolved some logistical challenges, it also presented new challenges.

I think when we were first looking at it there was a misconception that RI would solve all our problems. It doesn’t. It solves some of your problems for some learners taking certain types of tests. But I think as we’ve all touched on there, you’ve got a wide variety of learners taking a wide variety of tests. It’s not the golden bird that’s going to solve all your problems. But it’s another tool in your toolbox… AO1

Regulation and Ofqual guidance

The AOs we interviewed were aware of the information and guidance provided by Ofqual around remote assessment and invigilation. They expressed, however, an interest for regulatory terms to be more specific in relation to RI.

At the moment there’s no very specific regulation that says you must do X, Y and Z and this is what should be done. And obviously as an AO that’s kind of what we’re used to, as in that’s how we work. We’re regulated, so we know we need to follow the rules. But at the moment, because these are so new in terms of parameters, there isn’t quite as much regulation that just says right, this is best practice, this is what should be done. AO7

AOs were keen to emphasise the need for a common minimum standard across their sectors to ensure that RI was delivered with integrity.

I mean there’s certainly some products we’ve tried that we wouldn’t have gone ahead with, because we didn’t consider that we could have conducted tests with integrity with some of those solutions and the limited amount of features that they offer. We would equally want to know that our competitors are maintaining those kind of standards. […] It would be good to know that as a regulator you are keeping abreast of that, and nuances. AO1

Prior to the pandemic we were always under the impression, rightly or wrongly, that RI wasn’t something that Ofqual had a particular appetite for. Obviously we’ve all had to move with the situation. So it’s great that it seems like there is much more appetite. But I think just that understanding of what is absolutely up to the AOs, what’s within our gift and what is the regulatory piece, if anything, I think absolute clarity on that would be helpful; particularly because this is not something that we are just seeing as an adaptation because of the pandemic, this is absolutely something that we want to continue with in a post-COVID world. So really understanding where those lines are and if there is anything from a regulatory perspective that we need to absolutely build in, I think just that clarity for everyone I think is just really important. AO3

In many cases, RI was introduced as a solution to the challenges presented by the pandemic and the impossibility of administering normal examination series. There is, however, scope to implement it in ‘business as usual’ practice; therefore, AOs have been keen to understand Ofqual’s regulatory approach and how it is likely to evolve. This is further discussed in the final section of the report.

Discussion

Although RI has been around for over a decade, the need to implement it for assessment increased during the pandemic, with many AOs starting to offer RI to candidates for the first time. We aimed to understand the experience of a small sample of AOs who introduced RI, as well as some of the candidates that used it. The main findings of the research can be summarised as follows:

• AOs considered several factors when choosing between the ‘live’ and ‘record and review’ RI methods, each accounting for their qualifications and their candidates. Those AOs that implemented ‘live’ RI said they did so to provide a better service to candidates, in particular around live technical support. A ‘record and review’ approach was selected by AOs who wanted to provide greater flexibility to their candidates as to when they sat the assessment.
• ‘Record and review’ RI was generally considered to be cheaper to operate than ‘live’ RI. Depending on the details of the process, however, the proportion of videos that are flagged for further review by the AOs can be high, which can place a significant pressure on resource.
• Those we interviewed were confident that RI mitigated the risk of malpractice, explaining that remote assessment presented different challenges to paper-based assessment. The merits of several features, such as screen monitoring and a second camera were discussed, as were features which may not be strictly part of the invigilation system, such as those that prevent the candidate from accessing the internet or non-permitted software during the assessment. It would be helpful to undertake further research to understand the effectiveness of RI when tackling various forms of malpractice – the current study can explore only perceptions of effectiveness.
• AOs were aware of the general guidance provided by Ofqual (the Extended Extraordinary Regulatory Framework) and this was considered when developing their RI processes. AOs expressed an appetite for regulation to be further developed. Technical issues were identified as the main concern by both AOs and candidates. Issues with the set-up of the system, the cameras and the reliability of the broadband were seen to impact the experience of undertaking assessment with RI and in some cases meant that it was not available to the full candidate population. It is important to note that the most reliable broadband may not be evenly distributed across the country. Similarly, the socio-economic background of the candidate may influence their level of access to technology. These factors may well determine whether an individual is able to access RI, and the quality of their experience if they are. Understanding such equity issues is beyond the scope of this research, but they are important to consider as RI becomes increasingly prominent.
• All AOs were planning to continue using RI for some of their assessments in the future, with some looking to add additional assessments to their RI offer.

The main limitation of these findings is the self-selecting nature of the AOs and the candidates who participated. We used opportunistic recruitment in which AOs volunteered to take part, so it is likely that the AOs who were interested in engaging with the research were also those who had put more effort into introducing RI. Similarly, most of the candidates we interviewed were approached by one of the AOs and invited to contact us, and it may be that those we interviewed were more engaged with the course and the assessment than some of their peers.

Overall, this report illustrates the implementation process of RI for a small sample of AOs during the COVID-19 pandemic and informs Ofqual’s understanding of the advantages and challenges faced by AOs and candidates during this process. Currently, as an adjustment to address a student’s long- or short-term impairment, high-stakes exams can be taken at an alternative site (for more information, see JCQ Access Arrangements and Reasonable Adjustments booklet), however, (beyond the most exceptional of cases) this supervision is provided by an in-person invigilator. This work constitutes a first step in gathering evidence around the way RI might be considered more broadly in high-stakes assessments in England and the issues it could potentially raise. This research should not be interpreted as an endorsement of widespread use of this form of invigilation for high-states assessments. Ofqual is currently reviewing international standards and practices of RI adopted worldwide to identify approaches that could work safely and securely within the context of high-takes examinations in England.

References

Alessio, H. M., Malay, N., Maurer, K., Bailer, A. J., & Rubin, B. (2017). Examining the effect of proctoring on online test scores. Online Learning Journal, 21(1).

Alpha Plus. (2020). Remote proctoring.

Alpha Plus. (2021). Remote Invigilation - Lesson learned during the pandemic.

Berkey, D., & Halfond, J. (2015). Cheating, student authentication and proctoring in online programs. New England Journal of Higher Education, July 20.

Bohmer, C., Feldmann, N., & Ibsen, M. (2018). E-exams in engineering education-Online testing of engineering competencies: Experiences and lessons learned. IEEE Global Engineering Education Conference, EDUCON, 2018-April.

Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77–101.

Cluskey, G. R., Ehlen, C. R., & Raiborn, M. H. (2011). Thwarting online exam cheating without proctor supervision. Journal of Academic and Business Ethics, 4.

Cramp, J., Medlin, J., Lake, P., & Sharp, C. (2019). Lessons learned from implementing remotely invigilated online exams. Journal of University Teaching and Learning Practice, 16(10).

Davis, A. B., Rand, R., & Seay, R. (2016). Remote proctoring: The effect of proctoring on grades. Advances in Accounting Education: Teaching and Curriculum Innovations, 18.

Dawson, P. (2021). STRATEGIES FOR USING ONLINE INVIGILATED EXAMS.

Dendir, S., & Maxwell, R. S. (2020). Cheating in online courses: Evidence from online proctoring. Computers in Human Behavior Reports, 2.

Foster, D., & Layman, H. (2013). Online proctoring systems compared. Online Verfügbar Unter.

Gudiño Paredes, S., Jasso Peña, F. de J., & de La Fuente Alcazar, J. M. (2021). Remote proctored exams: Integrity assurance in online education? Distance Education, 42(2).

Hall, E. A., Spivey, C., Kendrex, H., & Havrda, D. E. (2021). Effects of Remote Proctoring on Composite Exam Performance Among Student Pharmacists. American Journal of Pharmaceutical Education, 8410.

Hylton, K., Levy, Y., & Dringus, L. P. (2016). Utilizing webcam-based proctoring to deter misconduct in online exams. Computers and Education, 92–93.

Jaap, A., Dewar, A., Duncan, C., Fairhurst, K., Hope, D., & Kluth, D. (2021). Effect of remote online exam delivery on student experience and performance in applied knowledge tests. BMC Medical Education, 21(1).

Jones, P., Tong, Y., Liu, J., Borglum, J. and Primoli, V. (2022). Score Comparability between Online Proctored and In-Person Credentialing Exams. Journal of Educational Measurement, 59: 180-207.

Karim, M. N., Kaminsky, S. E., & Behrend, T. S. (2014). Cheating, Reactions, and Performance in Remotely Proctored Testing: An Exploratory Experimental Study. Journal of Business and Psychology, 29(4), 555–572.

Kharbat, F. F., & Abu Daabes, A. S. (2021). E-proctored exams during the COVID-19 pandemic: A close understanding. Education and Information Technologies.

Langenfeld, T. (2020). Internet-Based Proctored Assessment: Security and Fairness Issues. Educational Measurement: Issues and Practice, 39(3).

Lilley, M., Meere, J. and Barker, T. (2016). Remote Live Invigilation: A Pilot Study. Journal of Interactive Media in Education, 2016(1).

Milone, A. S., Cortese, A. M., Balestrieri, R. L., & Pittenger, A. L. (2017). The impact of proctored online exams on the educational experience. Currents in Pharmacy Teaching and Learning, 9(1).

Moore, R., Vitale, D., & Stawinoga, N. (2018). The Digital Divide and Educational Equity: A Look at Students with Very Limited Access to Electronic Devices at Home. ACT Research & Center for Equity in Learning, (August).

Ofqual. (2021). Extraordinary regulatory framework: VTQ, COVID-19 Conditions and Requirements.

Sinha, P., Dileshwari, & Yadav, A. (2020). Remote Proctored Theory And Objective Online Examination. International Journal of Advanced Networking and Applications, 11(06).

Steger, D., Schroeders, U., & Gnambs, T. (2020). A Meta-Analysis of Test Scores in Proctored and Unproctored Ability Assessments. European Journal of Psychological Assessment, 36(1).

Tomasi, L. F., Figiel, V. L., & Widener, M. (2009). I’ve got my virtual eye on you: Remote proctors and academic integrity. Contemporary Issues in Education Research (CIER), 2(1), 31–36.

Vazquez, J. J., Chiang, E. P., & Sarmiento-Barbieri, I. (2021). Can we stay one step ahead of cheaters? A field experiment in proctoring online open book exams. Journal of Behavioral and Experimental Economics, 90.

Weiner, J. A. and Hurtz, G. M. (2017). A Comparative Study of Online Remote Proctored versus Onsite Proctored High-Stakes Exams. Journal of Applied Testing Technology, 18(1).

Woldeab, D.; Brothen, T. (2019). 21stCentury Assessment: Online Proctoring, Test Anxiety, and Student Performance. International Journal of E-Learning and Distance Education, 34(1).

Wuthisatian, R. (2020). Student exam performance in different proctored environments: Evidence from an online economics course. International Review of Economics Education, 35.