Dr Michelle Meadows speech at City of London Schools Conference 2021
Ofqual's Executive Director for Strategy, Risk and Research spoke to the City of London Schools Conference about online assessment.
Given my role at Ofqual it won’t surprise you that I am going to talk about qualifications. The move to remote teaching due to the pandemic has caused many to reflect on what this means for education. Will education spring forward in its use of technology or ping back to more traditional methods? Similarly, there is much discussion about educational assessment, the resilience of our exams system and whether the reliance on linear, paper-based exams is right.
The move to onscreen or online assessment has been mooted for a long, long time. And I’m not going to fall into the trap, that many others before me have fallen into, of predicting an impending move to e-assessment. But given that you have invited me to reimagine the world, there are many advantages that could come from such a move, advantages that go beyond the delivery of assessments during a pandemic.
E-assessment is undoubtedly more in line with how we live in our digital society. It would open up opportunities to improve the validity of assessments through inventive item design. Imagine the use of animation or interactive graphics to present concepts and information that would be hard to communicate on paper. The provision of large data sets, for students to analyse. Simulated science experiments allowing item types that would be impossible on paper.
In sum, assessment that is more authentic and valid.
The security of assessment could also be enhanced - no more wrong exam packets opened, no more security breaches with exam questions appearing on the internet. Although the risk of cyber-attack would need to be robustly managed and malpractice would of course evolve.
There is also evidence that some students, often those less engaged in education in general, find onscreen assessment more motivating and enjoyable. It might make testing a more positive experience for some.
Taking e-assessment a step further opens up other possibilities such as on-demand or adaptive, personalised assessment.
But I don’t want to spend my time repeating what is already known about the opportunities offered by e-assessment. I’m going to focus on what must happen to enable the move to e-assessment.
We recently conducted a review of the barriers to such a move for high stakes sessional qualifications such as A levels and GCSEs. We focused very much on the logistical barriers. We reviewed the literature, we spoke to stakeholders including exam boards, teacher leaders and the unions, and looked at how three international jurisdictions had introduced e-assessment in similar qualifications – Israel, New Zealand and Finland - each took a different approach.
So, what did we learn?
First of all, that government sponsorship, indeed leadership would be critical. International colleagues told us that there had to be strong political will to make the move. Not just because of the necessary investment in the school and college infrastructure.
Government must have a compelling vision that e-assessment matches wider societal changes - given how pervasive technology is in everyday life - and meets needs - including the needs of students and employers. And that the benefits justify the investment and the risks.
Interestingly, only one in three responding to our annual perceptions survey believe that onscreen assessment would be fairer and more manageable than the current system. This indicates the size of the job to convince the public of the benefits of the move. Young people were the most supportive although even among them, less than half responded positively.
Government would need to have an appetite for risk – international colleagues told us that there had to be an acceptance that things may go wrong, that there would be some learning by doing. In all the jurisdictions we looked at, things had gone wrong – power outages, internet failure - hiccups were seen as inevitable and robust system leadership was needed to weather the consequences. Of course, cross party consensus would be needed as implementation would inevitably span the electoral cycle.
Second, we heard about the significant infrastructure and resource challenges, and learned how these had been overcome elsewhere.
You will know far better than I, that IT provision and support within schools is variable, as is the quality of internet. There are hundreds of combinations of browser and operating systems across exam centres which makes delivering a consistent national approach challenging.
Plus, there is variation in devices which would also need to be overcome. For the move to e-assessment to be fair, there’d need to be a level playing field in terms of the quality and speed of devices and the ability of students to practice on new systems. Stakeholders feared that some students would be disadvantaged because of worse device or internet performance compared to students in schools with better equipment or better broadband.
We spoke to stakeholders in January 2020 and they warned of the different levels of IT expertise and confidence within schools. The need to teach remotely will have, of course, improved the situation but we shouldn’t dismiss this barrier. In New Zealand, only schools using digital systems in classroom teaching, and therefore with teachers comfortable with the software, opt into on-screen assessment.
New Zealand took a gradual, voluntary approach to implementation. This enabled public confidence to grow as uptake grew. However, they are now having to think about how they can switch to an opt out rather than an opt in approach. A dual system brings problems, for example ensuring that the demand of the two assessment modes is comparable. This is tricky, onscreen capability to plot graphs, draw diagrams or write in a foreign language through a non-roman alphabet is potentially difficult or different compared to using a pen.
And running a dual system means that it is impossible to access some of the real benefits that would come from e-assessment – there’s no opportunity to improve the validity of the assessments if the paper based test version is an anchor.
Some international systems have followed a bring your own device approach. In Finland a USB stick is given to every student which converts the devices to have the same functionality. This creates its own logistical and security challenges as thousands of USBs travel the country.
In England, there would also be an issue with finding sufficient space for online sessional exams to take place. There were more than 700,000 students entered for GCSE Maths and for GCSE English in 2019. In Israel other sites such as libraries are used if schools do not have enough devices, this is unlikely to work here but it shows the flexibility of thinking required to make the move possible.
To wrap up, the move to online assessment goes beyond issues of assessment validity which are Ofqual’s remit. This is about societal and cultural preferences, and the use of limited resource, now more than ever.
Here are some questions raised by this research -
Is this the best use of resource given the impact of the pandemic on learning? Do we have the shared vision required to make this happen? Do we have the necessary appetite for risk? How can we innovate in such a high stakes system? Change causes student and teacher anxiety – how would we manage that anxiety? Are we willing to be sufficiently flexible, to make compromises to make this happen? Would there be an appetite for a twin track, on-screen and paper-based system here? Or would it flounder on the rocks of comparability? If we can reimagine a future without paper then should we take a fresh look at the curriculum in the context of e-assessment?
Can we imagine a world where the stakes of qualifications are such that we’d take the e-assessment plunge?
Thank you.