Research and analysis

Summary of the policy advisory group's final report

Published 16 May 2013

Applies to England, Northern Ireland and Wales

1. The policy advisory group

The policy advisory group was primarily set up to provide support for strand three of the reliability programme, ‘investigating public perceptions of reliability’. The ten members of this group included a range of stakeholders:

• teachers
• students
• parents
• assessment providers
• employers
• assessment experts

The group was also tasked with considering the implications of the programme as a whole, and with making recommendations to Ofqual on future policy in relation to assessment reliability.

The reliability programme has produced a wealth of data and information concerning the reliability of public examinations in England, which is detailed and analysed in the reports that make up this compendium. The policy advisory group is less concerned with the detailed technical aspects of the reports, and more with the overall implications of the work, and on what is actually done with the information emerging from reliability studies. These may not be technical questions, but they are still challenging ones.

2. Increasing public awareness and understanding

Ofqual initiated the reliability programme to shine a light on all the issues relating to the consistency of examination results, including what stakeholders and the general public know and understand about assessment reliability. Ofqual wants to increase this level of knowledge and understanding - the question is, how? The technical detail of assessment reliability, as anyone will appreciate if they have read or even glanced at the reports which make up this compendium, can be overwhelming for non-specialists. But, while the technical detail may not be easily digestible by the general public, the reliability of public examinations has relevance for almost everyone. The question of how Ofqual can increase public awareness and understanding when the topic itself is so complex has been a preoccupation of the reliability programme from the outset.

The policy advisory group was specifically asked to provide advice on the work carried out under strand 3 of the reliability programme. This part of the programme investigated public perceptions of reliability through a series of surveys, workshops, and focus groups, all involving teachers, parents, students and general members of the public. Unsurprisingly, these studies found that it was only really teachers who had an understanding of the concept of reliability. However, it was also clear that the wider public does have a broad understanding of the assessment process, and of some of the issues which can affect the consistency of examination results. The studies also revealed an interesting aspect of the public’s attitude towards examination results: the public are generally aware that a certain amount of inconsistency or error is inevitable in a large examination system, and to a greater or lesser extent they seem to accept this. They are far less tolerant, however, of what they deem to be ‘preventable’ errors, such as errors in examination papers. They would expect the latter to be dealt with severely, but they considered the former to be an inevitable part of life. One focus group study even reported that the participants (teachers, students, employers, etc) felt that there was little to be gained by publishing reliability statistics, which would draw attention to such ‘inevitable’ errors. This last point, of course, raises the further issue that most often the public will not receive information about possible inconsistencies in examination results directly from Ofqual or the awarding organisations, but as a ‘story’ filtered through the lens of the media. With no universally accepted way to report the reliability of examination results to the wider public (although the report summarises some approaches from around the world), it is possible to begin to appreciate the issues that the policy advisory group grappled with as they considered what policy recommendations they should make on how to deal with assessment reliability.

3. The group’s recommendations

The final recommendations made by the policy advisory group to Ofqual are listed at the end of the report. In broad terms they propose that Ofqual should continue to work on assessment reliability in an open and transparent way, that it should publish its findings, that it should encourage awarding organisations to do the same, and that it should work to improve the public understanding of reliability. This compendium is one contribution to meeting those objectives. Anyone who has read some or all of the reports in this compendium should have a broad understanding of what is meant by the reliability of examinations results. It is to be hoped that they will also have a better understanding of the efforts that go into ensuring that, overwhelmingly, students do indeed receive an accurate grade, and that this process is repeated year after year after year.