No news is good news? Talking to the public about the reliability of assessment: Summary
Published 16 May 2013
Applies to England, Northern Ireland and Wales
© Crown copyright 2013
This publication is licensed under the terms of the Open Government Licence v3.0 except where otherwise stated. To view this licence, visit nationalarchives.gov.uk/doc/open-government-licence/version/3 or write to the Information Policy Team, The National Archives, Kew, London TW9 4DU, or email: psi@nationalarchives.gov.uk.
Where we have identified any third party copyright information you will need to obtain permission from the copyright holders concerned.
This publication is available at https://www.gov.uk/government/publications/reliability-of-assessment-compendium/no-news-is-good-news-talking-to-the-public-about-the-reliability-of-assessment-summary
1. Investigating public perceptions of assessment reliability
Ipsos MORI undertook two workshops, each with 36 members of the public, including teachers, students and parents. For these workshops the facilitators provided stimulus material about issues relating to the reliability of assessments, on which they attempted to gauge the group members’ views. A major finding from this work was the workshop participants’ apparent distinction between inevitable sources of variation in assessment marks and grades, and what they saw as preventable errors in marks and grades. Overall, participants seemed able to cope with the idea that ‘test-related errors’ arise and are to some extent inevitable; they were less tolerant of assessment errors arising from blatant human mistakes. The use of the word ‘error’ itself was also seen as problematic, especially when used to cover all aspects of unreliability - because it tends to produce negative reactions and assumptions of culpability on the part of assessors and awarding organisations.
2. Communicating with the public about assessment reliability
Blue Rubicon’s research (conducted exclusively with Ofqual staff) attempted to address the issue of how Ofqual could communicate most effectively about assessment reliability issues with the general public. Blue Rubicon is a ‘communications messaging consultancy’, and so is skilled at helping organisations frame communications strategies. In Ofqual’s case the question of helping the public understand the concept of reliability was seen as an important, yet problematic, challenge. A major recommendation from this strand of work was that the word ‘variation’ was a more constructive way of communicating about the uncertainties surrounding assessment results than either ‘reliability’ or ‘error’. This constituted one of a number of recommendations on how Ofqual staff and representatives might communicate about this issue with members of the public.
3. The implications of the research
Rather than give an exhaustive account of these two specialist strands of work within the Ofqual reliability programme, the conference paper reflects both on the implications of this work when viewed alongside the outcomes from other aspects of the programme, and also on the public’s current perception of assessment results in the UK. The media are responsible for a large part of the communication in this area, and in recent years they have thrived on so-called crises and disasters, whether these are to do with the annual production of assessment results at both GCSE and A level or with national curriculum assessments. Clearly, the media’s concerns are different from those of awarding organisations, naturally tending to raise doubts about assessment results, whereas awarding organisations will generally seek to increase the public’s confidence in the assessment results – which the agencies are responsible for producing.
The paper outlines the considerable technical knowledge about the limitations of assessment scores and grades as precise measures of achievement, and the very real difficulties faced by those who seek to communicate such insights to the public. It reiterates the view that awarding organisations should take central responsibility for estimating the degree of variability in assessment results. The paper concludes, positively, by asserting that a wide variety of media and analogies could be used to get the message of the complexity of the technical data across, and by holding out the hope that following such a strategy could make the UK public ‘more sophisticated’ users of assessment results.
Had the presentation returned to its provocative ‘No news is good news?’ title, it might have concluded that there are ways of conveying news about the limitations of assessment results that are not necessarily ‘bad news’. The presentation does, however, suggest that Ofqual is determined to ‘out’ this issue, which has, for many, been a ‘skeleton in the cupboard’ for far too long.