Research and analysis

A Quantitative Investigation into Public Perceptions of Reliability in Examination Results in England: Summary

Published 16 May 2013

Applies to England, Northern Ireland and Wales

1. Background

This report describes a study of the opinions of nearly 900 people who filled in an on-line questionnaire about their attitudes to the public exam system in England (GCSEs, vocational qualifications, and A levels). The study followed two others in which people were asked in small groups what they understood by error in public exams, and what might be done about it. In contrast with those two studies, this report involved a large number of respondents:

• 314 teachers of A level
• 358 A level students
• 210 employers

The topics targeted in the questionnaire were:

• the level of public confidence in the national exam system
• the factors influencing students’ performances in exams and the factors creating uncertainty in the results
• the public’s attitudes to different types of assessment error
• ways of improving the reliability of exams

The questionnaire also included some questions about the participants’ levels of trust in general.

The questionnaire provided people with statements about the exam system and asked them whether they agreed or disagreed with them. By comparing the responses of the different groups to the same questions the report gave an insight into the thinking of teachers, students and employers about the national exam system. This introduction to the report gives some of the main findings about the attitudes of those three groups.

2. Teachers’ opinions

Most of the teachers who completed the questionnaire (81%) thought that exams are fair to students and that most students are well-prepared for them: 89% thought that students get the results they deserve and 62% said they have confidence in the exam system. A quarter of the teachers said the system is doing a good job, but 61% said it could be improved and 12% that it needs to be reformed.

While 30% of the teachers said that exam results are essentially an estimate, 66% said that any kind of error is unacceptable. More than half of the teachers (56%) accepted that there is a difference between avoidable mistakes and certain kinds of inevitable measurement error, but 41% still claimed that all inaccuracy should be removed from the system. Nearly 80% of the teachers agreed that ‘feeling off colour’ on the day of the exam was ‘one of those things’ and that students are responsible for how they perform in an exam.

Most teachers (82%) thought that exam boards should be open about uncertainty in exam results, but 78% thought this would undermine public confidence. Less than 30% thought that inaccuracy in scores is ‘inevitable’ but nearly two-thirds were against giving information about the degree of uncertainty on students’ certificates. Giving their opinion on how to improve reliability in exams, nearly all the teachers said that marker training should be improved. Double marking was also advocated by 80% of the teachers. Only about 20% of the teachers advocated more use of multiple-choice questions and longer tests. Interestingly, only 31% of the teachers were in favour of more use of teacher assessments, compared with more than 50% of employers and more than 60% of students.

When the questionnaire asked about the respondents’ general levels of trust, the teachers turned out to have most trust in people and organisations they meet in the course of their work. They were also more trusting than the students and employers of professionals working for government agencies, but they were much less trusting of organisations they hear about through the media. Only a quarter of the teachers believed that government statistics are generally accurate.

3. Students’ opinions

The students in the sample were the least positive about exams. Less than 20% of them felt that their experience of exams was acceptable, and over 50% found exams ‘unreasonably stressful’. Only 42% of the students said they have confidence in the system, and only 17% thought that students get the grades they deserve. These figures are very much lower than the figures obtained in other similar studies. When it came to opinions about whether the exam system is doing a good job, and whether it needs to be improved or reformed, however, the students’ replies were similar to the teachers’ replies.

The students thought that any errors in the exam procedure were most likely to be introduced by the question papers or the marking, but difficulties with coping on the exam day also came quite high on their list. Generally the students were uncertain about how much grade misclassification there is in the system, though they all agreed that they were more likely to get a grade that did not reflect their ability in English than they were in maths and science.

Like the teachers, 30% of students accepted the approximate nature of exam results, but nearly two-thirds of them said that any kind of error was unacceptable. On the question of the inevitability of some kinds of error, the students were again similar to teachers in their attitudes, with 38% of them wanting all inaccuracy to be removed from the system. In contrast with the teachers and employers, a smaller percentage of students were willing to accept that students themselves are accountable for their exam performances (53%).

Most of the students (87%) thought the exam boards should be open about uncertainty in results, and 48% accepted that inaccuracy is inevitable; 72% thought that reporting this would, however, undermine public confidence. Nearly 50% were willing to consider the idea of indicating the degree of uncertainty on students’ certificates. Like the teachers, the students advocated more marker training and double marking to improve accuracy. Unlike the teachers, about 50% of the students were in favour of more multiple-choice questions and 65% in favour of more teacher assessment.

The students reported a much lower level of trust, generally, than teachers and employers, with 53% saying that you need to be very careful in your dealings with people, and only 30% saying you could trust people whom you meet for the first time. The students’ trust in the media was slightly greater than the other two groups: 40% rather than 30% for the teachers and employers.

4. Employers’ opinions

65% of employers thought that students get the results they deserve, but only 39% said they had confidence in the national exam system. Nearly a quarter of the employers wanted the system to be reformed, and 48% thought it needed to be improved, while saying that it was doing a good job. Less than 20% were prepared to say it is doing a good job without any improvement.

Most employers (60%) thought that problems with the occasion, the question paper, marking inconsistency and the way that grade boundaries are decided would all play a roughly equal part in introducing errors. The employers were the most tolerant of errors in the exam grades, with 46% saying that they were inevitable, and 50% saying that any error was unacceptable. Over 60% of the employers accepted that some inconsistencies are inevitable, with only 32% wanting all inaccuracy removed from the system. The employers broadly agreed (80%) with the teachers that the performance of the students on the day was the students’ own responsibility.

Most employers (85%) thought that there should be more public reporting about the uncertainty of exam results, although just over 50% said this could reduce public confidence. Four-tenths of the employers (40%) were willing to consider giving information on students’ certificates about how certain the final grades were, but 50% were against doing this. Employers supported more training for markers and double marking to improve consistency but less than 30% were in favour of longer tests and more multiple-choice questions.

A small majority of employers (56%) said that most people can be trusted, but 41% said you have to be very careful when dealing with people. They reported a fairly high level of trust in those organisations they come across professionally, but not in those they hear about in the media (only 30%). Less than 50% said they trusted professionals working in commercial enterprises; even fewer trusted those working for government agencies. Just over 20% of the employers trust government statistics.

5. Conclusions

Results from this study indicate that attitudes to unreliability in exam results vary between the three stakeholder groups investigated. The respondents showed different degrees of experience of the examination process and acceptance of measurement error in examination results. The majority appeared to understand the assessment process and the factors that affect students’ performances in exams. To a degree they also understood the factors that could introduce uncertainty into exam results. A substantial proportion of respondents from the three groups lacked awareness of some aspects of reliability. The report concludes that there is a need to educate the public to understand reliability concepts and the existence of uncertainty in examination results.

The report also notes that, to obtain a better idea about how the general public perceive the issue of inconsistency in public exams, a further study would have to involve a wider range of people than the present one, which surveyed the opinions of three particular groups very closely involved with the exam system.