Research and analysis

Responding to Criticisms of the CASLO Approach (Report B): Introduction

Published 18 November 2024

Applies to England

Authors

Milja Curcin, Paul Newton, Asteria Brylka and Latoya Clarke, from Ofqual’s Standards, Research and Analysis Directorate.

With thanks to

All the awarding organisations and their staff for their willingness and enthusiasm to participate in this research.

The CASLO Research Programme

This report is part of a series that arose from Ofqual’s 2020 to 2024 programme of research into the CASLO approach:

1. The CASLO Research Programme: Overview of research projects conducted between 2020 and 2024.
2. The CASLO Approach: A design template for vocational and technical qualifications.
3. How ‘CASLO’ Qualifications Work. (This was published in February 2022.)
4. Origins and Evolution of the CASLO Approach in England: The importance of outcomes and mastery when designing vocational and technical qualifications.
5. Responding to Criticisms of the CASLO Approach (Report A): A taxonomy of potential problems.
6. Responding to Criticisms of the CASLO Approach (Report B): Views from awarding organisations.
7. Responding to Criticisms of the CASLO Approach (Report C): Views from qualification stakeholders.
8. Responding to Criticisms of the CASLO Approach (Report D): Properties of qualifications from the CASLO research programme.
9. Understanding Qualification Design: Insights from the 2020 to 2024 CASLO qualification research programme.

List of acronyms

AO – Awarding Organisation
AC – Assessment Criteria
CASLO – Confirm Acquisition of Specified Learning Outcomes
DCS – Direct Claims Status
EQA – External Quality Assurance
GLH – Guided Learning Hours
GNVQ – General National Vocational Qualification
IQA – Internal Quality Assurance
LO – Learning Outcome
NOS – National Occupational Standards
NVQ – National Vocational Qualification
QCF – Qualifications and Credit Framework
RQF – Regulated Qualifications Framework
TQT – Total Qualification Time
VTQ – Vocational and Technical Qualification

Introduction

This study is part of a programme of research that Ofqual has conducted into an approach to qualification design which we call the ‘CASLO’ approach^{[footnote 1]}. The CASLO approach is defined via the following key characteristics (Newton & Lockyer, 2022):

1. the domain of learning is specified as a comprehensive set of learning outcomes (LOs), which tend to refer to elements of knowledge and skill
2. a standard is specified for each LO, via a set of assessment criteria (AC), which are used to judge student performances directly
3. a pass indicates that a student has acquired the full set of LOs specified for the domain (mastery requirement)

This approach gained national prominence within National Vocational Qualifications (NVQs), launched in 1986 (De Ville, 1986; Jessup, 1991), and it now underpins a large family of outcome‑based vocational and technical qualifications in England. The expansion of the CASLO approach was particularly facilitated by its adoption as the basis for a regulatory framework called the Qualifications and Credit Framework (QCF) (Ofqual, 2008), in operation from 2008 till 2015, which required most regulated vocational qualifications to conform to the CASLO design principles. The CASLO approach shares a lot of the above-mentioned characteristics with other education and training movements, for instance, outcomes-based education and training, competence-based education and training, and mastery learning. However, the combination of properties outlined above is specific for the family of qualifications in England that we are referring to as CASLO qualifications.

There is a large academic literature related to the CASLO approach, a lot of it emerging since around the late 1980s and associated with the introduction of NVQs and GNVQs (General National Vocational Qualifications). It identifies a variety of potential problems for the CASLO approach – some quite esoteric, relating to its philosophical roots; and some very pragmatic, relating to its impacts on teaching and learning or quality of assessment. This literature is fairly dated now, with few comprehensive large‑scale evaluations, and it seems to be skewed or polarised in the direction of criticism. Furthermore, the approach itself has evolved through repeated reforms and in response to changing accountability pressures over time. Therefore, it is not immediately obvious whether the academic criticisms raised in the literature (and the associated potential problems) are still relevant for the CASLO qualifications that Ofqual currently regulates and is likely to continue to regulate in the future.

With the aim of facilitating a more balanced consideration of the force of these criticisms from the literature, we invited awarding organisations (AOs) to reflect on and respond to them based on their experience of designing and certifying CASLO qualifications. Each participating AO nominated an ‘exemplar’ CASLO qualification – one that they considered to be particularly well‑suited to the CASLO approach, or that they believed to have high validity or positive impacts on uptake, teaching, or learning due to adopting the approach. These qualifications were the focus of our discussions with each AO, allowing us to consider their responses in light of the specific context of each exemplar.

Our overarching goal in this project was to explore the features and mechanisms that, according to awarding organisations, help to secure the quality and value of their CASLO qualifications in the face of varied academic criticisms and potential problems. The following were the main research questions we attempted to answer:

1. What are AO reasons for adopting the CASLO approach?
2. Do AOs recognise the potential problems associated with the CASLO approach in the literature?
3. What mitigations do AOs put in place to alleviate the risks associated with the potential problems?
4. What can AO responses tell us about the relative significance and severity of the potential problems?
5. Are there conditions (qualification contexts, purposes, design features) in which some of these problems are more or less relevant?

This study did not involve an evaluation of individual qualifications or the CASLO approach. The aim was to understand AO views about potential problems and draw conclusions about the applicability of criticisms from the literature to current CASLO qualifications and their contexts. Our analysis of AO responses to these potential problems, including the mitigations that they proposed, allowed us to describe and gain insights into the mechanisms that appear to underpin the technical quality (validity) of CASLO qualifications and their value (positive impacts). It also provided insights into the circumstances under which these mechanisms may operate more or less effectively. However, given the methodology of this study, we were not in a position to draw strong conclusions about their effectiveness. Where appropriate, limited commentary on the potential plausibility (or lack thereof) of the proposed mitigations and mechanisms is offered, based on logic rather than independent evidence.

For these reasons, it is important to emphasise that the following account captures the views of a sample of AOs concerning the features and processes that they build into their qualifications to help secure their validity and value. These insights provide a useful resource for reflecting on the sorts of problems that can affect CASLO qualifications, as well as the sorts of mitigations that can be put in place to confront them. However, it was not possible within the remit of this project to evaluate either the prevalence of alleged problems or the effectiveness of proposed mitigations. So, the publication of this report is not a formal endorsement of the validity or value of the 14 ‘exemplar’ qualifications that we studied in detail. The report is simply a foundation for reflecting upon how best to build a qualification – for a particular cohort of learners, situated in a particular context, with particular set of purposes in mind – that incorporates the high-level CASLO design template.

In the next section, we present the details of our methodology. This is followed by a section describing the perceived benefits of the CASLO approach and the reasons why it was adopted by our sample of AOs for their exemplar qualifications. We then go on to describe AO perceptions of the relevance of assessment-related problems to their exemplar qualifications and the mitigations that they proposed in relation to each assessment problem. A similar description is then presented for teaching, learning and delivery problems. The report concludes with a discussion section which offers reflections on our last 2 research questions as well as on our overarching question of what seems likely to underpin the quality and value of CASLO qualifications.

1. Until recently, this approach had no name. It was christened ‘CASLO’ in Newton and Lockyer (2022) because it is designed to Confirm the Acquisition of Specified Learning Outcomes. ↩