Research and analysis

Responding to Criticisms of the CASLO Approach (Report B): Appendices

Published 18 November 2024

Applies to England

Appendix 1: Exemplar qualifications

Table: Exemplar qualifications

Abbreviation	Title	Qual No.	Level	Primary purpose	Total Creds	GLH	TQT	Grading Type	Total certs until end of 2023	Certs for 2023 (Q1-Q3)	Operational start date
Adult care_L3	NOCN Level 3 Diploma in Adult Care (England)	610/0088/7	3	Confirm competence	58	372	580	Pass/Fail	95	5	01/05/22
Business_L3	Pearson BTEC Level 3 National Extended Certificate in Business	601/7159/5	3	Dual purpose	N/A	360	480	Graded	60,450	11,410	01/06/16
Chef_L2	iCQ Level 2 diploma in Professional Chef (England)	603/4270/5	2	Confirm competence	85	574	850	Pass/Fail	25	25	01/07/19
Construction_L1	ICM Level 1 Certificate in Construction Skills	Non-regulated	1	Dual purpose	16	160	160	Pass/Fail	N/A	N/A	N/A
Construction_L5	Pearson BTEC Level 5 Higher National Diploma in Construction Management for England	603/7859/1	5	Dual purpose	240	960	2400	Graded	N/A	N/A	01/09/23
Creative_L2	RLS Level 2 Certificate in Creative and Performing Arts (Specialism: Performing & Creating, Performing Arts Business & Production) (non-performance tables)	601/8614/8	2	Dual purpose	N/A	120	200	Graded	125	15	01/09/16
Creative_L3	UAL Level 3 Diploma in Creative Practice: Art, Design & Communication	603/5302/8	3	Dual purpose	72	540	720	Graded	16,193	6,032	01/09/20
End of life care_L2	IAO Level 2 Certificate in Principles of End of Life Care	601/6185/1	2	Confirm competence	16	135	160	Pass/Fail	70	0	01/06/15
Fenestration_L2	GQA Level 2 NVQ Diploma in Fenestration Installation	500/7825/2	2	Confirm competence	37	232	370	Pass/Fail	7,625	85	01/01/10
First aid_L3	ITC Level 3 award in Outdoor First Aid	601/7616/7	3	Confirm competence	2	16	20	Pass/Fail	16,725	2,315	01/10/15
Hairdressing_L2	VTCT Level 2 Diploma in Women’s Hairdressing	500/8976/6	2	Confirm competence	52	448	520	Pass/Fail	10,720	1,060	01/08/10
Housing_L5	CIH Level 5 Diploma in Housing	601/8133/3	5	Confirm competence	32	128	320	Pass/Fail	600	45	01/12/15
Skin peel_L4	GA Level 4 Award in Chemical Skin Peeling	610/0703/1	4	Confirm competence	12	60	120	Pass/Fail	10	0	21/03/22
Teaching support_L2	NCFE CACHE Level 2 Certificate in Supporting Teaching And Learning	603/2476/4	2	Confirm competence	32	225	321	Pass/Fail	17,520	3,340	01/01/18
Procurement_L4	CIPS Level 4 Diploma in Procurement and Supply	603/3924/X	4	Confirm competence	60	250	600	Pass/Fail	4,590	935	31/01/19

Appendix 2: Interview schedule

Exemplar qualification – rationale

First of all, we’d like to discuss your views about the CASLO approach in the context of your Exemplar qualification, so we’re keen to understand:

• why you’ve adopted the approach for this qualification, including
• why you think the approach works particularly well in this context, or
• why it might work better than a more classical approach

And, just as a reminder, by the ‘CASLO approach’ we mean:

• defining qualification content and standards in terms of detailed LOs and AC
• requiring students to achieve all specified LOs to pass the qualification (mastery)

1. 1) Why do you think the CASLO approach is particularly well-suited to your own Exemplar qualification?
   1. a) Is it particularly suitable given its purpose, cohort, context, or progression routes? In what ways?
   2. b) Is it particularly valued by stakeholders? Which stakeholders? Why?
   3. c) Does it work particularly well in terms of ensuring effective learning? How? How do you know?
   4. d) Does it work particularly well in terms of ensuring high-quality assessment? How? How do you know?
   5. e) Have you adapted the CASLO approach at all for your Exemplar qualification – maybe tweaked it, or hybridised it?
      1. i) How? Why? What effect do you think this has had?
   6. f) Have you built any extra controls into the CASLO approach for your Exemplar qualification?
      1. i) How? Why? What effect do you think this has had?
2. 2) How important do you think the mastery approach is to your Exemplar qualification (that’s the idea of requiring students to achieve all of the LOs)?
   1. a) Why? What are the consequences of adopting a mastery approach?
   2. b) Is the mastery approach applied strictly, or do assessors get to make any allowances, or is compensation used at all?
3. 3) Is entry to this qualification open to anyone, or do students have to satisfy formal or informal entry criteria? Is any prior knowledge or competence assumed?
   1. a) How do centres determine whether students are ready for the qualification, in terms of prior learning?

• We’re going to move on, now, to the main body of the interview and discuss some potential problems for CASLO qualifications from the literature, which we’ve grouped into 3 categories:

  • Assessment challenges
  • Learning and teaching challenges
  • Delivery challenges

Assessment challenges

The literature identifies various problems that involve assessors applying CASLO standards inaccurately or incorrectly:

• sometimes their judgements are too harsh or too lenient, but unintentionally so
• other times their judgements might be intentionally too lenient

We’ll explore each of these cases separately.

Section A: Inaccurate judgements

1) The literature identifies problems related to assessors making inaccurate judgements, meaning that some students pass when they shouldn’t, and some students don’t pass when they should.

According to the literature, that happens because assessment criteria are very hard to write precisely – and they’re very hard to interpret precisely – which is a big problem for the CASLO approach because assessors need to make heavy use of these written criteria when judging student performances.

Clarity and Range

1. 1) We’ve brought along an example of learning outcomes and assessment criteria for your Exemplar qualification.
   Some critics say that written statements alone – like these – are too imprecise to communicate the threshold between passing and not passing (that is, between satisfactorily achieving or not achieving).
   1. a) How do you make sure that your qualification standards are communicated with sufficient precision?
      1. i) Do you rely purely on written assessment criteria?
      2. ii) What steps do you take to ensure that your assessment criteria are communicated as clearly as possible (cf. our example) – including to, for example, new centres?
   2. b) [For qualifications with grades] Do you face any particular problems when trying to define thresholds between different grades?
      1. i) Do you rely purely on written grading criteria?
      2. ii) What steps do you take to ensure that your grading criteria are communicated as clearly as possible (cf. our example)?
      3. iii) When you use command verbs (from Bloom’s Taxonomy) to illustrate different levels of performance, how can you be sure that all assessors will interpret and apply them in the same way?
   3. c) Do you use any additional guidance to elaborate your assessment criteria (or grading criteria) for example, exemplars, guidance on sufficiency of evidence?
2. 2) Some critics say that assessment criteria need to be supplemented by range statements, to indicate the range of contexts across which students need to demonstrate competence (for each LO or for each AC).
   1. a) Do you use range statements (or anything similar) for your Exemplar qualification?
      1. i) how strictly are they supposed to apply (mandatory or illustrative)?

Assessor judgements

1. 1) Some critics say that there is more to having met a learning outcome than having satisfied a list of assessment criteria, but that this gets missed when assessors are reduced to ticking-off criteria lists, criterion by criterion.
   1. a) Do you recognise this as a potential criticism of your Exemplar qualification?
      1. i) [if YES then] what steps do you take to reduce the risk of arbitrary judgements that are constrained by criteria lists?
      2. ii) [if NO then] why doesn’t this criticism seem to be relevant?
   2. b) To what extent are assessors expected to apply assessment criteria holistically rather than atomistically?
      1. i) might assessors introduce an element of compensation at this level, not necessarily requiring all AC to have been ticked off?
      2. ii) what support do you provide to help assessors to make any holistic judgements as consistently as possible?

Assessor standardisation

1. 1) Some critics say that assessor standardisation tends not to be very effective for CASLO qualifications because
   • it happens too infrequently, or
   • it focuses more on procedures than standards, or
   • it’s just hard to develop effective guidance and exemplars
   1. a) Do you recognise ineffective standardisation as a potential criticism of your Exemplar qualification?
      1. i) [if YES then] what steps do you take to reduce the risk of ineffective standardisation? Do you have any specific requirements of centres? Is there any external standardisation (facilitated or organised by AO)?
      2. ii) [if NO then] why doesn’t this criticism seem to be relevant?
2. 2) Some critics argue that standards ultimately reside in the shared understanding of a community of practice (rather than in written criteria). So, it’s important for an assessor to be an active member of a sector-based community of practice in order to be able to apply standards consistently.
   1. a) Would you agree with that position?
      1. i) [if YES then] have you put anything in place to facilitate communities of practice relevant to your Exemplar qualification?
      2. ii) [if NO then] why don’t you agree with the position?

Assessment tasks or events

1. 1) Some critics say that having detailed assessment criteria to judge the quality of student performances makes it look like the assessment process is extremely straightforward. However, because assessors often fail to appreciate just how hard it can be to elicit the right kind of assessment evidence in the first place, CASLO qualifications are very vulnerable to being based on poorly conceived assessment events or poorly designed assessment tasks (that don’t elicit the right kind of evidence).
2. a) Do you recognise poorly designed assessments as a potential criticism of your Exemplar qualification?
   1. i) [if YES then] what steps do you take to reduce the risk of poorly designed assessments?
   2. ii) [if NO then] why doesn’t this criticism seem to be relevant?
3. b) If you devolve significant responsibility for designing assessment tasks or events to centres, then
   1. i) how do you ensure that those tasks or events always elicit high quality evidence?
   2. ii) how do you ensure that those tasks or events are sufficiently comparable in terms of the demands that they make of students?

Section B: Lenience and malpractice

1. 1) Some critics say that the imprecision of assessment criteria can act as a sort of smokescreen for assessors, allowing them to be intentionally lenient for students who haven’t quite reached the qualification standards (we’ll call this giving undue benefit of the doubt).
   This can be exacerbated towards the end of sessional courses, for students who are just about to leave, but who still haven’t quite achieved all of their LOs.
   1. a) Do you recognise assessors giving undue benefit of the doubt as a potential threat to your Exemplar qualification?
      1. i) [if YES then] what steps do you take to reduce the risk of this happening?
      2. ii) [if NO then] why doesn’t this criticism seem to be relevant?
2. 2) Some critics say that the line between formative and summative assessment gets blurred when using the CASLO approach, and this can lead to students being given too much support and then being assessed at a higher level than they have independently achieved.
   1. a) Do you recognise inappropriate support as a potential threat to your Exemplar qualification? How would you detect it?
      1. i) [if YES then] what steps do you take to reduce the risk of this happening?
      2. ii) [if NO then] why doesn’t this criticism seem to be relevant?
3. 3) Occasionally, assessors try to pass students who are a long way from meeting the qualification standards (we’ll call this malpractice). They can get away with this – according to some critics – because it’s extremely hard to detect and correct inaccurate assessor judgements under the CASLO approach.
   1. a) Do you recognise this kind of malpractice as a potential threat to your Exemplar qualification?
      1. i) [if YES then] what steps do you take to reduce the risk of this happening? How would you detect it?
      2. ii) [if NO then] why doesn’t this criticism seem to be relevant?

Learning and teaching challenges

The literature identifies a variety of problems that allegedly arise from the way in which CASLO qualifications are specified. They either criticise:

• what students end up learning, or
• how well students end up learning, or
• whether students end up learning what they need to learn.

We’ll address each of these criticisms separately.

Section C: What students learn…

1. 1) Because CASLO qualifications are extremely explicit about the learning outcomes that need to be acquired, this has led some critics to claim that they are too inflexible to respond to: • local economic needs, or • the bespoke needs of small employers, or • students with particular interests or aspirations, including demands made by higher level courses
   1. a) Do you recognise local or personal irrelevance as a potential threat to your Exemplar qualification?
      1. i) [if YES then] what steps do you take to facilitate local or personal relevance for your Exemplar qualification?
      2. ii) [if NO then] why doesn’t this criticism seem to be relevant?
2. 2) Some critics have said that the level of detail in CASLO specifications inevitably ties them to existing work functions, or to contemporary concerns, which limits their currency and means they provide poor preparation for future demands.
   1. a) Do you recognise a lack of currency as a potential threat to your Exemplar qualification?
      1. i) [if YES then] what steps do you take to reduce the risk of it arising?
      2. ii) [if NO then] why doesn’t this criticism seem to be relevant?
3. 3) Some critics have said that learning outcomes that are essential to a qualification – but that are very hard to put into writing – get left out of CASLO specifications, which means that students miss out on essential learning (for example,  outcomes like ‘independence’ or ‘autonomy’ or ‘problem solving’ or ‘professional judgement’).
   1. a) Does your Exemplar qualification include learning outcomes that are very hard to put into writing?
      1. i) [if YES then] how have you dealt with this challenge?
      2. ii) [if NO then] are you confident that students are not missing out on essential learning, given the difficulty of writing complex LOs?
   2. b) Are LOs harder to write and pin down for higher level qualifications (L4 and above)?
      1. i) Are there any challenges in achieve appropriate differentiation between qualification levels via LOs?

Section D: How well students learn…

1. 1) Because awarding organisations have to specify standards that can be achieved by all students, critics say that this puts a downward pressure on standards, meaning that no single learning outcome can be pitched at a level that is beyond the reach of the lowest attaining student (within the targeted cohort, that is).
   1. a) Do you recognise this downward pressure as a potential threat to your Exemplar qualification?
      1. i) [if YES then] have you been able to tackle, or mitigate, it? (How would you spot this?)
      2. ii) [if NO then] why doesn’t this criticism seem to be relevant?
2. 2) Critics say that because CASLO qualifications pay so much attention to learning outcomes – which can downplay the importance of an underpinning syllabus – many teachers fail to compensate for this and they fail to deliver coherent teaching programmes.
   1. a) Do you recognise the lack of a detailed syllabus – or the lack of guidance on how to teach the qualification – as a potential threat to your Exemplar qualification?
      1. i) [if YES then] have you been able to tackle, or mitigate, it? (How would you spot this?)
      2. ii) [if NO then] why doesn’t this criticism seem to be relevant?
   2. b) Do you design your unit specifications to capture anything important about progression in learning, or how learning is best sequenced?
3. 3) Because CASLO qualifications represent learning outcomes one by one – and without representing how those learning outcomes relate to each other – some critics say that students fail to learn holistically. This means that their learning is neither systematic, nor integrated, nor co-ordinated, which leaves them unable to apply their learning effectively.
   1. a) Do you recognise the failure to learn holistically as a potential threat to your Exemplar qualification?
      1. i) [if YES then] what steps do you take to reduce the risk of students failing to learn holistically? (How would you spot this?)
      2. ii) [if NO then] why doesn’t this criticism seem to be relevant?
   2. b) Do you believe that your Exemplar qualification does manage to specify learning outcomes holistically? How?
      1. i) Are there some units in your Exemplar qualification that are more difficult to specify holistically? Which ones? Why might this be the case?
   3. c) Is your Exemplar qualification designed to assess learning outcomes holistically, or synoptically? How?
      1. i) Does unitisation provide a barrier to this?
4. 4) Because CASLO qualifications specify learning outcomes one by one – and because they focus attention on detailed lists of criteria that need to be met for each learning outcome – critics say that this disposes students towards superficial learning.
   This might involve demonstrating the minimum possible performance on each criterion for each learning outcome – then moving on to the next learning outcome – and not revisiting learning outcomes that have already been achieved and therefore not consolidating their learning.
   1. a) Do you recognise superficial learning as a potential threat to your Exemplar qualification?
      1. i) [if YES then] what steps do you take to reduce the risk of students learning superficially?
      2. ii) [if NO then] why doesn’t this criticism seem to be relevant?
   2. b) Are there particular steps that an AO, or centre, can take to facilitate robust learning?

Section E: Whether students learn…

1. 1) Critics note that there is often a heavy burden associated with completing and documenting assessments, which can be demotivating. The requirement to achieve each and every learning outcome can also be demotivating – particularly when a student begins to fall behind – and this can result in disengagement and non-completion.
   1. a) Do you recognise demotivation and disengagement as a potential threat to your Exemplar qualification?
      1. i) [if YES then] what steps do you take to reduce the risk of it happening?
      2. ii) [if NO then] why doesn’t this criticism seem to be relevant?
   2. b) Are there particular steps that an AO, or a centre, can take to help engage students?

Delivery challenges

Section F: Burden

1. 1) We’ve already touched on this criticism to some extent. It’s basically the criticism that the mastery requirement forces students and teachers to spend so much time being assessed – and so much time documenting their assessments – that it ends up being a hugely burdensome process.
   1. a) Do you recognise this as a potential threat to your Exemplar qualification?
      1. i) [if YES then] what steps do you take to reduce the risk of undue assessment burden?
      2. ii) [if NO then] why doesn’t this criticism seem to be relevant?

Final observations

1. a) Now that we’ve reached the end of the interview, are there any final observations that you’d like to make, to help us to understand the difference between a ‘stronger’ CASLO qualification and a ‘weaker’ one?
2. b) Is there anything else at all that you’d like to say
   1. i) about your Exemplar CASLO qualification?
   2. ii) about the CASLO approach more generally?
   3. iii) or about how you’ve found the interview?

Appendix 3: Recognition status of potential problems by qualification^{[footnote 1]}

Table A: Recognition status of potential problems by qualification – assessment problems

Problem	Atomistic assessor judgements	Inaccurate judgements	Inappropriate support	Ineffective standardisation	Lenience	Malpractice	Poorly conceived assessment tasks/events
Adult care_L3	Yes	Yes	Not entirely	Yes	Yes	Not entirely	Yes
Business_L3	Yes	Yes	Yes	Not entirely	Yes	Yes	Yes
Chef_L2	Yes	Not entirely	N/A	Yes	No	Yes	Yes
Construction_L1	Not entirely	Yes	Yes	Yes	Yes	Yes	Yes
Construction_L5	Not entirely	Yes	Yes	Not entirely	Yes	Yes	Yes
Creative_L2	Not entirely	Yes	Yes	Yes	Yes	Yes	Yes
Creative_L3	Not entirely	Yes	Yes	Yes	Yes	No	Yes
End of life care_L2	Yes	Yes	Yes	Yes	Yes	Yes	Yes
Fenestration_L2	Not entirely	Yes	Yes	Yes	Not entirely	Yes	No
First aid_L3	No	Yes	Yes	Yes	Not entirely	Not entirely	Not entirely
Hairdressing_L2	Not entirely	Yes	Yes	Not entirely	Yes	Yes	Yes
Housing_L5	No	Not entirely	Yes	No	No	Yes	Yes
Skin peel_L4	No	Yes	No	Yes	Not entirely	Yes	Not entirely
Teaching support_L2	Yes	Yes	Yes	Yes	Yes	Yes	Yes

Table B: Recognition status of potential problems by qualification – teaching, learning and delivery problems

Problem	Content hard to pin down gets missed	Demotivation/ disengagement	Downward pressure on standards	Incoherent teaching	Lack of currency	Lack of holistic learning	Local or personal irrelevance	Superficial learning	Undue ass. burden
Adult care_L3	Not entirely	Yes	No	Yes	No	Yes	Yes	Not entirely	Yes
Business_L3	Not entirely	Not entirely	Yes	Yes	No	Yes	No	Not entirely	Not entirely
Chef_L2	Yes	Not entirely	No	Not entirely	Yes	Not entirely	Yes	Not entirely	Not entirely
Construction_L1	No	Not entirely	No	Yes	Not entirely	Yes	Yes	Not entirely	Yes
Construction_L5	Not entirely	Yes	N/A	Yes	No	Not entirely	Not entirely	Yes	No
Creative_L2	Not entirely	Not entirely	No	Not entirely	No	Yes	No	Not entirely	Not entirely
Creative_L3	Not entirely	Not entirely	Yes	Yes	No	No	No	No	Yes
End of life care_L2	Yes	Yes	Not entirely	Yes	Yes	Not entirely	Not entirely	Yes	Not entirely
Fenestration_L2	No	Not entirely	No	Not entirely	Not entirely	No	Not entirely	Not entirely	No
First aid_L3	No	N/A	No	No	No	Not entirely	No	Not entirely	No
Hairdressing_L2	Yes	Not entirely	Not entirely	Yes	Not entirely	No	Not entirely	Not entirely	Yes
Housing_L5	Yes	No	No	No	No	No	No	Not entirely	Not entirely
Skin peel_L4	No	No	No	No	No	No	No	No	Not entirely
Teaching support_L2	Not entirely	Yes	Not entirely	Not entirely	Not entirely	Not entirely	No	Not entirely	Yes

Appendix 4: Mitigation type by potential problem

Table A: Number of references to different mitigation types by potential problem – assessment problems

Mitigation type	Atomistic assessor judgement	Inaccurate judgement	Inappropriate support	Ineffective standardisation	Lenience	Malpractice	Poorly conceived assessment tasks	Total
QA	11	34	15	30	44	37	28	199
support and guidance	10	53	21	42	12	7	34	179
occupational /professional expertise	14	15	0	5	1	0	10	45
qualification /assessment design features	9	15	1	0	4	1	2	32
holistic aspects	21	8	1	0	0	0	1	31
standardisation	1	7	0	20	1	1	0	30
attitudes	0	0	0	0	15	7	0	22
contextualisation and relevance	4	5	3	0	2	1	6	21
Qualification /assessment design processes	0	13	0	0	1	1	3	18
hybrid aspects	1	3	3	0	5	1	0	13
prioritisation	0	0	1	8	1	0	2	12
supporting learning	0	0	0	0	9	2	0	11
communities of practice	0	3	0	2	0	0	3	8
context independence	0	3	0	2	1	0	0	6
operating on a small scale	0	2	0	1	1	2	0	6
assessment expertise	1	2	0	0	0	0	2	5
inputs	0	1	1	0	1	0	2	5
incentives and disincentives	0	0	0	0	1	2	0	3
implicit content links	1	0	0	0	0	0	0	1

Table B: Number of references to different mitigation types by potential problem – teaching and learning problems

Mitigation type	Content hard to pin down gets missed	Demotivation /disengagement	Downward pressure on standards	Incoherent teaching programmes	Lack of currency	Lack of holistic learning	Local or personal irrelevance	Superficial learning	Undue assmnt burden
holistic aspects	3	2	0	10	8	33	7	11	4
support and guidance	4	2	1	25	1	7	0	3	12
qualification /assessment design features	10	7	11	2	4	4	9	5	1
contextualisation and relevance	1	3	0	5	3	10	13	10	1
qualification/assessment design processes	17	0	4	0	12	0	6	0	0
attitudes	1	4	9	0	1	1	0	8	14
occupational /professional expertise	5	0	0	16	4	8	1	3	0
inputs	3	3	4	14	0	0	0	1	5
QA	0	2	2	9	2	5	0	0	5
supporting learning	0	11	6	0	0	0	1	0	4
implicit content links	0	0	0	10	0	6	0	1	0
context independence	0	0	2	1	3	0	5	0	0
prioritisation	0	1	0	0	0	0	0	0	5
hybrid aspects	0	2	1	1	0	1	0	0	0
assessment expertise	2	0	0	0	0	0	1	0	0
communities of practice	0	0	0	2	0	0	1	0	0
incentives and disincentives	0	1	0	0	0	0	0	0	0

1. Recall that our interview questions were framed in terms of ‘potential’ problems rather than ‘actual’ problems. Therefore, by saying that AOs ‘recognise’ a problem, that means that they recognise it as (at least) a potential problem (though some might also recognise it as an actual one). See explanation in the Analysis section about the use of the categories presented in the tables in this appendix. ↩