Responding to Criticisms of the CASLO Approach (Report B): AO views of potential teaching, learning and delivery problems

Question 1

Local or personal irrelevance and lack of currency

Accepted Answer

Because CASLO qualifications are highly specific about the LOs that need to be acquired, this has led some critics to claim that they are too inflexible to respond to local economic needs, the bespoke needs of small employers, or students with particular interests or aspirations. This lack of flexibility may then lead to content being taught that lacks local or personal relevance to students or qualification users. Relatedly, some critics have argued that the level of detail about the LOs and AC in CASLO specifications inevitably ties them to existing work functions, and to contemporary concerns. This may limit their currency and mean that CASLO qualifications provide poor preparation for the future.

Only a small number of AOs in our sample recognised these as potentially relevant problems for their exemplar qualifications, with 3 recognising local or personal irrelevance and 2 recognising lack of currency as potentially relevant. Except for the AO offering the ‘dual purpose’ Construction_L1 qualification, which recognised the potential problem of local or personal irrelevance, the other AOs that recognised these potential problems offered ‘confirm competence’ qualifications, suggesting that they might be less of an issue for AOs offering ‘dual purpose’ qualifications.

The discussions about these 2 potential problems highlighted the need to achieve a, sometimes difficult, balance between specificity and breadth of content in teaching and learning. This was often influenced by broader qualification purposes and attitudes of qualification users regarding how far that balance should tilt towards narrower occupational roles versus broader educational goals. Overall, the AOs did not think that these potential problems were specific to CASLO qualifications and argued that content specification for any type of qualification may face similar issues.

Most AOs flagged the flexibility of their qualifications in supporting contextualisation as a key aspect facilitating local or personal relevance. Several AOs also discussed certain context-independent aspects of their qualifications that are seen as essential, irrespective of personal or local preferences, and that largely retain longer-term currency. This was particularly the case in lower-level qualifications where the key focus is on core technical skills. In relation to the potential problem of lack of currency, qualification design processes such as qualification review were seen as a key mitigation. Several other mitigations, including some qualification design features such as broad Los or AC and optional units, were also mentioned in relation to both potential problems, as were supporting learning, practitioner CPD, and aspects of support, guidance and QA.

Contextualisation and holistic aspects

Most AOs thought that their qualifications achieved appropriate specificity through contextualisation and that, therefore, the potential problem of local or personal irrelevance was sufficiently mitigated. For instance, in the creative sector, one AO emphasised that their LOs and AC did not involve specific genre or stylistic requirements. Another AO pointed out that their LOs and AC are sufficiently broad to allow content important to individual employers to be mapped into the existing specifications to increase local relevance, although the extent to which this might be done had to be justified financially.

And I think, as we said earlier, there’s flexibility within the context, the way they can be assessed. […] we recognise that there are differences around the country in terms of terminology and, maybe, the way that people are doing things, but that’s why the assessment criteria is written as it is […] so we can take that into account.

Fenestration_L2

Sufficiently broad AC were also suggested as a mitigation in relation to potential lack of currency. Certain aspects of qualification content, particularly those related to legislation, policies, regulations or technology, were deemed to date more quickly as they advance or are updated. Therefore, AC usually refer to them as “latest” or “current” to allow for changes to be reflected in teaching or assessment without the need for qualification redesign, thus “future proofing” them.

Alongside contextualisation, most AOs discussed certain context-independent aspects of their qualifications that helped to limit the impact of the potential problem of local or personal irrelevance. In several exemplar qualifications, a significant proportion of content was deemed not suitable for tailoring to local or personal preferences. For example, in the qualifications that were mapped to National Occupational Standards (NOS), these were thought to ensure that important, nationally relevant, content, necessary to perform in a role, was included in the qualification irrespective of local or personal preferences. However, some AOs pointed out that qualifications mapped to the NOS had the potential to lose some of their currency by the time they are launched due to the lengthy process required to approve them. This suggested the need for implementing some currency‑related mitigations, which typically involved regular qualification reviews, as discussed in the next section.

[…] the treatment area is the treatment area, and they have to know things and be able to do things that are really specific. And that is, because it’s compliant with the NOS, it’s automatically complying with the national need […] So the learning outcomes are really specific and then the criteria can be a little bit more flexible.

Skin peel_L4

Qualification and assessment design processes and features

Some AOs flagged qualification design processes that helped to ensure relevant content is included in their qualifications from the outset and regularly reviewed to ensure its currency. Involving stakeholders, including employers, teachers or students in qualification development or review panels or consultations was among the most commonly mentioned aspects. Indeed, one of the qualifications in our sample was developed specifically in response to and with input from a particular centre to support the specific needs of their students. AOs also spoke about regularly collecting feedback from centres (often through EQAs) and other stakeholders about their qualifications, including in relation to how far contextualisation and local needs were supported. This then informed ad hoc or regular qualification revisions or reviews (with the latter usually taking place every 5 years). Regular qualification revisions or reviews were also used to mitigate a potential lack of qualification currency and to update qualifications in response to changes in the NOS, best practices, or regulations, where these were more directly referred to within the LOs or AC or in the guidance.

And, I think, […] the annual review process does mean that if something new comes out, for example, […] a critical piece of learning that every student in this particular sector has to have, then that can be built in. We try not to mess around with fundamental things in the specification within that 5‑year period, we tend to just do modifications, but sometimes a major amendment is needed, but we can respond to that and do that.

Construction_L5

In terms of specific qualification design features, one AO highlighted the importance of ensuring that command verbs are used appropriately in AC to allow for an optimal balance of specificity and contextualisation. This was related to specifying sufficiently broad LOs or AC, which can generalise to different contexts or student interests. Several AOs also spoke about optional units in their qualifications and their potential to additionally facilitate local or personal relevance. The optional units allow centres to tailor the qualification, so that they better suit local vocational settings. Some AOs said that they can sometimes develop specific units that might support individual employer needs for their staff continuous professional development (CPD), though these would not be certified as part of the broader qualification.

[…] learners can do a unit on stand-up comedy, they can do theatre and education, they can do devising, they can even do masks and puppetry. So, a lot of this stuff is quite targeted specifically to specific areas of industry that they might want to move into […].

Creative_L2

The AOs also noted certain constraints on personalisation of their qualifications despite the abovementioned mitigations. The AOs offering qualifications delivered in the workplace explained that the nature of the job and the workplace needs to sufficiently align with the content of the qualification, so that the student would have the opportunity to provide relevant performance evidence. For instance, as the Chef_L2 qualification includes units requiring students to cook meat, it would not be possible for students to take this qualification while working in a vegan restaurant. In such cases, the requirement would be for students to either take an additional job to enable them to cover that content or not take that qualification at all. Others suggested that the range of optional units that might be offered to students by centres is, to some extent, limited by centre resources rather than always involving free choice in relation to students’ interests or context.

AOs with lower-level qualifications additionally mentioned the fact that such qualifications necessarily include the core skills needed for the different occupations, which should be seen as universally relevant. This type of core content was also deemed less likely to lose its currency quickly.

The AO offering the level 2 hairdressing qualification pointed out potential challenges with ensuring sufficient student engagement in lower-level qualifications focused on core skills which do not simultaneously allow for a certain degree of personalisation through creativity or other aspects. However, it was also suggested that student expectations should be managed, as students need to accept that certain core skills need to be sufficiently embedded before they can be built on creatively or in other ways in higher-level qualifications. We discuss potential problems of student demotivation and disengagement later in the report.

Support and guidance, QA and CPD

Several AOs spoke about the need for CPD of their own staff, including EQAs, as well as of centre staff, to ensure the relevance and currency of qualification content in teaching, QA, development and review. One AO mentioned qualification writer training which was aimed at helping qualification developers to achieve the balance between specificity and breadth of the LOs or AC, to facilitate appropriate contextualisation without compromising reliability.

It’s a difficult line to walk, to make [LOs] clear enough through specificity but not so granular that they are inflexible, so that’s what we’re always trying to do and what we train writers to do, and we have a formula that we ask writers to follow, there are certain elements that they need to follow in how they define the learning outcomes.

Construction_L5

Some AOs also mentioned that EQAs are there to support and advise centres regarding changes in legislation, technology or other aspects. EQAs also collected feedback about how the qualification was being delivered, and if centres and students had any concerns regarding the relevance or currency of content.

[…] at every EQA interaction with a centre, there’s space within the documentation for them, for us to get their feedback and their views on the qualifications. […] are they still valid, are they still relevant, are they still fit for purpose. So, we do sort of encourage centres to have that input. […] talking to the assessors and ensuring they’ve got regular CPD, that’s part of it.

Fenestration_L2

Supporting students and learning

As another way in which local or personal relevance might be facilitated, one AO suggested that centres might teach content beyond the core specification of the qualification if this was considered appropriate. There may also be flexibility in delivery to teach (and assess) other things in addition to the specified qualification content if required by employers, keeping in mind that the extra content will not contribute to the qualification as such.

[…] the qualification is the qualification, that doesn’t mean you can’t teach other things. And you can assess those other things if you want, it won’t contribute to the qualification, but if you were a training provider or the employer wants the person to know this as well, there’s nothing to stop you doing things outside of the qualification.

Adult care_L3

As the flipside of potential increased relevance of qualifications to individual employers, some AOs recognised the risk that some of these qualifications might cover a relatively narrow range of LOs or be delivered in a narrow range of contexts, perhaps not allowing for sufficient transferability to new employment contexts. One AO described push back from some employers regarding the greater breadth of content in apprenticeships which did not align with their specific areas of work, and who, thus, did not see that as “adding value” to their business.

[…] I suppose, to flip this, that’s one of the criticisms now of the apprenticeship standard. Because within the apprenticeship standard, the individual has to understand all about plastic and aluminium and timber, and we’re getting kickbacks from employers – why is my training provider teaching them about plastic when we don’t do plastic […]. And we explain that’s the standard, […] the standards are about making someone fit for a broader industry than just their employer. So, it’s that balance really of giving the all-rounder knowledge.

Fenestration_L2

For some AOs, this was part of an ongoing debate concerning whether students are being trained to perform in a particular occupational sector or in a specific job. Some AOs suggested that good training organisations would be preparing students for the wider occupational sector, implying that this is what their qualifications enable them to do. Nevertheless, others specifically designed some of their qualifications to allow for delivery in specific employer-relevant contexts. For instance, the exemplar Fenestration_L2 qualification could be delivered and achieved within employer settings solely focused on installing plastic windows and not require students to also learn about installation of aluminium or wooden windows, unlike, perhaps, some of the qualifications that form part of apprenticeship on-programme training, which are required to be broader.

[…] ever since work-based qualifications were conceived, it’s always been that sort of dilemma, are they being trained to do that role or are they being trained for that particular job, and […] it does come down to the training organisation, because a good training organisation will deliver the training for the role, a poorer one might just do it for the job and to get them through.

End of life care_L2

Some AOs suggested that qualifications which are explicitly linked to the NOS were more likely to be transferrable to different jobs within an occupation rather than being too tied to individual employer contexts. However, others expressed concerns that the content specified in qualifications can be overly driven by a subset of influential employers that might be involved in their development, or the development of the NOS, thus threatening the local relevance of these qualifications or relevance to the wider sector. Ultimately, most AOs seemed to agree that these kinds of issues of content relevance and the balance that needs to be ensured between content specificity and content breadth were not unique to CASLO qualifications.

So, I just don’t think it’s something that’s CASLO specific. I think it’s something that’s specific to any type of qualification really. […] You know, it’s very difficult to make every single qualification meet the needs of every single person, and some people are going to end up inevitably learning something, a bit of something, that may not directly apply to them.

Housing_L5

In relation to qualification currency, several AOs argued that a qualification awarded at one point in time cannot be expected to “futureproof” someone’s career, despite the qualification being up to date when studied, and sufficiently broad to be transferrable. They suggested that this is mitigated by both students and employers accepting that there is a need for life-long learning and CPD, and investing time and resources towards this.

[…] when you achieve a qualification, you achieve it at that point in time. You’ve then got to look at yourself, or your employer looks at you in terms of keeping you up-to-date, you know, and we have particular qualifications like update qualifications. There are different requirements in different sectors for CPD and that sort of thing. The qualification is there to give you a grounding and introduce you into a particular occupation.

Fenestration_L2

Question 2

Content hard to pin down gets missed

Accepted Answer

Some critics say that LOs that are complex and difficult to put into writing in the commonly used format of relatively brief LO statements may, in consequence, get left out of CASLO specifications, even when they are essential to the qualification. This might mean that students miss out on essential learning. This kind of content might include, but is not limited to, relatively esoteric outcomes such as ‘independence’, ‘autonomy’, ‘problem solving’ or ‘professional judgement’.

Only 4 AOs acknowledged the potential relevance of this problem to their own qualifications, all of which were ‘confirm competence’ qualifications. The most common response (N=6 AOs) was to not entirely recognise this potential problem, while an additional 4 AOs did not recognise it at all. All AOs largely agreed that there are aspects in qualifications that are difficult, but not impossible, to capture in words.

They discussed some mitigations, revolving around having robust and intentional design processes, and the need for occupational or professional expertise of those involved in qualification development. Seeking relevant professional input and feedback from stakeholders was also viewed by the AOs as valuable.

Despite relatively little recognition of this potential problem in relation to what AOs considered to be the core qualification content, some comments suggested that the more esoteric content was, in fact, often considered non-essential to the qualification construct, and was, therefore, largely left implicit. These were skills or attributes such as communication, autonomy, resilience, collaboration, teamwork or problem-solving (variously referred to by the AOs as behaviours, attitudes or transferrable skills). Such content was typically considered to be a “value‑added” or a benefit of the teaching and learning process, its acquisition facilitated by the contextualised nature of CASLO delivery rather than missing. Several AOs which did feel that such content should be explicitly specified and assessed spoke about the benefits of the contextualised nature of assessment in CASLO qualifications in facilitating elicitation of such esoteric aspects. Where AOs spoke about the challenges of specifying and assessing such content, they tended to agree that this was not a CASLO-specific issue.

Qualification and assessment design processes and features

Most frequently referenced mitigations of this potential problem revolved around appropriate qualification design processes. AOs recognised that some LOs or AC are harder to write than others, but this was addressed by having robust and iterative design processes, including development and review panels, working groups involving experts and stakeholders, as well as consultations with centres. One AO emphasised their structured design process, with different layers of review across writers, reviewers and lead reviewers. Several AOs spoke about striving for language clarity, devoting a lot of time and attention to finding the right words to articulate complicated aspects of their qualifications. They also spoke about working closely with stakeholders and incorporating stakeholder feedback in the design process to ensure that no essential aspects are missed.

[…] we didn’t shy away from any difficult or complicated content or requirements in the design of the qualification. We started our development by […] talking to industry, talking to HE, talking to FE delivery staff members and understanding what they felt was the requirement from a qualification, and we wrote a qualification which encompassed all of those elements.

Creative_L3

I think there will be decisions to be taken when choosing the wording, not least because we do work with professional bodies and they’ll have their own set of standards that are perhaps very technically worded, so it’s finding a way to incorporate those without making it a bit of a Frankenstein’s monster of a unit, it’s finding a way to weave that in without losing what’s required by the professional organisation […]

Construction_L5

A few AOs emphasised the role of command verbs in helping to capture different aspects of content and requirements. Some AOs with lower-level qualifications believed that this potential problem was less relevant in their context as they thought that content at lower levels is easier to capture comprehensively in writing. Others did not think that higher qualification level was related to increased difficulty in communicating LOs believing this to be subject specific rather than level-related. One AO thought it was easier to capture the more esoteric aspects of content explicitly at higher levels, although, in their case, this seemed to be facilitated by the absence of certain design constraints present in some of their lower-level NVQ-style qualifications.

Yeah, it is a lot easier [at higher levels] because the qualifications aren’t so linked to the NVQs and, sort of, basic job competence, so there’s much more freedom and flexibility in terms of our assessment design.

Hairdressing_L2

Implicit content, contextualisation and holistic aspects

Despite being largely confident that robust and intentional design processes can help to mitigate the risks of essential content not being captured in qualification specifications, several comments suggested that there were various aspects that were either deliberately left implicit or were captured or communicated via means other than LO or AC specifications.

Some AOs noted that a great deal of learning of certain skills or attributes such as communication, autonomy, resilience, collaboration, teamwork or problem-solving might be happening due to the contextualised, holistic nature of delivery of these qualifications, though not all of these were directly assessed or could be reliably assessable. Some AOs saw these as “value-added” benefits of the teaching and learning process rather than part of the construct that was being assessed in their qualifications. However, AOs also recognised that such transferrable skills and attributes were often highly valued by stakeholders.

[…] we’re assessing the creative process. Autonomy and all those other transferrable skills, they are fantastic. They are by-products of the teaching and learning experience that are very much valued by industry, by stakeholders and students. But that particular component doesn’t need to be assessed, because it has no value in terms of us assessing the creative process.

Creative_L3

The AOs also emphasised the benefits of the CASLO approach in allowing aspects of such skills and attributes to be engendered more organically during delivery and more easily evidenced through assessment in contextualised situations, rather than being missed. Some AOs noted that there might be scope to make certain skills and attributes such as “employability skills”, “presentation skills” or “communication skills” more explicit in assessment.

[…] how do you assess resilience, but a student may well be getting very resilient over the course of their study, because they’re doing stuff with a local employer and it’s great growth for them, or they’re working with somebody and they’re having to adapt their style, or they’ve got a difficult challenge they’re having to work on as a group. […] so that could be more inherent, but there are certain units where communication skills are going to be assessed […] So, yeah, sometimes it is explicit and sometimes it can’t be because it’s just too hard to accurately or even close to accurately put on a piece of paper, so that’s another, I think that’s a challenge for all classical and CASLO type qualifications.

Construction_L5

Several AOs suggested that some of these more complex or esoteric outcomes were implicit in qualification levels, in the sense that level 3 would, for instance, imply a higher degree of expectation regarding autonomy than level 2. Instead of being an explicit LO at level 3, such aspects are captured in general assessment requirements to perform more autonomously than at the level below. It was also suggested that even where the LO specification might on its own be unclear as to the requirements regarding the more esoteric content, exemplar assessment materials and other resources such as guidance documents should help to form a clearer view about the overall intention of the qualification.

[…] for the most part, autonomy is what makes this qualification a level 3, for example. […] It’s all inferred across the qualification as opposed to being a distinct learning outcome, so there’s no learning outcome that says, the learner will be able to work autonomously on their own without the input of others, because that’s implied by the qualification as it is.

First aid_L3

Other AOs approached this differently, by making it more explicit in their LOs that autonomy is an outcome, for instance, by specifying that an outcome is about “realising a self-initiated project”, as in the Creative_L3 qualification. However, this AO did not go further in terms of defining their grade descriptors in relation to this outcome, leaving the assessment of autonomy to some extent implicit as an expectation rather than a clear criterion. In the Skin peel_L4 qualification, some more general pre-requisite units on performing specific treatments include LOs such as “maintain professional role and professional counsel” and refer to interaction with clients, professionalism, attitude, signposting, awareness of taking into account client’s mental health and so on.

[…] at unit 6, it’s an entirely self-directed, self-initiated extended project for the students. […] And you’ll see within, for example AO4, it’s “realising a self-initiated project”. So, we lean into this idea of autonomy within the words that we use. We might not necessarily say, you know, autonomous is not necessarily something that we define within our glossary for grade descriptors, etc., but the expectation is that the students are being autonomous.

Creative_L3

In relation to Hairdressing_L2 qualification, it was also pointed out that not only are some of these aspects difficult to write into specifications precisely, but that it is sometimes difficult to understand what stakeholders such as employers mean when they refer to “autonomy”, “initiative”, “commercial agility”, “decision making”, and so on. Although some aspects of these skills were incorporated into this qualification, the AO emphasised that it was important to manage employer expectations about the limited extent to which some of these were covered in their qualification. The AO suggested that employers should have an awareness of their own role in the continuing development of those that they employ. This AO also suggested that such broader skills and content are deprioritised partly because of the time pressure in this qualification and the need to focus on the core technical skills in the time available.

So, they are in there as sort of broad statements, and we sort of know what minimum looks like in terms of competence, but because they’re not graded qualifications and because […] you can’t really pin down what employers want, I don’t think the qualification really sort of meets that agenda, and we haven’t really identified what that agenda is either. So, there are limitations, […] but all in all we’re sort of comfortable with those limitations because it’s about being good enough and sort of reasonably competent in these communication areas to start in employment and then the employer takes it after that, certainly at level 2.

Hairdressing_L2

Occupational or professional expertise and attitudes

Several AOs made references to the importance of both AO staff and assessors having sufficient occupational or professional expertise to be able to understand, teach and assess the important aspects of the relevant domain. Assessor professional judgement was particularly emphasised as important by one AO, to the extent that it could replace explicit specification of esoteric outcomes such as autonomy or professionalism. The main reason given for this was that experts can recognise nuances such as how the student speaks to or treats somebody, which cannot be easily written down in specifications.

Particularly in care, for example, it’s interesting you’ve spoken to someone in that sector, because I think that one, in particular, again you absolutely cannot replace years of knowledge and experience in judging someone else’s suitability and ability at doing something. We could all read a 2‑page document on how to run a bath for a resident, but, actually, it’s in the nuances of how they speak to somebody, how they treat them, how they just have an understanding of what that person might be going through at that point […] that you can’t get from a list of potential judgement outcomes […].

Chef_L2

AOs also referenced the potential value of positive attitudes within centres related to teaching broader and enriching content, including transferrable skills, irrespective of whether these were specified in the LOs. One AO contrasted such “excellent practice”, including “meaningful assessment”, with other, more minimal approaches which might be detrimental to student progression, even where they may not prevent them from passing the qualification. It was implied that this AO expected centres to teach such broader content.

There are certainly things that centres do that are examples of really excellent practice. So, for example, there is a centre that delivers a level 3 [qualification] […] where all the evidence is presented as a website that the learners create. So, they effectively create a portfolio for themselves that includes a CV […], and they’ve got video excerpts that have been properly edited rather than just the camera lift up and focus on learners in the corner of a classroom. […] But they’ve also created something that’s embedded wider skills because they’ve designed websites, they’ve thought about presenting themselves to industry and, to an extent, it’s something they can take with them and they could direct people to as part of applications for further, higher education or even gaining work or auditions. […] that’s a really good example of meaningful assessment, isn’t it, which is obviously what we’re trying to encourage centres to do, and it’s quite stark, the contrast between the centres who you can see it’s just one learner in a classroom with somebody playing piano as a backing track for them [versus] centres that are putting on a gig, a show with parents and other staff members and other learners all in the audience who give feedback and they get an actual proper experience out of it.

Creative_L2

Question 3

Downward pressure on standards

Accepted Answer

Because awarding organisations have to specify standards that should be achievable by all students in a target cohort, critics have said that this puts a downward pressure on qualification standards. This means that no single LO can be pitched at a level that is beyond the reach of the lowest attaining student within the targeted cohort, especially where, with a strong mastery model, not achieving even one LO would mean not achieving the entire qualification.

Only 2 AOs, both offering ‘dual purpose’ qualifications, recognised this problem as potentially relevant to their exemplar qualifications, with others not entirely recognising it (N=3) or not recognising it at all (N=8). Most AOs emphasised that standards in their qualifications are necessarily linked to occupational requirements and are thus non-negotiable, irrespective of student ability to achieve them. However, this position was slightly more flexible in one AO offering a ‘dual purpose’ qualification. There, a more inclusive approach to specifying standards was deemed to be required, while managing qualification user expectations about the standards that students were likely to achieve. This was because these qualifications often prepare students for entry‑level jobs or progression to education, in which further learning was both required and expected.

AOs spoke about ensuring sufficiently robust design processes and involving stakeholder feedback so that appropriate standards can be set in their qualifications and that they are appropriately pitched to the relevant qualification level. Beyond that, it was necessary to ensure appropriate and sufficient teaching, which some AOs encouraged their centres to do. Certain flexibilities in some CASLO qualifications, such as absence of constraints on learning time and opportunities to resit or retake assessment, were deemed helpful in providing sufficient opportunities for students to achieve appropriate standards. This removed the pressure to set low qualification standards by design. Some AOs also mentioned several protective attitudes that helped reduce the risk of this potential problem arising, such as sufficient student engagement with teaching and learning process, and AO integrity.

Qualification and assessment design features and processes

Getting the demand right for the qualification level was emphasised as important in the development process, alongside pitching different elements of the qualifications appropriately to that level, including the total qualification time, the guided learning hours and the credits. Some AOs described different mechanisms that helped them determine the appropriate standard within the level, including their notion of the “scope” or the “range” of a level and their understanding of typical students undertaking qualifications at a particular level. The AOs again referred to the way they used command verbs to indicate the standard of the AC in their qualification (for instance, not including “evaluation” or “analysis” in level 2 qualifications). The AOs that mentioned this suggested that the standard is set “in the middle” of the typical range of a level. In contrast, another AO noted that, due to the combination of the low level of their exemplar qualification and the absence of grading, the standard that was being set was minimal, but still ensured basic safety and sound technical skills and was, thus, acceptable to stakeholders. They emphasised that this minimum threshold was adhered to and that students still sometimes failed if unable to reach it. Several AOs spoke about making sure that relevant stakeholder feedback was consulted to ensure that the appropriate standard was set in their qualifications.

[…] when we’re developing qualifications, you’re looking at the framework you’re developing against, the level you’re developing for, the scope within that level and where you’re pitching those, the TQT and the Guided Learning Hours, your credits, you don’t want to be at the bottom end or the top end of that range, you’ve got to get it right to allow the movement, […] and you’ve got the balance right. […] because there’s many qualifications that I’ve seen over the years where they’re just on the cusp of about to tip into another level and they become too hard then, and unreachable for those that might be at the cusp of the level below, so you’ve got to pitch it right.

End of life care_L2

Asked specifically about whether it was more challenging to specify appropriate standards for knowledge-related content, one AO suggested that it was not necessarily more difficult to specify this, but that, due to variable cohort ability, it was more difficult to ensure that standards were adhered to by centres when knowledge‑related evidence was collated in portfolios. However, they emphasised that the evidence from externally set knowledge tests, observations of performance and portfolios triangulate, and represent multiple hurdles, providing greater assurance overall about whether the appropriate standard was reached. Another AO suggested that, in their qualification, knowledge and skills were intertwined to the extent that a potential reduction in standards for knowledge content would manifest in weaker performances in related skills areas. This mitigated the risk of potentially lowering the knowledge‑related standards to match lower cohort ability.

So, I think the knowledge and the skills are intertwined, you have to have the knowledge in order to be able to master the skills. […] I don’t think one could be more than another and I think they’re equally as important and if you’re expressing that theoretical knowledge at too low a standard then they’re not going to be able to perform and learn those skills, are they?

Construction_L1

Several AOs emphasised that if, despite sufficient teaching and opportunities to be assessed, a student still was not able to pass, the employer or centre would need to consider removing them from the course. For these AOs, the occupational standards involved the core content essential to the qualification, particularly in safety‑critical domains, which had to be achieved and were not negotiable. Alternatively, if students were unable to meet the standards at a particular qualification level, they could attempt lower-level qualifications. This appeared to be a common position among the AOs that offered ‘confirm competence’ qualifications.

[…] if someone comes into this and takes one of our qualifications and […] after 4, 5, 6, 7, 8 times of teaching or being assessed and they can’t meet it, then there’s a decision for the employer to take there, to say – look, you just can’t do it. So, it’s necessarily a fail. […] I don’t think it’s, you know, as the criticism states. The qual is written for an occupational role. If they can’t carry out that occupational role, then they can’t achieve the qual.

Fenestration_L2

[…] I think maybe because it’s in an employment context. Nobody’s ever fed it back in a development meeting. They’ve never said I’ve got a learner that wouldn’t be able to do that. You know, it is what it is, that’s the expectation, and if they can’t meet it, they shouldn’t be on that qualification, maybe they need to look at a lower-level qualification.

Housing_L5

On the other hand, one AO offering a ‘dual purpose’ qualification suggested that, because such qualifications prepare students for either progression to education or entry‑level jobs, a more inclusive approach to specifying standards was required and, perhaps, more negotiations between stakeholders about the appropriate standard for different qualification purposes and intended progression routes. This AO seemed to prioritise ensuring that the qualification was pitched at a standard which was accessible to a range of students, ensuring that the qualification included content which was relevant and important to the students’ next steps. The AO emphasised the need to manage expectations of qualification users about the standards that students were likely to achieve, as these qualifications prepare students for entry-level positions, in which further learning was both required and expected.

Inputs and supporting learning^{[footnote 2]}

Most AOs emphasised the need for appropriate and sufficient teaching to enable students to achieve the standards in their qualifications and that it is “about raising people to the standard” rather than bringing the standard down to the lowest common denominator. One AO noted that they expected their centres to be “ambitious in their delivery” and to set projects and themes that are appropriate for the level, which the AO monitors as part of their QA process. Several AOs pointed out that absence of time constraints on learning as well as multiple assessment and resit opportunities help to ensure that appropriate standards are reached by students, including space to receive additional support for more demanding areas of the qualification.

I don’t think the qualification standards do get lowered to the lowest common denominator. It is about raising people to it; hence you don’t fail, you just need to learn more. It’s not about let’s lower the standards, so people don’t fail. It’s about, there’s the standard, you’ve got as much time as you need to get to that level.

Chef_L2

I don’t think I’ve ever heard anyone say, oh well that’s quite a difficult bit of the job, so we’ll exclude it. I think what we’d say, that’s quite a difficult bit of the job so the learners might need a bit extra support to get there, or it might take them a bit longer.

Adult care_L3

Despite most of the exemplar qualifications in our sample not having specific entry requirements, it was implicit in some AO comments that centres should consider how appropriate and achievable the qualification may be for the students, given their starting point. This could be done through some form of initial assessment or by “recruiting with integrity” and in the best interest of the students in terms of them having the potential to achieve the qualification. In some cases, the initial assessment would revolve around making sure that students work in jobs which would enable them to achieve the desired qualifications where these require appropriate workplace evidence.

I know it’s a Level 1 qualification […] But the students still have to work independently and if they’re not capable of working independently then the centre isn’t recruiting with integrity and are […] setting students up to fail rather than the qualification being pitched at a level that is too high or is too low and if it’s too low so therefore it has no value.

Construction_L1

Attitudes

Alongside appropriate and sufficient teaching, AOs emphasised the need for sufficient student engagement in the learning process. Where students were engaged, there would be a higher chance of them reaching the appropriate standard.

You can put on all the teaching and support in the world, if the learner’s not engaging, they will not pass. With engagement in place then the rest of it remains true in that, if the learner is engaging and they’re on the right qualification and they’ve got teaching and support, they should be able to pass the qualifications.

Creative_L2

One AO emphasised that the nature of the construct certificated by their exemplar qualification – the creative process – creates a natural barrier to low standards, potentially in combination with the mastery model. It ensures that even a minimum level of achievement still requires a great deal of continuous student engagement, thus countering the criticism that there is a downward pressure on standards in such qualifications.

What we’re talking about here is a whole entire rehearsal process leading up to the performance, the planning around that and the review after that. […] So, the minimum level of engagement there is still quite high, […] there’s not like this crazy low threshold on learners […] It actually does take quite a lot of work […] And also, you’ve got something which is a creative output which we’re assessing as well and looking at. […] You cannot just show up not knowing your craft in any way and perform. […] So, with creative subjects I don’t see it as much as a problem just because there’s so many natural barriers in the way of them just attaining it without any effort.

Creative_L2

Several AOs spoke about their integrity as organisations, being very aware of perverse incentives such as funding pressures or pass rates that might incentivise lowering standards to achieve larger volumes and, therefore, profit. The AOs also spoke about their awareness of the impact that inappropriate standards at lower levels can have on student progression to higher level qualifications, and this was another reason why they did not consider deliberately lowering the standards.

[…] more broadly, yes, […] we’re always aware that there is the race to the bottom, as we call it, and […] going to the lowest common denominator. […] I don’t recognise that for our qualifications because that’s something that we’re actively trying to fight against. […] personally, that’s not what I’m about and I certainly wouldn’t want us as an organisation to be about that either. You just lose your integrity, I think, and integrity is a big thing […], but not everybody’s like me.

First aid_L3

I don’t think that’s the case and I don’t think that’s something that we would even think about that this is too hard […]. I just think even at Level 1, there are standards that have to be met otherwise what are you equipping them for? […] How are you going to allow them to progress if they are [only] meeting the standard because you’re making it too easy?

Construction_L1

Question 4

Incoherent teaching programmes

Accepted Answer

According to criticisms in the literature, because CASLO qualifications pay so much attention to LOs – which can downplay the importance of an underpinning syllabus – many teachers fail to compensate for this, and they fail to plan and deliver coherent teaching programmes. This might be partly affected by the failure of the LO-based units or qualifications to capture important aspects of learning progression or how learning is best sequenced in a particular domain.

This potential problem was the most frequently recognised among teaching, learning and delivery problems, with half of the AOs (N=7) believing it was potentially relevant to their exemplar qualifications. The other half did not think that this potential problem was (entirely) relevant to their exemplar qualifications. It was recognised by 4 of 5 ‘dual purpose’ qualifications while only 3 of 9 ‘confirm competence’ qualifications recognised it.

The most frequently referenced mitigations involved offering support and guidance to centres or students, aspects of which involved providing guidance related to, what might be classed as inputs to the teaching and learning process, that is, pedagogy and/or schemes of work. Inputs in the form of mandatory or indicative content lists were also suggested as helpful, as were certain holistic aspects of qualification delivery and/or assessment. Occupational or professional expertise of teachers was mentioned equally frequently as support and guidance as an important mitigation alongside teacher ability to see and make use of implicit content links, based on the nature of qualification content or context. All AOs except one mentioned at least 3 of these mitigation types in different combinations and some mentioned all of them.

One of the themes that emerged in relation to this potential problem was the need to ensure a balance between prescriptiveness and flexibility or contextualisation of teaching and learning in these qualifications as a certain degree of flexibility was seen as one of the key benefits of the CASLO approach, despite associated challenges. Views related to the possibility of achieving this balance in their qualifications appeared, at least in part, to shape the position of the AOs in relation to how much and what type of content specification and delivery guidance they thought was appropriate to provide.

Flexible delivery and contextualisation imperative

Whether or not the AOs provided any specific inputs to teaching and learning, they tended to emphasise the need for flexible delivery and contextualisation, which they thought were very important in their context. This was a strong theme that permeated most of the discussions in relation to teaching and learning problems.

So, what you’ll see against each of the learning outcomes is some specific general mandatory content. So, we’re not going into specifics. What we’re saying is that you need to teach the students material that is relevant to their specialist practice, but we don’t specify exactly what those materials are. So again, that’s allowing individuality to come out depending on the cohort, the students’ interests, and the resources a centre has available.

Creative_L3

Flexible delivery and contextualisation were particularly emphasised by the AOs that did not see incoherent teaching as potentially relevant in their context. They expressed fairly negative views in relation to prescribing delivery approaches or schemes of work to centres. Some of them also argued that it was necessary to allow certain content areas to be de-emphasised for some students, where they may have already acquired these in a previous job, for instance. Some of these AOs thought that, despite being more challenging for the centre, the opportunity to provide tailored delivery in this kind of vocational qualification was largely beneficial for the majority of the centres and students.

[…] in terms of saying to a centre – and this is how we want you to deliver it, we want you to deliver this on this day, I’d say well that’s not really for us to say. Leave that to the people who are qualified teachers who know the learners, who know how the centre works.

Adult care_L3

I think it’s fair to say that delivering qualifications such as these is a lot harder than delivering a GCSE because you don’t have a textbook to deliver from. So, I think these qualifications actually force you to engage more with the content of the syllabus and come up with something which is right for your learners […] using the resources at that centre which is relevant to the part of the country that they’re in.

Creative_L2

Inputs: content lists or syllabi

Despite emphasising the need for contextualisation and flexibility, the majority of the AOs in our sample specified some form of teaching inputs, such as mandatory or indicative content lists, to supplement the LOs and AC in their qualifications. Though, as noted above, these tended to be specified at a relatively high level of generality. AOs suggested that this should help provide an indication to teachers about what to include in their teaching programmes, as well as to inform and to help standardise teaching programmes in terms of coverage.

AOs delivering Business_L3, Construction_L1, Construction_L5, Creative_L3 and Hairdressing_L2 qualifications – which all thought the problem of incoherent teaching was potentially relevant in their context – specified core mandatory content for their exemplar qualifications. In Business_L3, this content was referred to as the syllabus. In relation to the Creative_L3 qualification, it was emphasised that the introduction of core mandatory content was in response to HEIs, who wanted to have confidence that students who take this qualification know an appropriate range of materials that are relevant for their specialist practice. This was thought to support the comparability of student experience and the use of results for selection into higher education. Other AOs, however, noted that despite specifying core mandatory content, this still left room for variation across centres in terms of the quality of the learning experience that they provided to students in relation to broader content that they may or may not be willing or able to teach.

So, we’ll have unit content which is the minimum that needs to be covered to address all of the learning outcomes and assessment criteria. But you would expect that, beyond that, they do bring in more of the teaching and learning. But again, we don’t measure that and that can vary. […] some learners and some centres might get a better learning experience and cover more than others would, but they would all have had to deliver the absolute minimum that’s required to achieve the assessment outcomes.

Hairdressing_L2

Fewer of the AOs which did not recognise this problem provided explicit inputs in terms of content lists or syllabi, in addition to the LOs and AC. Where they did provide additional content, this tended to be indicative, although some of these AOs did specify mandatory content too, sometimes including range statements.

Implicit content links as implicit curriculum

Most AOs did not explicitly specify the order of teaching the LOs or units, nor any other links between them. However, many of them suggested that implicit content links, or the implied best order to teach units or within-unit content, emanated from the nature of the construct of their qualifications, such as the creative process, or the context in which learning and/or assessment takes place.

We have a syllabus, I believe. It’s an implicit syllabus in the sense of our qualifications all lending themselves to the creative process. The creative process is our syllabus. That’s the underlying learning that anybody delivering our qualifications has to cover, it’s about the creative process. The activities and themes that they’re setting for the students to engage with in order to explore the elements of the creative process are where the centres’ autonomy comes in.

Creative_L3

[…] sector-based content tends to build on each other. You don’t have content which is completely standalone and not feeding into anything else. If you’re thinking about business, if you’re thinking about creating a marketing plan, that’s got to be in context with the size of the business, their purpose, whether they’re a sole trader or not, so all of that learning comes into play. So, while it’s probably not obvious in terms of visually in the specification, we have got this unit and that unit separately, […] the content does work together because of the nature of the content.

Business_L3

Some AOs suggested that the construct of their qualifications, such as, for instance, in Creative_L2, is mostly related to acquiring relatively distinct, discrete skills, like building up a “toolkit”, but demonstrating in assessment that these can be applied in holistic situations with inherent structure or logic. Therefore, there was no need to specify clear progression in learning or a specific order for teaching when outlining the LO specifications in these qualifications.

Others, however, observed that there is often a natural or implicit progression in the complexity of some of the tasks or skills that a student might encounter. They suggested that this progression reflects a “natural learning curve” and, thus, can be considered a “common sense” progression, rather than something that needs explicit elaboration in qualification specifications to inform teaching approaches. One AO spoke about the way the structure of its Creative_L3 qualification, which includes 2 “developmental” units and a final, synoptic unit, which also determines the overall qualification grade, give a clear developmental trajectory for teaching and learning. The content from units 1 and 2 feeds into and is “pulled together” in the final unit, with the overall structure resulting in a “cohesive learning experience”.

[…] whether you want to call it best practice or […] common sense, you know. Someone’s got to have experience of installing normal window and doors before you move them onto sliding folding doors and things like that and bay windows. […] for me it’s just that natural learning curve, isn’t it. […] I think there is an accepted common-sense approach that has served the industry well.

Fenestration_L2

The assessment outcomes [in unit 3] really pull together a lot of that information from units 1 and 2. […] So there is a clear follow-through and narrative, which is about that transparency for the learner. […] Obviously, they need to understand the language, the destination that they’re journeying to, but they understand the relationship between units 1 and 2 and unit 3, because they understand that the learning that they are doing within those 2 formative units will be applied, but in a more autonomous and self-directed fashion for unit 3.

Creative_L3

Some AOs also discussed the benefits and drawbacks of implicit alignment between teaching and assessment, where assessed curriculum provides insights into taught curriculum. For instance, there was a suggestion that the nature of assessment, as apparent in assessment brief templates or examples, can provide insights for centres about the nature and complexity of the construct that will be assessed and, therefore, the nature of the skills that they are meant to teach. However, this might also lead to the content that is assessed being overly driven by what is assessed, potentially limiting the breadth of the curriculum.

In the Business_L3 qualification, explicit effort was made to reduce such potential negative assessment washback into teaching and learning. This was done by, on the one hand, providing a syllabus separate to the specification of the LOs and AC, and on the other, by blurring the mapping between AC and the syllabus to some extent by writing relatively broad AC. This AO also noted that the alignment between assessment requirements and the content of teaching was still preserved and captured in their grading criteria. The grading criteria imply and “pull together” the range of content and skills that need to be taught to enable students to achieve those criteria in their assignments in a holistic way.

[…] the idea is that holistically they would need to have learned that content to be able to approach that assessment. While you can’t do a one-to-one mapping from the grading criteria back into the content, from a learning point of view they would need to have that bucket of learning […] so it would be hard for them to say I’m only going to teach this bit, because that’s the bit that you are assessing, because the grading criteria pulls together quite a lot of learning to be able to say, can you apply that in this context?

Business_L3

Occupational or professional expertise and communities of practice

For teachers to be able to recognise these implicit links, including those that emanate from the nature of assessment or the grading criteria, they need to possess occupational or professional expertise. The AOs thought that, in addition to content specifications and other support tools, it was equally, if not more, important for centre staff to have specific occupational and professional expertise to deliver and expand on the content and produce coherent schemes of work or programmes of learning. For instance, it was suggested that, even where qualification specifications did not specify the order in which units or LOs should be taught, sector experts would know that certain aspects of content cannot be taught unless some other content had been introduced. There was no difference in this sense between the group of AOs that recognised the incoherent teaching problem and those that did not. Community of practice in relation to pedagogy around a long-standing qualification also had the potential to bridge a gap between what is in the qualification specification and how best to deliver it to students.

[as] the starting point, these units, these qualifications can’t be delivered by non-specialist. That’s simply not an option. So, if we talk, if we sound like we’re saying that this should be obvious to the teachers, it really should because they are practitioners in this so they will have gone through the qualifications, and they’ll obviously be qualified at degree level, most likely in the performing arts. Quite often in FE, they are also working in the industry as well as teaching […].

Creative_L2

Some AOs pointed out that there was a limit to what can be specified by way of content by the AO, and that this needed to be translated into programmes of learning by teaching specialists. One AO suggested that providing a level of support to centres was necessary whether or not the qualification was based on LOs or a detailed syllabus. This recognises the challenge and importance of alignment between curricula and pedagogy, irrespective of the specific way in which the curriculum may be specified.

[…] there have certainly been examples recently with very extensive sort of syllabus style content where providers have had the challenge of trying to get to grips with that and teach it in the appropriate way, you know, even with many hundreds of pages of syllabus type content. So maybe it’s more about the nature of the support that’s given for any particular and especially new qualification rather than something that’s inherent in the content itself.

Teaching support_L2

Support, guidance and quality assurance

Most AOs thought that their direct impact on the level of occupational or professional expertise of centre staff was limited. However, they suggested that they still had some indirect control over it through QA processes, including checking for centre staff expertise as part of their centre approvals process, which helped mitigate potential issues caused by inadequate teacher expertise.

While several AOs also reported seeking feedback from students and centres about student learning experiences in the context of their qualifications, the AO delivering the Construction_L5 qualification, which needed to support progression within HE, emphasised this aspect of QA more strongly. They explained that their qualification must align with competitor qualifications in terms of not only level and content, but also other quality standards, including the quality of student learning experience. This seemed to drive the AO to put more explicit effort into quality assuring those aspects. They aligned their quality assurance processes with the HE Quality Code as well as complaints procedures, incorporating these types of quality checks more explicitly within their EQA.

More generally, support and guidance for centres in relation to teaching and learning featured more prominently than EQA monitoring processes in this area. Most AOs mentioned mitigations related to providing support and guidance to centres on how to teach their qualifications or how to avoid negative washback of assessment into teaching, irrespective of whether they recognised the problem of incoherent teaching. In general, support and guidance were seen as integral to the AO qualification offer, with AOs being responsive to centre requests for support.

The AOs used a mix of approaches for communicating or implementing support and guidance for centres. Several of them provided guidance documents, while AOs delivering the First aid_L3 and Construction_L5 qualifications offered explicit training or CPD for centres. Such training, which, in some cases, included optional video training content, covered teaching styles and approaches to be used for delivering the qualifications. All the AOs also involved their EQA staff in providing regular or on‑request support to centres. Some of them thought that a particular advantage of the CASLO approach was the opportunity to provide centres with regular support and advice, as well as early and timely intervention, if needed. Continuous support and regular supportive surveillance allowed the AOs to notice issues with a failing centre promptly and was seen as particularly protective against poor teaching practices.

[…] it’s almost like questioning our CASLO quals, do they allow for poor teaching, well everything allows for poor teaching. […] With CASLO quals, I think you’re more likely to pick it up because we’re looking through the year and it isn’t just that end of year assessment where, all of a sudden, it’s like, hang on, how come we’ve only got three people who have got a C. Here we know through the year if something’s happening. So, I think, with the CASLO it’s actually protecting against bad teaching […]

Creative_L2

Support and guidance for centres seemed to focus on 2 somewhat separate areas within or across the AOs. Some AOs provided tentative guidance about aspects of qualification delivery, sometimes involving exemplar schemes of work. These might include a suggested order and approach to teaching the units (for instance, teaching “long and thin” versus discretely) or example timescales which would be more likely to allow for sufficient skill acquisition ahead of summative assessment. This was sometimes done in recognition that centres may not have sufficient time or resource to develop schemes of work from scratch. On other occasions, these materials were provided in response to centre demands for additional resources, which was suggested by some AOs to ebb and flow to some extent, with more demand for resources from centres nowadays than in the past.

[…] the centre has to come up with their own schemes of work and they have to think about how they’re going to make it relevant to their centre […] that’s a barrier for some centres and teachers because teachers aren’t people who have loads of spare time but that’s why we do have support in the way of providing exemplar schemes of work and having support from an EQA to talk through what they’re thinking about doing.

Creative_L2

Other areas of guidance focused on certain aspects of pedagogy. These included, for instance:

promoting holistic, project-based delivery or structuring teaching and/or assessment around relevant workplace-based practices
moving centres away from “chalk and talk” approaches to delivery, deemed potentially inappropriate for some vocational qualifications
suggesting teaching discrete units focused on related knowledge and practical skills in tandem, or
suggesting ways for students to recap and consolidate learning.

The quotes below illustrate some of these aspects of pedagogy guidance.

[…] we’re always thinking about guidance on how to teach the qualification, not least because we work in so many different countries around the world […] and […] sometimes you’re on a journey to get teachers to think in a more practical and applied way and move away from more chalk and talk type models of teaching and learning. […] we have elements of our training packs that focus on teaching and learning styles and approaches […]. We talk about teaching and learning in the spec, we talk about teaching and learning in delivery materials that we produce, so we definitely want the teaching and learning approach to be appropriate as we see it for the sector and for the type of qualification […]

Construction_L5

[…Unit] 1 is a theoretical based unit, so it’s focusing in on the skills around research, exploring a theme, planning, etc. And then you’ve got a unit 2 which is a more practical unit, so it’s about materials, techniques and processes. Now the expectation is that a centre will deliver those 2 units in tandem, so a student will explore the thematic enquiry of project, you know, develop those research skills, methods and methodologies, and then they will apply those and explore them through practical activities.

Creative_L3

A few AOs suggested that direct collaboration between the AO and centres to incorporate an existing teaching approach into qualification design can help bridge the potential gap between what is in the qualification specification and how best to deliver it to students. In the First aid_L3 qualification, the order in which the qualification content was specified was explicitly mapped onto the order that it would normally be taught in a specific pedagogical approach involving “progressive teaching” and “deep active learning”. However, this AO did not enforce this teaching approach and thought that teaching methods were ultimately the prerogative of the centres.

The AOs with qualifications primarily assessed in the workplace appeared to provide least explicit guidance in relation to schemes of work or pedagogy. They saw their role mostly in providing guidance for assessors and enabling them to carry out appropriately holistic assessment to avoid creating negative washback into workplace learning, which could affect its implicit coherence. These AOs thought that assessors should be trained and empowered to assess in holistic ways, making the best use of naturally occurring events to accumulate evidence about student competence, rather than following atomistic specifications or any prescribed schemes of work. To support this, these AOs provided training for centres on how to deliver qualifications and assessment in holistic ways and ways that are appropriate for individual students.

[if assessors] just go unit by unit by unit, […] not assessing holistically, they’re not engaging with the learner, they’re not talking to the learner about which bits they can cover, and which bits they’ve already done. […] that’s where you get things not being taught in order or in a natural course of events. […] So, it’s about having that skill of looking at what the learner needs to start with and then wrapping the assessment and teach around it, not having a standardised one size fits all.

Chef_L2

Despite their primary focus on assessment, some of these AOs, when prompted, also noted that a more explicit alignment between the AOs and other actors in the teaching and learning process was needed for students to receive sufficient support. Misalignment was seen to occur in contexts where employers might not support the student sufficiently, treating them as “part of their headcount”, or where assessors did not assess efficiently or engaged in ineffective communication and planning with employers or students. Developing positive relationships between AOs and employers was also mentioned as valuable, so that employers could see the benefits of teaching and learning not just for students but for their businesses too.

[…] after a while, the employer comes to rely on that person as part of their headcount and part of their productivity. But it’s got to be a 3 if not 4-way partnership, hasn’t it, between the provider, the employer, the learner, obviously we play a part in that as well. So [learners] get mixed levels of support from an employer, I guess.

Fenestration_L2

Few AOs spoke about providing specific learning resources for students. One AO emphasised the need to support employers in providing training facilities, especially where qualifications are delivered in college settings, where these are less likely to be available. A couple of AOs said that they provided supplementary reading or optional video training content.

Question 5

Lack of holistic learning

Accepted Answer

Because CASLO qualifications represent LOs one by one – and without overtly representing how those LOs relate to each other – some critics say that students fail to learn holistically. This risks learning being neither systematic, nor integrated, nor coordinated, leaving learners unable to apply their learning effectively.

While this problem was recognised as potentially relevant for 4 of the exemplar qualifications (3 of which were ‘dual purpose’), this was not the case for the majority of our sample. Irrespective of whether the potential problem was recognised, the AOs suggested a range of mitigations and protective factors that they thought helped to reduce potential issues. These mostly revolved around using holistic assessment (that is, tasks or events integrated across multiple AC, LOs or units), as well as relying on the occupational or professional expertise of teachers and their ability to exploit implicit links across LOs or units. Some AOs also mentioned implementing certain design features in their qualifications which promote holistic approaches to teaching, learning and assessment. Support, guidance and QA directed at promoting and monitoring for holistic approaches were also mentioned by a few AOs.

The mitigations proposed for this potential problem largely mirrored those proposed in relation to atomistic assessor judgement, perhaps reflecting the implicit view of many AOs in our sample that there is a strong interaction between assessment and teaching and learning in CASLO qualifications. When discussing mitigations for a lack of holistic learning, AOs repeatedly conveyed their views that addressing the ways in which CASLO qualifications were assessed would influence how they were taught and how the content was learnt. Therefore, even though the AOs largely did not explicitly recognise the problem of lack of holistic learning as arising from atomistic specification of LOs directly, the potentially negative impact of atomistic assessment seemed to be more recognisable to them.

Holistic aspects, implicit content links and contextualisation of assessment and delivery

Most AOs discussed holistic assessment approaches which utilise or reflect contextualised or real‑life work situations as key mitigations for this problem. This was argued to give assessment events or tasks implicit synopticity, requiring integration of learning as a matter of course to complete the task. Learning and assessment in realistic contexts were thought to place students in situations where successful execution of the task would implicitly represent proof of holistic competence across all task features. It was also suggested that this required assessors to make an effort to assess holistically, across AC, LOs or units, making the best use of already holistic situations while minimising negative washback effects on learning. A similar mitigation was mentioned in relation to the potential problem of incoherent teaching.

This was particularly emphasised in qualifications such as Skin peel_L4, Hairdressing_L2 or Fenestration_L2, which involve job roles that provide a service from start to finish, with an outcome of satisfactory quality. Here, expertise must be applied holistically, but also flexibly, given that each client receiving the service will be different.

It’s very difficult to do it in separate pieces because you’ve got a whole person that wants a whole service done and so it would be impossible to sort of, you know, do any of the steps in a different order or not bring them all together because it’s all about the total look, the haircut, the style, if there’s anything off or not done properly they all relate to each other, the client’s texture of hair, it’s sort of implicitly integrated by the nature of the context because you’re dealing with a person and their preferences […].

Hairdressing_L2

AOs in other occupational areas which do not involve tasks with a natural start and endpoint also argued that assessment that focuses on skill application within a wider task or project mitigates the risk of atomistic, non-integrated learning.

[…] so you can’t show that you can plan a lesson or work with children on a specific task without being an effective communicator and respecting the expertise of the class teacher and so on, so it brings together all of those components […] [to pass it] you’re having to have an impact and not just show up with a planned activity but implement it and interact and know what to do when the child or young person is refusing to participate. All of those things that you need to do, I think, cuts through that level of isolated achievement, I just don’t think it’s possible in this context.

Teaching support_L2

Contextualised holistic assessment, sometimes in the form of project-based assessment covering several LOs or sometimes units, was also used in primarily college-based qualifications (Construction_L5, Creative_L3, Business_L3) to help both promote integrated learning and elicit holistic application of knowledge and skills. In this context, an additional challenge that was recognised by the AOs (already mentioned in relation to the potential problem of poorly conceived assessment tasks or events) was how assessments might be designed to holistically cover various aspects of knowledge and skill across LOs or units to promote holistic learning while still referencing individual AC. This, in the words of one AO, “does take skill” on the part of centres.

In general, AO discussions mostly revolved around ensuring sufficiently holistic assessment, rather than specific holistic teaching and learning strategies. This implied that there were expectations of appropriately holistic assessment having positive washback on learning. A few AOs, nevertheless, specifically discussed holistic delivery (teaching) as helpful in mitigating the potential problem of a lack of holistic learning. Some AOs saw this as a hallmark of good teaching practice. However, one AO pointed out that in lower-level qualifications, for instance at Level 2, it may be inappropriate to implement teaching across multiple units to promote understanding of the connections between them, as that would be too demanding for that level.

It’s about making sure that when we’re monitoring centres that they’re approaching the delivery in the way of projects, rather than individual criteria, and not standalone learning outcomes which don’t provide context to the learner. So, it’s about that replicating, emulating what the industry does, in that you wouldn’t do research on its own for no purpose. So, you’re contextualising the learning outcome within the overarching purpose of a whole project, and then your research informs that project, rather than today we’re doing research, tomorrow we’re doing evaluation.

Creative_L3

[Referring to the L2 version of the qualification] I think, as well, we have to remember it’s a level 2, I don’t think at a level 2, you would necessarily be teaching 5 units all at the same time, so the learners understand the connections between them. That feels like you’d be way beyond level 2 at that stage.

Adult care_L3

Again, it was implicit in many comments that qualification specifications broken down into AC and LOs were to some extent used as an aide memoire or a tool to lay out the important aspects of knowledge and skills to ensure higher consistency and to avoid missing certain aspects of content, rather than there being an expectation that teaching and learning should proceed in a list-wise fashion. There was a shared expectation among the AOs that teachers, owing to their expertise and experience, can see implicit links across the atomised specifications that would enable more holistic approaches to teaching and assessment.

Qualification and assessment design features

Some AOs emphasised that their qualifications explicitly incorporated design features that encourage holistic delivery and assessment. These were aspects such as the “plan-do-review” unit structure of the Creative_L2 qualification that was repeated across different units, despite their different context and focus. This was meant to instil a general and transferrable reflective approach in students towards their creative practice. In the Housing_L5 qualification, a final reflective “professional practice skills” unit was used to promote holistic learning, while the LOs were intentionally grouped in specific ways within preceding units to make the implicit links among them more overt.

Some qualifications, namely Creative_L3 and Business_L3, included explicitly synoptic units. These AOs argued that this mitigates the risks of atomistic learning because assessment in these units requires purposeful knowledge and skill integration in the context of a broader task to address a range of relevant criteria or content across the qualification. However, some AOs that did not include synoptic units expressed certain reservations regarding the feasibility and potential effectiveness of these their qualifications. Some suggested that overarching synoptic units might not work in qualifications that cover a wide range of topics due to potentially high complexity of the construct and the difficulty of defining it sufficiently comprehensively for assessment. Also, holistic assessment across multiple units was perceived by some AOs to increase assessment burden for the student.

One of the things that this qualification doesn’t have is an overarching synoptic type assessment, so one might associate that type of assessment with a more holistic approach, the learner has to think about what they’ve done across the qualification, but you could make the argument that it would be hard to define something of that nature for a qualification that […] covers the range of things that this qualification covers.

Construction_L5

Occupational or professional expertise and experience

AOs again emphasised the importance of teacher and assessor occupational expertise to be able to teach and assess holistically. Several AOs suggested that teaching experience is required for more holistic teaching, giving teachers confidence to teach across and beyond atomistic specifications. One AO suggested that the pressure to make sure that students achieve the qualification might incentivise teachers to deliver the qualification more atomistically, LO by LO. This echoed one of the drivers for atomistic assessment or judgement discussed earlier.

So, I think a confident teacher would probably deliver it more holistically and I think a less confident teacher might do it more systematically. As I say, I don’t think this qualification necessarily produces that teaching, but I can see how it would. And again, when people are under pressure to get learners achieved, it feels safer to almost deliver it learning outcome by learning outcome and then provide the evidence of attainment as they go through.

Adult care_L3

Support, guidance and quality assurance

As in other areas, several AOs discussed the importance of support, guidance and QA to ensure that centres are delivering holistic teaching and assessment. Most AOs said that continuous support is typically provided by their EQAs during their regular interactions with centres, where they might provide examples of scenarios describing how holistic delivery can be achieved. AOs also said that they encouraged centres to reach out to EQAs to ask for advice and guidance in relation to this. Some AOs also provided specific guidance documents explaining holistic delivery and assessment design. One AO noted the need to orientate centres towards more holistic delivery if their previous experience of other qualifications involved more atomistic delivery approaches.

An aspect of EQA monitoring in some AOs includes checking the appropriateness of assessment briefs that centres produce in terms of whether they will enable students to meet a range of LOs within one holistic assessment brief. These AOs also check that qualification delivery more generally revolves around contextualised project‑based teaching, learning and assessment. However, despite providing support for centres, as well as conducting some monitoring of holistic approaches, the AOs again pointed out that their impact on how qualifications are delivered in centres is to some extent limited. There seemed to be a broad agreement that attempts to raise the level of prescriptiveness might jeopardise the flexibility that is highly valued in CASLO qualifications.

You can only recommend what is the best way to deliver a qualification and people will decide […] whether they take your advice or not. So, I do think that’s quite difficult, although you are telling people it can be assessed holistically, some people will pick it up and they will, I guess, go through it, assessment criteria by assessment criteria.

Construction_L1

Question 6

Superficial learning

Accepted Answer

Because CASLO qualifications specify LOs one by one – and because they focus attention on detailed lists of AC that need to be met for each LO – critics have said that this disposes students towards superficial learning. This might involve demonstrating the minimum possible performance on each AC for each LO – then moving on to the next LO – and not revisiting LOs that have already been achieved and therefore not consolidating their learning.

This problem was recognised as potentially relevant for only 2 exemplar qualifications: a ‘dual purpose’ one (Construction_L5) and a ‘confirm competence’ one (End of life care_L3). Several AOs interpreted this potential problem as analogous to the problem of “teaching to the test”, which they saw as a universal assessment washback problem, irrespective of the specific qualification design approach. AOs also thought that having a mix of highly motivated and less than motivated students, with the latter more likely to be prone to superficial learning, was inevitable in most qualifications, whether CASLO or not. Some AOs also noted that what indeed may demotivate students and orientate them towards a superficial approach to learning in some CASLO qualifications was the absence of grading.

All AOs suggested several mitigations and protective factors that they thought helped to reduce the risk of superficial learning arising. These, again, mostly revolved around using holistic, contextualised approaches to assessment and delivery, which increase the sense of relevance of learning and assessment for students. Several AOs also suggested that positive student and teacher attitudes to learning and teaching, including vocational and professional passion, as well as student agency and availability of choice were also helpful. Some mitigating design features were also mentioned, as were support and guidance directed at promoting and monitoring for holistic approaches to teaching and assessment.

Holistic aspects, contextualisation and relevance of assessment and delivery

Most AOs discussed holistic assessment approaches which utilise or reflect contextualised, real-life work situations, as key mitigations for this problem. Several AOs highlighted the opportunities that the use of holistic delivery and assessment creates for knowledge and skills to be revisited and consolidated in the context of different practical tasks and holistic situations. For instance, in construction or hairdressing, even though certain skills may be practised in isolation, their application on different sites or clients, as part of a broader task or service, will often call for these skills to be repeatedly applied, necessarily alongside other skills that each situation might require. Some AOs suggested that the presence of implicit content links across units, particularly in qualifications where content embodies certain broader processes, such as the creative process, help to ensure that students return to and consolidate different areas of knowledge and skills.

Within this, some emphasised that it is the sense of relevance for students in such contextualised assessment or delivery approaches that guards against a superficial approach to learning. Furthermore, the practical nature of the content in several exemplar qualifications, irrespective of whether this was knowledge or skills, helped to reduce the risk of superficial learning that is only aimed at passing the assessment.

[…] they’re adult learners who are in a job, and actually most of the things that they’re learning within the qualifications they are then going to go on and use within their job. So, it’s not something that they’re just learning for the sake of learning and then forget about. Potentially, their assignments could actually be about something, an example that they’ve done in their job.

Housing_L5

This position echoed AO comments in the context of the potential problem of poorly conceived assessments, and their insistence on educating centres to create vocationally relevant and engaging assessments. Assessments that are relevant, involve an appropriate degree of challenge, and that hold some stakes for students to become invested in, were thought to reduce the risk of a superficial approach with the sole aim of meeting the criteria. According to some respondents, such assessments would be less likely to drive students to “work to the learning outcomes” per se. They would work to the assessment or project brief, which would simultaneously lead to coverage of LOs.

[…] it’s a lot to do with the centre not using assessment in the way that’s intended and they’re not maximising the use of assessments. So, if you’re putting on a whole school show, you’re going to get very few learners who don’t put in any effort. I can guarantee that. If you’re asking the learner to just perform in the corner of a classroom with no one else around then, yeah, […] there’s no stakes there, is there? They can just do it in front of the teacher. They don’t care. They’ll just get it over and done with. So, it goes back to that question of setting up the centre to make sure that they’re maximising their delivery and make sure they’re doing it properly and making assessment vocational and worthwhile. If they do that, then it kind of solves its own problem.

Creative_L2

Attitudes

The idea that vocational relevance reduces the risk of superficial learning is closely related to a frequently mentioned protective factor involving student positive attitudes towards their area of learning. In some cases, this involved a sense of vocational passion for their subject, which was deemed protective against wanting to learn solely in order to pass a test or meet an AC. The AOs offering creative practice qualifications also suggested that their subject-matter creates a “natural barrier” against superficial learning because delivering poor creative pieces, even where these might meet the minimum standards of the qualification, would be an unpleasant experience for the students.

I think there’s a deep-rooted passion in the way that learners take [these qualifications] and go forward with them that negates that teach to the test idea, because that isn’t really what people want as an outcome.

Business_L3

You’re not wrong though, you can do a very minimal performance. So, you could do something which is pretty awful, but we don’t find that happens too often because it’s kind of self-punishing because it’s not a very pleasant experience when it comes to performance and creative outputs.

Creative_L2

Some AOs also suggested that professionalism and the vocational passion of teachers, who are passionate about doing the best for their students and not limiting their prospects by teaching to the test, is also helpful in reducing the risks of superficial learning. Indeed, a couple of AOs argued that it is the role of the tutors to understand the broader relevance of qualification content beyond the classroom and assessments, and to “light the fire” under students and explain why it is worth learning that content, especially where students might not be naturally ambitious or fully engaged.

Some suggested a further protective factor related to student agency and choice. As students typically choose certain vocational qualifications because they are engaged by the subjects and find them interesting, superficial learning is believed to be less likely to occur than in the qualifications that are compulsory. Some AOs offering qualifications that support progression to HE suggested that students tend to know what grades they will need to achieve to progress to specific higher education courses and, therefore, are not willing to “settle with just scraping through” as lower grades would not allow them to enter their chosen courses. It was suggested that this attitude enabled students to challenge poor or superficial teaching where it might be present and where teachers may just be aiming to get them to achieve minimal standards to pass rather than achieve higher grades.

However, within this, it was suggested that different levels of engagement among students, both those that naturally strive for excellence and those that are willing to meet only the minimum standard, is common in all types of qualifications, not only in the CASLO ones. In relation to those students that might be less engaged, CASLO qualifications that did not involve grading were sometimes suggested to further limit aspirations and opportunities for stretch and challenge beyond just meeting the minimum standard to pass. This perspective was tempered by views suggesting that despite this potential limitation, the focus on highly relevant content, optimised for certifying certain specific skills, justified the use of a binary pass/fail threshold. Student motivation was bolstered through content relevance in such cases. Relatedly, in qualifications that do involve higher grades, some AOs suggested that striving for just the pass grade in certain circumstances, especially with adult students who might have other priorities, may not in itself be problematic, nor a problem with the CASLO approach per se.

Qualification and assessment design features

In relation to the apparent absence of stretch and challenge through a lack of grading, which might predispose students towards superficial learning, some AOs pointed out that these are single-level qualifications. Therefore, where students are demonstrating performance beyond the level of their current qualifications, there is a possibility of progression to a higher level instead of a higher grade, which might be equally motivating to students.

[…] some people will come into a qualification, and they’re really fired up, they really want to learn, they’ve done the basics, but they’re also producing evidence that is […] of a higher standard or more in-depth. […] They’ve met the threshold. The assessor might be saying – that’s all fantastic evidence. We need to look at how we progress you onto the level 3 because that’s where you can put all that into practice and use that evidence for that.

Fenestration_L2

The notion that certain CASLO qualifications involve only highly relevant, core content, that has been distilled to “get to the crux” of what students need to know and should be able to do was also suggested as a design feature that helps to guard against superficial learning. This was suggested to reduce the burden of assessment and, thus, leave sufficient time for more robust learning of the content that is considered fundamental in the qualification. This was described as being particularly important for specific types of students, such as adult learners, who might be studying part-time alongside a full-time job, family and other commitments.

Some AOs suggested other mitigating features such as pass standards that are not minimal, combined with demanding content that requires a high level of engagement and perseverance to ensure its comprehensive coverage and assessment due to the mastery model. For example, in one creative practice qualification, it was suggested that even minimal engagement by students would require substantial effort to reach the endpoint of a rehearsal process to finally deliver a performance. In some qualifications where different areas have to be evidenced on multiple occasions over time to cover the range, this aspect of the mastery model was said to further mitigate this potential problem. Relatedly, continuous assessment (rather than terminal assessment) was considered to be more protective against “rote learning”, focused on just passing the assessment.

I think it’s the fact that the minimum in those, in many of the cases are not very minimal. You know, so I can demonstrate that I’ve been through a rehearsal process which takes place over 8 weeks, you can have minimal engagement with that, but you’re still engaged for 8 weeks of rehearsals and […] we’d expect to see evidence of the rehearsals and we’d expect to see your plans for each rehearsal, what happened at each rehearsal. And yeah, you can still be a fairly surface level on that. […] but […] they’re having to engage in quite a large way to get to that point.

Creative_L2

One AO suggested that where units can be completed in any order, this can exacerbate the potential problem of superficial learning as it makes it more difficult to utilise certain links between units to revisit and embed knowledge and skills further. Several AOs implicitly agreed with this view, arguing that their exemplar qualifications benefit from “organic”, implicit links between units, which feed into each other. One AO suggested that it was necessary, where possible, for qualifications to be designed to allow “units to integrate”, thus enhancing the implicit links between units and holistic delivery, as previously discussed. The same AO suggested that repeating similar criteria across units can help achieve this aim, and mitigate the potential problem of superficial learning, particularly in larger qualifications with many related components.

Somewhat mixed views were expressed across our respondent sample in relation to the advantages and disadvantages of a high degree of alignment between teaching and assessment. Some AOs suggested that where teaching and assessment work seamlessly together, and where assessment “naturally occurs” as part of the taught programme, there is less focus on the AC as such and less risk of superficial learning. This was suggested despite some of these qualifications having highly explicit and detailed AC (and the mastery principle operating at the AC level, that is, with no compensation permitted across AC).

[…] because we essentially just translated it into a regulated format, it was all broadly understood and the assessment process already existed, so the assessment and the teaching and learning are all interwoven, so that it’s not driven by the assessment process, the assessment is something that naturally occurs as part of the taught programme […] So, for us, it’s not an issue because the course and the qualification was designed for everything just to work seamlessly together […].

First aid_L3

Other AOs recognised that the explicitness of AC, especially in relation to the command verbs used, may encourage superficial learning as students could become unwilling to go beyond what the AC requires. For instance, they might not be motivated to engage in “analysing” some content where the AC might only require them to “describe” it. To mitigate this risk, some AOs strived to disrupt this alignment to some extent, such as in the Business_L3 qualification, by making the mapping between AC and teaching content less direct than in earlier versions of their qualifications, with the aim of focusing attention away from assessment and more onto teaching and learning.

Support and guidance

There were few direct references made to support and guidance as a mitigation for superficial learning. However, because many of the other mitigations just discussed are the same as those discussed earlier in relation to other potential problems – for instance, various holistic aspects of delivery and assessment – it stands to reason that support and guidance provided to implement those mitigations might indirectly address this potential problem, too. In particular, as with several other potential problems, it was suggested that continuous support and multiple touchpoints with centres throughout the delivery of a qualification help to identify and minimise issues that might lead to superficial learning, alongside other teaching and learning problems.

One AO also mentioned the potential value of explaining to tutors the relevance of content that might at first sight appear to be irrelevant, so that they can then pass this on to their own students to enhance engagement.

So, sometimes there are things in the qual which may look superficial to students, but by supporting the tutor and educating the tutor as to the relevance of things, they then bring that to life and can show the students why it’s important to have those skills […].

Construction_L5

Question 7

Undue assessment burden

Accepted Answer

According to criticisms in the literature, the mastery requirement in CASLO qualifications forces students and teachers to spend a great deal of time being assessed and documenting their assessments, resulting in an unduly burdensome assessment process.

Five AOs acknowledged the potential relevance of this problem to their exemplar qualifications (2 of which were ‘dual purpose’). Six AOs did not entirely recognise it and an additional 3 did not recognise it at all (the latter, all being ‘confirm competence’ qualifications). Where the problem was recognised, the AOs mostly agreed that burden tended to fall on centres and tutors more than students, though some believed that this could vary and that it could affect students, too.

Overall, most AOs thought that those involved in assessing or taking these qualifications tended to accept assessment as integral to their experience, and necessary for the qualification approach to achieve its broader purposes. AOs also discussed providing support and guidance to centres to help them optimise their assessment and QA processes, through the use of e-portfolios, holistic approaches to assessment, flexible assessment, and streamlining of the amount of evidence collected. Appropriate planning of delivery by centres was also seen as important in mitigating the potential problems with assessment burden. Continuous support, engagement and feedback from tutors, and flexibility in delivery and providing sufficient guidance to students were also deemed helpful in encouraging students to take ownership of the assessment process rather than seeing it as burdensome.

Attitudes

Positive attitudes of practitioners and students towards the assessment process and its position in the broader context of qualification delivery were the most frequently referenced protective factors reducing the perception of undue assessment burden and its potential negative impacts. Most AOs thought that assessment was accepted as integral to CASLO qualifications by both teachers and students.

Some AOs saw assessment burden as unavoidable irrespective of the qualification approach, and as the price one has to pay for achieving a qualification, which is “something substantial” that “you’ve got to earn”. In relation to the CASLO approach, the AOs thought that a balance needed to be struck between the potential risks from assessment burden and the broader purposes of these qualifications, which called for sufficiently exhaustive assessment.

The AO offering the Construction_L5 qualification emphasised that they did not see evidence of students reporting issues with administrative burden of assessment when conducting their annual student survey. The AO suggested that, rather than being perceived as a burden, the continuous and formative nature of assessment was seen by students as motivating and engaging. It offered multiple opportunities for them to attempt assessment and improve based on feedback, instilling a “sense of achievement” from getting results along the way, and motivating them to keep going. An AO in the creative sector pointed out that formative assessments exist in the creative industry, too, and are an inherent part of carrying out creative projects for customers. Thus, it was natural to incorporate that kind of “iterative” approach into their qualifications. Student agency and choice in pursuing the qualifications that they are interested in were also seen as helpful in reducing a sense of burden.

A lot of these risks are real risks, or they’re potential risks if you don’t identify them and mitigate. […] And it’s reflective as well of the industry. You don’t get a brief from a client to go away for 12 weeks, deliver them a set of graphics that they’ve requested, and they say thanks very much, and that’s it. It’s an iterative process. You know, feedback and assessment exist in the industry, so it exists in the way that we’ve designed the qualification as well.

Creative_L3

Several AOs that recognised assessment in the CASLO approach to be “extensive” nevertheless thought that teachers saw it as fundamentally important to the qualification and ensured that this significant part of teaching was appropriately planned. Tutors were believed to be passionate about assessing student work in several qualifications, including those in the creative, business and beauty sectors, and to take pride in both their assessor role and in seeing the achievement of their students. One AO also suggested that while external assessment might create less burden for teachers, it might increase their anxiety levels.

Assessment and feedback [are] fundamentally important to the success of this qualification. They [teachers] see assessment as a significant part of teaching. It’s not just about delivering the projects in the workshops; assessment is at least 50% of the work that they’re doing. So, it’s planned in I would say, it is extensive, but it’s planned for.

Creative_L3

I think for teachers it is always going to be more work to continually assess rather than just have a final exam, […] but also assessors understand the need for that. And I don’t think an assessor in this subject area would stand for just giving a student an exam at the end.

Skin peel_L4

Teachers, many of them, really take a lot of pride and enjoy that element of seeing the work, seeing the artefacts, seeing the achievement.

Business_L3

Support, guidance and quality assurance

Assessment burden was largely seen as more directly affecting tutors and centres than students, though some AOs thought that this might vary depending on centre practices or student attitudes, with students sometimes experiencing a greater burden. One AO felt that the burden for students increased at higher qualification levels but was low in their level 1 exemplar qualification. Several AOs discussed different support mechanisms that could help centres streamline the assessment process.

Most AOs made references to e-portfolio systems which provide electronic platforms to capture assessment evidence and maintain records. Although these systems have been in relatively wide use for more than 2 decades, some AOs noted that not all centres used them. In some cases, centres lacked the necessary technology, while others preferred at least some aspects of the evidence to be collected on paper (for instance, client signatures in Hairdressing_L2). AO comments suggested that the use of such technology was optional for centres.

But the other thing we’ve done to try and alleviate that is we have a really good e‑portfolio system now where they can scan things in […].

Hairdressing_L2

I don’t think we would specify that a provider must use an e-portfolio for example, but we would certainly allow a provider to use an e-portfolio if they had a platform. I think most providers will and it would work for a qualification like this, I’ve no doubt.

Teaching support_L2

There was a suggestion from some AOs that centres may at times over‑assess or collect and submit large amounts of unnecessary evidence for moderation, owing to nervousness about missing important aspects of AC. This was seen to increase burden for everyone involved in the process, including the moderators who, in such circumstances, were more likely to miss key pieces of evidence. Collecting unnecessarily large volumes of evidence was also sometimes attributed to “weaker assessors”. Several AOs mentioned different forms of training and guidance that they provided to centres to support them with using the e-portfolios, but also, in relation to sufficiency of evidence, assessing holistically via tasks covering multiple AC, LOs or units. Support included example assignment briefs as well as EQA support.

Some AOs saw administrative burden related to various QA processes, including those for their own EQAs, as greater than assessment burden specifically. To address this issue, in their support for centres, some AOs included guidance about efficiently managing centre IQA processes. As part of centre approval and EQA, AOs also check that centres have sufficient staff numbers for the required workload and monitor and support centre assessment planning.

Inputs and supporting learning

Appropriate delivery and assessment planning was seen by several AOs as another important mitigation that helps to reduce the risk of assessment burden for both students and tutors, to ensure that there is sufficient time to collect evidence, as well as to assess and QA it.

Because if you know you’ve got a year, you don’t wait until day 364 to mark it and go, actually we haven’t done that bit, sorry. They should be doing it all the way through planning, and effective planning to ensure that that doesn’t happen. So, again, for me, that comes back to a weakness of its delivery, not the approach itself, poor planning and delivery by whoever is doing it.

Chef_L2

We say to them that they need to be forward thinking in their planning for the year. They need to leave sufficient time for that. They’re required to implement some rigorous IV processes. So, they’ll often have double marking, blind marking, they’ll have benchmarking activities.

Creative_L3

Regular formative feedback and tutor engagement with students were also deemed important because, if centres “get their formative assessments right”, this leads to a “natural” and less burdensome progression through the qualification. Several AOs also emphasised the need for tailored approaches to delivery, making potential time savings by focusing less on areas where students might have prior expertise due to their background, for instance. For some, this included flexibility in the type of evidence that might be collected, which, where accepted by the AO, reduced the burden on centres and students. One AO pointed out that the sense of burden can be alleviated by promoting student choice and autonomy in relation to which evidence was provided and in which format.

For some AOs, the transparency of qualification specifications and how these were captured on their electronic systems was seen as helpful to students to keep track of their progress. One AO suggested that there might be benefits in providing guidance to students and then encouraging them to take more ownership of documenting the assessment process. This was believed to help build student confidence in self‑directed learning and getting themselves organised, instilling important skills that might serve them well in their future careers.

[…] as an AO we’ve tried to overcome that with our i-learner system that learners can log into any time they like. They can see not only the standards they’re working on, […], the shrinking list of things to do, they can view that. They can mail their assessors any time they like to ask for advice. They can submit their evidence whenever they like.

Chef_L2

Holistic aspects and relevance

A holistic or project approach to assessment was mentioned as helpful in optimising the assessment process and, thus, reducing potential assessment burden.

What gets in the way of teaching is start at unit 1, 1.1, show me how to do this. That, and not assessing holistically, is where the time gets taken up. And it’s where either the centre’s staff just are not appropriately equipped really, or their awarding organisation is giving them a great big folder of workbooks they’ve got to go through, and they’re more worried about filling out the workbooks and filling out the funding forms than teaching.

Chef_L2

[…] we do encourage a holistic approach to assessment, so trying not to have lots of tiny assessments, but maybe having some bigger ones and maybe with repetition of the learning outcomes, so if it’s not met in one it could be met in another, so you can see that the student has gained that knowledge.

Construction_L5

A sense of relevance of the assessment process to students’ current or future jobs was also deemed to contribute to a reduced perception of burden. This was flagged in relation to qualifications in the beauty and creative sectors, in particular. For instance, collating portfolios consisting of aftercare leaflets and other relevant paperwork – based on current practices and potentially useful in students’ future practice – was one example of this. Some centres reportedly capitalised on this by “selling” the process of building a portfolio to students as useful and helpful down the line when looking for jobs, thus, making it more motivating. Evidence that is “naturally occurring” in the process of delivering a service to a client was also deemed by some AOs to be less onerous to collect than evidence that needs to be collected through specifically constructed assessment tasks.

For the students, they aren’t really doing much more than they would be doing anyway. And having all of those processes in place and having a really good induction pack for their clients, having really good aftercare leaflets, going through that process and having all of their paperwork in place, I think learners find that very reassuring that they know that they’re doing all the right things and they’ve got everything ready to go in their business.

Skin peel_L4

So, I don’t know, I think a lot of evidence that you would collect for this is naturally occurring so you’re not having to set it up and organise it and document it […]

Construction_L1

Question 8

Demotivation or disengagement

Accepted Answer

Critics have noted that the heavy assessment burden associated with CASLO qualifications can lead to students experiencing demotivation or disengagement from learning. Furthermore, the requirement to achieve every LO because of the mastery model might also be demotivating, particularly when a student begins to fall behind, potentially leading to non-completion.

Four AOs acknowledged the potential relevance of this problem to their exemplar qualifications (one of which was ‘dual purpose’), whereas 2 AOs with ‘confirm competence’ qualifications did not recognise this potential problem at all and an additional 7 did not entirely recognise it. One AO was not directly asked about this potential problem.

The most prevalent mitigations suggested by the AOs included aspects of flexible delivery such as not imposing time constraints on learning, and providing an opportunity to resit assessments and achieve unit-level credit. These aspects were perceived to provide a “safety net” for students and to reduce the likelihood of disengagement due to a potential sense of failure of an entire qualification. Inputs such as regular feedback and clarity about the learning trajectory were also mentioned. The option to contextualise students’ learning and assessment and make it personally relevant was also seen as helpful in keeping the students engaged despite assessment burden. Some AOs also pointed out the value of continuous support and multiple touchpoints with centres, providing AOs with opportunities to detect potential for non-completion early in the delivery. This was in addition to internal centre monitoring of student progress as part of IQA. EQAs were also tasked to encourage centres to use evidence in formats other than written to help reduce assessment burden and increase engagement where writing might present a barrier to some students.

In contrast to the abovementioned criticism, the mastery model was deemed by most AOs to contribute to student engagement rather than cause disengagement from learning. Nevertheless, some AOs introduced certain design features to their qualifications to reduce the chances of student disengagement that might be related to the mastery model.

Inputs and supporting learning

In relation to this potential problem, the most prevalent mitigation type suggested by the AOs involved different aspects of supporting learning and students. Flexibility in delivery was mentioned by several AOs, including no time constraints on learning, multiple assessment and resit opportunities and unit‑level achievement or credit. Some AOs emphasised that flexible delivery with no time constraints protects against dropout arising from challenging life circumstances. As such, students can leave and come back to continue their qualification when they are able to do so, rather than abandon it completely. They emphasised that this sort of flexibility was also less likely to lead to disengagement due to a sense of failure, which some AOs thought was common in other qualification approaches.

I think, actually, when you look at other types of qualifications, with a CASLO, you’re less likely to get dropouts. […] you’ve done 3 units, and you need 4, you’re struggling with the fourth, there’s no reason why you can’t pause that particular unit at that time and then go back to it. […] I think you get that sense of failing with an exam, or a multiple-choice test or whatever, and, actually, that in itself would potentially make the learner think what’s the point, I’ve failed.

Housing_L5

However, resits or resubmissions were limited in some exemplar qualifications, and the maximum grade achievable was then usually capped at pass. One AO where this was the case emphasised the importance of centres evaluating the impact their teaching is having on student outcomes and whether students might be summatively assessed too early.

More generally, there was a view from several AOs that certain aspects of teaching or delivery approaches helped to support learning and combat the potential problem of disengagement. These included transparency in communication and guidance from teachers and assessors, as well as teacher ability to track individual student progress and to motivate students with tailored approaches and individual support.

You could retake, but it would have to be a new assignment, and it would have to be supported by your centre to do that. […] There is a cap I think still on these quals in terms of how many times you can attempt that exam. […] I think, first of all, the centre would really need to be reflecting on whether […] it’s anticipated that learners will pass on the coursework grades. I think the centre at that stage would really need to be assessing whether the assessment was too soon, whether there was enough teaching and learning, whether there was something else at play that needed to be evaluated.

Business_L3

One AO agreed that disengagement arising from not being able to keep up with the work due to longer-term absence was possible in these qualifications. However, there were mechanisms that helped students to catch up, such as extending the course time, special consideration policies and calculated grades for missed units in some cases. The same AO thought that if students were present and engaging with learning, they would tend to progress, and achieve the qualification. Piecemeal achievement throughout the qualification, alongside regular formative feedback, was also deemed to help with engagement, as it provided a “safety net” for students unlike in qualifications with terminal assessment only. Regular formative feedback was considered essential in managing student progress and ensuring that students are clear about their trajectory throughout the qualification, especially where there was a strong mastery requirement across the LOs, where the lowest graded LO would determine the qualification grade. In such instances, regular feedback, and involving students in assessing their own work against the AC alongside their tutors, would ensure that the final result does not come as a surprise and, thus, negatively affect student engagement.

I think disengagement or falling behind for us equates to a longer-term absence I think more than it does anything else. If they’re present, they tend to, they’ll make the progress, I think.

Creative_L2

[…] actually they [qualifications] can be motivating in terms of achievement all the way through. […] I think these do allow for certain students to know and be safe in the knowledge that they are achieving as they go along and it’s not all going to be done at the end and it’s not going to be a massive disappointment […].

Construction_L1

There were somewhat mixed views among a few AOs about the impact of workbooks that students might be asked to complete throughout their qualification. One AO thought that centres recognised that these can negatively impact on student engagement and looked for alternative methods to assess them and collate evidence. Another AO thought that providing students with (non-compulsory) workbooks at level 1 can offer structure and support in tracking one’s own progress. E-portfolios were again suggested as more engaging as tools for collecting evidence.

I think there was a trend a few years ago for training providers to rely heavily on workbooks, especially for lower level qualifications, but I think training providers are recognising that it does cause disengagement, so I think we’re seeing a move away from that and more innovative ways of assessing people and online learning has also kind of helped because it meant that you can make that much more engaging.

End of life care_L2

Qualification and assessment design features

As noted in the section on the benefits of the CASLO approach, several AOs thought that the mastery model motivates students to be engaged and keep up a good standard of work throughout the qualification, because they know that they cannot allow themselves to perform poorly on some outcomes as this will affect the entire qualification result. However, several AOs recognised the potential demotivating effect of students getting lower grades in units that they take early in their programme, which they may not be able to compensate for later on as their competence develops.

For this reason, and to allow students room to develop their knowledge and skills sufficiently before attempting assessment that would determine their qualification grade, one AO implemented “summative grading” that derives the overall qualification grade from the final (synoptic) unit that is taken towards the end of the course. The aggregation model in this unit, however, involved no compensation, meaning that the qualification grade would be determined by the lowest graded LO in that unit. The other 2 units of the qualification were seen as “formative” – students were required to pass them before attempting the final unit, but they were not graded and did not contribute to the overall qualification grade. These units were deemed to allow space for students to experiment, act on formative feedback, learn from their mistakes and develop, given a less pressured aggregation model which guards against disengagement.

I think if this approach was taken for the formative units before they got to this summative one, it might have an impact on student engagement, because they would have thought, you know, if I get a pass here, there’s no way I’m going to get a merit at the end of the qualification. But this happens at the end point, you know, it’s a high stakes assessment. So, it’s in their best interest to be engaged throughout the entire qualification, because they know that if they don’t, they’re not going to get the grade that they want.

Creative_L3

Other AOs incorporated other aggregatory principles in deriving higher grades for their qualifications (beyond the passing grade)^{[footnote 3]}. For instance, some AOs applied a “charity” aggregation principle when determining the overall qualification grade, where the overall result represents (or tends towards) the highest level of proficiency across the domain. One AO justified the use of that aggregation approach by the desire to match the approach adopted within comparable university level degrees rather than operate a “more difficult” model. These mitigations were particularly intended to address possible disengagement among students that received lower grades in units taken early in their programme and who, without some degree of charity in the model, would be unable to access higher overall qualification grades.

[…] the charitable approach. There is an element of that as well when it does land on a true medium. It gives the learners the benefit of the doubt. […] So, that comes when you’re calculating the overall grades for the learning outcomes. […] So, it’s […] unclassified, pass, merit, distinction for each of the assessment criteria and then you calculate an average of those criteria to form your learning outcome grade.

Creative_L2

[…] that’s a broadly compensatory, slight compensation, charity, if you like, model, but the top line on that is that we’ve looked at comparable grading practice in other HE qualifications […]. We didn’t want to make [this qualification’s] overall grading model more difficult than what you see in a comparable university level degree, which really is the comparable qual in this example.

Construction_L5

The AO offering the Hairdressing_L2 qualification expressed some concerns regarding time constraints in its college-based exemplar qualification. It explained that, given the length of time required for students to master the practical skills to a safe degree, this core content had to be prioritised over some other, more peripheral, areas, such as communication or commercial skills, for instance. Thus, on the one hand, the burden for students was reduced, and they were provided sufficient time to master the core skills rather than risking disengagement and dropout. On the other hand, this required managing employer expectations regarding the amount of additional training on the more peripheral skills that might be required. This AO did not think that any dropouts were related to assessment or administrative burden in their exemplar qualification.

I think the NVQ model certainly was outcomes-based and achievement-based but not necessarily […] time-based. And anything that eats into that time is a big concern for employers certainly. It’s just the type of skill and qualification that you do need time to sort of practice. So yeah, it’s a difficult challenge, you know, and that’s where I think these qualifications and classroom-based college learning will never really stand up against the apprenticeships where they’re sort of learning on the job, and they have the time on those skills.

Hairdressing_L2

Contextualisation and relevance

The option to contextualise learning and assessment and make it personally relevant was seen as helpful in keeping students engaged and countering the possible sense of burden arising from the amount of assessment. One AO suggested that in qualifications where some content was there for apparently no good reason, students’ sense of relevance might be weaker and sense of burden stronger. This then required stronger engagement and creativity from tutors to ensure that students do achieve those LOs, too. The sense of relevance and engagement was also deemed to increase when assessment was more practical.

I think some of the things we find with these qualifications is that there’s quite a lot of room for the learners to follow their own intrigue, which is […] a great source of engagement. […] you could be looking at […] performers you’re interested in, or you might be directed to an array of different other performers that you’ve not seen before and are interesting to you. So that’s good. It’s very personal because […] it’s their dance, their acting so it’s a very personal thing to them so that feels quite engaging and it’s very different from what they’d be doing elsewhere […].

Creative_L2

Support, guidance and quality assurance

Some AOs pointed out the value of continuous support and multiple touchpoints with centres, providing AOs with opportunities to detect potential for non-completion early in the delivery. Their EQAs undertake “formative sampling” of student work during qualification delivery and can also advise centres on best practice or note any issues with the nature of the evidence being collated, potentially helping to alleviate unnecessary assessment burden. Some AOs mentioned that student progress is tracked through a regular contact between the assessor and student and if a student has not progressed in a certain amount of time, this it is flagged to the IQA and dealt with.

It’s useful having that constant relationship with the centres, isn’t it, and what they’re doing and knowing about their delivery and then having, having their assessments running, unit by unit throughout the year, so you do have that build‑up of evidence along the way. So, any other eventualities you can see what’s going on but also you can catch things early as well when you’re doing your quality assurance work.

Creative_L2

So, part of the reason we provide our e-portfolio system to our centres is so that we can see assessments, we can see progress. We can see the evidence generating. So, our EQAs can do formative sampling. And then as we were talking about earlier, then guide the IQAs.

Chef_L2

Some AOs, through their EQAs, encouraged centres to use alternative evidence collection methods such as making videos, or conducting a professional discussion, or questioning, instead of insisting on students producing written evidence across the board. The AOs perceived this to help to alleviate assessment burden for both students and assessors, and to help promote engagement, particularly at lower qualification levels, where writing might present a barrier for some students. Some AOs also noted that, while they advocated for holistic assessment, there were benefits in breaking things down for some students who might feel disengaged when faced with a requirement to complete a large written portfolio.

I think we try to get people to think creatively about how they capture evidence. My experience has been that a lot of people think they have to write it down in order for it to be valid, well that’s not the case. And certainly Level 1 learners, they might not be able to write it down […], but they can probably explain it to you. […] So actually, there are lots and lots of other ways of collecting evidence, everyone’s got mobile phones, the kids love making little videos of themselves, we need to encourage the use of technology more, it’s something we try to do.

Construction_L1

I can see the sort of negatives of a huge written portfolio right, fill that in for me. It’s going to turn some students off. So, whilst we encourage holistic, you know, you can still break holistic down into bite size. And you can still use alternative methods, you know, it doesn’t have to be a load of written work. […] In a half hour discussion tease bits of out that candidate.

Fenestration_L2

Some AOs also monitor completion rates and would review the qualification if these were not aligned with their expectations. Similarly, grade profiles are monitored for consistency over time in those AOs that have graded qualifications, helping to detect potential problems with achievement and completion.

The total number of AOs against some problems differs because not all AOs were explicitly asked about some problems. ↩
In the analysis, we grouped several mitigations under the mitigation type we call ‘inputs’. These included aspects such as syllabi or content lists, pedagogy guidance, schemes of work, learning resources, as well as sufficient or appropriate teaching. ↩
See Newton (2018) for an analysis of common aggregatory principles used in VTQs. The following 4 were observed in that study: mastery (where overall result represents (or tends towards) the lowest level of proficiency across a specified domain, or subdomain); compensation (where overall result represents an average level of proficiency across specified domain, or subdomain); configuration (where overall result represents a particular pattern, or configuration, of proficiencies across a specified domain, or subdomain); charity (where overall result represents (or tends towards) the highest level of proficiency across a specified domain, or subdomain). ↩

Cookies on GOV.UK

Applies to England

Local or personal irrelevance and lack of currency

Contextualisation and holistic aspects

Qualification and assessment design processes and features

Support and guidance, QA and CPD

Supporting students and learning

Content hard to pin down gets missed

Qualification and assessment design processes and features

Implicit content, contextualisation and holistic aspects

Occupational or professional expertise and attitudes

Downward pressure on standards

Qualification and assessment design features and processes

Inputs and supporting learning[footnote 2]

Attitudes

Incoherent teaching programmes

Flexible delivery and contextualisation imperative

Inputs: content lists or syllabi

Implicit content links as implicit curriculum

Occupational or professional expertise and communities of practice

Support, guidance and quality assurance

Lack of holistic learning

Holistic aspects, implicit content links and contextualisation of assessment and delivery

Qualification and assessment design features

Occupational or professional expertise and experience

Support, guidance and quality assurance

Superficial learning

Holistic aspects, contextualisation and relevance of assessment and delivery

Attitudes

Qualification and assessment design features

Support and guidance

Undue assessment burden

Attitudes

Support, guidance and quality assurance

Inputs and supporting learning

Holistic aspects and relevance

Demotivation or disengagement

Inputs and supporting learning

Qualification and assessment design features

Contextualisation and relevance

Support, guidance and quality assurance

Is this page useful?

Help us improve GOV.UK

Help us improve GOV.UK

Inputs and supporting learning^{[footnote 2]}