Responding to Criticisms of the CASLO Approach (Report B): AO views of potential assessment problems

Question 1

Inaccurate judgements and standardisation

Accepted Answer

The literature identifies problems related to assessors making inaccurate or inconsistent judgements about whether or not students meet the relevant AC in CASLO qualifications, meaning that some students pass when they should not, and some students do not pass when they should. According to the literature, that happens because AC are very hard to write and interpret precisely. Therefore, AC alone cannot communicate the threshold between passing and not passing, or between different grades, precisely enough. This can potentially present significant problems for the CASLO approach because assessors need to make heavy use of these criteria when judging (and grading) student performances directly.

Of the 14 qualifications in our sample, this problem was recognised as potentially relevant for 12. Two AOs thought it was not entirely relevant for their exemplar qualifications (both with the ‘confirm competence’ purpose), though they still discussed a range of mitigations and protective factors that supported the interpretation of AC.

None of the AOs relied on AC alone to communicate the threshold between passing and not passing or between different grades. They discussed multiple mitigations, across at least 4 different types in each case, referring to them as a “package” which mitigates potential problems with unclear AC. These were often discussed with reference to resource challenges and the need to ensure a balance between prescriptiveness and flexibility or contextualisation in qualification delivery. Resource challenges, in particular, appeared to permeate and, to some extent, shape the way that the mitigations were put in place.

The following quote describes some of the typical layers of quality assurance (QA)^{[footnote 2]}, support and guidance, as well as some other mitigations, such as standardisation and occupational expertise of practitioners that, as a package, help to address this potential problem:

I think what the criticism doesn’t account for […] is the quality assurance that sort of wraps around that and the standardisation practice. […] So I think it’s highly unlikely that, if the sampling strategy is working well, the assessors are all sort of qualified and have those standardisation activities and training and planning, and the same as our EQAs have constant standardisation and they’re all qualified in that process, I think it’s highly unlikely that learners would get through an assessor, an IQA and an EQA without having that consistent accuracy. I mean, it is going to be picked up across that. […] So, it basically does rely on industry codes of practice, guidance material, training from the awarding organisation, internally at the centre, engagement, there’s a whole machine outside of it, because if that wasn’t there, it’s hugely open to different interpretation and inaccuracies.

Hairdressing_L2

All AOs flagged QA as a key mitigation, followed by support and guidance, which was mentioned by all except one AO. The number of references to different aspects of these 2 mitigation types was higher than the number of references to all the other mitigations combined. Of other mitigation types for this potential problem, the most referenced were occupational or professional expertise, communities of practice and qualification or assessment design features and processes.

Contextualisation and holistic assessment

The challenge of finding the balance between ensuring sufficient specificity of the AC to support consistent judgement and allowing for their sufficient breadth to enable flexibility and contextualisation of assessment underlined most of the discussions on this potential problem. Evidently, the broader the AC, the more open to interpretation they might be. The AOs described different ways of mitigating this risk while striving to write broader AC^{[footnote 3]} to allow for sufficient contextualisation. Different AO views on where this balance should tilt depending on the specific context in which assessment takes place probably explain some of the diversity of approaches that the AOs described for mitigating this potential problem. For instance, the AOs took distinct stands on providing support and guidance to centres and appeared to have varied degrees of reliance on occupational expertise and judgement of assessors.

[…] it’s very easy for assessment criteria to become too specific, so specific that they effectively become tasks or restricts centres’ abilities to use them effectively. If they’re too prescriptive, you end up [with] very constricted limited opportunities to evidence across centres, and you get a lot of the same stuff year-on-year and across centres, and that brings with it its own risk, of course. So, we’re very mindful to ensure that the assessment criteria are fit for purpose but don’t restrict innovation and variation within centres.

Creative_L3

Simultaneously, contextualisation was seen by the AOs as necessary to support AC interpretation. As the AC are often written relatively broadly, so they can apply to a range of contexts, their precise interpretation partly depends on the individual contexts to which they are applied. This then limits the range of meaningful interpretations and rules out irrelevant answers or actions. Some AOs emphasised the need for, and encouraged, assessor autonomy in interpreting the AC in the specific employer or other contexts. Necessarily, these AOs also strongly argued that one of the key requirements for appropriate AC interpretation is occupational expertise of assessors and QA staff. Some AOs also suggested that holistic assessment and ensuring that judgements are formed taking into account a wide range of evidence, help to make assessor judgements more confident and accurate.

[…] when I spoke earlier about the building regulations, the specifications, industry recognised best practice, the assessors will also be aware if there’s anything that might have a specific, you know, this is how we do it as a business. So, it’s being aware of that and knowing what the employer’s looking for. Because again, whilst they’ve got to meet the criteria of the qualification, it’s all about using the qualification for its intended purpose, and that’s to raise the level of skills and knowledge, and if that can tie in with what the employer needs specifically, then they’ll incorporate that into the assessment.

Fenestration_L2

While most AOs emphasised the need for the AC to be sufficiently broad to allow for contextualisation, some AOs suggested that certain types of skills are essentially context-independent and, thus, easier to assess consistently. For some, these were basic technical skills. For others, these were creative process skills that lead to a final product, for instance in creative arts, which are equally applicable across different specialisms and art forms. One AO suggested that standards based around principles rather than the specifics of individual tasks help to make AC applicable across contexts. Where the focus of the qualification was on such constructs, the AOs apparently had an easier task of achieving the balance between AC specificity and contextualisation. Other context-independent aspects that the AOs mentioned were industry best practice, or protocols, that are captured by the National Occupational Standards (NOS), such as in some beauty sector or first aid qualifications.

I think in our qualification, it’s so rigid that there is no flexibility of variation, everybody knows what has got to be achieved and that is it and that’s been set as a national standard for certain things. So, to be very specific, everybody probably up and down the country knows for CPR, it’s 30 compressions to 2 breaths to the tune of the Bee Gees Stayin’ Alive and that has got to be shown and demonstrated and there is no flexibility, but I think we are in a very specific world here that applies to our qualification and first aid qualifications and not necessarily all other CASLO qualifications.

First aid_L3

Quality assurance

All AOs in our sample described complex and multi-faceted QA processes and strategies that, in their view, significantly mitigated the potential risks of inaccurate judgements. Most AOs emphasised the importance of establishing, from the start, through the centre approval process, that centres have required staff expertise. This included holding relevant assessor and quality assurance qualifications. Requirements regarding staff occupational and professional expertise seemed particularly prominent in AO comments, reflecting the views that this in itself significantly mitigates inconsistent interpretation of AC.

[…] a part of the approval process is to make sure that they do have the vocational specialisms in the centre when they’re delivering. So, they are mostly people who have either industry experience or who are still active in the industry […].

Creative_L2

Centre approval also requires centres to establish internal quality assurance (IQA) processes, which most AOs emphasised as the linchpin in the broader QA package of measures. IQA processes and remit were described by the AOs as being quite broad. They involved training of centre staff in qualification and assessment delivery, standardisation of assessors to consistently interpret and apply AC, providing feedback and a second opinion where assessors are not sure about how to judge certain student performances, and so on. Some AOs expressed awareness of how skilled and time-consuming IQA process can be, requiring centres to have it properly resourced to ensure a reasonable amount of IQA.

I think it’s only ever as good as the quality system in the centre. So, I think if there’s a system in the centre to ensure that there is a standardised approach across assessors and that standardisation happens quite regularly. So, it is reliant on the IQA within centres, as well, to ensure that any weaknesses or inconsistencies within assessors are picked up, flagged, returned and that development’s given back to assessors. […] if there are those instances where […] things are more borderline, then that does go to the IQA as the person who oversees that assessment process and is able to have that oversight of the standard of the practice against the centre.

Adult care_L2

AOs described external quality assurance (EQA) as another key aspect of QA that involves monitoring of centres and ensures continued adherence to AO requirements across all aspects of qualification delivery, including interpretation of standards. All AOs spoke about their centres having dedicated EQAs with appropriate occupational expertise, whose checks include moderation^{[footnote 4]} of samples of assessment decisions to ensure consistent interpretation and application of AC in assessment.

[…] we have our army of chef examiners, senior external moderators and moderators who engage actively throughout the year with centres to ensure that that interpretation is articulated clearly to them […] so that we can assure ourselves that there’s understanding across the board, within centres and across centres of what we mean by those command words, and how those assessment criteria should be read and applied. So, I think that’s a fundamental part that sits along the assessment criteria.

Creative_L3

Most AOs noted that EQAs also check that IQA processes and standardisation activities within centres are taking place regularly and are effective. Several AOs explained that EQAs do not just engage in checking documentation about IQA processes. They gain deeper insights about these by considering evidence about the competence, dedication and enthusiasm of centre staff, and especially IQAs. EQAs have conversations with IQAs to understand the nature of the practices they put in place, whether they devise bespoke materials for use in their centres rather than using generic ones available online to support standardisation and other activities, and so on. The nature of the feedback IQAs give to assessors is also considered in terms of whether it is superficial or genuinely evaluative.

While some AOs described their EQA process as involving checks of assessor judgement accuracy and consistency, others questioned how far the EQA checks should focus on judgement accuracy, rather than focusing upon broader assessment and IQA processes, reflecting different views about the extent of AO responsibility. One AO questioned whether it was EQA’s role to “second-assess”, suggesting that, partly due to the resource-intensive nature of EQA, its role was more to check that all processes were in place to support correct assessment decisions rather than the details of the decisions as such.

It’s a fine line between quality assuring and second assessing. And we’re not there to second assess. […] we’re looking at things holistically. […] so, not necessarily drilling down into individual specific criteria on every candidate for a number of reasons. One – that’s not the best use of our time. If the centre have got the processes in place, the assessor’s made his decision, they’ve been through the internal quality checks, we’re then looking more at process.

Fenestration_L2

Most AOs explained that they implemented risk-based sampling when monitoring centres, based on various criteria, including how experienced the centre is in delivering their qualifications, its past track record and so on. Within this, some AOs emphasised that they focused more extensive moderation/QA on particular units, which might be deemed more high stakes, such as summative synoptic units that determine the overall qualification grade or units that otherwise provide key evidence of competence. Others suggested that their relatively small cohorts allowed for most assessment decisions to be moderated. Each of these increased the chance of inaccurate assessment decisions being spotted. Some AOs emphasised that not giving direct claims status (DCS)^{[footnote 5]} to centres, and thus not confirming certification of qualifications unless moderation had taken place, provides further opportunities to check and assurance that assessment decisions in their exemplar qualification are correct.

Overall, there was a sense from most AOs, summed up in the comment below, that they felt confident that the QA processes as a package sufficiently guarded against the potential problem of inaccurate judgements.

[…] we feel that as a package, you know, we have a strong and rigorous oversight of how the qualification is delivered and assessed, and we’re confident that the required standards are consistently met by centres across the country.

Creative_L3

Support and guidance for centres

Support and guidance that the AOs provide to centres to mitigate the potential problem of inaccurate judgements were referenced equally frequently as the QA processes in the interviews. Indeed, AOs often described their QA as “a dual process” involving both monitoring and support in relation to interpretation of standards and other aspects of qualification delivery. This integrated approach to QA and support appears to aim to get the centres to the point where they can operate qualification delivery with little additional involvement from the AO. Achieving this ideal may be seen as the most effective mitigation against many of the potential problems with the CASLO approach, including inaccurate judgements.

[…] under our model what we’re looking to do is get the centre in a position where we can say we are confident you’re doing everything correctly, […] what our ideal is, we are just looking to say yes, you’re managing your affairs correctly. We don’t need to start delving in and looking at all the detail to a great extent, […]. We don’t want to be doing it for you.

Adult care_L3

Most AOs provided continuous support involving multiple opportunities for centres to ask for help and clarification and for AOs to get insights about how centres are interpreting qualification standards. Continuous support was mostly provided by EQAs who, in addition to their monitoring role, also take the more supportive role of a “critical friend”, advising on best practice, including the interpretation of the AC. One AO suggested that the whole CASLO approach is based on giving and receiving feedback, which leads to incremental improvement, including in relation to interpretation of standards. They expressed confidence that the centres and practitioners are generally receptive to feedback and ready to learn from each other. Some AOs also emphasised that the timing of their support and QA is of key importance, as continuous monitoring allows them to spot issues early on and provide early intervention to ensure appropriate interpretation of AC.

A lot of that [is] done very early in the delivery of the qualification so that we’re able to identify those misalignments perhaps with the interpretation of the assessment criteria and achieve alignment early on before it’s embedded in the qualification, so I think the timing of that is key.

Creative_L3

Most AOs also provided guidance documents with different types of information to support interpretation of AC, recognising that the AC on their own are open to interpretation. AOs delivering occupational qualifications often referenced documents that sit outside of the qualification and involve industry codes of practice, building regulations or treatment protocols, depending on the specific sector, which help to situate the AC in the broader context of relevant best practice and, thus, help with their accurate interpretation. One AO with a ‘dual purpose’ qualification mentioned that the core and indicative content in their qualification provides an indication of the level of response or evidence required for each of the LOs to achieve a pass. Most AOs also provided specific guidance which includes definitions of key command verbs used in AC and AO expectations about their interpretation. Others provided guidance documents with elaboration of grade descriptors where relevant or glossaries that explain key qualification terminology. Providing guidance was deemed to be helpful even where the interpretation might be seen to be fairly constrained due to links to the NOS or due to being rooted in the long-established standards in a sector, perhaps in particular where assessors might be new or less experienced.

So, we give guidance to centres that says if it starts with ‘state’ this is the type of answer we’d expect. If we’re saying ‘list’ it is literally bullet points. If we’re saying ‘explain’ or ‘describe’ then they’ve got guidance from us as to what type of answer to expect.

Fenestration_L2

Several AOs spoke about different kinds of training activities that they provide to centres, some compulsory and some optional, helping to ensure appropriate interpretation of standards and assessment requirements, often within the broader training about qualification delivery. These were sometimes delivered online to promote wider uptake. Within this, some AOs took into account how their assessors might interact with the guidance provided and diversify the media they use to ensure assessors engage appropriately by, for example, including training videos about different aspects of assessment practice. Training centres in how to run their own internal standardisation or related activities was also a relatively common upskilling activity undertaken by AOs, with some providing guidance and materials that centres can use to support these activities.

[…] and we actually have […] training videos talking about how to conduct reviews, on how to conduct planning, those types of things to say well here’s what we’re looking for when you’re conducting a review and here’s the hallmarks of what we want to see at the higher grades as well.

Creative_L2

Several AOs provided specific exemplars of student work to centres to support interpretation of standards and standardisation processes. One AO mentioned that, for knowledge questions, centres are expected to have indicative answers to demonstrate the pass grade threshold requirements that are reviewed by the EQA, who provides relevant feedback. However, a few AOs suggested that there are resource issues in providing centres with exemplars for every unit and every AC simultaneously, which would be “an impossibility”. To deal with this, some provided exemplars for different units or AC incrementally over time, creating “banks of exemplars”. Others explained that exemplar materials tend to be available for core or otherwise higher priority units or criteria (for instance those known to be harder to assess consistently). However, the AOs emphasised that the units which are not as frequently or as thoroughly exemplified were still sampled and moderated through EQA, or that EQAs might request exemplification for certain optional units if they felt this was needed.

Yeah, we do give exemplar work out. […] we provide standardisation materials that they can use in standardisation sessions that includes an assessment commentary or an explanation why the grades were given and why we would support those grades or agreeing those grades at moderation.

Creative_L2

[…] we try to circle round the qualification to pick up different units or particular learning aims every year. So that then builds up a bank of exemplars that assessors can then refer back to. So, it varies depending on the qualification, but we’d usually have a unit’s worth of assignments and exemplars. […] our focus is really on the core. […] But I believe that we do have optional units that the SSV will ask to interrogate particularly […] if they feel that that’s an area that they want us to interrogate or exemplify for centres.

Business_L3

Despite most AOs providing some form of guidance or exemplars, there were somewhat different views on the extent of detail that should be provided or whether these materials should be provided at all. This tended to reflect an awareness of the difficulty of achieving the balance between flexibility of contexts that assessment might take place in and the need for a level of reliability and consistency in AC interpretation without being too prescriptive. The AOs emphasised that any guidance needs to be interpreted in the broader context of the task by occupationally competent assessors. Some AOs also saw value in centre ownership of assessment and encouraged centre and assessor development in this respect, as well as creativity in using different kinds of tasks, which might be stifled if centres relied too much on exemplars and detailed guidance.

[…] we do have exemplars and we do go through those with centres. But again, we’re cautious of the risk there of being too prescriptive and then ending up centres just picking those examples and replicating them. So, we guard against that […] through our entire engagement with centres, where we’re encouraging variation and creativity, we’re encouraging them to be diverse.

Creative_L3

Some AOs deliberately did not provide exemplars of student work nor extensive guidance documents to help with AC interpretation because “it tends to just drive more questions”. Rather, they provided support and feedback to centres through their EQAs and said they could supply some exemplars or clarification at centre request. In contrast, Construction_L1 described providing extensive guidance to make sure that even tutors who may not have the relevant sector background would be able to teach and assess.

It was also suggested that centre requirements for guidance seem to ebb and flow, and the AOs appear to be responsive to this in relation to both assessment and teaching. Some AOs said that their more recent qualifications (though not the exemplar ones in our sample) tended to involve more explicit guidance on how to interpret the AC. This was sometimes introduced in response to growing demands from centres, where assessors or IQAs might find that having more explicit guidance reduces the pressure they are under and supports their decision making. In other cases, this appeared to reflect the growing experience of the AOs in relation to the extent of guidance that is likely to lead to better outcomes. Some comments also suggested that there might be individual assessors or centres that do not see the value of detailed guidance, training or exemplars, or feel that they do not have time or sufficient resource to engage with them.

Standardisation as a challenge and a mitigation

Despite suggesting that standardisation is an important mitigation in relation to incorrect judgements, most AOs recognised that conducting standardisation in the context of CASLO qualifications was challenging and resource intensive. This was mainly due to the multitude of contexts that their centres typically functioned in and the demand for flexibility to meaningfully address these diverse settings in standardisation. Nevertheless, most of the AOs appeared confident that they were doing enough to ensure sufficient standardisation was taking place.

All AOs in our sample said that they expect centres to conduct internal standardisation, typically run by IQA staff, and would be checking that this was taking place through EQA. However, there were flexible requirements regarding standardisation frequency, format and formality in the centres, recognising their different size, contexts and circumstances. Thus, standardisation might happen once a year in some centres and monthly in others.

We tend to do it on their numbers and how they deliver, so where we work with colleges, we do tend to say a couple of times a term because that’s how they work, and they work on those calendars. Where it’s independent training providers, again, it’s a numbers thing, some do it as regularly as monthly, others do it 4 times a year, that kind of thing. We encourage a minimum of 4 times a year for a standardisation event, but other than the fact that we actually say standardisation has to happen and we do tell people what we expect to happen at standardisation and what the purpose of it is, we’re slightly more lenient on stipulating timings and how often, because it does depend on how they deliver and what size of organisation they are, because if there’s only 2 or 3 trainers and they’ve only got a dozen learners each, for them to meet and standardise every month might well be overkill.

End of life care_L2

Some AOs said that they also require centres to attend compulsory annual standardisation events organised by the AO. However, cross-centre standardisation did not appear to be common, with AOs typically using EQAs to monitor between‑centre consistency. This was sometimes said to be related to the difficulties with standardising across diverse approaches to assessment across centres, as well as due to their geographical dispersion. Not all AOs suggested specific reasons as to why such standardisation events were not organised, implying that internal centre standardisation alongside EQA was sufficient. A few AOs suggested that, while AO‑organised standardisation events may not happen sufficiently frequently, there is demand from centres for such events.

[…] we used to do quite a lot of cross-centre standardisation led by us, we’ve not done it in quite a few years now, obviously COVID pushed a lot of things back, but we do actually get that feedback from centres quite a lot that they do want to attend cross-centre standardisation led by us. So, I think that is something that we are aware of, and we would like to do.

Adult care_L3

AOs also said that they conduct standardisations of their EQAs and/or external assessors. Several AOs described the way that they conduct EQA standardisation, explaining that it involves explicit consideration of student work and application of standards rather than just focusing on assessment administration processes. Others described similar processes undertaken by their centres, where the focus is on scenario-based standardisation and discussion of exemplar portfolios and borderline performances.

[…] they all have to attend and pass standardisation. They have a pre-event task, which is on the element of a unit, that they do a couple of weeks before the actual training event and that gives the senior external examiner a bit of an idea of […] what the external examiners are thinking. Then there’s the professional discussion aspect, looking at units, having that discussion about, here’s some student work, has that met P6 of Unit 1, yes/no, the senior will then explain, after that discussion, why it has or hasn’t, depending on the scenario. So, those are the professional discussions and then it ends with a standardisation exercise, which is […] – now you need to go away and actually produce your external examiner report based on this particular student work, this particular unit, and that’s really them seeing that they can apply the standard.

Construction_L5

Among various resource and other challenges of implementing standardisation in CASLO qualifications, the scale of some of these qualifications and the number of units which might require standardisation appeared to be 2 of the more prominent concerns. Several AOs pointed out practical challenges about organising standardisation events for their EQAs in terms of gathering them all at the same place or allowing for sufficient time for them to collect relevant exemplar materials to discuss in standardisation meetings. Some AOs said they started to encourage centres to conduct standardisation online to overcome some of the resource and cost issues of in-person standardisation.

Some AOs mitigated the resource challenges by prioritising higher stakes or core units or those that are known to be more difficult to judge consistently. Some suggested that they focus cross-centre standardisation on the graded qualifications, rather than those that only involve pass/fail decisions, possibly because of the perception that graded decisions are more challenging to make consistently. This mirrored the AOs’ rationale for prioritising certain units where they provided exemplar performances, discussed in the previous section.

So, more of a challenge is the resource, without shadow of a doubt. You’ve never got enough people to do what you want to do you’re always managing and balancing the risk with the resources that you have available. In terms of getting it into people’s heads, that’s what we’re reasonably good at […].

First aid_L3

Some AOs pointed out that assessment methods producing relatively constrained types and nature of evidence are more likely to lead to consistent judgements that are easier to standardise. This was the case with the assessment of practical tasks in Hairdressing_L2 and the assessment of assignments in Business_L3 qualifications. In contrast, standardisation of portfolio-based assessment, where portfolios might contain varied types of evidence, was deemed to be more problematic.

It was also noted that while there is an appetite to have more one-off training or standardisation events for centres as well as EQAs, standardisation is often a continuous process taking place during other centre‑based activities and outside of formal standardisation events. Some AOs also felt strongly that EQA standardisation happening twice a year was sufficient.

Occupational or professional expertise and experience

Although all AOs spoke about the importance of support and guidance as well as QA and standardisation as key mitigations of inaccurate judgement, most AOs also emphasised the need for the practitioners involved in assessment of CASLO qualifications to have relevant occupational expertise. Such expertise enables assessors, IQAs and EQAs to interpret the AC and their specific requirements and terminology in the context of the relevant occupation and associated standards that they should, by implication, be familiar with. Indeed, one AO thought that additional guidance about how to interpret the AC should not be needed if assessors are occupationally competent.

So, I think it’s really important to recognise that […] the whole context of this is delivered by professionals who are artistic practitioners who understand [for instance] problem solving from that perspective. […] It’s practitioners who are professionals at each level, both from us and both within centres, who understand this terminology.

Creative_L3

I think it is quite obvious to assessors, […] they just know if it’s safe or not based on their experience. Because the assessors are at the top of their game in terms of these treatments, they will have massive amounts of experience in carrying them out.

Skin peel_L4

Several AOs also noted that, in addition to occupational expertise, centre staff involved in qualification delivery should have specific assessment or QA expertise. However, some AOs described providing additional training for assessors to support them in conducting IQA or EQA duties, sometimes expressing concerns about the effectiveness of assessor or QA qualifications.

[…] we’ve taken on EQAs that we’ve had to retrain in how to make judgements and how to apply what it is they’re supposed to be doing, despite them holding the qualification and have been lead EQA for someone else for 5 years. […] what we really want is someone who knows the subject area and we’re going to teach you how to EQA and EQA properly, because that serves us better.

Chef_L2

Another AO suggested that nowadays, due to changes in policy and funding arrangements in centres, AOs have less direct influence on centres with respect to assessor continuous professional development (CPD) or the length of their prior industry experience. The AOs can monitor these, but cannot enforce specific requirements, suggesting that this might have negative consequences for consistent interpretation of standards.

Again, it’s back to sort of the change in policy. […] it was there as an NVQ requirement [that] colleges absolutely funded these 30 hours [of CPD] […] We still recommend 30 hours […] and we monitor it when EQAs go in, but in terms of sort of the sanctions, unless it was completely absent, there’s not really too much that we can do. And the other thing is the 5 years commercial experience, we used to be able to dictate that as part of the qualifications, but of course now that’s really up to colleges on their own recruitment practices […] So those 2 things probably don’t help in terms of maintaining standards.

Hairdressing_L2

Communities of practice

Most AOs agreed that strong communities of practice, especially in relatively niche or small sectors, where practitioners might work across different centres, help to promote consistent interpretation of AC. Some AOs highlighted the sense of shared ownership of the standard amongst practitioners, and practitioners’ wish to protect the reputation of their sector and to “self-police”, as important factors in ensuring that standards are not allowed to slip or be frivolously interpreted.

[…] the centres that deliver this qualification for us, they’re quite geographically disparate but they also share trainers, so there is a national standard because they actually all work together to maintain a standard because it’s used by the outdoor sector in general for different outdoor activities, so it’s there informally […] and it’s a nationally agreed standard. […] so they tend to self-police, so we tend to be able to get good intelligence on centres that are maybe not playing the game and […] the outdoor education sector is a very, very small world, which means, again, […] they’ve all got a vested interest because none of them want to look like they’re letting the side down, as it were.

First aid_L3

While most AOs recognised the potential value of strong communities of practice, some highlighted certain limitations of these. It was noted that where qualifications are delivered internationally, there can be less reliance on communities of practice because there are too many practitioners working across very diverse contexts. For this reason, they relied more on written requirements as a “grounding” or baseline for the interpretation of AC.

Another AO noted that strong communities of practice and a strong sense of “sector expertise” can lead assessors to think that they have internalised the standard to the extent which might result in impressionistic judging without sufficient reference to the relevant criteria. For this reason, there was a need to still consider the AC and the written requirements and ensure that these are adhered to appropriately through training and standardisation.

[…] I think there is a bit of a danger, an overreliance firstly on a community of practice where it’s so strong that people internalise the standard so much so that sometimes they feel like they can disregard the criteria. But actually, what we’ve had to do is refocus them and say go back and look at the assessment criteria, […] because they forget what we’ve actually specified in the specification.

Creative_L3

Despite potential limitations, several AOs said that they actively promoted networking and community building among their centres, though some acknowledged that there was more that they might be able to do in this respect. A range of ways of engendering and promoting communities of practice were suggested across AOs, including in-person events such as conferences, forums or other meetings, online groups, etc. Most AOs also saw their normal training events or EQA support as contributing to the development of communities of practice across their centres. Some AOs noted that networking opportunities are sometimes created via centre initiative, additionally helping to strengthen communities of practice.

It was observed by some AOs that communities of practice take a long time to foster and that sufficient engagement with centres is required to achieve a level of common understanding of AC that can be relied on. The AO offering the Hairdressing_L2 qualification further emphasised the need for investment in these communities, which, with reduced funding for bodies such as sector skills councils, which used to support many of these communities through activities and forums, might be threatened in some sectors. This AO suggested that there was an onus on the AOs in those sectors now to do more to engender and support communities of practice.

And [as] the awarding body, we’ve been going for well over a decade now, and I think the understanding of our expectations has got momentum and I think that carries through. […] And as a community of practice, […] there is understanding attached to some of these words and some of these phrases […], and that kind of underpins the consistency as well, which we draw on. […] that’s not something you can just create overnight. That’s the legacy of our engagement with those centres and the momentum of that community of understanding […].

Creative_L3

Qualification and assessment design processes and features

Several AOs emphasised the need for language precision and clarity in writing AC to ensure that there is a good chance of them being interpreted accurately. They described certain qualification design processes which help to ensure that this was achieved, typically involving multiple rounds of development and review by expert and stakeholder panels before the qualification is launched. Some AOs said they involve centres that currently deliver their qualifications in development meetings and take their views into consideration. Some AOs also spoke about regular qualification reviews, which might sometimes be initiated for certain AC that have been found through regular monitoring to be difficult to interpret. However, some AOs reflected on the high levels of resource required to support detailed reviews across qualifications.

Several AOs discussed the use of certain design features which, in their view, help to guard against inaccurate judgements. Most AOs flagged carefully selected command verbs which help signal different aspects of standards within AC, including the expected nature and complexity of the performance required to achieve a grade. However, some AOs also recognised the challenges with using command verbs to differentiate between the qualification levels.

[…] that has been an ongoing challenge from QCF days, what the particular command words mean, how do you identify levelness, and obviously there was this whole thing with the old QCF writing guidelines, where particular command words were reserved for particular levels, so there is a bit of that in here, you know, where we’ve gone back to that old-school benchmarking and said, you know, what does a ‘describe’, […] what level was that traditionally aligned to on the QCF framework? Of course, I know we’re on the RQF framework but that was our additional basis. The other thing that we always do when we think about command words that we use, such as propose, we also benchmark it against other level 3 qualifications.

Creative_L3

Several AOs also talked about using grade descriptors to support the interpretation of the AC. These are provided separately from the AC rather than being attached to each individual AC. These grade descriptors also rely on the use of command verbs through which the AOs strive to adequately capture different complexity levels and demand across different grades.

One AO suggested that common structure across different units of their Creative_L2 exemplar qualification, which encapsulates aspects of the creative process that they cover (“plan-do-review”), can help mitigate the potential impact of limited resource to conduct standardisation. The common structure helps to ensure consistent AC interpretation across the board irrespective of the specific focus and context of each unit, even if no explicit guidance was provided or if standardisation was not conducted on each unit or AC.

[…] we could look at how to plan that unit and how to review that unit and then you could go away and deliver a different unit, but you’d still have a good understanding of what you’re expecting learners to do in relation to the planning tasks and the review tasks. So, it’s not like every single unit is an island that you can’t deliver because you haven’t had a standardisation in that unit.

Creative_L2

Question 2

Atomistic assessor judgements

Accepted Answer

Another criticism from the literature is that atomistic CASLO specifications encourage atomistic and/or mechanistic judgements, with assessors often reduced to ticking off AC lists, criterion by criterion when assessing. This can lead to the potential problem of arbitrary and, therefore, deficient, judgements as, according to criticisms, there may be more to having met an LO than having satisfied each individual AC. In particular, if competence requires the integration of elements of knowledge, skill, and understanding – yet these elements are only ever assessed discretely, criterion by criterion (and potentially via discrete tasks/events) – then this raises the potential problem of not assessing comprehensively and authentically.

Only 5 AOs in our sample recognised these as potentially relevant problems for their exemplar qualifications (all except one of these were ‘confirm competence’ qualifications), while the majority did not entirely recognise it or did not think it was relevant at all. Even though most AOs did not see these as potential problems for their exemplar qualifications, this did not appear to be because they believed that their qualification AC specifications necessarily afforded holistic judgement or holistic assessment (see below). Instead, it was because of assessors being able to, in some ways, see beyond atomistic AC, and, sometimes, due to mitigations or protective factors embedded in their qualifications. Everyone discussed at least 2 but mostly several mitigations and/or protective factors that helped reduce the risk of this potential problem arising.

It is helpful to draw an explicit distinction between holistic judgement and holistic assessment at this point, as these were both discussed in response to questions about atomistic assessor judgements. Holistic assessment describes an assessment scenario – a task or a naturally occurring situation – that elicits evidence related to multiple AC simultaneously, typically because the scenario calls for an integration of relevant knowledge, understanding, and skill. Holistic judgement, on the other hand, describes an approach to evaluating assessment evidence, where assessors do not focus on individual LOs or AC independently, each in their own right, instead deferring to a higher-level judgement of competence. In terms of the logic of the CASLO approach, holistic judgement is potentially problematic, as it seems to open the door to a compensatory approach, depending on how it is implemented in practice.

When discussing mitigations for the potential problem of atomistic judgements failing to assess integrated competence adequately, the AOs used the notion of holistic assessment to refer to either holistic assessment or holistic judgements as defined above, not making a clear distinction between them in all cases. In some cases, they referred to holistic assessment procedures or tasks, which integrate multiple AC, LOs or even units – that is, holistic assessment. Otherwise, the notion of holistic assessment was used to refer to holistic judgement in the sense of evaluative, contextualised, generalisable judgement although still anchored by specific AC. Such judgement was thought to be based on varied evidence and multiple performance conditions, rather than being formed based on a single instance of successful performance against a single AC. This understanding of holistic judgement seemed also to be implied where AOs referred to assessors “making a judgement” rather than “ticking boxes” during assessment. Where it is sufficiently clear from the respondents’ comments that they are referring to holistic judgement when saying holistic assessment, we will use the former in our commentary throughout this section.

The quote below describes some of the mitigations that were often suggested, such as use of holistic assessment (that is, tasks or events) across AC or LOs and contextualisation. It also highlights an awareness of the potential washback of atomistic assessment on learning, as well as increased burden of assessment, which we discuss later in the report. Finally, it highlights a tension between the benefits of holistic assessment and the need for a level of specificity and clarity in the mapping of the more holistic assessment tasks to AC. This was a prominent theme in our discussions with the AOs in relation to the atomistic assessor judgements, but also some other potential problems, including poorly conceived assessment tasks/events and lack of holistic learning.

Yeah, I think it is about seeing it as a high overall level of what that learning outcome is […] we try and encourage that more because it probably really burdens them to actually look at each assessment criteria individually and say, right, we’re going to do an assessment on this, we’re going to do an assessment on this, […]. And is the learner then joining those dots up about what they’re actually learning, […]. I think it is kind of contextualised in that doing it as part of bigger assessment pieces holistically across learning outcomes and assessment criteria, but obviously ensuring that when you do develop those assessment tasks or materials that they are clearly mapped in the background to all the assessment criteria.

Adult care_L3

Overall, the main mitigations proposed by the AOs included holistic delivery and/or holistic assessment or judgement, often coupled with the requirement for occupational or professional expertise of assessors. Some AOs also discussed mitigating effects of certain qualification design features and spoke about providing support and guidance to centres, and monitoring for atomistic assessment or judgement through their QA processes.

Perceived drivers of atomisation

While some AOs did feel that atomistic judgements and tasks may be prompted by highly atomistic AC, this was not always seen as the only or the main reason why assessors might feel compelled to apply a tick-box approach in assessment (in judgements or when devising assessment tasks), potentially rewarding deficient performances as competent. It was suggested by some AOs that additional pressures can exacerbate the effect of atomistic AC lists, which otherwise should be seen more as a tool to ensure higher consistency and to avoid missing certain aspects of content. Despite the use of atomistic specifications, there was no expectation from the AOs that assessment, teaching and learning should proceed in a list-wise fashion, nor without using more integrated assessment tasks or events or without reference to a higher‑level holistic/professional judgement when appropriate.

For example, it was suggested that mastery requirements at AC level, where not achieving even one AC may threaten the overall qualification result, might create nervousness in assessors. Assessors might be worried about missing something and disadvantaging students if not directly assessing against each and every AC. Relatedly, an AO suggested that pressures on teachers to ensure that students pass can also lead teachers to a “path of least resistance” approach of assessing atomistically to ensure each AC is at least minimally met under the mastery approach even where genuinely integrated or satisfactory performance may not be in evidence. However, this might result in missing (or ignoring) the wider point of why aspects of a task are performed or whether they are performed in the most engaging way, turning assessment and performance into a “cookie cutter exercise” or a “sausage machine”. However, it was emphasised that this was a rare occurrence that can be detected by EQAs, and not a desirable outcome. These comments resonated with the potential problem of superficial learning, which we discuss later in the report.

[…] I think the pressure on them sometimes makes them think – what is the easiest way I can get the kids through it – and […] the easiest thing to do is to do X, Y and Z and what happens is the learners don’t necessarily understand why they’re doing it or they’re not using it in the most engaging way. But nevertheless, they meet the criteria, they all pass […] This isn’t just CASLO, this is all quals now, but it is something you see because of that nervousness […] with having to get numbers and the pressures on learner achievement.

Creative_L2

It was also implicit in some comments that, in some qualifications, securing grades at the lower end of the achievement spectrum, such as a pass, might, in fact, solely require a student to meet the individual AC in a fairly mechanistic way. This suggests that demonstrating evidence of integrated performance at the level of a pass may not be essential in some cases, therefore indirectly allowing for atomistic assessment and judgements. On the other hand, there may be more explicit requirements for integration at higher grades like merit or distinction, which may be captured in the AC or grade descriptors.

[…] So, that’s a practical piece of work, you have seen that a learner has automated. They have not done it creatively, they have not done it in an interesting way […] they’ve just slapped automation on a track, that’s pass. It’s then the merits and distinctions where they have made a creative use of that and they can sort of, we ask them to explain why they’ve done what they’ve done as well.

Creative_L2

One AO pointed out that assessors might feel under pressure to conform to apply a tick-box approach when being observed by inspectors or EQAs, even when they might be assessing more holistically otherwise and then ticking the AC off at a later point. Therefore, it may be difficult to observe this holistic professional judgement in action during brief EQA or inspection visits. Finally, another AO suggested that the tick‑box approach can be attributed to a trend of the assessor role and confidence being more generally undermined by accountability and funding pressures across broader education context rather than just in relation to assessment.

Holistic delivery and assessment, contextualisation and real-life task setting

Some AOs suggested that, despite the atomistic look of CASLO specifications, a holistic or project approach to delivery (in terms of teaching and learning) helped to support holistic assessment, which was likely to align with holistic delivery where tutors were also assessors. More specifically, it was suggested that teaching in their qualifications does not focus on individual AC and is more likely to relate to the LOs, which are more holistic in terms of covering a broader topic.

The unit should be delivered holistically, […] you don’t do 1.1 and then 1.2 and then 1.3. I think it’s set out in that way because it’s probably the easiest way to set it out than to define what it is that you expect. I think the learning outcomes provide an umbrella that gather all those assessment criteria up and looking at learning outcome 1 [in this unit], it’s about accidents in the construction environment. And you’d certainly learn about cause and effect at the same time, rather than looking at common causes of accidents and then assessing that and then moving on to what you would do to prevent them and the consequences of those. […]

Construction_L1

Other AOs considered a holistic or project approach to assessment to be more of an explicit mitigation of the potential risks of inauthentic assessment based on deficient atomistic judgements. Even though AC are atomistic, in some qualifications assessments happen at the level of a wider practical task. It was implicit in various comments that it was beneficial for assessor judgements to be situated in the broader context of tasks or activities rather than following the specification breakdown into AC, LOs or even units, because this approach helps form more confident judgements based on wider evidence. Thus, when making judgements, assessors have to take into account how the activities that happen during those tasks fit together, how students address different requirements of the tasks, how they justify their decisions, and so on. This partly capitalises on the presence of implicit links across LOs within or across units in some qualifications. This was especially the case where a unit with its multiple LOs might correspond to a complete task that is commonly carried out in the workplace and where activities may happen in quick succession, making it more difficult to assess each one discretely.

[…] obviously you have got to focus on the assessment criteria but not just on those […] I think it is about having that holistic overall judgement about meeting that learning outcome, […]. Because seeing it in isolation, yeah that might be enough evidence, but sometimes it’s stronger evidence if that is joined with other assessment criteria as part of one assessment activity that is seeing something being completed from start to finish, for example, that you can take more context from the learner completing an activity that might […] meet the assessment criteria on paper, but as a strong piece of evidence you can actually see it holistically across the piece.

Adult care_L3

That word holistic. That’s what we encourage […], that’s why they’re not assessing against a unit during observation. They’re looking at the whole [task], completing their evidence, gathering their evidence, and they will cross-reference that against the criteria. You know, we saw that one, we didn’t see that, we can pick that one up next time.

Fenestration_L2

Different AOs discussed different parts of their qualifications that tasks or assessment events should integrate. Some spoke about LO level integration, while some suggested that assessment situations can sometimes straddle different units and that this should be embraced by assessors (as in the Fenestration_L2 quote above). There were also AOs that suggested the unit level as the appropriate level at which assessment should be integrated if desired, though they also said that centres tended to use more discrete assessments than that, with several assignments within a unit. Any residual risks related to potentially deficient atomistic judgements in the absence of cross-unit synoptic assessment were seen as a necessary trade-off against the benefits of unitisation by some AOs.

While synoptic assessment across units could be considered as a mitigation of the risks around inauthentic assessment based on deficient atomistic judgements, some AOs flagged that in this type of assessment the mastery model requirements can adversely interact with challenges in assessment design, especially when assignments are not externally set. If a broad cross‑unit synoptic task happens not to be sufficiently well designed or mapped to AC, students that would fail it would also potentially fail to achieve multiple units at once, instead of failing just one unit as in more traditional unit-based CASLO assessments (see the next section for more on the potential challenges around the design and use of integrated, holistic tasks or events).

Instead of developing explicit synoptic units, some AOs suggested a reliance on the perceived inherent validity of assessments which are typically conducted in a real‑life setting as a protective factor. In some cases, where a qualification embodies a construct such as the creative process, the AOs flagged that assessors can only reach a judgement about the student’s grade having seen the whole process, therefore reducing the likelihood of atomistic judgements being applied despite atomistic AC specifications.

The qualification that we’re discussing, the whole process is part of the assessment. […] most of the units follow the “plan, do and reflect” model but built into that there are things like rehearsal skills and how well can you work with others and how well can you collaborate. And the assessor really needs to see the process of all that happening in order to be able to award a grade.

Creative_L2

Discussions about holistic judgement were also partly related to the notion of judgement generalisation which is an expectation that assessor judgements about whether certain AC have been achieved would be formed over a period of time during which the evidence is accumulated. Thus, rather than just ticking off an AC when it has been evidenced for the first time, assessors should consider the AC to be met only after having seen the student perform adequately on several occasions. Relatedly, it was suggested that assessors should not ignore additional, potentially contradictory, evidence in forming an overall view of whether a student has achieved a standard. This might arise in situations where something had been observed and “ticked off” before but then might appear again in the context of another observation with a different focus. In their comments, the AOs implied that integrating evidence from different assessment events and, thus achieving judgement generalisation, tends to be easier when assessment is carried out in the context of holistic situations, as opposed to when it is atomistically focusing on individual AC or even units.

[…] I think it is the fact that if an assessor is to assess via units, and they may decide after a couple of visits that FI3 all signed off all done. Well, no because next time you go out to see something, you’re going to see some elements of FI3. So, you know, not on about over-assessing but you need to keep that in mind that that is still part of the overall process.

Fenestration_L2

While the requirement for generalisable judgements was implicit in most qualifications, thus potentially guarding against arbitrary atomistic judgements, exact requirements in this respect were seldom specified. Only Skin peel_L4, Hairdressing_L2 and Chef_L2 specified the so-called range statements, outlining the range of conditions in which performances need to be demonstrated and how many times this needed to happen for summative assessment. Furthermore, the latter AO said that these were often indicative, and it was more important to get a holistic judgement of achievement across different AC, as well as units. Some AOs also suggested that the range statements were not needed in their exemplar qualifications because the skills assessed are largely generalisable to different contexts and do not need to be explicitly demonstrated across different contexts that each elicit different aspects of the skill. In most cases, however, it seemed that it was the professional judgement of assessors that AOs relied on to determine the scope of generalisation or how much evidence was enough to make a reliable judgement across relevant contexts and situations. It seemed, based on AO comments, that being able to make such determinations would only be possible based on holistic judgement, rather than applying a mechanistic, tick-box approach.

[…] the guidance that’s given is that they need to produce evidence that shows the candidate can meet the criteria consistently over an appropriate period of time. Now I know that’s quite a broad brush, but we don’t want […] the old NVQ world where everything had to be done 3 times. That was it. It was almost carved in stone. […] it’s not necessarily that. It’s the assessor using their judgement which is something we encourage because they’ve got the right background. […] some people may get 2 on-site assessments, some may get 5, and it’s not over-assessing. It’s just making sure that they’ve reached the decision for the right reasons as we say, not just ticking them off to the next one.

Fenestration_L2

Occupational or professional expertise and experience

Most AOs emphasised the need for the practitioners involved in assessment of CASLO qualifications to have relevant occupational expertise to be able to see the bigger picture and the significance of certain aspects of performance to meeting the AC (also illustrated in the quote above). Some AOs specifically emphasised assessment expertise as a potential mitigation. It was suggested that the way assessors had been trained to assess during their own assessment qualifications will to some extent shape their own approach thereafter.

[…] they are highly experienced assessors in that field, and they know what a safe treatment looks like. So, then I would say they’re not going to get bogged down into a tick list, they’re going to look at the observation holistically and see if it is all safe, which is kind of the underpinning of everything really. So, they aren’t just looking at, have they done this, have they done this, have they done this? They need a holistic understanding of the range and of the treatment protocol and of the learning outcomes and indicative content to ultimately decide if that learner is providing a safe treatment or not.

Skin peel_L4

It was also suggested that getting enough sufficiently occupationally expert people in the system to assess can be a challenge in some sectors, which is why AOs sometimes resort to over-specifying the content and assessment. It was noted that this tends to be more of an issue in college than work-based settings. Less experienced assessors were deemed particularly vulnerable to potential atomistic judging, “clinging to the bit of paper” with AC lists rather than taking a more holistic approach that more experienced assessors might take. At the same time, it was recognised that AC specifications can help support the judgement of newer assessors early on.

Support, guidance and QA

Several AOs spoke about the value of support and guidance, sometimes in the form of training including videos of assessment in practice, to encourage or enable assessors to take a more holistic approach to assessment. One AO described using exemplars of student work to illustrate to centres that the AC do not have to be assessed or evidenced one by one. This was, again, interrelated with their EQA monitoring, typically resulting in developmental feedback for centres rather than punitive actions.

[…] we do sometimes experience centres that are focused on assessment criteria only […] and our response to that is usually through the EQA process. They would get feedback on the way to develop that, and this could be followed up at a centre visit or in training […].

Creative_L2

So, it’s not something that we specifically put in and say that they have to do it, but it is something that we encourage, and we strongly encourage it. In fact, one of the most common conversations our EQA staff have is, when they go out and observe assessment being delivered in a very rigid list-like way, they will encourage that centre to go and take a look at our guidance, go and take a look at our videos and talk to us and set up a follow-up meeting about holistic assessment opportunities.

First aid_L3

One AO advocated not providing AC checklists as recording mechanisms in order to encourage more holistic judgement that would, in this way, be more tailored to individual students. Instead, this AO required assessors to produce a “summative statement” when assessment for a unit is complete, to give an overall judgement of the student and their competence across the entire unit content. This was intended to encourage assessors to look at evidence in the round and form an overall judgement at unit level.

Some AOs suggested supplementing AC checklists as recording mechanisms with additional evaluative comments about how judges interpreted and judged the AC, including the rationale for their decisions. However, some AOs did not see the benefit of asking assessors to provide descriptive comments where judgements are just binary achieved or not achieved, arguing that such comments are more appropriate where qualifications are graded.

Given the significance of occupational and professional expertise of centre staff as a mitigation for these potential problems, the AOs also spoke about the importance of establishing and monitoring that through their approvals process. In addition, some AOs expected centres to address holistic assessment across units as part of their internal standardisation.

The IQA process was also mentioned as important to discourage atomistic assessment and judgements. The need for IQAs not to just review assessment records, which are necessarily atomistic, but to shadow and observe assessors to get an insight into how they are forming their judgements in real time was highlighted. One AO emphasised the usefulness of unannounced EQA visits to centres which allow them to see assessment practices in action, including whether holistic assessment within a broader scenario is being used. However, this contrasts with the point made by the same AO, discussed earlier, regarding how external observation can influence assessor performance, potentially inducing assessors to approach assessment more atomistically than they normally would to demonstrate their adherence to written specifications.

Qualification and assessment design features

In proposing a holistic or project approach to assessment as a mitigation of potential risks around inauthentic assessment based on deficient atomistic judgements, several AOs implied that assessment should focus as much on the overall performance, and how any discrete activities which might correspond to individual AC are integrated and fit within it, as on the discrete activities/AC themselves. This suggested that AOs believed there was a need to ensure that their qualifications testified to the overall coherence and effectiveness of integrated performances as evidence of competence. While this goal was often implied in the broader purpose of some qualifications, it was not easy to capture this in the specification of the AC or LOs. This then left the possibility that all the AC and LOs might be demonstrated and ticked off over time, but potentially not in the context of integrated performances, despite the intention of the qualification designers.

In some exemplar qualifications within safety critical domains, when assessing occupational tasks such as cosmetic procedures on clients, performing an integrated procedure as a single process was seen to be critical. In these qualifications, a mechanism that is employed to ensure that the integrated character of the task is captured in assessment involves a strong task-level mastery requirement across all the relevant AC. That is, while the AC might correspond to individual activities, they are to be jointly met each time in the context of a broader procedure, arguably amounting to overall successful and integrated performance. In such cases, a holistic judgement of how effectively and consistently individual activities within a broader task were carried out and integrated is in effect imposed by the mastery requirement at procedure or task level.

No, the treatment has to be safe, and they have to follow the treatment protocol every time. It’s a little bit like saying that if we applied that compensatory model, […] if you were in a clinic and someone was going to stick a needle in your face, actually, if they turned up on time, they were wearing the correct uniform, they wore the gloves and they did all the PPE and they sterilised everything, they didn’t stick the needle in the right place, but they did everything else OK, we cannot let them pass at that stage. […] they have to do everything right, all of the time, effectively.

Skin peel_L4

The AC specifications, as the structural feature of all CASLO qualifications, were discussed from slightly different perspectives by different AOs. Several AOs spoke about the need for a level of specificity in the way the AC are written and used to support reliability and consistency in judgements, despite this requiring an analytic (and atomised) approach. To mitigate a potential negative washback effect on teaching and learning, these AOs encourage holistic delivery or assessment design, as previously discussed.

Just to clarify, we don’t have holistic assessments; we have a holistic approach to delivery […]. Obviously, a project will require them to research, to develop, to refine, to present, to analyse, to evaluate, you know, that’s the nature of a creative project. So that’s a holistic delivery approach. But the assessment is learning outcomes and criteria. They are producing evidence which is assessed against those specific things.

Creative L3

Others deemed AC lists to be useful for assessors to help with tracking assessment progress and for “gap identification”, as well as to help make the requirements transparent for the students. Some AOs suspected that some assessors were assessing holistically – then ticking off the criteria “to meet the paperwork” – though this was not easy to observe or quantify.

[…] it’s not quantifiable. What people do every day to what anybody actually verifies them doing can be very different things and there’s probably far more holistic assessment going on where they’re watching people on a course demonstrating things multiple times, […] but what you write down as an assessor and tick the boxes for will be to meet the paperwork and I think this is something that’s not as easy to reconcile.

First aid_L3

Interestingly, one AO in our sample effectively tried to ‘quantify’ some of the complexity involved in judging practical performances in their qualification. They described a hybrid^{[footnote 6]} approach to capturing a sense of contextually justifiable partial performance via assigning mark tariff to AC in some of their assessments and allowing for partial credit. They also used mark tariff without partial credit to, in effect, assign higher weighting to the AC that required full demonstration and where contextual factors should not play a role. As an example of a justifiable partial performance, they described a situation in which, on one occasion, a student might be observed carrying out a full consultation with a new client, but then later when that client becomes a regular, the consultations may become less extensive. They judged this to be acceptable because the student will already know the client and their relevant circumstances, and the assessor will take that into account by assigning partial credit rather than making the student go through detailed consultation each time just to tick boxes.

This AO also explained that, although this approach appears to involve a degree of compensation when judging, this was essentially contextual compensation that naturally occurs when judging anyway, and its presence was only made more explicit through an overt mark scheme. This type of contextual compensation was clearly distinguished from compensation that allows for certain skills not to be exhibited at all, which was not seen as appropriate in this qualification.

Another AO advocated that holistic professional judgement should sometimes involve an element of compensation across AC (or range) and that this may be legitimate in some instances even though there has been a tendency for AOs to operate mastery at the AC level as well as the LO level. Although this was not their current practice, they argued that this should be allowed to support more meaningful judgements even where certain skills might not be exhibited, if the overall weight of the evidence suggests a sufficient degree of competence in relation to each individual LO.

One AO suggested that the use of grade descriptors that encompass and apply across the relevant AC, rather than applying at individual AC level, helps to promote holistic assessment. Although they did not see this as directly allowing for compensation, this approach was likely to inherently invite judgements that consider the overall weight of evidence across the AC. This might, to some extent, de-emphasise individual AC and invite some compensation, as might the mastery requirement at LO level described above.

Others suggested that broader LOs or AC can help drive more holistic assessment by requiring students to demonstrate their knowledge or skill by looking at the wider picture. Relatedly, having broader LOs or AC was thought to promote the use of professional judgement by assessors, which was likely to be holistic if made against broad criteria. However, it was also suggested that such LOs can be more easily implemented in assessments that focus on the knowledge constructs rather than practical skills.

Question 3

Poorly conceived assessment tasks or events

Accepted Answer

Some critics say that having detailed and apparently transparent LO and AC specifications as standards to assess against, makes it look like the assessment process is extremely straightforward. However, assessors often fail to appreciate how hard it can be to elicit construct-relevant assessment evidence. For this reason, CASLO qualifications are vulnerable to being based on poorly conceived assessment tasks or poorly conceived assessment events that do not elicit the right kind of evidence against the specifications.

In our sample, 10 AOs recognised this as a potentially relevant problem for CASLO qualifications (including for all ‘dual purpose’ qualifications). Whether or not they entirely recognised the problem, all AOs discussed mitigations and protective factors that helped reduce the risk of this problem arising. The most prominent types of mitigations were support and guidance for centres, alongside QA, with some references to occupational and professional expertise, communities of practice and some qualification design processes and features, too. Real‑life or highly realistic task setting, contextualisation and the holistic nature of assessment situations, were also discussed. The latter were sometimes also mentioned as aspects that present further challenges to assessment design, particularly to comparability.

In discussing different mitigations, the AOs provided us with some insights into the different facets that they perceived as challenges to designing or implementing effective assessment tasks or events in their contexts. This showed an awareness that the assessment design process is anything but straightforward.

Perceived challenges in assessment design

Several AOs pointed out that different assessment methods may suit different types of constructs. Therefore, one of the key challenges of assessment design, especially given the amount of flexibility typically offered to centres, is to make appropriate choices as to what assessment method is the most suitable for each construct. Some AOs suggested that they would prefer externally set assessments where these met the purpose of the qualification, as they recognised the challenges that can potentially arise when centres are selecting assessment methods and devising their own assessment tasks.

[…] you say to the centre, well you use loads of methods in the most appropriate way and […] if they’re really good at it and they love the flexibility, they’re going to love it, but actually that’s pretty much a blank sheet of paper to be starting from and that can be quite daunting.

Adult care_L3

As to why developing assessments that are well aligned with their purposes may be challenging to centres, the AOs mentioned a lack of expertise in assessment design among centre assessors, and constraints in the resources that are accessible across centres. Some AOs offering ‘dual purpose’ qualifications, which are often delivered in college settings, recommended creating assessments such as assignments that would be not only construct relevant but also sufficiently vocationally aligned and engaging. Again, these require effort and thought to design. It was also suggested that it is more difficult to create (vocationally) relevant tasks at lower qualification levels. This was said to risk centres habitually using the same method for assessing a particular type of AC, despite the alignment between AC and method at times being questionable. There were also views that tasks in lower-level qualifications might be more difficult to pitch appropriately in terms of demand, sometimes resulting in these tasks being too demanding for the level.

I would say probably the area that might be more prevalent is not having relevant vocational scenarios or not linking what’s being asked of them to a vocational scenario. […] So, I think that is the area that can certainly be missed and can be more damaging than, I think, centres can understand at the time.

Creative_L2

One AO highlighted complexities in apparently straightforward and commonly used assessment methods, such as professional discussion (for instance, the use of leading questions), which need to be addressed for the assessment method to be implemented effectively. In the context of workplace-based assessment, challenges around identifying suitable real-life situations for assessment that would enable students to evidence all the relevant AC were also mentioned. Several comments suggested that assessing (theoretical) knowledge aspects was more challenging within the CASLO approach than assessing practical and other skills.

I’m not sure people always understood the difference between an assessment method and an assessment task. So, the assessment method might be professional discussion, well that’s fine, but then, as a centre, if I’ve got to go away and write the structure for professional discussion that hits all the assessment criteria, that asks the questions that aren’t leading, it suddenly becomes quite difficult to do it well.

Adult care_L3

In the previous section, we described a range of comments highlighting some threats to assessment authenticity that might arise if close alignment between atomistic lists of AC and assessment tasks or events results in the tasks or events failing to elicit integrated evidence of competence. However, the AOs also discussed the need to ensure that, while assessment tasks or procedures were able to elicit integrated evidence, they were still being sufficiently linked to the individual AC, which might be a particular challenge when designing cross-unit synoptic assessments. It was also suggested that AC which are too prescriptive can limit opportunities for contextualisation and innovation in assessment design by centres, highlighting the importance of specifying AC in the right level of detail to support optimal task design.

Throughout, the AOs suggested tensions between ensuring sufficient contextualisation of assessment and a degree of consistency or comparability between centres or students. While mostly accepting the likely trade-off required in this respect, some AOs recognised that more might need to be done to promote greater comparability, without necessarily giving up on contextualised assessments.

Support, guidance and QA

Support and guidance that the AOs provided to centres were referenced most frequently in our interviews as helpful in mitigating the potential problem of poorly conceived assessment tasks or events. Within this, the AOs discussed different support mechanisms, including early intervention and continuous support. They emphasised their availability and that of their EQAs to provide advice on the choice of assessment methods for different AC, as well as more specific advice on how to design assignments throughout the delivery of a qualification. As part of their early intervention and support, some AOs offered a specific service for centres which involves checking of assessment tasks (typically assignment briefs) designed by centres before administration. Most AOs also spoke about providing guidance documents on how to write assessments targeting specific constructs, or how to choose the most appropriate assessment methods. Some AOs also provided guidance and templates for checking the appropriateness of assignments during IQA and some optional training for centres focusing on assignment writing.

Overcoming the potential issue of assessment being too atomistic was frequently mentioned as the focus of AO support and guidance for centres. However, given the recognised challenges in relation to the clarity of mapping onto individual AC in holistic approaches, some AOs also said that they provided guidance in relation to assessing holistically whilst ensuring each AC is met. This was considered especially challenging where the same practical tasks covered multiple LOs or units but might be used to provide evidence for AC which might have a different focus in relation to different LOs or units. Some AOs provided guidance to ensure that the holistic tasks which centres might use do not become overly complex or demanding for students.

[…] what we try and provide guidance for our centres to do […] I’m talking about hitting multiple assessment criteria with nice scenarios where learners can apply what they’ve learnt and can apply multiple different skills in order to reach an outcome […] It’s not an easy thing to do, but it’s something that we think benefits the learners in the long run.

First aid_L3

Several AOs spoke about providing different types of exemplar assessments or templates, such as assignment or project brief templates. As with examples of student performances discussed previously, AOs mentioned resource limitations that drive some of their decisions about the extent of exemplar assessment materials that they might provide. For instance, one AO described developing exemplar materials for selected units that could then be used by centres to develop further assessment materials at a similar standard for other units. Another focused on what they perceived to be more challenging constructs to assess appropriately, such as knowledge aspects. However, some AOs chose not to provide exemplars at all, particularly where they could not be easily contextualised, but also where there was risk of templates not being sufficiently adapted by centres.

We have a project brief template, which we developed as part of the qualification development […] and is accessible for centres to use. And obviously when they put evidence or tasks in all the relevant sections, we feel that that constitutes a really solid accessible and progressive project brief for students.

Creative_L3

It was also apparent in AO comments that, while they sometimes encouraged and advocated certain approaches to assessment design, these were rarely compulsory. The quote below expresses what was implicit in many AO comments, which is that there are potential benefits in AOs investing in upskilling centres, so that centres can reap the benefits of flexibility of the CASLO approach while still ensuring high quality assessment. Nevertheless, some AOs suggested that centres often have a preference for off-the-shelf assessment materials to support them, potentially because they do not have the expertise to develop appropriate assessment materials, or time to develop the required expertise.

[…] by allowing a centre to design its own brief, that gives that flexibility and that ability to meet local needs but still showing that the learning outcomes have been satisfied. I think the approach was more a case of, if we give the tutors the skills they need to design really good assignment briefs, that’s better than almost spoon-feeding and saying, this is the way to do it, so that was the approach that we took […].

Construction_L5

AOs also described different aspects of their QA processes that are aimed at mitigating potential problems with inappropriate assessment tasks or events by more explicit monitoring of centre practices. Several AOs spoke about the centre approval stage, when they have a discussion with centres about centre expertise gaps, whether they might need further support in any areas, and what their proposed approach to assessment would be.

IQA processes were flagged as an important mitigation, as there is an expectation that assessments would be internally discussed and verified before administration. Some AOs said that IQA checks would cover a range of assessment design aspects, including relevance to the local context and construct targeting, as well as accessibility to students and any modification to AO-set tasks.

We’d also then expect the IQA within the centre to pre-verify those assessment materials, so to do that check before they’re used to make sure they’re appropriate, that they’re inclusive, that […] a learner completing that assessment, they would meet the required learning outcomes, assessment criteria and that would then be delivered.

Adult care_L3

Another important mitigation that most AOs spoke about concerned EQA processes. Unsurprisingly, most of the EQA checks that related to assessment quality would be conducted by AOs upfront rather than just before or after certification, to prevent students from being assessed using inappropriate tasks. Here, again, it was not always easy to separate EQA monitoring from the continuous support and guidance that the EQAs provide. However, some AO comments suggested more explicit attempts to sample and monitor assessment development at different stages of the delivery cycle, rather than just providing feedback at centre request. As part of their EQA checks, several AOs spoke about scrutinising not just the tasks as such, but also centre assessment development and IQA processes.

We ask to see 25% of the assignment briefs. […] But it’s prior to them using the assignment brief. And because we’ve seen 25% and approved them, we deem that they’re able to write at that particular level. But it may be that, obviously, one of those assignment briefs aren’t pitched at the right level, and at that point we’d ask to see the assignment brief for that particular unit where there is potentially an issue, and we would give them feedback that way.

Housing_L5

And to us, monitoring the effectiveness of the brief writing and the internal verification process for the centre that checks their own briefs, that is key in that, I think that’s really important.

Creative_L3

Nevertheless, some AOs also spoke about issues with task appropriateness emerging during final moderation, for instance, where the task did not elicit performances that could earn higher grades. At that point, the centre would be asked to provide further opportunity for the relevant students to be assessed on more tasks. In addition, centres would be assigned a higher risk rating and, therefore, provided with additional support and additional monitoring in the next academic year.

Occupational or professional expertise, experience, attitudes and communities of practice

While most AOs discussed practitioner occupational and assessment expertise as a requirement for avoiding potential issues with poor assessment design, it was also suggested that this expertise was complex and was not easy to develop. Some AOs recognised that there are going to be inevitable differences in assessment expertise between centres, with strengths and weaknesses in different areas. Interestingly, very few AOs specifically mentioned assessor qualifications as a way for assessors to enhance their assessment design expertise. Those that mentioned them suggested that these qualifications were useful and effective in increasing assessor competence, but that they were not a requirement in all sectors nor with all AOs.

Some comments revealed that, for AOs, an important aspect of assessor expertise was to be flexible and able to tailor assessment to individual student needs. This involved recognising the most appropriate way to elicit knowledge from students where some assessment methods might present a barrier, for instance, using oral questioning rather than written tests with some students in the workplace context. While in most contexts the AOs discussed the need for tutors or assessors to ensure appropriate targeting of the relevant constructs when designing assignments, in the qualifications assessed in the workplace, effective planning was additionally often pointed out as an important facet of assessor expertise. The ability to plan assessment events effectively was deemed to facilitate appropriate assessment that does not negatively interfere with other activities, including teaching and learning, and, thus, the overall learning context.

It’s down to the assessor to devise an assessment. […] it’s about going right back to the beginning with the learner, sitting down and doing a plan. And so, it’s down to the assessor really to play as they go along with the learner how they’re going to be able to demonstrate that evidence. […] [and] arrange with the employer, not just with the learner, where are we going to get the opportunities to get this sort of evidence […]

Chef_L2

It was also suggested that assessor expertise and effective assessment strategies and resources are often developed over time, as centres become more experienced and familiar with specific qualifications, which, in turn, helps to mitigate the problems with assessment design. Sometimes, resources developed in the context of related or precursor qualifications may also be helpful. This links to the idea of communities of practice. The AOs that flagged communities of practice in this context actively promoted them by facilitating meetings or forums for centres to share best practice and resources. Some AOs also pointed out that assessors often have positive attitudes towards creating engaging and high-quality project briefs, and towards their own professional development. This is because assessors feel professionally invested in sharing their expertise and skills, both as practising professionals with their students and as assessment practitioners with other assessors.

We’ve got lots of really capable centres that write really superb assignments, and our focus has been on those centres sharing with one another, and creating those communities where tutors are able to do that. Obviously, there are some that don’t really want to share some of the stuff that they’ve worked really hard to build and make. But there are many that do share. […] We have subject advisers that manage forums where people do share and there is that bit of professional discussion around good assignment writing. […] fundamentally teachers are in a community of practice where professional development is really important, and really active engaged tutors will push one another to make the best assignments that they can.

Business_L3

Real-life task setting and contextualisation

Several AOs, particularly those delivering the ‘confirm competence’ qualifications, which are often assessed in the workplace, discussed the mitigating effect of real‑life task setting. In their view, the fact that assessment is not simulated, and thus not “designed”, in itself helps overcome some of the potential issues with poorly conceived assessment, and inherently ensures a high degree of validity as well as more holistic assessment.

I think that’s a strength of our qualifications because they are delivered in the workplace. You know, back to the college environment, yeah you could set up a poorly designed assessment task, here’s a 600 by 600 straightforward, bog‑standard window, take that out and put it back in, right, you can fit windows. So, I think because it’s workplace assessment. It’s live installations, it’s all the bits we spoke about and customers, dogs running about and yapping, having to move children out the way, vulnerable adults, and all that. You couldn’t get that in a college, in a simulated environment […].

Fenestration_L2

More generally, many AOs discussed both the need for and the advantages of real-life contextualisation to make assessments more effective in eliciting appropriate and relevant evidence. Through that relevance, assessments were also thought to be more supportive of “less academically minded” students whom they believed tended to perform better on such assessments than on more abstract ones. While generally strongly arguing in favour of contextualised assessments, the AOs also recognised the potential challenge this brings to ensuring an appropriate degree of consistency and comparability between different centres or students within centres. Some AOs argued that a degree of inconsistency is inevitable as well as, to some extent, acceptable as a trade-off for the advantages of contextualisation.

I think it is a fair point, a fair criticism at times, but […] this isn’t academic assessment, this is vocational assessment, so this is people being assessed about how well they can do a job. And people do jobs in different ways, employers do expect different things from their employees. You know, things will be done differently, learners will be assessed differently. […] In some ways it’s a strength, up to a point where you might cross a border and go – it’s getting a bit inconsistent now. But I’m not sure every learner has to be assessed in exactly the same way for it to be a valid assessment.

Adult care_L3

Some AOs suggested that, in their qualifications, contextualisation should not affect comparability because the fundamental skills which are assessed are, essentially, context‑independent. Others, offering qualifications typically delivered in college or school settings, explained that their assessment tasks typically involve cohort‑level contextualisation, rather than contextualisation at the level of individual students. Assessment tasks would thus be the same across the cohort rather than catering for individual student interests or particular familiarity with certain employers, for instance. This would reduce the range of assessment task variation and mitigate the potential issues with comparability.

I have seen for sub-cohorts there being something that’s very specific to a learner, the skills and capabilities that they bring, or their desires post getting their qualification, maybe they already have a connection with an employer. But I think that’s the exception, that’s not the rule. Centres are delivering to large cohorts. And so that would really create quite a lot of assessment and teaching burden if they were to do that.

Business_L3

AOs also flagged certain parameters that imposed a degree of consistency in relation to the range of specific skills and conditions under which those needed to be exhibited for assessment. This was mentioned in the context of qualifications with a strong emphasis on client safety, such as those in the beauty sector. However, these AOs also noted challenges with ensuring that an appropriate range of conditions is available for all students and suggested that there can be some flexibility to allow students to complete qualifications in a reasonable amount of time. In some qualifications in the beauty sector, this might involve students demonstrating skills on non‑fee‑paying clients, thus, somewhat relaxing the real-life context. Other AOs spoke about allowing students to be questioned about situations that might not easily arise in the workplace, or presenting witness testimonies or other types of indirect evidence where it was difficult for assessors to observe them carrying out a practical demonstration of skills.

Qualification and assessment design processes and features

Some AOs flagged aspects of their assessment design processes that help to ensure that exemplar assignment briefs provided to centres are of appropriate quality. These included involving employers, as well as international representatives where qualifications are delivered outside of England, in qualification development or review panels working on the development of assignment briefs to ensure their vocational relevance. Others flagged regular reviewing of assessment effectiveness in centres and amending assessments where issues are found.

Several features of the CASLO approach were also mentioned as helpful in mitigating potential problems of poor assessment design. Some AOs mentioned the relative transparency of the AC as going a long way towards ensuring that the tasks targeting these AC are appropriate. Another AO noted that command verbs in the AC provide some pointers about the nature of the assessment tasks, though recognised that the mapping between command verbs and specific design choices is not entirely transparent. One AO emphasised the need to ensure sufficiently broad AC, so that they would not overly restrict the potential for contextualisation and creativity when designing tasks to elicit different kinds of evidence.

Yeah, a couple of reasons why I don’t entirely agree with that one is that the assessment criteria themselves form the basis of tasks. So, […] there’s plenty in there for them to create a project out of as well in many cases.

Creative_L2

One AO suggested that the mastery model in CASLO qualifications helps to motivate centres to design appropriate tasks, rather than “skimp” or “play fast and loose trying to hit assessment criteria”, given the stakes that mastery imposes if students did not meet certain AC due to task inadequacy. There were also views that skills-related constructs, perhaps particularly basic technical skills, are more straightforward to assess reliably and validly than constructs such as knowledge. This implied that the potential problem of poorly-conceived assessment is less of an issue for CASLO qualifications that largely deal with skills constructs.

[…] you can either titrate in chemistry or you can’t, it’s a fairly basic skill. However, ask someone to explain the theory of an atom and what you get and what you ask them to provide can be so diverse, what level of knowledge somebody has achieved is very, very different against the same assessment criteria and that’s where they fundamentally, I think, fall down and the criticism is acceptable […]

First aid_L3

Hybrid aspects

Despite apparent awareness of potential challenges to centres designing and administering assessments, few AOs went down the route of introducing external assessment into their qualifications. Those that did have some externally set and/or marked assessments explained that this was due to stakeholder or accountability requirements. They argued that such assessments are perceived by employers or other stakeholders to be more reliable in confirming aspects of competence that were seen as essential for certain sectors.

One such example was Hairdressing_L2, where health and safety-related knowledge was assessed by a compulsory AO-set and internally marked multiple-choice test. If a student does not pass that test, the AC that are not achieved need to be met via other means, usually through questioning or other evidence collated in their portfolio.

[…] It goes back to this thing around the employers are just absolutely obsessed with […] safety and basic skills, if they’re, sort of, putting somebody working on a client, there are insurance factors, there’s all sorts, but even if something goes wrong with a client once, that’s sort of a real loss of business, so they’re very much attached to this making sure the technical skills are basically competent and safe, and they really do want these assurances or they just won’t employ […] the learners […].

Hairdressing_L2

One ‘dual purpose’ qualification in our sample incorporated some AO-set and marked assessments to enable this qualification to be included in the Department for Education Performance Tables. The AO offering this qualification suggested that their external assessment, which involves complex, creative and practical tasks, challenged the common notion of external assessment as involving only “paper and pen tests” or multiple-choice tests. They suggested that it would be beneficial to think more widely about different ways in which external assessment could be designed to retain sufficient validity and relevance to students, which are both deemed inherent in the more traditional, internal and contextualised task-based assessments within the CASLO approach.

External assessments are designed, the team’s designed them in a way that we felt would be accessible and meaningful to vocational learners. So, we have tasks in many of the assessments. They’re not just 2 hours in an exam hall with one paper booklet kind of stuff. […] I think the challenge from a design point of view is finding the right balance, and it’s finding the assessments that we classify as external which is right for individual subjects, because what’s right for this one may not be right for that one. So, it’s thinking a bit wider about that.

Business_L3

Question 4

Lenience and malpractice

Accepted Answer

Some critics say that the imprecision of AC can act as a smokescreen for assessors, allowing them to be intentionally lenient towards students who have not quite reached the qualification standards, giving undue benefit of the doubt. This can be exacerbated towards the end of sessional courses, for students who are just about to leave, but who still have not quite achieved all their LOs. Occasionally, assessors may try to pass students who are a long way from meeting the qualification standards, resulting in malpractice. They can get away with this – according to some critics – because it is extremely hard to detect and correct inaccurate assessor judgements under the CASLO approach.

Most AOs in our sample recognised both of these as potentially relevant problems. The AOs were slightly more likely to recognise the potential malpractice problem (around 78%, N=11) than the potential problem of lenience (around 64%, N=9). The latter was recognised for all 5 of the ‘dual purpose’ qualifications in our sample. However, it was recognised for only 4 out of 9 ‘confirm competence’ qualifications.

The following comment reflects some of the commonly held views about the potential sources of lenience, as well as commonly used mitigations.

I think you’re always going to have that, because if you’ve, let’s say you’ve been teaching on a programme for 2 years, you’ve got to know those learners and you will always have some that are not going to achieve the qualification in full […] despite how hard they may have worked through that time, and that human element kicks in where you think, oh, they’ve worked so hard, […] there’s got to be here somewhere where we can let them, but I think the measures are in place thereafter with centres, if they’ve got internal verifiers, you’ve got your external quality assurers, there’s those checking-in points just in case that […] does occur.

End of life care_L2

These potential problems were generally discussed against the backdrop of potential perverse incentives related to funding, accountability and other pressures, that were seen as likely to influence centre behaviour. Most AOs also implied that a certain extent of unreliability in the system arising from lenience or malpractice would inevitably remain despite best efforts to eradicate it. This was due to limited resources to QA every single student result, as well as due to the complex nature of the judgements being made by everyone involved, including assessors, IQAs and EQAs.

Most AOs discussed at least 3 different mitigation types for these potential problems. All the AOs flagged QA as a key mitigation. It was interesting to observe that practitioner attitudes, including practitioner integrity, sense of pride or vocational passion, were mentioned almost as frequently as QA. Support and guidance for centres, but also aspects of these qualifications that were deemed to support learning, such as absence of time constraints or flexible delivery, were also frequently mentioned. Several other mitigations such as various disincentives external to qualifications, as well as some qualification design features, were also discussed, though by fewer AOs.

Incentives and disincentives

The AOs mentioned a range of potential incentives that might influence centres and their staff to be lenient or commit malpractice. The presence of these incentives, often related to funding or accountability measures, was deemed to complicate the task of quality assuring qualification results. In contrast, the perceived absence of such incentives helped AOs to manage risks around lenience and malpractice in their qualifications. In addition, qualifications that did not provide a licence to practice were seen to be more resilient to these 2 problems.

While most AOs were clearly aware of different potential perverse incentives, some also included potential factors that might drive perverse incentives into their risk‑based sampling models, helping to ensure that centres deemed susceptible to these issues receive extra monitoring.

Turnover of staff, Ofsted inspections, there are probably others, registration volumes, if centres are registering on time […] Financial health could also be an indication, because that might create an incentive to do the wrong thing, if the financial health of an organisation is poor. That’s maybe not schools and colleges […]. That’s more work-based learning […].

Business_L3

Schools and colleges faced potential perverse incentives in the shape of accountability pressures, performance-related pay or funding patterns focused on achievement rates. Private training providers were also mentioned as potentially vulnerable to fraud for financial gain, especially where they might not be able or willing to invest sufficiently in robust IQA systems nor employ full-time IQA staff. It was also mentioned that certain roles that are normally fundamental for QA in the CASLO approach, such as IQAs, are potentially under a lot of pressure to ensure expected achievement rates across all types of provision.

Speaking about [this qualification] though, which is primarily delivered in schools and colleges, and therefore maybe […] isn’t so at risk of assessors maybe taking advantage of that for financial or other gain, it’s really tricky would be my answer, because there are incentives and motivations for centre assessors to maintain achievement levels for example, […] That is set at centre level, or it’s other initiatives around performance related pay and things like this.

Business_L3

QA and support and guidance for centres

Given the awareness of these different incentives, as well as the overall recognition of the potential for assessors and IQAs to be lenient or commit malpractice, the AOs outlined a range of QA practices that served to mitigate these potential problems. The comments below sum up most of these practices as well as AO perceptions about the likelihood of such problems arising in practice given these mitigations, especially with extensive EQA (particularly with small cohorts) and with monitoring of internal centre processes via IQA.

There’s always going to be human judgements, human errors, but having quite a robust moderation process with internal moderation, assessors supporting each other if there’s multiple assessors, and then the external moderation as well, and not releasing certificates until there has been moderation and scrutiny, this mitigates the risk of that happening, I would say.

Skin peel_L4

[…] I suppose, in theory, if you had an assessor that was determined to take that approach with one learner then possibly the chances of them getting away with that are fair. If that was to happen with the majority of a cohort, I think it would be highly unlikely that they would be able to get away with that. And especially if that was something that was happening regularly over time […]. We have a lot of check in points […]. There might only be a handful of learners so the sampling can look sometimes almost at the entire cohort. […] it would be a bold and brave decision I think on the part of the assessor to try and get away with that given the risks if we catch them.

Creative_L2

As in relation to the potential problem of inconsistent judgement, multiple layers of QA, alongside standardisation, were deemed necessary to mitigate the potential risks of lenience or malpractice. Within this, centres were expected to have their own responsibilities for checking for malpractice, for having appropriate conflict of interest policies and procedures and for observing relevant AO’s appeals and complaints processes. IQA was seen to be fundamental in this, and some AOs suggested that the main focus of EQA is the IQA practice rather than student work. Were these to raise alarm bells, then the centre risk rating would be increased, and they would be subjected to more monitoring.

Most AOs spoke about different aspects of their EQA processes as deterrents against lenience and malpractice. These included a risk-based sampling approach, which also includes some random sampling of student work within each centre, potential for unannounced EQA visits, as well as potential punitive measures to address lenience or malpractice where it is discovered. Most AOs also mentioned not giving centres DCS status, so that student work had to be moderated prior to certificates being given to students. The potential burden and workload for centres if students had to be recalled and reassessed (were issues to be spotted during moderation) might also serve as a deterrent against committing malpractice.

[…] and there’s the risk that if they are conducting malpractice that they’ll lose their jobs, that’s a really harsh reality that actually happens. And not just their jobs, their entire colleagues’ jobs, so we could shut that centre down and everyone in that centre is out of a job. So, there’s quite a lot of deterrents there.

Adult care_L3

While most AOs agreed that lenience or malpractice could occur in practice, some AOs did not think that these are difficult to spot in assessment evidence by EQAs. As EQAs are seen as experts in the field and know where the standard should be, they are expected to be able to assess the appropriateness of the nature of evidence provided to support assessment decisions and whether this evidence was generated by the students. However, several comments suggested a need for triangulation of evidence from different sources in addition to student work, including looking at assessment management processes, speaking to staff and students, observing assessment taking place or gaining intelligence from centres that operate in a tight, “self-policing”, community of practice. It was also pointed out that EQAs may sometimes be faced with the need to weigh up evidence across the entire qualification and attempt to form a holistic view of its sufficiency and appropriateness, as opposed to querying potentially weaker evidence for each AC or LO. These comments suggest that spotting evidence of lenience and malpractice based on one source of evidence only may not be entirely straightforward.

I think in terms of the leniency, again, that’s where the external examiner comes in, having been standardised themselves, to spot and, ultimately, not allow certification where that’s seen. […] we’re looking not just purely at student work, but those things like the assessment management processes that go alongside it, speaking to staff, speaking to students, so it’s not purely the student work, there’s some triangulation there as well, the management of the academic standards…

Construction_L5

In a similar vein, some AOs suggested that making judgements on the borderline between 2 grades, and being able to argue a position on that, is difficult for both assessors and QA staff. Some comments implied that AOs might be more concerned about more significant discrepancies, such as where a distinction grade was given where moderation suggests it should be a pass, than about the borderline between adjacent grades, which might be more difficult to spot or challenge. Either way, the AOs emphasised that the focus of moderation was to ensure that assessment decisions made by the centre were appropriate and justified, supported by evidence, made faithfully, and that any borderline cases were acknowledged.

[…] it’s difficult because everything is judgement based, isn’t it? But I think if an assessor makes a judgement that might be weak, but does cover what is needed, I guess you’ve got to have a clear justification as to why you would argue that that’s not enough […]. But I think it’s whether an independent person looking at that work could challenge that in a way that would make the assessor actually rethink that decision. I think that’s how it usually is and the EQA is the final arbiter of that really because they’re the one who does see the standards across all the centres. […] But it would be for the assessor or IQA to convince the EQA really of their judgement and whether that’s appropriate.

Adult care_L3

Given the complexity of the judgements and decisions that EQAs have to make, it was unsurprising that some AOs suggested that there were challenges associated with the nature of EQA work. While some challenges related to ensuring consistency of EQA practices and processes across centres, others were about ensuring that EQAs remain unbiased and objective in their judgements towards centres that they might know well. These issues were partly mitigated by implementing a hierarchy of EQAs, with more senior ones monitoring more junior ones. Another mitigation that was suggested was rotating EQAs across centres, so that no EQA would monitor a centre for more than 3 to 4 years.

A few AOs spoke about analysis of results patterns as a way to monitor changing trends in centre results that can indicate potential lenience or malpractice, although any action based on such analyses would happen only following the release of results. Some mentioned monitoring stability of grade distributions in centres over time (such as the proportion of distinction grades), whereas others compared results of internally and externally assessed components and interrogated radical differences between the two. As these AOs pointed out, such analyses needed to be interpreted sensitively, as there could be other reasons for potentially aberrant patterns which may not reflect lenience or malpractice, such as improved teaching in centres or learners performing better under some assessment models. The AO offering the First aid_L3 qualification suggested that high pass rates in their context may not necessarily be indicative of lenience in standards, because a large proportion of their cohort already have this qualification but are obliged to refresh it every few years for legal or insurance reasons. One AO suggested that monitoring of dwindling registration numbers over the lifecycle of a qualification might suggest that the qualification might not be perceived as valuable and imparting genuine skills needed for the workplace, potentially due to lenient application of standards.

As with other potential problems, support and guidance for centres was seen as an important mitigation of potential lenience or malpractice, in particular where the centre was new. The likelihood of these problems arising was also reduced by continuous support through regular EQA visits and feedback on various aspects, including interpretation of standards. Some AOs suggested that timely planning by centres and EQA support with planning can help to reduce the chance of last-minute pressure to collect evidence or make assessment decisions that do not meet the qualification standards. AOs recognised that sometimes there can be problems with how centres capture and present evidence of AC achievement rather than with intentional lenience or malpractice. To address this, some AOs provided guidance for assessors (to support later EQA) on how to write up observation reports and how to elaborate on the evidence that supports their judgements on whether a student meets certain criteria.

[…] we’ve had some chuckles over the years with some things that you see. “In my opinion, this man can fit windows.” Well, that’s not what we’re looking for, is it. So, they’re given freedom but with quite a tight rein on them so to speak. We know when to pull that rein in a bit.

Fenestration_L2

Attitudes

Whether or not the AOs explicitly recognised the potential problems of lenience or malpractice, they emphasised the protective effect of certain practitioner attitudes that would likely minimise the risk of these problems arising. Several AOs spoke about the need for a degree of trust to exist between the actors in the qualification system. Relatedly, it was also felt that there had to be some reliance on practitioner integrity and professional standards to guard against malpractice, and a recognition that some of the practitioners occasionally did not live up to those standards.

This professional relationship and trust that we have with our providers has to be there, because we depend on that, otherwise we would have to moderate everything and touch everything. And […] we don’t want to move away from the design of the qualification or the benefits of the qualifications in doing that, so this professional trust has to be there.

Business_L3

I think we all know that the assessment process has to be built on an element of trust. You know, we can’t be there at every assessment on every occasion.

Fenestration_L2

Some AOs also pointed out that assessors and tutors are often invested in their students and, therefore, strive to ensure appropriate qualification delivery in their best interest. It was noted that they often have a sense of professional passion and pride in relation to their teacher or assessor role and their sector, and are protective of their reputation, which also reduces the chance that they would assess in a way likely to lead to lower standards. This, in turn, may sometimes lead to high expectations from students and assessing to higher standards than required in the relevant qualification.

Positive attitudes that mirrored those of teachers and assessors were also recognised among students. Students’ passion and pride about their qualification, with students being keen to prove their skills and not appreciating being assessed on trivial tasks or against lower standards, were suggested by some AOs as another protective factor against lenience and malpractice. Adult professionals, in particular, who might be taking higher level professional qualifications, were perceived to be protective of their professional reputation, which for them was a strong disincentive against cheating or malpractice.

[…] but I think, within this particular subject area, when people get to the point at which they are signed off as being allowed to deliver training or assessment, they’re so incredibly proud of that and so aware of their own reputation within the industry, that actually the idea of standards slipping is not, I’d say it’s almost the opposite problem we have, that people want trainees to be better than actually they’re expected to be, […] and the standards might be higher rather than lower.

Skin peel_L4

This means a lot to a lot of learners. They’ve been working hard for 12 months, they’ve got to this [final culinary challenge unit], [and if assessor said] – just make us some Marmite on toast, that will be fine – they’d be gutted.

Chef_L2

Relatedly, some AOs noted that despite perceptions that employers who deliver qualifications to their own staff (“employer centres”) might be incentivised to pass students that did not meet the standard, they believed that this was unlikely to be the case. This was attributed to the positive attitudes of employers towards employee upskilling and delivering qualifications “for the right reasons”. Employers were perceived to genuinely want to improve their employees’ skills, so that they would become “more effective and productive for the business” rather than just to meet the AC to get a formal certification. It was also implied that where employers relied on private training providers to deliver qualifications, they had expectations that the training provided would lead to employees operating at the appropriate level. If this was not the case, this might threaten the reputation of the provider and the relationship between the provider and the employer.

Additionally, a few comments suggested that potential biases arising from assessors personally knowing the students might be reduced where assessment is carried out by external (visiting) assessors^{[footnote 7]}, which was common in some workplace‑based qualifications delivered by private training providers. In such cases, it was relatively uncommon for students to be assessed by their own managers or supervisors. However, this still did not preclude external assessors becoming familiar with and invested in the students that they might assess on multiple occasions.

Supporting students and learning

The AOs mentioned the mitigating role of supporting students and learning in terms of flexibility in delivery as frequently as support and guidance for centres. In their comments, AOs most often referred to aspects of flexible delivery such as absence of time constraints on learning, multiple assessment and resit opportunities and unit-level achievement or credit.

[…] we don’t put any constraints on learners or centres to say you must finish in X number of weeks. […] So, if someone hasn’t met the standard then they’ve not met the standard yet. There’ll be more assessments planned in. I can see the criticism in terms of if it’s a college delivered qualification, because they’ve obviously got finite timescales, but in a workplace assessment it doesn’t really impact on us.

Fenestration_L2

[the approach] allows for students to resit the assessment where required. It’s not all end loaded. So, I think that in itself provides opportunity for the student to be allowed to fail and opportunity [to] resit the assessment.

Teaching support_L2

If they’ve met the standard of the first unit but not the second, […] they can get a unit certificate […], but they won’t achieve the full qualification if they haven’t met the standard, no matter how nice a person they are […].

First aid_L3

These were believed to have a disincentivising effect on centres, making them less likely to rush students through qualifications before they genuinely met the standards. Among these, absence of time constraints on learning, and assessment when ready, were most frequently mentioned as helpful in mitigating potential problems of lenience or malpractice, particularly in occupational qualifications delivered in the workplace.

However, some AOs also noted that there were certain pressures that can impose time constraints even if the design allows for the qualification to be delivered in less constrained time scales. These were employer requirements to complete the qualification by a certain time or funding arrangements which might affect decisions of providers in relation to length of the programme. This resonated with various comments mentioned earlier referencing potential perverse incentives that might affect behaviours.

Qualification and assessment design features

In the context of the potential problem of lenience, the requirements of the mastery approach and its multiple hurdles were mentioned as mitigations, particularly in relation to the pass/fail threshold. Some AOs considered continuous internal assessment to be more resilient to malpractice than terminal internal assessment where students might take an internally administered exam at the end. This was because continuous internal assessment involved build-up of evidence about student competence over time, which could not be easily counterfeited.

[…] the pass/fail threshold is effectively protected through an inbuilt protection mechanism, which is the fact that the students need to pass those hurdles of the developmental units […] so we can be confident that they’ve met that minimum competence threshold for a pass before they’ve got to that final unit […]

Creative_L3

[…] where it’s exams at the end, we’re seeing more malpractice there because a lot of the onus is taken off the centre there, and we see them trying to push learners through where they shouldn’t, rather than on these where it is part of that whole entire process that they’ve got to make sure it’s followed right, because it does come back on them more, so yeah, don’t tend to get many issues.

Adult care_L3

Interestingly, some AOs delivering ‘confirm competence’ qualifications seemed to suggest that the nature of the construct of these qualifications meant that there were “no borderline performances” at the pass grade boundary. In these qualifications, the threshold between knowing and not knowing how to do something, or whether someone addressed or did not address pass criteria, was thought to be clear, and thus unlikely to serve as a smokescreen for lenience. Where qualifications are graded, some AOs suggested that there was more “intellectual angst” around how well AC were addressed in relation to higher grades than whether they were addressed at pass grade boundary. This implied that it would be relatively straightforward to detect lenience or malpractice, at least at the pass grade boundary, in CASLO qualifications.

I think as well […] that with competence qualifications there are some things in there where subject matter experts, the tutors, the practitioners who are really understanding, will say well there isn’t really a borderline, they’ve done it right or they haven’t.

Adult care_L3

Question 5

Inappropriate support

Accepted Answer

The literature identifies problems related to the blurring of lines between formative and summative assessment in the CASLO approach, which typically uses internal, and often continuous, assessment in both workplace and college settings. This can lead to students being given too much support by tutors or assessors (intentionally or unintentionally) and then being assessed as having achieved a higher standard than they would have achieved independently.

The AOs discussed different mechanisms through which inappropriate support might arise which included:

overly directive feedback and “handholding” through assessment, often in conjunction with repeated assessment
overly scaffolded tasks
tutors completing or providing answers to summative assessments for students

While the latter mechanism was considered to be clearly in the realm of malpractice, the other 2 were deemed to, at least in part, spring from the best intentions of teachers to support their students. They were also deemed to be related to deficiencies in teacher expertise in providing appropriate feedback or designing appropriate tasks.

These different ways in which inappropriate support could arise suggest that this potential problem interacts in important ways with the potential problems of lenience and malpractice, as well as with the potential problems around poorly conceived assessment tasks or events, which we discussed in earlier sections. Ultimately, all these can be seen to be related to the imprecision of AC, which allow some leeway for assessors both to design the tasks and interpret the standards in ways that may, inadvertently or intentionally, reduce the level of demand to which students are assessed.

In our sample, of the 13 AOs that were explicitly asked, 11 recognised inappropriate support as a potentially relevant problem while one did not entirely recognise it, and one did not recognise it at. The problem was recognised as potentially relevant for all ‘dual purpose’ qualifications and for 6 of 8 ‘confirm competence’ qualifications.

The proposed mitigations mostly focused on QA alongside support and guidance for centres, and often involved advice and guidance about providing appropriate level of feedback, and in some cases also about writing appropriate assessments that are not overly scaffolded. Potential mitigating effects of assessment contextualisation were also discussed, as were some restrictions to flexibility of assessment administration such as limited resit or resubmission opportunities.

QA and support and guidance for centres

All except one AO that recognised this potential problem, and one that did not entirely recognise it, spoke about IQA and EQA as important mitigations. Within this, some AOs suggested that, during their monitoring, EQAs consider the appropriateness of the feedback given to students during their formative and/or summative assessments. Some AOs suggested that the feedback should not be overly directive or prescriptive, and where it is, the AOs might intervene to address that. Others suggested that EQAs “get to know the staff within centres” and how they work with students, which gives them insight into whether they are likely to be over‑supportive.

Obviously when we do moderation […] centres have to have available to us their course folders for the whole qualification. We are able to say, we’d like to see assessment feedback for formative units, IV documentation, bits and pieces, other evidence. […] So, we’re able to pick up if they’ve been overly directive in their feedback, overly prescriptive, and we will then take interventions to mitigate that.

Creative_L3

One AO suggested that inappropriate support can be detected by EQAs by noticing similar answers given by different students to the same AC. Nevertheless, the occurrence of similar student responses in an assessment should always be carefully scrutinised without prejudice, as the same AO also noted that in lower-level qualifications this may be a consequence of the purpose of such qualifications, which is to engage and support students as much as possible.

In addition to EQA monitoring, support and guidance for centres on how to give appropriate feedback and avoid inappropriate support when conducting different kinds of assessments, including observation in the workplace, was also mentioned by the majority of the AOs. As in relation to other problems, continuous support and early intervention was typically provided by EQAs through regular visits and feedback to centres. Some AOs mentioned specifically advising centres about assessment and delivery planning, so that students are not assessed summatively until the centres are confident that they have all the skills needed to complete the assessment, rather than attempting multiple resits. This was intended to help centres with maintaining a degree of separation between formative and summative assessment. One AO emphasised the need for centres to strive towards separating formative and summative assessment as far as practically possible, and they provided feedback to centres regarding that regularly. They recognised that sometimes summative events can turn into formative ones when students do not achieve all the criteria, and were keen to ensure that summative records do not contain formative feedback that might represent too much support for the next summative assessment.

But we do make it clear to all of the centres, you know, there’s coaching and training and there’s assessment and the two are different. While the same person can do both, you can’t do both at the same time. […] So, it’s a fine line but they are aware of the difference.

Fenestration_L2

[…] we tell centres as well they shouldn’t be putting assessments in front of learners until they’ve got all the skills needed to complete the assessment. So that, along with the fact that we tell them not to reassess and reassess is, hopefully, you’ve got a learner who’s in the correct place and isn’t going to be assessed multiple times in order to achieve.

Creative_L2

In First aid_L3 qualification, which is delivered over the course of 16 guided learning hours, the AO also encouraged assessors to separate instruction and assessment opportunities. However, the “progressive teaching methods” that they spoke about and advocated to their centres appeared to fundamentally involve continuous assessment, with multiple opportunities to achieve the AC during the (typically 2‑day‑long) training course. This potentially presented a challenge to the effectiveness of mitigations based on separation of formative and summative opportunities.

[…] the assessors within our centres, especially for this qualification, tend to be pretty good at drawing a line as to when supported practice is over and when you are doing it for real […] And the standard practice that we encourage is that if the assessor sees somebody get something out of sequence, they’ll just tell them to stop, take a moment, let their partner who they’re working with have a go and then come back and do it again, but there’s multiple opportunities during the day for people to meet each criteria and, […] it tends to be accumulated throughout the day and that’s because of the progressive teaching methods that many of our centres are using.

First aid_L3

The AOs also mentioned guidance documents detailing their expectations in relation to what constitutes appropriate feedback and support, and separation of formative and summative assessment, and how internal assessments should be carried out so that they are not overly scaffolded. However, one AO suggested that there might be grounds for introducing more explicit controls around how feedback is provided in the context of summative assessment, in addition to guidance. Some AOs mentioned that they provided training on aspects of formative feedback or exemplars of feedback sheets and good practices for providing feedback for centres to use.

Hybrid aspects

Several AOs discussed mitigations such as a restricted number of opportunities for resit or resubmission of evidence as a way to potentially increase the distinction between formative and summative assessment and reduce the opportunity for students passing based on inappropriate feedback. In some cases, tighter resubmission rules were put in place to increase parity with academic qualifications because of perceptions that constant resubmission brings into question the level of demand of these qualifications.

These kinds of restrictions stand in contrast to some of the mitigations related to lenience and malpractice, such as absence of time constraints on learning. To the extent that introducing additional constraints on learning time and resit opportunities might present incentives for lenience and malpractice, there may be a need for additional mitigations for these risks in qualifications with such restrictions.

We […] have developed the resubmission rules […], which means that if a learner misses the distinction by a little bit and wants to retake, resubmit, there is a lead internal verifier who has to confirm that the learner is able to do that on their own accord and they haven’t been given specific feedback. […] So, they would have to resit an assignment if there was no cause for the lead IV to agree a resubmission. So, there are some stronger rules around that, which [are] monitored through SV. […] Those rules are put in to try and bring some parity to academic qualifications, because [these qualifications] came under scrutiny about, if you keep resubmitting, resubmitting, resubmitting, then what kind of level of demand is there?

Business_L3

Well, we’re certainly really clear that although resubmission is a matter for centre policy they shouldn’t be, learners shouldn’t have […] more than one resubmission opportunity.

Creative_L2

Some AOs explained that in their qualifications, which are typically delivered in the workplace, assessment was often carried out by external (visiting) assessors, rather than students’ own supervisors. In such cases, summative assessment is not continuous, with summative assessment situations being more clearly separated from learning and formative assessment, helping to mitigate the potential problem of inappropriate support.

Contextualisation and real-life task setting

Some AOs noted that one potential source of inappropriate support can be assessment tasks that are overly scaffolded. To mitigate this, the AO delivering Creative_L2 qualification discussed how contextualisation of assessment tasks helps to mitigate this because tasks anchored in a specific context lend themselves less to scaffolding and potentially formulaic “cookie cutter type” responses. They spoke about how they “successfully moved a lot of centres away from templates” that provided too much scaffolding, and which might also restrict opportunities for students to show higher levels of understanding and access higher grades. This comment implied that some effort on the part of AOs is required to educate centres and tutors about how best to write tasks that address the relevant AC most effectively, including avoiding inappropriate scaffolding.

A few AOs thought that certain constructs such as practical skills are “naturally safeguarded” from the potential problem of inappropriate support during assessment. This was especially the case where assessment happens in real time, is part of a larger contextualised process (for instance, a theatre production rehearsal), and is usually recorded on video. These tasks or events were also less likely to lend themselves to repeated assessment to allow students to eventually achieve a pass or a higher grade. In some qualifications, where assessment is carried out in the real‑life setting of a workplace situation, where a student might be delivering a service for a client, the assessors would not provide feedback or guidance as this would be antithetical to standard practices in the workplace. This, then, also mitigated the risk of inappropriate support being given in the context of summative assessment situations.

Some AOs suggested that detecting inappropriate support in knowledge assessments might be difficult. Relatedly, some recognised the risk that certain types of assessment tasks used summatively, such as open-book assignments, might allow potential reproduction of knowledge that was not, in fact, independently learnt and embedded. However, these risks were thought to be mitigated in their qualifications because this knowledge ultimately needs to be applied and assessed in the context of practical tasks and work environments rather than just in a one-off summative assessment situation. This allowed for the absence of knowledge to be detected and ensured that the knowledge is, in fact, embedded and retained.

[…] apart from [the Health and Safety] Unit 1, a lot of it is skills based and, in terms of Unit 1, it’s something that actually threads throughout […], health and safety applies across all of the units, and learners’ behaviour has to change in response to the knowledge that they gain from Unit 1 as to how they work in a construction environment. So, I think if learners are not displaying that knowledge in their practical assessments, I think that’s quite easy to pick up.

Construction_L1

Some AOs seemed to feel strongly about the benefits of the integration of learning and assessment into the real-life environment, arguing that implementing a more artificial, decontextualised, terminal assessment would reduce the validity of their qualifications, even though it might prevent inappropriate support for students.

I think it’s almost like, and the question back to those people is, well how would you like us to do it? And I suppose you could follow that journey and almost end up with end point assessment of which is certainly not fault free. You know, when I’m working, putting someone in an end point assessment situation is a completely false situation for them.

Adult care_L3

Incentives and disincentives

Two AOs suggested certain resource-related disincentives that may act as protective factors against inappropriate support, related to limited time, pay, and resources available for assessment in centres. These were believed by the AOs to disincentivise approaches which might involve spending a lot of time repeatedly assessing students until they achieved the qualification or assessors investing their own time writing students’ work for them, thus providing inappropriate support.

I was thinking that possibly most centres don’t have the time to be constantly assessing like that, which is another aspect of this that takes care of itself to some extent. […] they are quite pressured quite often because, obviously, the arts are not necessarily everyone’s priority, so, in some instances, they only have the learners for three-quarters of an hour or an hour a week.

Creative_L2

I just don’t think it’s something that a tutor-assessor wants to do. A lot of tutor‑assessors nowadays get paid for the hours they turn up and teach face‑to‑face. They don’t get paid for the time that they spend doing assessment work. So, again, I just can’t see why there’d be any incentive for them to want to write a learner’s piece of work for them […].

Housing_L5

The total number of AOs against some problems differs because not all AOs were explicitly asked about some problems. For instance, participants for 13 out of the 14 qualifications were explicitly asked about inappropriate support, of whom 11 recognised this potential problem, 1 did not entirely recognise it and 1 did not recognise it. ↩
The quote below mentions IQA and EQA. IQA, or internal quality assurance, involves QA processes implemented within centres. AOs variously referred to this as internal quality assurance (IQA), internal verification (IV), internal moderation, etc. We will use IQA throughout to refer to centre-internal quality assurance processes, though alternative terminology might feature in quotes. EQA, or external quality assurance, involves QA processes that are operated by the AOs. AOs typically referred to this as external quality assurance (EQA) or external verification (EV). We will use EQA throughout to refer to these processes. Note that throughout the report we also use the terms IQA and EQA, often in plural, to refer to the individuals that carry out the IQA and EQA processes. In this use, IQA and EQA stand for Internal Quality Assurer and External Quality Assurer. ↩
In this overarching section and in the section about teaching, learning and delivery problems later, we use bold to highlight individual mitigations and protective factors within each higher-level mitigation type. ↩
In Ofqual regulations, the term ‘moderation’ has a specific meaning and refers to checking that assessment decisions are appropriate BEFORE certification. In our interviews, some AOs appeared to use the term moderation to mean checking of assessment decisions which might happen before or after certification. Throughout this report, we therefore also use the term moderation loosely as it was not always clear which meaning it had in AO comments. ↩
As noted in Newton & Lockyer (2022, pp. 14-15), although this term is widely used in the industry, DCS is operated in different ways by different AOs. The term does not appear in the Ofqual regulatory framework. DCS may be conferred upon centres by AOs when they are satisfied that a centre can effectively deliver a qualification and assess it with consistent accuracy. Once granted DCS, the centre can request certification for an individual or group of students without the need for their assessment decisions to be externally quality assured before the award of each certificate. Recent Ofqual regulations permit DCS as long as some form of Centre Assessment Standards Scrutiny (CASS) is undertaken, and certain baseline requirements are met. ↩
There were several qualification design features that we coded as ‘hybrid’ in our analysis and grouped under the overarching ‘hybrid aspects’ mitigation type. These were typically the features which are more commonly used in the classical approach, such as externally set assessments, use of marks rather than direct grading, terminal (rather than continuous) assessment, use of external assessors, and restrictions on certain aspects of assessment delivery such as number of resit opportunities. Where multiple such features were mentioned in relation to a particular potential problem in our interviews, these were grouped and discussed in sections called ‘Hybrid aspects’ in this report. ↩
These are normally employed by independent training providers rather than AOs, so they are “external” not in the sense of working for an AO, but in the sense of not simultaneously being students’ supervisors or tutors that might assess informally or continuously during a working day or a lesson in college. ↩

Responding to Criticisms of the CASLO Approach (Report B): AO views of potential assessment problems

Applies to England

Inaccurate judgements and standardisation

Contextualisation and holistic assessment

Quality assurance

Support and guidance for centres

Standardisation as a challenge and a mitigation

Occupational or professional expertise and experience

Communities of practice

Qualification and assessment design processes and features

Atomistic assessor judgements

Perceived drivers of atomisation

Holistic delivery and assessment, contextualisation and real-life task setting

Occupational or professional expertise and experience

Support, guidance and QA

Qualification and assessment design features

Poorly conceived assessment tasks or events

Perceived challenges in assessment design

Support, guidance and QA

Occupational or professional expertise, experience, attitudes and communities of practice

Real-life task setting and contextualisation

Qualification and assessment design processes and features

Hybrid aspects

Lenience and malpractice

Incentives and disincentives

QA and support and guidance for centres

Attitudes

Supporting students and learning

Qualification and assessment design features

Inappropriate support

QA and support and guidance for centres

Hybrid aspects

Contextualisation and real-life task setting

Incentives and disincentives

Is this page useful?

Help us improve GOV.UK

Help us improve GOV.UK

Cookies on GOV.UK

Applies to England

Inaccurate judgements and standardisation

Contextualisation and holistic assessment

Quality assurance

Support and guidance for centres

Standardisation as a challenge and a mitigation

Occupational or professional expertise and experience

Communities of practice

Qualification and assessment design processes and features

Atomistic assessor judgements

Perceived drivers of atomisation

Holistic delivery and assessment, contextualisation and real-life task setting

Occupational or professional expertise and experience

Support, guidance and QA

Qualification and assessment design features

Poorly conceived assessment tasks or events

Perceived challenges in assessment design

Support, guidance and QA

Occupational or professional expertise, experience, attitudes and communities of practice

Real-life task setting and contextualisation

Qualification and assessment design processes and features

Hybrid aspects

Lenience and malpractice

Incentives and disincentives

QA and support and guidance for centres

Attitudes

Supporting students and learning

Qualification and assessment design features

Inappropriate support

QA and support and guidance for centres

Hybrid aspects

Contextualisation and real-life task setting

Incentives and disincentives

Is this page useful?

Help us improve GOV.UK

Help us improve GOV.UK