Research and analysis

Improving attainment among disadvantaged students in the FE and adult learning sector: evidence review (HTML)

Published 21 January 2020

Applies to England

Improving attainment among disadvantaged students in the FE and adult learning sector 

An evidence review 

Research report January 2020 

1. Executive summary

The Further Education and adult learning sector (FE sector) plays an important role in improving socio-economic outcomes and supporting social mobility. Individuals from disadvantaged backgrounds are significantly less likely to participate in learning or training and tend to have lower attainment but more likely to participate in FE and adult learning compared to other post-16 routes. For government reforms designed to increase adult learning to be effective and for outcomes to improve, it is crucial to develop the evidence on what works and share knowledge and best practice. Yet there has been relatively limited activity focussed on bringing together the evidence on what works to improve outcomes in the sector. This report aims to help fill this gap by reviewing and mapping the evidence on what works to improve attainment for disadvantaged students in the FE sector.

1.1 Key findings

The review found 63 studies across 4 countries, 9 of which are from England. Fifty­-two of the studies are rated as ‘high quality’ of evidence. This contrasts with the evidence in the education sector (3 to 18 years) given it is anticipated that more than 3,000 studies will be included in Education Endowment Foundation’s new teaching and learning toolkit.

Programmes^{[footnote 1]} designed to improve attainment among disadvantaged students can have a positive impact but some appear to only have marginal effects or no impact. While some research shows that programmes can significantly improve attainment (for example, obtaining a Level 2 qualification), other studies found the improvement to be modest. This indicates that the programmes are only slightly more effective than ‘business as usual’.

Programmes can improve students’ progress in learning, although these improvements may not be sustained over the longer term. Programmes have been used to support achievement in basic skills and support progress on to further and higher education. Despite encouraging short-term effects, however, improvements may not be sustained over the longer term or lead to higher levels of attainment.

Overall, the evidence on what types of intervention work to support disadvantaged learners is mixed. Much of the evidence included in this review suggests that different programme types have mixed outcomes. However, the most effective programmes appear to be those which offer comprehensive support and those that integrate with course curriculum (for example, embedding basic skills provision in a vocational pathway).

There is a lack of evidence on what works for specific groups of disadvantaged students. While there are a few studies that consider what works for specific groups, such as care leavers, lone parents or adults with an ESOL need, most programmes are designed to support disadvantaged students in general – individuals with low skills or on low incomes.

Most evaluations are relatively short-term, with only a few studies tracking longer-term outcomes. Most of the studies in this review evaluate the impact of a programme over a relatively short timeframe. Fewer evaluate outcomes several years post-randomisation. As attainment may be a longer-term outcome, studies that have a short evaluation period may not capture this outcome, or other related socio-economic outcomes.

Overall, the review found a lack of UK-based casual evidence. While there is a breadth of research and evidence in the FE sector, it tends to be small-scale and qualitative rather than being focused on impact and policy effectiveness. Most causal evidence is from the US which presents limitations for transferability to the UK context.

1.2 Recommendations

Government should invest £20 million over 5 years to establish a What Works Centre for FE. The policy brief that accompanies this review provides more detailed information on this investment and priorities needed.^{[footnote 2]}

Government and the Centre should focus on what works across all stages of the learner journey, from participation to longer-term socio-economic outcomes. This will require investment in long-term evaluations that track outcomes over several years.

Government and the Centre should also focus on what works for specific groups of learners. Disadvantaged groups, such as care leavers and adults with ESOL needs, face specific barriers to learning. Yet there is limited evidence on what works for whom. 

2. Introduction

The Further Education and adult learning sector (FE sector) plays an important role in improving socio-economic outcomes and supporting social mobility. A disproportionate number of learners in the FE sector are from disadvantaged backgrounds compared to other post-16 education routes: nearly a third of students are from the 20% most deprived areas in England.^{[footnote 3]} FE students are also likely to be on low incomes, with 25% claiming out-of-work benefits before they started their course.^{[footnote 4]}

FE and adult learning can improve employment outcomes at various stages in individuals’ working lives, alongside health, wellbeing and social integration outcomes.^{[footnote 5]} It also provides opportunities for individuals to gain new qualifications and progress out of low pay, as individuals with low or no qualifications are more likely to get stuck in low pay and employees with no formal qualifications earn around 20% less than employees with GCSEs.^{[footnote 6]} The sector will also play an increasingly important role in helping workers retrain or upskill as technological developments change demand for skills.

Yet, despite the FE sector’s role in tackling these challenges, there has been relatively limited activity focused on collating the evidence on ‘what works’ to help practitioners close the attainment gap between disadvantaged pupils and their peers. This report aims to help fill this gap by reviewing and mapping the evidence on what works to improve attainment for disadvantaged students in FE and adult learning.

3. Policy context

This section looks at the challenges relating to FE and adult learning as they relate to key stages in the ‘learner journey’, from engagement and participation to the socio-economic returns from FE and adult learning, before setting out the latest policy developments.

3.1 Policy challenges

Low and unequal participation

There has been a long-term decline in the level of adult participation in further education and adult learning. This is visible in the number of adults taking funded courses, with the number of adults aged 19 and over taking a non-apprenticeship further education course falling by nearly two-thirds.^{[footnote 7]} It is also evident in adults’ self-reported participation in learning. The latest data from Learning and Work’s (L&W)’s adult participation survey found that just 1 in 3 adults (35%) had taken part in learning in the last 3 years, the lowest level in the survey’s 22 years.^{[footnote 8]} Participation in adult English and maths provision is also falling when more than 5 million adults lack both functional literacy (below Level 1) and numeracy (below entry Level 3).

Beyond the headline figures on adult participation, there is also evidence of stark inequalities in access to learning, with disadvantaged adults being significantly less likely to participate.^{[footnote 9]} The adult participation survey and the Social Mobility Commission’s adult skills gap report highlight the extent to which the adults who could most benefit from access to education and training – based on social class, level of education and employment status – are the least likely to be taking part, with gaps in many areas having grown in recent years.^{[footnote 10]}

The inequalities in participation are partly caused by employers investing in training workers who are already highly skilled. According to the Office for National Statistics’ (ONS) annual population survey, adults with a Level 4 qualification or above are over 50% more likely to have received training in the last 3 months than adults who do not have a qualification at that level.^{[footnote 11]} Employers appear to see more of a ‘business case’ for upskilling their already well­ qualified workers, rather than supporting less well qualified workers to progress.

Variation in attainment

Disadvantaged students tend to achieve lower level qualifications. The Longitudinal Education Outcomes (LEO) data shows that students who achieved Level 2 or below in FE as their highest qualification were more likely to be disadvantaged than students who achieved Level 3 or above.^{[footnote 12]}

Achievement rates vary across different courses, learner groups, institutions and local areas. Achievement rates for adult basic maths and English courses were 65% compared to an average of 89% across all qualification types.^{[footnote 13]} Overall achievement rates for adult learners (aged 19+) vary from 83% in the Swindon and Wiltshire LEP area to 91% in the North East LEP area.

Variation in returns from FE and adult learning

Disadvantaged students are less likely to progress on to higher level earnings compared to non-disadvantaged students. Non-disadvantaged students were more than twice as likely to progress on to higher level earnings (over £25,000) by age 26 compared to disadvantaged students, 32% and 14% respectively.^{[footnote 14]}

3.2 Developments in government policy

There have been a number of recent policy interventions which aim to help adults access education and training opportunities.

Apprenticeships have increasingly been seen as a tool for upskilling adults, rather than just a route for young people into the labour market, with those aged 25 and above accounting for two-fifths of apprenticeship starts in 2017/18. The apprenticeship levy- a 0.5% levy payable by employers with a pay bill of over £3 million -was introduced in 2016 in an effort to encourage more employers to invest in skills. With the apprenticeship levy account set to be overspent this year, there have been growing concerns that spending on expensive higher and degree level apprenticeships may come at the cost of opportunities for young people and disadvantaged adults.^{[footnote 15]}

From this year, the government has devolved the adult education budget (AEB) to 6 mayoral combined authorities and delegated the budget to the Mayor of London. The AEB funds non-apprenticeship further education for adults aged 19 and over. The aim of devolution of the AEB was to allow local areas to better align adult education and skills provision with local needs, so that funding supports local residents and delivers on local economic priorities. Mayors have expressed a desire to use the devolved funding to narrow inequalities and support adults with lower levels of qualifications to progress into higher skilled jobs. However, this budget has reduced significantly in recent years, with spending on adult education – excluding apprenticeships – falling by 47% since 2009/10.^{[footnote 16]}

More recently, the government has been developing the National Retraining Scheme (NRS). This new programme aims to help adults retrain into better jobs, and to be ready for future changes in the economy, including the impact of automation. The scheme is focused on adults aged 24 and over who are in work on low to middle incomes, and without a degree level qualification. This group has been identified as both being particularly vulnerable to economic change, and relatively poorly served by current government support. The government announced £100m of funding for the NRS in the autumn budget 2018.

The Treasury also recently announced a £400m funding package for 16-19 provision, marking a funding rate increase for the first time since 2013. This includes £45m aimed at supporting the delivery of T levels and £35m for targeted interventions to support students on Level 3 courses to resit GCSE/level 2 maths and English.^{[footnote 17]}

For these reforms to work and outcomes to improve, it is vital to improve the evidence on what works, and to share knowledge and best practice.

4. The review

4.1 Aims of the review

This review draws together and maps causal evidence and associated implementation evidence to better understand what works to improve attainment for disadvantaged learners in FE and why.

The primary focus is on interventions that can prove causality and aim to improve attainment (gaining qualifications or credentials) for disadvantaged students aged 16 and over, excluding apprenticeships. The approach to the review means that other outcomes including enrolment, engagement/attendance, achievement, and social and economic outcomes are captured too as they link to attainment but were not the primary focus.

Further work would be needed to fully assess what works for disadvantaged learners throughout the learner journey (access, participation, and progression into employment or further learning). This would offer lessons on what works to improve accessibility and engagement through to progression outcomes including economic and social.

4.2 Methods

Criteria for review

Inclusion criteria were developed to ensure that the studies are relevant to the parameters of the research. The criteria for the review are summarised in Table 1.

Table 1: Criteria for review

Population:

Disadvantaged students aged 16+. Primary factors of disadvantage include socio-economic background (for example, eligibility for free school meals), and having a basic skills need (for example, students whose highest qualification is below Level 2). Disadvantaged groups include young adult carers, care leavers, Black, Asian, and minority ethnic (BAME) learners, learners with learning difficulties and disabilities, and learners with an English for Speakers of Other Languages (ESOL) need.

Interventions or programmes:

Initiatives that aim to improve attainment. Eligible interventions must include an educational/learning component. Types of intervention include financial incentives, comprehensive student support services, use of technology in the classroom, revised curriculum or behavioural nudges and communication.

Outcome:

The primary outcome of interest is attainment (defined as education qualification (vocational or academic) or levels achieved). Secondary outcomes include increased motivation, attendance and labour market outcomes.

Study design:

Impact evaluations that seek to understand the causal effect of interventions, including experimental, quasi-experimental and non- experimental designs. Studies must include the use of a control group.

Geography:

Studies from OECD countries published in English

Date:

1990 onwards

Search strategy

Using the inclusion criteria, we identified precise search terms and synonyms to conduct a comprehensive search of a wide range of sources. This included academic databases (for example, the Applied Social Science Index, the British Education Index and ProQuest (including ERIC), specialist research institutes (including Centre for Vocational Education Research, National Foundation for Educational Research, MDRC, National Centre for Postsecondary Research), government websites (Department for Education, Department for Business, Energy and Industrial Strategy and Ministry of Housing, Communities and Local Government) and think tanks and other organisations with an interest in improving outcomes for disadvantaged students (L&W, Nuffield Foundation, Joseph Rowntree Foundation, Impetus, and NCVO).

We also scrutinised reference lists of studies retrieved to identify additional papers and hand searched contents pages of specific academic journals. For example, Journal of Youth Studies, Journal of Vocational Education and Training and Research in Post-Compulsory Education. We issued a call for evidence to identify missing evidence.

Study selection

Titles and abstracts of studies returned through the search strategy were screened against the inclusion criteria. This created a long list of 304 studies.^{[footnote 18]} Studies on the longlist were assessed for eligibility based on the full text and a final shortlist of 63 studies was created. Most of the studies were excluded based on their study design; studies were based on qualitative research methods and therefore did not robustly assess the effectiveness of an intervention.

Quality assessment

The shortlisted studies were quality assessed using the Maryland scale which measures the quality of a study based on methodology, risk of contamination, attrition rate, sample size and date.^{[footnote 19]} The higher the quality rating, the higher confidence we have in the effectiveness of the intervention.

4.3 Quality and nature of the evidence

Much of the literature (both systematic reviews and impact evaluations) on what works to improve attainment among disadvantaged students in the FE and adult learning sector note the lack of robust evidence.^{[footnote 20]} The authors of a Skills for Life study note that most studies on literacy and numeracy provision have been small-scale and qualitative. Moreover, the quantitative studies that do exist tend to measure change and progress among learners, but do not use a comparison group to identify the causal effect of the intervention.^{[footnote 21]} Given this evidence gap, there is a general call from the research community to prioritise an evidence-based system to build knowledge of how best to support disadvantaged learners.^{[footnote 22]}

The majority of studies in this review, 52 out of 63, are evaluated through randomised control trials (RCTs). These studies offer the most robust evidence on the effectiveness of interventions to improve disadvantaged learners’ attainment. RCTs – where individuals are randomly assigned either to a treatment or a control group – are considered the ‘gold standard’ because random assignment helps to ensure that any changes in programme participants’ outcomes are caused by the intervention, as opposed to other factors. These studies give us a high degree of confidence that outcomes can be attributed to participation in that specific programme. Most of these studies are from the United States.

At the next quality level down are studies which used a matched control group to create a ‘counterfactual’. The counterfactual was used to test the difference between the group that underwent ‘treatment’ and the group that did not, in order to assess the level of impact. A small number in this review use this type of methodology.

The majority of the literature returned on the initial search met the inclusion criteria in terms of population, programme type and outcome, but not the criteria for the study design. They offer relevant, in-depth data about participant barriers to, experience of and outcomes of learning for disadvantaged groups, but do not provide causal evidence on what works to improve attainment.

Considering comparability and transferability

Given that most of the studies in this review are from the United States, it is important to consider the extent to which they are transferable to the UK context. As the focus is on practical and pedagogical interventions (for example, mentoring, basic skills provision, occupational training) and not systemic or institutional (for example, education policy or institution-wide management reforms) changes, lessons can be drawn more easily from the approaches taken. However, further investigation is required to assess the transferability of some interventions, such as residential training.

The outcome measures used in US studies (primarily General Educational Development (GED) receipt and credits earned) can be roughly equated to qualifications levels in the UK skills system. However, it is important to caveat that the systems cannot be compared like for like, placing limitations on transferability. These are explained in more detail in Box 1.

A second limitation to consider is that many of the US studies are set in community colleges which tend to serve the post-18 cohort, presenting a gap in evidence on the 16-18 cohort.

Box 1: Outcome measures

Attainment:

Definitions of attainment are primarily driven by education qualification (vocational or  academic) or levels achieved. For example, receipt of a qualification or credential that 

indicates the learner has reached a certain level. 

Attainment measures in this review include GCSEs, and Functional Skills qualifications (Level  1 and Level 2). Additionally, the GED is a primary attainment measure in many US studies. The GED is a set of tests that when passed certify the test taker (American or Canadian) has met high-school level academic skills. The UK equivalent is a Level 3 or A level qualification. 

Achievement:

Achievement means distance travelled or progress made. For example, many studies in the review use number of credits earned as a positive indicator of academic progress. Earning credits can either contribute towards a ‘credential’ (a qualification or certificate) or towards transferring to a 4-year degree level programme. Other studies use scores on standardised literacy and numeracy tests to measure achievement or progress.

Box 2: Types of interventions

Multiple interventions are programmes that comprise 2 or more of the following: tutoring, counselling and/or advice, financial support, basic skills provision and occupational training. Examples include CUNY ASAP, lnstituto del Progreso Latino’s Carreras en Salud programme, The Valley Initiative for Development and Advancement programme and Project QUEST.

Basic skills interventions are programmes delivering language, literacy and numeracy provision (English, maths and ESOL). In England, these courses are taught at Level 2 (approximately GCSEs of grades A* to Cl 9 to 4) and below. Examples include the Community-based English Language (CBEL) programme, the Skills for Life programme and the Welfare to Work programme. Basic skills interventions include remedial or development programmes in the US. Examples include the GED Bridge programme.

Accelerated learning interventions are intensive forms of provision delivered over a short-time period, for example a semester (3 to 4 months). They aim to prepare students for college-level courses. Examples include the CUNY Start programme.

Occupational training interventions are primarily focused on delivering occupational training with the aim of increasing participants’ employment and earnings. Examples include Integrated basic education and skills training (I-BEST), The Workforce Training Academy (WTA) Connect Program, Madison Area Technical College Patient Care Pathway Program, Traineeships and Greater Avenues for Independence (GAIN) Program.

Behavioural interventions are interventions that include relationship building, adapting the environment, managing sensory stimulation, changing communication strategies, providing prompts and cues, using a teach, review, and reteach process and developing social skills. Examples include the Behavioural Research Centre for Adult Skills and Knowledge (ASK) interventions.

Digital interventions are provisions which use Information Communication Technology (ICT), online and blended as a learning approach or medium. Examples include ModMath and the INVEST programme.

Financial interventions are monetary stipends or support. Examples include the MAPS programme and the 16-19 bursary fund evaluation.

Innovative pedagogy interventions test a pioneering approach to teaching. Examples include the Dana Center Mathematics Pathways (DCMP).

Mentoring is provision that is delivered on a one-to-one basis or in groups no larger than 2-3 students. An example is the Beacon Mentoring programme.

Learning communities are designed to give students a chance to form stronger relationships with each other and their instructors, engage more deeply with the integrated content of the courses, and access extra support. Examples include Kingsborough College Learning Communities programme.

Residential interventions involve participants living at the programme site and participating in various support elements including basic skills provision, occupational training and mentoring. Examples include Jobs Corps and the National Guard Youth ChalleNGe programme.

Interventions and educational outcomes: evidence and gap map

The map below illustrates the evidence identified for interventions aimed at improving the attainment of disadvantaged learners. The y-axis shows the type of intervention and the x-axis illustrates the type of attainment measured.

From the cluster of circles in the centre of the map, it is clear that there is a predominance of US studies. Whereas the gap in evidence on the left-hand side of the map demonstrates the overall lack of research from the UK context.

Studies included in the review predominantly explore the effectiveness of behavioural interventions, financial incentives, occupational training (that has a basic skills component), and multiple interventions – those that have a comprehensive support offer.

There is less evidence on the effectiveness of innovative pedagogical approaches, one-to-one or small group support. There were also only 2 studies exploring residential interventions, although these programmes have limited transferability to the UK context.

What works for whom: evidence and gap map

The map below illustrates the evidence identified for interventions aimed at improving the attainment of different groups of disadvantaged learners. It plots types of intervention against different learner groups.

The majority of interventions target low-skilled individuals, with the largest number of interventions investigating the effectiveness of behavioural approaches for this group. Not surprisingly several interventions for low income learners examined the impact of financial incentives on attainment, which provided mixed results. Most of the programmes designed to support low-income groups involved occupational training programmes, which all produced positive results.

This map clearly shows the number of gaps in the evidence relating to other disadvantaged groups. No studies were identified that concentrated solely on Black, Asian and minority ethnic (BAME) groups. It is also noteworthy that there are only single studies relating to individuals with special educational needs (SEN), lone parents, adults with an ESOL need and youths not in education, employment or training (NEET). Furthermore, there was no evidence found on examining what works to support young adult carers or care leavers. This shows that further research, which targets these groups of disadvantaged students, is needed.

5. The findings

5.1 How effective are the interventions?

This section considers the effectiveness of programmes designed to support disadvantaged students. Given the scope and aims of this review, effectiveness or impact is primarily measured by attainment. Other outcomes measured include achievement (or progress}, attendance and retention, and wider outcomes such as soft skills (for example, confidence and social skills) and labour market outcomes.

Attainment

Programmes designed to support disadvantaged students improve attainment can have a significant impact on individuals’ attainment of basic skills qualifications. Studies found significant improvements in learners’ basic skills attainment including English and maths Functional Skills and GCSE attainment and the equivalent attainment levels in the US. A behavioural intervention that used texts and social support improved Level 2 attainment rates by 24% (5.1 percentage points, from 21.1% to 26.2%). In the same study, reflective writing exercises improved attainment in maths and English by 25% (4.2 percentage points, from 16.7 to 20.9%). These behavioural interventions measured attainment and attendance. The results for attainment were statistically significant whereas some of the effect sizes for attendance were not.^{[footnote 23]} Similarly, a study testing innovative approaches to delivering basic maths courses found that attainment among the participant group improved with a mean difference score that was nearly twice that of the control group.^{[footnote 24]}

However, some studies show little to no improvement in attainment. The evaluation of the Pathfinder Extension (a comprehensive model of basic skills provision) for example, found no positive impact on receipt of basic skills qualifications.^{[footnote 25]}

Programmes designed to support disadvantaged students can improve attainment of qualifications, including GED and degrees. Findings from 3 studies show that programmes can also have a significant positive impact on GED completion and receipt.^{[footnote 26]} In the GED Bridge programme which tested a new approach to teaching the GED preparation course, Bridge students were more than twice as likely to pass the exam than those on the traditional course (53% and 22% respectively). Three-year effects of a CUNY ASAP (see table 5.2 for more detail) found that 40% of the programme group received a degree, compared with 22% of the control group.^{[footnote 27]} Additionally, the evaluation of Project QUEST (a comprehensive support programme supporting low-income adults) 6 years post-randomisation shows that participants were significantly more likely than control group members to earn a postsecondary qualification – 75% and 57.2% respectively.^{[footnote 28]}

There is some evidence that while interventions do have an impact on attainment, the effect size can be modest. A long-term study of a learning communities programme (see Box 3.2 for details on the learning community approach) estimates that the programme only increased graduation rates by 3.3% (with 39.5% of programme participants and 36.2% of control group participants earning a degree).^{[footnote 29]} Four years after a programme aiming to prepare students for college by providing enhanced courses and student services began, only 7.6% of programme students had earned a degree or certificate, compared to 6.4% of the control group, representing a small effect size on attainment.^{[footnote 30]}

Achievement

Financial incentives and basic skills programmes have been found to have a positive effect on skills development. There is evidence from the US that programmes have a significant impact on developmental maths and/or English credit attainment (increased number of credits earned), test scores in literacy and numeracy, and completion rates of basic skills courses.^{[footnote 31]} For example, the US National Workplace Literacy Program which measured changes in learners’ basic skills by using pre-and post-intervention skills assessment tests, generally found significant improvements in learners’ basic skills.^{[footnote 32]} The CBEL programme – an intervention aimed at adults with very low levels of functional English proficiency – found similar levels of effectiveness, with programme participants achieving speaking and listening scores that were double that achieved by the control group.^{[footnote 33]}

Improved achievement can support progression in to further and higher education. For example, CUNY Start (an accelerated learning programme aiming to prepare students for college) found that by the end of the programme semester, 57% of programme group students were college-ready in maths, compared with 25% of control group students.^{[footnote 34]} Additionally, by the end of the programme, 38% of programme group students were college-ready in all 3 subject areas (maths, reading and writing), compared with 13% of control group students. There is also evidence that programmes are effective in supporting adults to achieve college-level credits and pass college-level courses.^{[footnote 35]} An innovative pedagogy model – the DCMP programme – found that 27% of programme group students enrolled in college-level maths and 18% passed the course, rates more than double those of students taking traditional courses.

These results were sustained over 3 semesters.^{[footnote 36]}

Despite encouraging short-term effects, some studies demonstrate that achievement over the longer-term may not be maintained or lead to improved attainment. For example, ModMath – a basic maths course delivered via computer-based instruction allowing for self­ paced learning – found that while students made greater progress early on by earning more credits that control group students, there was no discernible difference in course completion, becoming college-ready and/or passing the first college-level maths course.^{[footnote 37]} Most students either did not attempt all required developmental math courses/modules, failed one or more courses/modules, or did not re-enrol. Similar findings were found in other programmes – despite encouraging short-term effects, there is less evidence of meaningful improvements to students’ long-term academic progress.^{[footnote 38]}

Enrolment, attendance and persistence

A range of different programme types have been found to have a positive impact on enrolment. Four studies provide evidence on the effectiveness of programmes increasing the number of participants enrolling on a course.^{[footnote 39]} Often, studies found that programme students may be more likely to take up a full-time course than control students who opt for part-time study.^{[footnote 40]} They may also be more likely to enrol on more than one new course at the start of the programme.^{[footnote 41]} However, it is noted that increased enrolment is most likely attributable to the programme starting at the same time.^{[footnote 42]}

Programmes have been found to have a mixed impact on attendance. The ASK behavioural interventions (see Box 2 for more detail) found significant improvements in attendance on basic skills courses.^{[footnote 43]} Overall, the PACE programmes found a significant increase in the amount of education or training received (hours attended). One study testing the effect of a £5 incentive for every class attended found it had an adverse effect on learner attendance. The authors suggest that the programme should be replicated with a larger incentive.^{[footnote 44]}

Short-term programmes may not improve persistence, retention, and/or progression into further education. There are a number of studies evaluating programmes that last a semester (3 to 4 months long). Overall, short-term interventions tend to lead to modest impacts that are concentrated in the duration of the programme. The evidence suggests that these by themselves are not typically sufficient to boost re-enrolment or lasting changes to students’ progression.^{[footnote 45]} For example, the DCMP study (see Box 2) found that over 85% of students from both groups were registered for courses during their first semester in the study, but less than 50% were still enrolled in college by their third semester, underscoring the challenge that community colleges face to supporting students to persist from semester to semester.^{[footnote 46]}

Studies that track long-term outcomes often find that programme students do not maintain progress. For example, an evaluation of a PBS found that credits earned, and rates of persistence declined over the programme’s lifetime.^{[footnote 47]} In the first year, programme students earned an average of 15.9 credits, compared with an average of 14.2 credits in the control group - an estimated impact of 1.7 credits. In the second year of the study, after the scholarship ended, programme group students earned an average of 0.7 credits more than control group students. Finally, in the third and fourth years, the number of credits earned was, on average, the same for both groups. Studies that find positive enrolment effects tend to be presenting early findings rather than tracking outcomes over the longer term.^{[footnote 48]}

Wider impacts

Interventions can impact on learners’ soft skills including their self-confidence, employability and social integration. There is evidence of programmes having a positive impact on students’ self-management, independence, self-awareness, interest in lifelong learning, emotional intelligence, and engagement in college.^{[footnote 49]} The CBEL programme found a significant impact on social integration measures, including confidence interacting with services and social interactions.^{[footnote 50]} The ASK interventions found that building non-cognitive skills as part of a course can have an impact on whether an individual succeeds in education.^{[footnote 51]}

Studies show positive impacts on employment outcomes. Some of the studies in this review are evaluations of sectoral or occupational training programmes for low-income, low­ skilled adults, and therefore measure employment outcomes in addition to attainment and achievement. Some show a clear impact on participants’ earnings. For example, results from the Year Up intervention (a comprehensive support model including occupational training, basic skills provision and financial support) show a 53% increase in earnings for the treatment group (their average quarterly earnings were $5,454, compared to the control group members earning $3,559 on average).^{[footnote 52]} Another of the PACE programmes showed a 9-percentage point increase in employment in a healthcare occupation (25% for treatment group members versus 16% for control group members).^{[footnote 53]} However, employment progression was into low-paying positions.

It is argued that progression into higher paying jobs and/or higher-skill jobs may take more time to achieve, highlighting the importance of tracking outcomes over the longer-term. The Welfare to Work programme which provides GED preparation courses, basic maths, English and ESOL courses to highly disadvantaged adults, showed that as students’ skills levels increased and reached certain attainment benchmarks, they appeared to have substantial benefits in terms of employment, earnings, and self-sufficiency.^{[footnote 54]} The latest evaluation of Project QUEST shows it had a large positive impact on career-advancement over a 6-year period.^{[footnote 55]} For example, QUEST participants earned $5,080 more than control group members in the sixth year after study enrolment.

Further work would be needed to fully assess studies that examine economic and social outcomes.

5.2 What types of approaches are likely to be effective?

This section explores the types of interventions that work to improve outcomes for disadvantaged adult learners.

Evidence on the effectiveness of basic skills interventions is mixed. Both the CBEL and US workplace literacy programmes sizably improved learners’ language and literacy skills compared with the control group. In the latter programme, more than 85% of workers who attended a workplace literacy course scored higher than the typical worker in the control group.^{[footnote 56]}

As discussed in the previous section, the Welfare to Work Program, which included a range of basic skills provision, improved participants’ skills levels, GED receipt and enrolment into further postsecondary learning compared to the control group. However, only a small proportion of the programme participants increased achievement and attainment. Those that did pursue further learning appeared to experience substantial benefits in terms of increased earnings and self­ sufficiency, indicating a positive snowball effect for a small proportion of participants.^{[footnote 57]} The Skills for Life evaluation found modest improvements in programme participants’ literacy and numeracy skills, but overall no significant difference between them and the control group.^{[footnote 58]} One study exploring the effects of a basic skills programme in the US shows positive impacts on achievement and short-term persistence, but no evidence that the course increased completion of college-level credits or degree completion.^{[footnote 59]}

There is evidence that suggests interventions that include multiple support strands can work to improve outcomes. There is evidence in the 3 evaluations of CUNY ASAP, which included financial support, personal advisers, flexible provision and career specialists, to suggest that the comprehensive support model offered to low-income, low-skilled students works to improve enrolment, persistence and attainment.^{[footnote 60]} The first ASAP programme substantially improved students’ academic outcomes over 3 years, almost doubling graduation rates, increasing the number of credits earned and increasing enrolment on higher education courses compared to rates among the control group. Given these unprecedented findings, the model was rolled out and evaluated in other community colleges, with findings that corroborate. Other programmes such as the Valley Initiative for Development and Advancement and Project Quest, which offered combined provision support these findings.^{[footnote 61]}

However, the Opening Doors programme, which provided 2 elements of support – enhanced student services in the form of regular and intensive support from a college counsellor and a modest stipend – found that any positive impact was limited to the programme lifetime.^{[footnote 62]}

Innovative pedagogical approaches can work to improve attainment and persistence. All 3 studies included in this review under the category of ‘innovative pedagogy’ resulted in a positive impact. The DCMP report found that programme participants were more likely to pass the basic maths course than the control group.^{[footnote 63]} Gains for the programme students continued to be observed (measured by enrolment in and passing a college-level maths course) in the following 2 semesters. These findings indicate that this model can help students reach a critical college milestone: making it to and through a college-level math course. Innovative pedagogy was also effective in teaching adults basic maths in FE colleges in England and delivering a basic literacy course to low level readers in the US.^{[footnote 64]}

Occupational training interventions that embed basic skills content can improve attendance, attainment and progression into further learning. The Traineeships programme in England increased the likelihood of programme participants moving into further learning: 42% of trainees were in further learning 12 months later, compared with 29% of the matched comparison group.^{[footnote 65]} There was also a significant impact on gaining any positive outcome within 12 months.^{[footnote 66]} The Jobs Corps programme, I-BEST and Pima Community College Pathways to Healthcare programme were also found to increase college course enrolment, credits earned and credential attainment.^{[footnote 67]} However, results from other PACE programmes which are categorised as occupational training such as the WTA Connect Program and Madison Area Technical College Patient Care Pathway Program show limited impact on educational outcomes.^{[footnote 68]}

Similarly, there is evidence to suggest embedding vocational content in essential skills provision works well for students preparing for the GED exam – programme participants had higher course completion rate, exam pass rates and were more likely to be enrolled in college than the control group.^{[footnote 69]}

Behavioural interventions have mixed results. The ASK interventions found that programmes that provide learners with encouragement, social support, the opportunity to reflect on why they value learning and feedback to highlight effort and performance can be particularly effective – improving attainment and attendance.^{[footnote 70]} One of the most effective interventions incorporated both weekly text messages of encouragement to learners and updates to their social supporters (family and friends of the learner). Further, a short writing exercise, in which learners reflected on their personal values and why they are important to them, also improved attainment.

A US study which explored the impact of a programme providing low income individuals with personal assistance for student loan applications found that this financial ‘nudge’ led to increased college attendance, persistence, and aid receipt compared to those who received information about the loan but no personal assistance.^{[footnote 71]}

Other studies have evaluated ‘student success’ courses that focus on changing students’ behaviours and attitudes, including increasing their awareness of their and others’ emotions, understanding their own learning styles, improving time management skills, and recognising their responsibility for their own learning to prepare them for community college.^{[footnote 72]} Although findings demonstrate a positive impact on students’ behaviour (for example, self-management, interdependence, self-awareness and interest in lifelong learning), there is little evidence to suggest that this approach improves educational attainment or achievement long-term. Four years after one of the studies began, programme and control group students had made similar academic progress.

Digital interventions may improve engagement for some students but do not appear to be any more effective than traditional basic maths and literacy courses in raising completion levels or attainment. A US evaluation of adult basic education programmes that used ICT products to facilitate learning of basic skills found that most instructors and students reported positive experiences of using ICT, suggesting a positive impact on engagement.

However, the impacts on skills assessment results were mixed.^{[footnote 73]}

Other evidence suggests that digital interventions have a modest impact on student outcomes. Modmath (see Box 2) and a UK study examining the effects of a computer-assisted instruction intervention on basic numeracy skills for adults with learning disabilities found that the approach was no more or less effective than the traditional course which was teacher-led.^{[footnote 74]} Similarly, a 4-week educational programme with computer assisted instruction designed to raise prisoners’ achievement on maths and reading tests found no significant difference in scores between the programme and control group.^{[footnote 75]} Two studies from the early 1990s found computer-based learning to have similar limited impact.^{[footnote 76]} Although the second, the INVEST programme, shows that computer-based approaches may work more effectively for improving maths skills than literacy skills.

Accelerated learning or intensive courses may produce short-term improvements, but the evidence suggests impacts may decline over time. A bridge course designed to support students with basic skills needs build competencies over the course of several weeks before entering college had some impact on first college-level maths, reading and writing course completion.^{[footnote 77]} However, by the end of the 2-year follow-up period there was no difference in attainment or persistence. For example, the programme group enrolled in an average of 3.3 semesters, and students in the control group enrolled in an average of 3.4 semesters, a difference that is not statistically significant.

A study exploring the effectiveness of accelerated learning on developmental students’ outcomes in English classes found that programme participants were more likely than the control group to attempt and pass developmental English courses and persist to the next year to attempt and complete more college-level English courses than the control group. However, the 2 groups were equally likely to earn an associate degree or transfer to a 4-year higher education college.^{[footnote 78]}

Financial incentives can work to improve educational enrolment, attendance, attainment and persistence. Three of the studies in this review evaluate the impact that Performance­ Based Scholarships (PBS) have on community college learners’ engagement, enrolment and earning of college credits (see Box 2 for more details). The studies have positive results including increasing number of credits earned, having a positive effect on full-time enrolment and decreasing the time it took students to earn a degree.^{[footnote 79]} Strong support for the value of financial incentives also comes from the ASK behavioural intervention trial which found that a cash ‘buddy incentive’ – incentives that were only awarded to learners where both buddies achieved the attendance target – achieved higher attendance than those in the individual incentive group, showing that an incentive that incorporates a social dimension can be even more effective.^{[footnote 80]} Similarly, positive results were found in a programme offering financial incentives for low-income parents attending community college. Programme participants had higher pass rates and earned more course credits than the control group. Programme participants also had higher rates of registration in colleges in the second and third semesters after random assignment compared to the control group.^{[footnote 81]}

Financial incentives and support can also work for younger adults. The impact evaluation of the 16-19 bursary fund (following the removal of the Education Maintenance Allowance) found that participation rates in full-time learning dropped by 1.6 percentage points. Estimates show that there was a 2.3 percentage point fall in the Level 2 achievement rate by age 18, leading to a 1.1 percentage point fall across the whole Year 13 cohort, from 88.3% to 87.2%. The authors attribute these changes to the removal of the maintenance allowance.^{[footnote 82]}

Conversely, a study testing whether bi-weekly instead of termly disbursement of financial aid works to improve students’ academic outcomes, found no substantial impact. This indicates that the timing of receipt of financial aid may not have a significant bearing on student outcomes.^{[footnote 83]}

One-to-one, mentoring and small group support can significantly improve persistence and completion and may work to support ‘at risk’ groups’ attainment. While the Beacon Mentoring programme (see Box 2 for detail) had no impact on attainment overall, sub-group analysis shows that ‘high-risk’ students (learners referred to developmental maths rather than college-level, therefore at higher risk for poor outcomes) were significantly less likely to withdraw from the maths class and earned a significantly higher percentage of credits per attempt compared to the control group.^{[footnote 84]} Another study testing 3 different types of educational intervention: team approach (approximately 14 students, a teacher and a counsellor), small group approach and individual tutoring found that the small group approach (4-6 students to 1 tutor) had the most significant impact on completion rates among ‘at risk’ adults and adults who had previously dropped out of adult basic education.^{[footnote 85]} 60% of students in the small group completed 3 months or more compared to a 40% completion rate for those in the team group, and a 20% completion rate for those being tutored individually.

Evidence on the effectiveness of learning community interventions is mixed. Learning communities are a popular strategy in the US to boost low success rates, particularly among students who take up basic English and math provision. Five of the studies in this review explore their impact, but with mixed results. One study found that learners earned significantly more credits than their control group counterparts and maintained this result for 7 years post-randomisation, indicating the long-term effect on their academic progress.^{[footnote 86]}

However, some of evidence shows that while learning communities appear to have an impact on students’ achievement (measured by pass rates of developmental classes) during the programme semester, this effect diminishes over time with no impact on college enrolment or cumulative credits earned in the semester following the programme.^{[footnote 87]}

5.3 What works for whom?

There is limited evidence on what works to improve attainment for different sub-groups of disadvantaged students. Most studies included in this review present analyses for the sample as a whole, telling us what works for the cohort in general. Fewer studies conduct sub-group analysis. Nevertheless, the evidence that does exist provides some learning on the following student characteristics.

Distance from learning

Some studies combine sub-group analysis findings to demonstrate the programme’s impact on ‘non-traditional’ students. For example, Project QUEST found that impacts were greatest among participants aged 25 and older, and those who had a GED rather than a high school diploma.^{[footnote 88]} This indicates that QUEST was particularly successful in reaching into the community to engage people who were unlikely to navigate their way through postsecondary training to a good job on their own. Using proxy indicators for distance from learning based on age and date of receiving High School Diploma or GED (since there was no measure of ‘time since taking last maths class’ available), the MAPS study – a PBS programme that offered access to a maths lab -also found that the intervention had a more positive impact on adults who were considered less likely to take up education upon enrolment. This suggests that a financial incentive may be effective to motivate disadvantaged adults to take up basic skills provision. Another study found that teaching adults in small groups had the most significant impact on completion rates among adults who had previously dropped out of adult basic education.^{[footnote 89]}

In contrast, the Skills for Life learning evaluation found that older learners, those with children and those who believed they had a literacy problem were less likely to achieve Level 2 English and maths qualifications. This suggests that attainment appeared to be linked to distance from learning. As QUEST adopted a multiple intervention approach, whereas the Skills for Life programme was the delivery of basic skills courses, these findings indicate the effectiveness of comprehensive approaches to supporting disadvantaged students.^{[footnote 90]}

Age

Evidence shows that the effectiveness of interventions differs for different age groups. Two programmes targeting young adult learners (16 to 24-year-olds) found that the intervention had a stronger impact on the older sub-group of the cohort. For example, for participants aged 19- 23, Traineeships had a positive impact on being in employment 12 months later. In contrast, there was no significant impact for the younger cohort of those aged 16-18.^{[footnote 91]} The ChalleNGe programme found similar results that staff attributed to higher levels of motivation and focus among the older group.^{[footnote 92]}

Previous educational attainment/level

Some studies show that interventions worked particularly well for lower level learners. For example, ASK behavioural interventions and the CUNY Start programme were most effective for students who had the highest skills need. ASK may have had a more powerful impact on lower level learners because it introduced elements that met lower level learners’ needs.^{[footnote 93]} For example, social support, or an intervention which addresses anxiety about learning or self-doubt about their ability to keep learning. Similarly, the Beacon Mentoring programme targeted students enrolled in lower level maths courses, who had a high degree of failure.^{[footnote 94]} While the programme had no impact on attainment overall, the introduction of mentors had a significant impact on progress of students who had the lowest skills levels.

In contrast, other studies found that positive outcomes were more pronounced for participants who had higher skills levels. For example, GAIN (see Box 2) and a Learning Communities programme found that educational gains were concentrated among participants who had relatively high levels of literacy when they started the programme.^{[footnote 95]} Similarly, the CBEL trial found that previous higher educational attainment was a significant predicator of improvement in proficiency, which could indicate a familiarity with class-based study, or aptitude to learn. An evaluation of digital learning for adult basic literacy and numeracy uncovered challenges in using computer-based learning approaches with low-skilled adults. In recognition of this, the authors concluded that for those with the lowest skills, blended and hybrid models of learning with instructors delivering 50% or more of instruction would possibly be more effective.

The Skills for Life evaluation presents mixed results. Progression increased as course level decreased, suggesting a positive impact on the least skilled students. However, students with Level 3 qualifications were also more likely to progress, as were those who had stayed in education beyond the age of 18.^{[footnote 96]}

Financial dependency

The MAPS intervention found that the intervention was more impactful for financially independent students, compared to those who were financially dependent. The authors argue that this indicates that a monetary incentive may be a more salient motivating factor for adults who are in charge of their own financial situation.^{[footnote 97]}

The impact evaluation of the 16-19 bursary found that the move from the Education Maintenance Allowance (EMA) to the bursary most negatively impacted the most deprived students.^{[footnote 98]}

Ethnicity

The MAPS analysis also suggests that the programme was most effective for BAME students (except those who responded as Hispanic) and less effective for those who identified as white. A subgroup analysis showed that CUNY Start has positive effects for students across all racial and ethnic groups, and, in particular, for Asian students enrolled in English courses, and black and Hispanic students enrolled in maths. A preliminary exploration of achievement gaps suggests that CUNY Start may effectively shrink achievement gaps between black and white, and Hispanic and white students in maths.^{[footnote 99]}

5.4 Cost-effectiveness

The overwhelming majority of studies included in the review do not provide evidence on programme costs, or cost-effectiveness. A small number of studies presenting early impacts of programme note that it is too soon to produce such analysis and that future reports will focus on cost-benefit analysis. This section presents evidence that does exist on intervention cost and cost-effectiveness.

Evidence suggests that high-level investment can produce large enough impact to make the programme cost-effective. CUNY ASAP generated significantly more graduates than the usual college services. Therefore, despite the substantial investment required to operate the programme (the college invested $16,284 more per ASAP group member than it did per control group member, or 63.2% above the amount spent on the typical student receiving usual college services), the increase in the number of students receiving degrees outpaced the additional cost.^{[footnote 100]} Similarly positive results were seen in the MAPS programme. The investment of between $1,394 and $1,863 more per student in the programme than per control group resulted in a 5.6 percentage point increase in the likelihood of completing a college-level maths course. This impact is large enough that it lowers the cost per course completion compared with the usual college services without the programme.^{[footnote 101]}

Evidence suggests that some programmes are not cost-effective. There is some evidence that resource-intensive programmes may not be cost-effective. For example, Job Corps cost $16,500 per participant. Results of the cost benefit analysis over 4-year survey period found measured benefits were less than $4,000; therefore, programme costs exceed programme benefits.^{[footnote 102]} Implementing biweekly disbursements of financial aid was also found to be burdensome and costly. The more frequently biweekly payments were recalculated to account for changes in students’ circumstances (for example, numbers of courses taken), the more expensive it was to maintain the system of incremental disbursements.^{[footnote 103]}

The evaluation of the 16-19 bursary fund estimates that despite short-term savings, the policy made an overall loss to the exchequer of £84 million (which authors state is likely to be an underestimate).^{[footnote 104]}

Some studies estimate high ‘start up’ costs of provision but argue that this cost would reduce over time. An evaluation of 8 summer bridge programmes estimated that the cost per student ranged from $835 to $2,349. However, the authors state that some costs may be interpreted as “start-up” costs, which are unlikely to be needed if the programmes are run in subsequent years. They also suggest embedding support programmes such as these into the regular high school or college schedule would reduce costs.^{[footnote 105]}

Interventions that use text messages to encourage learners’ attendance and achievement were found to be particularly cost-effective methods. The ASK behavioural interventions trials found that the estimated costs of the text message intervention in FE colleges was less than £5 per learner, including the cost of the messages and staff time. An intervention that texted parents to explain how easy the enrolment process was found that at £10 per additional enrolment for the most effective message, it was a cost-effective way to increase enrolment.^{[footnote 106]}

Learning community interventions appear to be low-cost relative to costs overall. The cost per student of learning communities appeared low when compared with how much community colleges spend on average per student. A recent analysis of national postsecondary education expenditures in the US estimated that community colleges spent an average of about $12,000 per year to educate each full-time student. When the cost of learning communities is assessed against this value, the costs appear incremental and may be justifiable to the college, if value is derived from running the programme. However, there are implications on the value given that this approach has minimal impact on student outcomes.^{[footnote 107]}

5.5 Delivering interventions

This section draws on the findings from the evidence, which offer lessons for the design and delivery of learning interventions.

Recruiting learners

Proactive and innovative approaches can help to overcome challenges in recruiting disadvantaged learners. The PACE programmes, which were successful in meeting their recruitment targets, took a proactive approach to engagement and developed new recruitment methods and tracked referral sources to improve target methods.^{[footnote 108]} For example, one of the PACE initiatives extensively marketed the programme through media, partners and word-of­ mouth referrals.^{[footnote 109]} Potential applicants wanting further information prior to applying were able to attend an optional monthly orientation session to learn about the services, eligibility and participation requirements. Potential applicants were also given information on the certificate or degree programmes the initiative supported.

Appropriate selection of key referral partners is vital. An example of the importance of the role of referral partners is illustrated in the Youth Contract programme, which aimed to support 16-24-year-olds to participate in education, training and work. One of the delivery models devolved the funding to 3 core city areas, where 6 local authorities determined the shape and nature of delivery. This meant that the eligibility criteria were determined locally, based on local priority and need. The evaluation found that engagement with local authorities was crucial as they held data, which was an important source of information for targeting appropriate local populations.^{[footnote 110]} Furthermore, delivery agents conducted outreach to support the identification and engagement of learner populations who are hard-to-reach, such as those not in education, employment or training.

However, not all partnerships resulted in increased recruitment. One programme where learning took place in the workplace found that employers responded to applicants with GCSEs, rather than those with equivalent level Functional Skills qualifications.^{[footnote 111]} The authors also noted that voluntary work experience of potential learners did not improve response rates from employers. The authors concluded that more research is needed in this area.

Enrolment

Pre-enrolment sessions that raise awareness of student services and build social networks can be beneficial. The ModMath Programme (a modularized, computer-assisted, self-paced developmental maths course) sought to address these barriers, by offering potential students preparation, or taster, courses prior to enrolling on the full course.^{[footnote 112]} The information provided via the ModMath initiative included briefings about the educational setting (for example, student services and campus navigation and soft skills, such as study and problem­ solving skills). This approach also fostered a sense of community among the learner cohort.

Registration to the programme also opened early, to enable students to choose the courses they wanted to enrol on well in advance of taking up learning. This not only enabled students to create a schedule that was convenient to them, it also increased the likelihood of enrolment on their chosen courses.

Encouraging persistence

Regular and accessible communication can encourage and motivate learners, leading to improved persistence and attainment. Communication interventions can take a number of forms, for example, one-to-one or group face-to-face sessions with an adviser, text messages, emails, phone calls. For example, The Bridge programme provided regular communication from an adviser about transition processes, such as researching future learning options and understanding entry requirements that improved learners’ knowledge about their next steps.^{[footnote 113]} The Bridge intervention successfully built learners’ skills for achieving the GED test, as well as supporting progression into further learning.

Participation in support networks can increase learner engagement and persistence. The increase in support networks, as a result of being part of a Learning Community (which consists of other students and staff), can improve the persistence of underprepared students.

Additionally, researchers suggest that the isolation of low-skilled learners is reduced by linking their developmental course to another course, because they will be more engaged than if basic skills are taught in isolation in a stand-alone course.

Providing learners with a consistent single point of contact, such as an adviser, mentor or key worker, can be beneficial. Participants involved in a number of basic skills programmes noted the important role a consistent point of contact played in their experience of the programme and in helping them to remain engaged in courses. For example, the Moving Forward programme which provided each student with an adviser, addressed the reluctance of Latino men to ask for help because of strong notions of manhood, independence, and self­ reliance.^{[footnote 114]} Higher levels of contact can also foster soft and academic skills development.

Crucial to this approach is action planning to set, record and measure goals.

Student services should be comprehensive and personalised. Student services may include academic support such as tutoring, study skills training, or practical support such as on­ campus childcare, or support with transport. For example, the Opening Doors Programme, assigned students to a counsellor who they met at least twice each semester for 2 semesters to discuss academic progress and to resolve any issues that might affect their schooling.^{[footnote 115]} Researchers also recommend allocating far fewer students than the regular college counsellors to facilitate more frequent, intensive contact with the students on the programme.^{[footnote 116]} When these services are offered via a trusted source (an adviser) and well-marketed, the evidence shows that students are more likely to access them.^{[footnote 117]}

Provision

Provision should consider the aspirations of the target group. The GAIN in California was an occupational training intervention emphasizes large-scale, mandatory participation in basic education, in addition to job search, training, and unpaid work experience, for welfare recipients.^{[footnote 118]} The evaluation found that GED test score gains were concentrated in the site that created an innovative adult education programme tailored to the needs of people on welfare.

Other sites where the programme was not delivered using this approach saw less impact.

Another example of the need to tailor provision is the use of digital interventions for supporting the learning of adults with basic literacy and numeracy skills. In one study, one fifth of participants did not enjoy using ICT to learn and preferred to work directly with instructors, rather than learning on-line.^{[footnote 119]} The researchers commented that depending on the scaffolding techniques used to embed the ICT products and the availability of support from instructors, some learners could become stuck in the digital learning environment and experience frustration. This is an important consideration, particularly for learners with the lowest skills levels. Therefore, blended and hybrid models involving tutors delivering at least half of the instruction was suggested as most effective for basic skills programmes. It is therefore key that programmes need to offer additional student services for disadvantaged students to engage in.

Evidence suggests that provision aimed at improving adults’ literacy and numeracy skills is particularly effective when it embeds vocational content that aligns with students’ career aspirations and/or pathways. The National Workplace Literacy programme was designed to emphasise contextualised, job-specific instruction in basic skills for adults who lack basic skills.^{[footnote 120]} The provision which included English, maths and ESOL was workplace-based. Results indicate that this intervention improved literacy skills, interest in further learning, engagement in literacy tasks at home and self-rated literacy skills of learners. There was also a positive impact on teamwork skills and communication skills at work.

Unlike the other PACE programmes, the I-BEST programme used a ‘team teach’ approach, meaning at least 50% of occupational training class time was delivered by both a basic skills instructor and an occupational instructor. Basic skills instructors used concepts from students’ occupational coursework as a vehicle for building basic academic skills; that is, customizing the content and instructional delivery to draw on examples from occupational content. This approach had positive results – the treatment group members received an average of 13 more credits compared to control group members, and there was a 32-percentage point increase in credential receipt.

There are beneficial outcomes for learners when training addresses both occupation­ specific and basic educational skills. The PACE programmes in the US included a number of studies which incorporated both educational and occupational training.^{[footnote 121]} For example, one study focused on training for low-income Latinos for employment in healthcare occupations (primarily Certified Nursing Assistant and Licensed Practical Nurse).^{[footnote 122]} The intervention increased the number of hours of occupational training and basic skills instruction received, as well as attainment of educational credentials within the 18-month follow-up period. Additionally, the intervention increased employment in the healthcare sector and led to a reduction in the number of participants reporting financial hardship.

Accelerated and intensive programmes can help learner engagement and developmental learning. There have been mixed results from accelerated, intensive learning programmes, however positive behavioural results have been noted. For example, an impact study of a student success course, at a US technical community college, provided intensive focus on changing students’ behaviours and attitudes (such as increasing awareness of their own and others’ emotions, understanding their learning styles and recognizing their responsibility for learning).^{[footnote 123]} The intervention had a positive impact on attributes such as students’ self­ management and awareness but did not result in meaningful gains in academic achievement.

The size of learning groups can affect completion rates. Research examined 3 different types of educational intervention: a team approach, a small group approach and individual tutoring.^{[footnote 124]} The study found that the small group approach had the most significant impact on completion rates among ‘at risk’ adults and adults who had previously dropped out of adult basic education.

Designing courses that relate to career aspirations support learner engagement and continued college participation. Interventions that take a contextualised approach have been found to have a significant motivating factor on learners.^{[footnote 125]} Rather than developing maths, writing and reading through generic, basic skills exercises, students learnt by using materials that were specific to the healthcare or business track they were considering pursuing. One year after enrolling in the intervention, students were far more likely to have finished the course, passed the GED exam and enrolled in college, than students who had a more traditional preparation course.

Contextual approaches help make learning basic skills more relevant and meaningful. For example, one study conducted in a prison environment, used science and the natural world as a way of making reading more contextually relevant and meaningful.^{[footnote 126]} Learners practiced and used reading skills and behaviours to learn more about a topic that was of interest to them. The experimental curriculum evaluated in the intervention contributed to reading gains while stimulating and sustaining learners’ interest, motivation, and enthusiasm.

Sustaining outcomes

Providing learners with ongoing support and access to services can improve sustained outcomes. A number of researchers report that the beneficial impacts of interventions were not sustained in the long term, and yet there were few suggestions as to how this problem might be addressed. This is a highly relevant question, since disadvantaged learners will most likely continue to face barriers beyond the programme lifetime; this leads authors to raise the question of extending the length of time support interventions are available.^{[footnote 127]}

Measuring outcomes

When measuring outcomes of basic skills interventions, those delivering the programmes need to consider what outcome measures would most effectively indicate success. Studies note that for learners the impacts of the interventions go more broadly than skills attainment alone, for example self-awareness, self-esteem and mental health.^{[footnote 128]} Research using a Community-based English language intervention was designed to improve functional English provision for individuals with an ESOL need.^{[footnote 129]} Analysis of the social integration outcomes indicated that the intervention led to improvements in social interaction and bond forming. There was also evidence of increased confidence in independently going shopping and engaging with health professionals. The research results indicate that Community-Based English Language courses increase English language ability and encourage wider social integration.

6. Conclusions

In line with what is highlighted in existing literature reviews, a key finding from this review is the overall scarcity of evidence on what works to improve attainment among disadvantaged students in FE and adult learning in the UK. This finding strengthens the case for investment in research which tests interventions to build our knowledge of how best to support disadvantaged groups achieve outcomes (as set out in the policy paper that accompanies this report). Future research should explore the entire learner journey to build our understanding of what works to improve outcomes in the FE sector, from participation to related longer-term socio-economic outcomes.

The studies in this review explore the impact of interventions designed to support disadvantaged students. However, very few explore the effectiveness of an intervention for specific groups of adults. For example, the review found very few studies exploring what works to support ESOL learners, lone parents, prisoners or NEET young people and did not find any studies which targeted BAME learners, young adult carers, care leavers or learners with LDD. There is also a scarcity of evidence on the cost-effectiveness of programmes. To improve understanding of how best to support specific groups of disadvantaged adults, further research is needed that explores what works for whom.

6.1 Recommendations

Government should invest £20 million over 5 years to establish a What Works Centre for FE. The policy brief that accompanies this review provides more detailed information on this investment and priorities needed.^{[footnote 130]}

Government and the Centre should focus on what works across all stages of the learner journey, from participation to longer-term socio-economic outcomes. This will require investment in long-term evaluations that track outcomes over several years.

Government and the Centre should also focus on what works for specific groups of learners. Disadvantaged groups, such as care leavers and adults with ESOL needs, face specific barriers to learning. Yet there is limited evidence on what works for whom. 

7. Annex 1: The evidence 

7.1 Study 1

Department for Education, Improving engagement and attainment in maths and English courses: insights from behavioural research, 2018, accessed 24 September 2019

Department for Business, Innovation and Skills funded the creation of the Behavioural Research Centre for Adult Skills and Knowledge (ASK). ASK aimed to use behavioural science to examine different ways of supporting learners, with Level 1 and Level 2 English and maths, to improve their skills in these subjects.

Interventions were aimed at addressing dispositional barriers – for example, learners’ beliefs, attitudes, perceptions of learning, and changing behaviours towards learning.

The interventions took place at FE colleges, workplaces, community settings and a laboratory setting. Interventions tried were weekly text messages to improve learners’ attendance (19+ year group}, social support texts to family and friends about learners’ progress, weekly text messages and social support to learners (16-19-year-old group}, reflective writing exercise (beliefs about learning}, and feedback (highlighting effort and performance).

Randomised control trials (RCTs) were designed to evaluate a variety of behavioural interventions. The participants were aged 16 to 19 years and adult students (ages of adults not stated}, who were enrolled in maths and English courses at FE colleges in England. There was an excess of 22,477 participants (not all projects stipulated numbers).

Weekly text messages improved attendance of adult learners (19 years and over) by 22% and achievement rates by 16% (GCSE grade A* to C); social support texts improved attendance rates 5% and achievement rates by 27% (GCSE grade A* to C); weekly text messages and social support to learners aged 16 to 19 years improved attainment rates by 24% (GCSE grade A* to C); the reflective writing exercise improved attainment in maths and English by 25% (GCSE grade A* to C).

Most successful trials were those aimed at people already involved in traditional learning programmes (for example at FE colleges). Those individuals in employment were more difficult to engage in behavioural interventions.

The estimated costs of the text message intervention in FE colleges was less than £5 per learner, including the cost of the messages and staff time. An intervention that texted parents to explain how easy the enrolment process was found that at £10 per additional enrolment for the most effective message, it was a cost-effective way to increase enrolment.

7.2 Study 2

Department for Education, The relative effectiveness of blended versus face-to­ face adult English and maths learning, 2018, accessed 24 September 2019

Pedagogical intervention, using blended learning, delivered by Adult Education providers in 13 centres. Participants had Entry Level 1, Level 1 and Level 2 English and Maths. The aim was to test whether blended learning and face-to-face learning differ in their effectiveness at increasing skills. The primary outcome of interest was a measure of skill change using Item Response Score for Reading and Maths and Total Score for Writing.

Randomised control trial in England to assess the impact of delivery mode on skills. Learners were aged 19 years and over, and the original sample size was 863 (472 for English and 391 for maths).

There was a substantial sample loss, which resulted in the collection of pre and post test data for only 58 writing learners and 75 maths learners. Owing to the sample size the ability of this RCT to detect differences was greatly reduced and the findings were non-significant.

No details on cost or cost effectiveness.

7.3 Study 3

CfBT, Raising maths attainment through enhanced pedagogy and communication: Results from a ‘teacher-level’ randomised controlled trial, 2013, accessed 24 September 2019

The programme explored the impact of training teachers on a Neurolinguistic Programming Model (NLP) and learners’ maths attainment in FE colleges in England.

RCT with 173 learners aged 16 to 71 years, who required adult basic skills provision.

The addition of NLP training produced a significant improvement in maths attainment, as measured by the mean percentage score on the single level maths test. The increase in mean difference for this group was over 3 times that of the control group and approximately one and a half times that of the ‘maths training only’ teacher group. Results suggest some NLP training may be helpful to maths teachers, where a baseline of effective pedagogy is in place.

A total of 24 teachers (6 in the control condition, 9 in each of the other conditions) completed the full research cycle. 173 adult learners completed the post-treatment maths test (43 with teachers who had not had NLP training, 67 with a teacher with training in innovative maths pedagogy, and 63 with training in NLP and innovative maths).

Cost effectiveness was not mentioned.

7.4 Study 4

MDRC, Moving Forward: Early findings from the Performance-Based Scholarship Demonstration in Arizona, 2013, accessed 24 September 2019

The aim of this programme was to test an alternative incentive structure on performance at a community college in the USA. Students eligible for the Performance Award Programme could receive up to $4,500 in total over 3 semesters. Payments were contingent on their meeting academic benchmarks throughout the semester and participating in support services (for example, tutoring and workshops).

1,028 students were randomly assigned to the treatment group (611), or the control group

(417). The target group were young Latino men, with low income status (defined as having an Expected Family Contribution of $5,273 or less).

Participants had a net increase in financial aid, including less reliance on loans than students in the control group and were more likely to stay in college for a second semester, to enrol full-time, and to earn more college credits over the first year of the programme. The programme increased full-time enrolment in students’ second semester. In their second semester in the programme, students in the programme group were 13.2 percentage points more likely to enrol full time compared with a control group mean of 48.8% (a 27% increase). The programme increased the number of credits earned. Students in the programme group earned almost 2 full credits more than those in the control group over the first year of the programme.

This RCT did not allow researchers to separate the effects of one programme component from another. This study could only determine whether the entire Programme package was effective.

Cost effectiveness was not stipulated.

7.5 Study 5

MDRC, Mapping Success: Performance-Based Scholarships, Student Services, and Developmental Math at Hillsborough Community College, 2014, accessed 24 September 2019

Mathematics Access Performance Scholarship (MAPS) provided an incentive for low-income students at community colleges, who were referred to developmental maths, to take their maths courses early and consecutively, and to have help with maths in an on-campus Math Lab. The aim was for students to pass with the necessary grades, in return for a modest scholarship contingent on performance.

A US RCT with a sample size of 1,075 students with low income status (having an Expected Family Contribution of $5,273 or less), who needed basic maths support.

More than 90% of programme group students earned a MAPS award (payment) in the first semester, but scholarship receipt rates declined in the second and third semesters. MAPS students were much more likely to visit a Math Lab than control group students (engagement). MAPS helped move students further along in the maths course sequence, and MAPS students accumulated more credits overall, partly because of their greater progress in maths. The programme had no discernible impact on student retention.

The cost analysis showed that the $1,394 to $1,863 of additional investment in each programme group member resulted in a 5.6 percentage point increase in the likelihood of completing a college-level maths course. This impact was large enough that when costs (direct programme costs plus the cost of credits attempted by students over 2 years) were tied to the number of college-level math course completions, the programme lowered the cost per outcome achieved in comparison with the usual college services without the programme. Specifically, the cost per college-level maths course completion for the programme group is between $126 and $1,594 less (or up to 5% less) than the cost per college-level maths course completion for the control group.

The programme resulted in students earning 0.7 more maths credit and 1.6 more credits overall; these impacts were not large enough, however, to lower the cost per maths credits or total credits earned.

7.6 Study 6

MDRC, Four-Year effects on Degree Receipt and Employment Outcomes from a Performance-Based Scholarship Program in Ohio, 2015, accessed 24 September 2019

Randomised controlled trial involving 2,285 community college students, aged 18 years and older and focusing on low-income parents, the majority being low-income mothers. The intervention was financial support, that is, a performance-based scholarship.

The study included 4 years of post-random assignment data to examine the long-term impact of a performance-based scholarship programme, since financial aid is contingent on academic performance, on degree receipt, employment, and earnings.

Findings provide evidence that the programme decreased the time it took students to earn a degree, but do not provide evidence that the programme increased employment or earnings by the end of the study period. More programme than control group participants met full-time credit benchmarks and programme students earned more credits overall.

No programme cost information.

7.7 Study 7

American Economic Corporation, Does Job Corps Work? Impact Findings from the National Job Corps Study, 2008, accessed 24 September 2019

This study examined the effectiveness of the Jobs Corp programme on improving outcomes for disadvantaged youth. Participants were those receiving welfare or food stamps, or with very low income, or who lived in a disruptive environment and needed additional education, training or job skills. The intervention was residential academic and occupational training at Job Corps Centre.

RCT in USA. Disadvantaged 16-24-year olds; 15,400 treatments and controls.

Programme participation increased educational attainment, reduced criminal activity and increased earnings for several post programme years.

Cost $1.5 billion dollars and with 60,000 new participants per year, so the cost equals

$16,500 per participant. Results of the cost benefit analysis over 4-year survey period found measured benefits were less than $4,000; therefore, programme costs exceed programme benefits.

7.8 Study 8

MDRC, Staying on Course: Three-Year results of the National Guard Youth ChalleNGe Evaluation, 2011, accessed 24 September 2019

To test the effectiveness of ChalleNGe, which aimed to support disconnected, at-risk youth (high school dropouts) in a quasi-military setting. First phase was physical orientation and second phase was primarily classroom based, aimed at supporting students to prepare for the GED exam. Third phase was a placement, which could include employment, education and military service.

RCT with a sample size of 1,200 individuals, aged 16 to 24 years in USA.

Members of the programme group were much more likely than those in the control group to have obtained a GEDcertificate or a high school diploma and to have earned college credits.

No information provided on costs.

7.9 Study 9

MDRC, Enhancing GED Instruction to Prepare Students for College and Careers, 2013, accessed 24 September 2019

To test the effectiveness of the programme, which aimed to prepare students and adults without a GED exam to pass the GED, and also continue on to college and training programmes. Whilst it was an essential skills provision intervention, rather than developing maths, writing and reading comprehension skills through generic exercises, students learnt by using materials specific to the health care or business track they were considering pursuing. In-class time was 108 hours over 12 weeks.

RCT with a sample size of 369 in a community college setting in the US. Participants were post-18 years, average age was 26.6 years

One year after enrolling ‘Bridge’ students were far more likely to have completed the course, passed the GED exam and enrolled in college than students in a more traditional GED preparation course.

No information on cost analysis.

7.10 Study 10

Washington, DC: Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services, Valley Initiative for Development and Advancement: Implementation and Early Impact Report (PACE), 2017, accessed 24 September 2019

The aim of this evaluation was to examine the actual implementation of the intervention. Did it deviate from plans or expectations? What were the students’ participation patterns and experiences with programme services? What were the main effects of the Valley Initiative for Development and Advancement (VIDA) on educational attainment, including credentials received, and other educational outcomes? The intervention itself was occupational training, with counselling, financial support and accelerated basic skills intervention.

RCT in US a community-based organisation. Post-18-year-old low-income students, who had to be unemployed, underemployed, meet federal poverty income levels, or be on public assistance. 958 study participants (478 in treatment group and 480 in control group).

Training for nursing and allied health professionals were the most common programmes attended, followed by education, social services and specialised trade programmes. The treatment group earned significantly more college credits than the control group (33.1 and 27.5 respectively) and VIDA significantly increased the rates of full-time college enrolment and enrolment more generally. The treatment group members earned significantly more college credentials.

Although these effects on credits may not immediately appear to be large, these impacts are among the largest reported to date from random assignment tests of programmes aiming to increase college success among low-income individuals.

VIDA’s direct assistance with tuition and related training expenses averaged $7,000 per participant

For all 9 of the PACE programmes, the most relevant element was the ‘College Prep Academy’; a 16-week, accelerated, full-time basic skills (‘bridge’) programme for those not college ready, but who achieved 10th grade skill levels or better. The course met daily to prepare individuals to pass college entrance exams.

The research team pose the question whether this intervention is cost beneficial. At the time of the report no cost analysis had been undertaken. The authors comment that the research team will examine this question at both 36 and 72 months after random assignment. They state that given the programme requires substantial up-front investments in both participant time and programme resources, the initial answer will be quite tentative and the later more definitive.

7.11 Study 11

Washington, DC: Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services, Bridging the Opportunity Divide for Low-Income Youth: Implementation and Early Impacts of the Year Up Program (PACE), 2018, accessed 24 September 2019

US intervention, which provided 6 months of full-time customized training in the IT and financial service sectors, followed by 6-month internships at major firms. For first 6 months of the programme students attend classroom-based courses at Year Up full time. Training addressed both occupation-specific and general skills.

Programme aimed to support low-income young adults aged 18 to 24 years who had a high school diploma, or equivalent, to overcome challenges and build successful careers in fast­ growing technical occupations. Evaluation team randomly assigned 2,544 young adults to treatment (1,669) and control (875) groups.

Average quarterly earnings, in the sixth and seventh quarters after random assignment, were $1,895 higher for the treatment group ($5,454) than for the control group ($3,559), a 53% impact. Impacts diminished in the following year but remained at 40%.

Cost effectiveness is forthcoming (but not available currently).

7.12 Study 12

Washington, DC: Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services Washington, Des Moines Area Community College Workforce Training Academy Connect Program: Implementation and Early Impact Report (PACE), 2018, accessed 24 September 2019

One of 9 PACE programmes for participants who already had a high school diploma or equivalency credential, basic skills remediation was provided online, alongside occupational training. For participants who did not have a high school diploma or equivalency credential, they were enrolled in the GED exam or the High School Equivalency Test (HiSET) preparation courses at the Des Moines Area Community College, rather than in the Workforce Training Academy (WTA) Connect basic skills remediation.

RCT at a Training Academy in USA to assess the effectiveness of a career pathways programme. 943 study participants (470 to the treatment group and 473 to the control group), who were low income, low skilled students. Age range was post-18 years; over 40% of participants were older than 35 years.

WTA Connect resulted in a modest increase in attainment of credentials by participants within the 18-month follow-up period, but there were no other educational or career impacts.

No cost analysis provided in the report.

7.13 Study 13

Washington, DC: Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services Madison Area Technical College Patient Care Pathway Program: Implementation and Early Impact Report (PACE), 2018, accessed 24 September 2019

One of the 9 PACE programmes to support academic preparation for, and accelerate entry into, Madison’s College 1-year and 2-year healthcare programmes. The intervention was occupation training, accelerated basic skills introduction with credit-bearing courses and advice.

RCT in a US community college setting with 500 study participants (251 randomised to the treatment group and 249 to the control group). Low income students, aged post-18 years, of whom 55% were 25 years and over.

The intervention did not increase the number of college credits earned. The programme increased the likelihood of enrolment in occupational training but did not increase hours of occupational training received or the attainment of education credentials within and 18- month follow-up period.

No cost analysis provided in the report.

7.14 Study 14

Washington, DC: Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services, Pima Community College Pathways to Healthcare Program: Implementation and Early Impact Report (PACE), 2017, accessed 24 September 2019

One of the 9 PACE programmes, this particular intervention was occupational training. The treatment group accessed the accelerated basic skills college readiness class, or the open lab for those who did not test into their training of choice.

RCT at US Community college with 1,217 study participants (609 in treatment group and 608 in control group). Low income students aged 17 years and older.

The treatment group attended significantly more hours of occupational training (the primary outcome measured in this report) than the control group. This was primarily because the treatment group enrolled in occupational training more often (in healthcare related training). The treatment group was significantly more likely than the control group to participate in advising such as career counselling, to receive help arranging supports, and to receive job search assistance. The treatment group earned significantly more college credentials (degrees, certificates) than the control group.

No cost analysis provided in the report.

7.15 Study 15

Washington, DC: Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services, Washington State’s Integrated Basic Education and Skills Training (I-BEST) Program in Three Colleges: Implementation and Early Impact Report (PACE), 2018, accessed 24 September 2019

One of the 9 PACE programmes, the signature feature is team teaching by a basic skills instructor and an occupational instructor during at least 50% of occupational training class time. The specific intervention for this study was occupational training.

RCT at Community College/Technical Colleges in USA; 632 participants (315 treatment and 317 controls). Participants were low income students, aged 18 years and over.

The Integrated Basic Education and Skills Training (I-BEST) programmes at the 3 colleges increased participation in college-level courses, number of credits earned. and credential attainment.

Future analyses will explore the costs of the programme relative to the benefits it produces for participants and society.

7.16 Study 16

Washington, DC: Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services, Workforce Development Council of Seattle-King County Health Careers for all Program: Implementation and Early Impact Report (PACE), 2017, accessed 24 September 2019

Occupational training programme that supported 3 levels of skills training: Foundation Level, Entry Level and Advanced Level.

RCT at Community College in USA with post-18, low income students or those who have a barrier to training and employment (for example low basic skills or disabilities). 654 participants (328 in treatment group and 326 in the control group).

Health Careers for All increased the percentage of participants enrolling in healthcare related training over an 18-month follow-up period. However, there was no impact overall on receipt of a credential, or total hours of occupational training.

No cost analysis provided in the report.

7.17 Study 17

Washington, DC: Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services, lnstituto de/ Progreso Latino’s Carreras en Sa/ud Program: Implementation and Early Impact Report (PACE), 2018, accessed 24 September 2019

This PACE programme research had a focus on training for low-income Latinos for employment in healthcare occupations, primarily Certified Nursing Assistant (CNA) and Licensed Practical Nurse (LPN). The Carreras en Salud Program consists of 5 elements:

• a structured healthcare training pathway, starting at low skill levels
• contextualised and accelerated basic skills and ESL instruction
• academic advising and non-academic supports
• financial assistance
• employment services

RCT in USA at the Institute del Progreso (an independent training provider charity), providing occupational training for post-18, low income students. 800 study participants (402 in treatment group and 398 in the control group).

The programme increased the hours of occupational training and basic skills instruction received, and the attainment of education credentials within an 18-month follow-up period. The programme also increased employment in the healthcare field and resulted in a reduction of participants reporting financial hardship.

Future analyses will explore the costs of the programme relative to the benefits it produces for participants and society.

7.18 Study 18

MDRC, Incremental Disbursements of Student Financial Aid: Final Report on Aid Like A Paycheck, 2019, accessed 24 September 2019

Programme examining a financial support intervention. The setting was a US Community College and tested whether changes to the timing of student aid disbursement could help students stretch their financial aid to cover their expenses throughout the term; also, whether such a policy could improve students’ academic and financial outcomes.

RCT with a sample size of 8,840 low income students, aged 18 years and over (average age 22 years). Academic outcomes are measured by credits earned.

Bi-weekly disbursements reduced students’ debt to the college in the first semester, but this reduction was no longer evident at the end of the fourth semester. On average there was no evidence of bi-weekly disbursements improving students’ key academic outcomes. Little evidence that participating colleges, or the government, saved money by implementing bi­ weekly disbursements.

Implementation of the policy was costlier than (and not as simple as) expected, even when implemented without the constraints of the RCT.

There is little evidence that the participating colleges or the government saved money by implementing biweekly disbursements. Generally speaking, the more frequently biweekly payments were recalculated to account for changes in course loads, or changing financial aid availability, the costlier and more burdensome it was to maintain the system of incremental disbursements. In broad terms, over time the colleges found that conducting less frequent and less exacting adjustments eased the burden (and lessened the costs) of calculating and clearly communicating about biweekly disbursements. One college typically conducted major recalculations only when students withdrew; other colleges generally recalculated more frequently during the study period but reported that over time they reduced the number of these recalculations in order to lower their burden and costs. In summary, the authors concluded that if biweekly disbursements were to become the standard policy at other colleges in the future, it is likely that while some of the costs would not be as high for future adopters, other costs would remain.

7.19 Study 19

MDRC, A Randomized Controlled Trial of a Modularized, Computer-Assisted, Self-Paced Approach to Developmental Math, 2018, accessed 24 September 2019

This trial’s intervention was essential skills provision; the aim of the evaluation was to test the effect the opportunity to enrol in ‘ModMath’ had on students’ likelihood of completing the developmental maths course sequence, when compared to the opportunity to enrol in the college’s ‘traditional’ maths courses.

USA RCT with a sample size of 1,403. Participants were post-16 adults who lack basic skills; the majority in the sample were 19 to 24 years. The trial was in a community college setting. The outcomes were measured by participation (enrolment in college and enrolment in any maths class) and progress on/completion of developmental maths

‘Modmath’ not found to be superior to a ‘traditional’ maths class. After 3 semesters, 23% of programme group members had completed the developmental math sequence, the primary outcome of interest. Similarly, 22% of the control group had achieved the same milestone.

No cost analysis provided in the report.

7.20 Study 20

MDRC, Doubling Graduation Rates: Three-year effects of CUNY’s Accelerated Study in Associate Programmes (ASAP) for Developmental Education Students, 2015, accessed 24 September 2019

The researchers wanted to test whether Accelerated Study in Associate Programmes (ASAP) – a Community College reform – could support more students in remedial education to graduate and to graduate quickly. City University of New York (CUNY) ASAP consisted of a multifaceted, integrated, and long-lasting programme for community college students, providing a range of services and support to help more students graduate and to help them graduate sooner. The programme aimed to address multiple barriers to student success over 3 full years. ASAP required students to enrol full-time and take developmental courses immediately and continuously. The programme offered comprehensive support services such as closely monitored advice, career development, and tutoring. The intervention also offered financial support, including tuition waivers for students in receipt of financial aid, textbook vouchers, and monthly MetroCards for use on public transport. The programme also provided structured course enrolment to support academic momentum, with block and consolidated course schedules and support for course taking in winter and summer sessions. The ASAP intervention represented both an opportunity and an obligation for students, because they needed to continue enrolling full time and participating in the programme to receive the programme’s benefits.

RCT at New York Community College for low income, low skilled, post-16 students. A total of 896 students were randomised, 451 in the programme group and 445 in the control group.

ASAP substantially improved students’ academic outcomes over 3 years, almost doubling graduation rates.

At the 3-year point, the cost per degree was lower in ASAP than in the control condition. Because the programme generated so many more graduates than the usual college services, the cost per degree was lower despite the substantial investment required to operate the programme. While it cost more to run ASAP, the increase in the number of students receiving degrees outpaced the additional cost.

7.21 Study 21

MDRC, Bringing CUNY accelerated study in associate programmes (ASAP) to Ohio, 2016, accessed 24 September 2019

This was an RCT with post-16-year-old, low-income individuals requiring developmental education. The study examined whether the Accelerated Study in Associate Programmes (ASAP) intervention could support more students in remedial education to graduate and to graduate quickly. The intervention required students to enrol full-time and take accelerated developmental courses immediately and continuously. ASAP also offered comprehensive support services, such as career development and tutoring and financial support.

The outcome of the evaluation of ASAP, which looked at 2 cohorts from 3 community colleges, found that after 3 years 40% of ASAP programme group students had graduated, compared with 22% of control group students. Additionally, the students who were offered the intervention had better short-term outcomes, including positive effects on enrolling full time, earning more credits, and persisting into the second semester.

Authors note that the evaluation will continue tracking longer-term academic data. In addition, the evaluation will examine the implementation of the Ohio programmes, including fidelity to the model and the key differences between the programmes and standard college services, and analyse the cost-effectiveness of the Ohio programmes.

7.22 Study 22

MDRC, Becoming College-Ready: Early findings from a City University of New York (CUNY) start Evaluation, 2018, accessed 24 September 2019

An essential skills provision intervention to test whether City University of New York (CUNY) Start programme boosts success rates of least prepared college students (those who are referred to remedial maths or English to build their basic skills). The programme delays college matriculation (enrolment in a degree programme) for one semester, and CUNY Start provides intensive instruction in maths, reading, and writing during that semester with a prescribed instructional approach. It also provides advising, tutoring, and a weekly seminar that teaches students skills they need to succeed in college. Students pay $75 for the programme and cannot use financial aid.

US RCT with 3,835 post-16 students who lack basic skills on entry to Community College, or who are assessed as not prepared to work for college level coursework.

During the first semester in the study, programme group students made substantially more progress through developmental education than control group students, while control group students earned more college credits. During the second semester, programme group students enrolled at CUNY colleges (that is, participated in CUNY Start, or enrolled in any non-CUNY Start courses as matriculated students) at a higher rate than control group students. In each of the subgroups of students examined, the programme group made more progress in developmental education and earned fewer college credits in the programme semester than the control group.

A report on longer-term effects and cost-effectiveness is scheduled to be published by MDRC in 2020.

7.23 Study 23

MDRC, Keeping Students on Course: An impact study of a student success course at Guildford Technical Community College, 2012, accessed 24 September 2019

Intervention was a Community College Reform, more specifically, a student success course for developmental education students aimed at improving psychosocial awareness and academic achievement. Intervention is modelled on Skip Downing’s On Course philosophy and curriculum. The initiative placed an intensive focus on changing students’ behaviours and attitudes, including increasing their awareness of their and others’ emotions, understanding their own learning styles, improving time management skills, and recognising their responsibility for their own learning.

US RCT with the aim of examining the impact of the programme (a success course) on students’ achievement. A sample total of 911 students, 458 in the treatment group and 453 in the control group. All students were at least 18 years old and were lacking basic skills such that they were required to take remedial/developmental courses.

The course had a positive impact on students’ self-management, interdependence, self­ awareness, interest in lifelong learning, emotional intelligence, and engagement in college among students with low levels of these attributes. But the gains in efficacy did not lead to meaningful effects on students’ academic achievement during the programme semester, or in post-programme semesters. Despite the absence of an overall effect, the programme did have positive effects on the first cohort of students enrolled in the study, with students demonstrating improved grades, retention in college, and credits earned.

No cost analysis provided in the report.

7.24 Study 24

MDRC, Serving community college students on probation: 4-year findings from Cheffey College’s Opening Doors Program, 2011, accessed 24 September 2019

The programme intervention was a Community College Reform. The target group of students were those on academic or progress probation, had earned fewer than 35 credits toward a degree or credential, did not have an associates’ degree (or higher) from an accredited college or university and had a high school diploma or GED certificate. The ‘College Success’ course, taught by a counsellor, provided basic information on study skills and the requirements of college. As part of the course, students were expected to complete 5 visits to ‘Success Centers,’ where their assignments, linked to the College Success course, covered skills assessment, learning styles, time management, use of resources, and test preparation.

US RCT with 444 students (224 in treatment group and 220 in control group). Participants were aged 18 to 34 years.

The college’s Enhanced Opening Doors programme had positive short-term effects; when the 2 programme semesters were complete, students in the programme group had earned more credits than students in the control group and were nearly twice as likely as control group students to be in good academic standing. Despite the programme’s encouraging short-term effects, it did not meaningfully improve students’ long-term academic outcomes. Four years after the study began, programme and control group students had made similar academic progress. Strikingly, during that time, only 7% of all students in the study had earned a degree or certificate. The authors comment that by making some classes optional the participation rate dropped, and that future programmes should consider the message sent regarding required and optional courses if a higher rate of overall compliance is to be achieved.

When the cost is considered in comparison with outcomes, the Enhanced Opening Doors programme produced mixed results in the short term and was likely cost-neutral over the long term. Notably, the programme was cost-effective at moving students into good academic standing in the short term. Additionally, the cost per earned credit during the 2 programme semesters was slightly lower for programme group members compared with control group members, but this positive observation is seen only when all credits are considered and fades away when considering only degree-applicable and developmental credits. Finally, impacts on key long-term outcomes, such as degree-applicable and developmental credits earned, were not statistically significant. As a result, the programme was not able to display definitive cost effectiveness over the long term; it seems more likely that the programme is simply cost-neutral, meaning that over the long term, the cost per key outcome is essentially the same with or without the programme.

7.25 Study 25

MDRC, Learning Communities for Students in Developmental Math Impact Studies at Queensborough and Houston Community, 2011, accessed 24 September 2019

This research used Learning Communities as an intervention. In its most basic form, a learning community model co-enrolls a cohort of students into 2 classes together. Proponents believe that when students spend time together in multiple classes, they are more likely to form social and academic support networks that in turn help them persist and succeed in school.

Students were only eligible for the programme if they needed developmental education in the subject area targeted by the college.

US RCT with 1,034 participants at Queensborough Community College and 1,273 at Houston Community College. Participants were aged 16 years and older.

At both colleges, students in learning communities attempted, and passed, their developmental maths class at higher rates than students in the control group. However, this impact generally did not translate into increased cumulative progress in maths by the end of 2 or 3 semesters.

No cost analysis provided in the report.

7.26 Study 26

NPCR, Learning Communities for Students in Developmental Reading An Impact Study at Hillsborough Community College, 2010, accessed 24 September 2019

This research used Learning Communities as an intervention. Students were only eligible for the programme if they needed developmental education in the subject area targeted by the college.

US RCT with 1,071 participants at Hillsborough Community College. Participants were aged 16 years and older.

Overall, for the full study sample, Hillsborough’s learning communities programme did not have a meaningful impact on students’ academic success.

No cost analysis provided in the report.

7.27 Study 27

NPCR, Learning Communities for Students in Developmental English Impact Studies at Merced College and The Community College of Baltimore County, 2012, accessed 24 September 2019

This programme’s intervention was focused on Learning Communities.

US RCT with 1,424 programme and control students in the Merced College sample, and 1,083 students in the Community College of Baltimore County sample. Participants were post-16 and in need of developmental English support.

At Merced College, Learning Communities students attempted and earned significantly more developmental English credits, than students in the control group during the programme semester. At the end of the subsequent semester, they had passed significantly more English courses than their control group counterparts. At the Community College of Baltimore County, there were no meaningful impacts on students’ credit attempts or progress in developmental English. On average, neither college’s Learning Communities programme had an impact on college registration in the post-programme semester, or on cumulative credits earned.

The researchers conclude that at both colleges, the cost per student of learning communities appeared low when compared with how much community colleges spend on average per student. A recent analysis of national postsecondary education expenditures estimated that community colleges spent an average of about $12,000 per year to educate each full-time student. When the cost of learning communities is assessed against this value, the costs appear incremental and may be justifiable to the college, if value is derived from running the programme.

7.28 Study 28

MDRC, A Random Assignment Evaluation of Learning Communities at Kingsborough Community College: Seven Years Later, 2014, accessed 24 September 2019

The research intervention was Learning Communities at Community Colleges. Students were eligible if they were first-time incoming freshmen planning to attend college full time during the day, and they tested into either developmental English or college-level English. These students are not low-income, nor are they necessarily in remedial English. The aim was an evaluation of the impact of learning communities on short-term and long-term academic progress and graduation rates.

US RCT with 1,534 participants, of whom 769 were randomly assigned to the programme group and 765 to the control group. Participants were aged between 17 and 34 years.

Findings indicate that the programme’s positive effects on short-term academic progress (credit accumulation) were maintained 7 years after random assignment. In addition, the analysis provides some limited evidence that the programme positively affected graduation rates, particularly for those students without remedial English needs, over this period. At the same time, however, there is no discernible evidence that the programme improved economic outcomes.

No cost analysis provided in the report.

7.29 Study 29

MDRC, Paying for Persistence: Early Results of a Louisiana Scholarship Program for Low-Income Parents Attending Community College, 2006, accessed 24 September 2019

The intervention was financial support at a Community College. The aim of the evaluation was to test the effectiveness of a financial incentive on progress. Students were aged 18 to 34. To be eligible, students had to meet the following criteria: Be a parent of at least one dependent child under age 19; family income below 200% of the federal poverty level; a high school diploma or GED certificate and a passing score on a college entrance exam, and no degree or occupational certificate from an accredited college or university.

US RCT with a sample size of 1,019. The programme group received the Opening Doors scholarship and counselling, and the control group received whatever regular financial aid and counselling was available to all students.

The outcome of the intervention was a positive impact on full-time enrolment and a positive impact on pass rates and course credits. There were also higher rates of registration in colleges in the second and third semesters after random assignment.

No cost analysis provided in the report.

7.30 Study 30

MDRC, Improving Basic Skills: The Effects of Adult Education in Welfare to Work Programmes, 2002, accessed 24 September 2019

The programme intervention was essential skills provision for post-16 students at adult education providers. Classes were adult basic education (ABE) classes, programmes preparing students for the GED exam, regular high school classes, and ESOL classes. The programme population were highly disadvantaged students who were low-income, mostly jobless, single-parent women who lacked a high school diploma and were receiving welfare. Other evaluation aims were to what extent, and for whom, do welfare-to-work programmes increase participation in adult education services and increase educational attainment and achievement? And, do education-focused welfare-to-work programmes improve education outcomes?

US RCT, sample size 5,863.

Gains in reading skills appeared to vary with the length of time spent in the adult education programmes. Improvements in maths skills and GED receipt were associated with shorter spells of adult education. As students earned GEDs, increased basic skills, or subsequently participated in postsecondary programmes, they appeared to have substantial benefits in terms of employment, earnings, and self-sufficiency.

However, relatively few adult education participants received a GED, increased their basic skills, or entered postsecondary programmes. Receipt of a GED credential was an important predictor of subsequent enrolment in postsecondary programmes. Participants in basic education programmes who went on to postsecondary education or training programmes appeared to experience substantial benefits in terms of increased earnings and self­ sufficiency.

No cost analysis provided in the report.

7.31 Study 31

Economic Mobility Corporation, Escalating Gains: The Elements of Project QUEST’s Success, 2018, accessed 24 September 2019

The aim of the evaluation was to test Project Quest’s impact on earnings. The Quest project provides comprehensive support and resources to help individuals complete occupational training programmes at local community colleges and professional training institutes, pass certification exams, and obtain jobs in targeted industries.

US RCT with 410 participants of whom 207 were in the treatment group and 203 in the control group, aged between 16 and 64 years.

Project Quest students earned $5,000 a year more than control group students. QUEST participants were significantly more likely than control group members to earn a postsecondary credential and to work in the targeted health-care field. QUEST recruited individuals for the study who were not enrolled in college, 63% of whom needed remedial maths and/or reading instruction before they could enrol in their occupational programmes; this group typically has low degree completion rates. Impacts were greatest among participants aged 25 and older, and those who had a GED rather than a high school diploma, usually non-traditional college students who were unlikely to navigate their way through postsecondary training to a good job on their own.

No cost analysis provided in the report.

7.32 Study 32

Community College Research Center, New evidence of success for Community College remedial English students: Tracking the Outcomes of Students in the Accelerated Learning Program (ALP), 2012, accessed 24 September 2019

This intervention was essential skills provision for post-16 underprepared students in a community college setting.

Regression analysis and propensity score matching, with a sample size of 592.

Results suggest that among students who enrol in the highest-level developmental writing course, participation in ALP is associated with substantially better outcomes in terms of English 101 completion and English 102 completion (college-level English courses). The authors also compared college-ready students enrolled in ALP sections of English 101 with their counterparts in wholly college-ready sections and found that those in ALP sections had equivalent performance within English 101 itself, but slightly lower subsequent college-level course enrolment and completion.

No cost analysis provided in the report.

7.33 Study 33

NCPR, Bridging the gap: An impact study of eight developmental summer bridge programmes in Texas, 2012, accessed 24 September 2019

The aim of the evaluation was to test the effectiveness of alternative remedial education with underprepared students and recent high school graduates, aged 16 years and older, in community college settings. The programmes involve intensive remedial instruction in maths, reading and/or writing and college preparation content for students entering college with low basic skills.

US RCT with 793 in the treatment group and 525 in the control group.

While programme participation did not appear to increase college enrolment, there is evidence that programme students were more likely to pass college-level courses in math and writing, when measured by credits gained and semesters registered at college.

Across the 8 sites, per student costs ranged from $835 to $2,349. The average cost per student across all 8 sites was $1,319 (with a standard deviation of $502); this figure includes the stipend of up to $400 per participant. Some costs may be interpreted as ‘start­ up’ costs, which are unlikely to be needed if the programmes are run in subsequent years –  or complete a little more than one additional college course (typically worth 3 credits) – for the programme to provide students with credits at the typical cost per credit in Texas. The developmental bridge programme model was relatively expensive and as such would require the availability of funds from outside of regular ‘K-12’ (average daily attendance) and higher education (FTE) public funding mechanisms. The authors conclude that educators may want to consider ways to reduce costs by embedding support programmes such as these into the regular high school or college schedule.

7.34 Study 34

NCPR, The impact of postsecondary remediation using a regression discontinuity approach: Addressing endogenous sorting and noncompliance 2008, accessed 24 September 2019

The intervention was essential skills provision for post-17-year-old underprepared students and those referred to remedial education. The aim of the evaluation was to test the effectiveness of this alternative remedial education in a community college setting.

US study using a regression discontinuity design with a sample size of 100,000. Instead of having a treatment group this design uses a sample of students close to the cut-off.

The results suggest maths and reading remedial courses have mixed benefits. Being assigned to remediation appears to increase persistence to the second year and total number of credits completed for students on the margin of passing out of the requirement, but it does not increase the completion of college-level credits or eventual degree completion. Taken together, the results suggest that remediation might promote early persistence in college, but it does not necessarily help students on the margin of passing the placement cut-off to make long term progress toward earning a degree.

No cost analysis provided in the report.

7.35 Study 35

Department for Education, The 16-  19 bursary fund: Impact evaluation, 2015,

accessed 24 September 2019

Report for the Department for Education in England, which examined the effect of financial support for low income students aged 16 to 19 years. The aim of the evaluation was to provide statistical impact analysis of the removal of the Education Maintenance Allowance (EMA)/lntroduction of 16 to 19 bursary fund and cost benefit analysis. The report provides estimates of the policy’s impact on participation and attainment of Level 2 and Level 3 qualifications during the 2011/12 and 2012/13 academic years, and provides a cost-benefit analysis of the policy considering effects on lifetime earnings and exchequer tax receipts and benefit payments. The estimated impacts should be interpreted as the changes in participation and attainment rates compared to a hypothetical no-reform scenario where the EMA had remained in place and unchanged in 2011/12 and 2012/13. It is not an estimate of its effect compared to no financial support for learners aged 16 to 19.

Sample size from the National Pupil Database records of 4,4047,194 individuals.

The outcomes were measured by: full-time (FT) participation in post-16 education and whether learners had achieved the Level 2 or 3 attainment threshold by the end of the academic year in which they turned 18. The outcomes: the estimated effect of the implementation of the policy led to a 1.6 percentage point (ppt) fall in FT participation amongst Year 12 students who would otherwise have been eligible for the full EMA award. The impacts on attainment are largest for the poorest students, defined as those who would have been eligible for the full EMA award. The authors estimate that there was a 2.3 ppt fall in the Level 2 achievement rate by age 18, leading to a 1.1 ppt fall across the whole Year 13 cohort, from 88.3% to 87.2%. For Level 3 achievement, the effect is much smaller at 0.5 ppts for those who would have been eligible for the full grant, with no statistically significant impact across the whole eligible group or the cohort as a whole.

The short-run savings associated with the policy are in aggregate outweighed by the long­ run losses; the authors estimate an overall loss to the exchequer of £84 million (which they state is likely to be an underestimate).

7.36 Study 36

Department for Education, The Youth Contract for 16-17-year olds not in education, employment or training evaluation: Research report, 2014, accessed 24 September 2019

Report on an aspect of the government reforms programme, designed to reduce unemployment among 16 to 24 year olds, this report focused on intervention of intensive support for 16 to 17 year olds, which aimed at offering targeted support to disengaged young people, so they participate in education, an apprenticeship or a job with training. The programme combined a variety of schemes including a wage incentive for all employers, but specifically SMEs who recruited a young person via the Work Programme (WP), Apprenticeship Grant for Employers (AGE), work experience (8-week placements prior to the young people entering the WP and sector-based Work Academies (work experience placements combined with work focused training).

By end September 2013 (latest available figures),11,920 young people had started the national YC and 4,114 were recorded as initially re-engaged in a positive outcome. Owing to time-lag involved in achieving these outcomes authors expected smaller numbers to show in sustained re-engagement data, however by end September 2013, 489 participants were recorded as having sustained re-engagement. The payment model for providers was considered too low to identify and support young people furthest away from re-engagement; this was exacerbated by lower than expected volumes of young people entering the programme.

Authors recommend that a coordinated local response is required to identify, support and meet the needs of young people who are NEET or ‘at risk’ of disengagement.

Cost estimates were hampered because the DfE was unable to disclose to the evaluation the actual cost per participant, therefore the cost analysis was based on authors’ assumptions and as such the cost estimates are likely to be overstated. The internal rate of return varied extensively; nationally it was 64.6%, in Leeds, Bradford and Wakefield it was 45.8%, but in Newcastle and Gateshead it was only 19.3% and as such the data suggests the delivery of YC nationally was more cost effective. The net overall benefits per participant of YC were concluded to be higher for young people with low educational outcomes under the national model, but the authors highlight that there is a trade-off between value for money and total social benefits.

7.37 Study 37

Department for Education, Impact evaluation of 24+ Advanced Learner Loans Research Report, 2018, accessed 24 September 2019

To encourage and facilitate participation of individuals aged 24 years and over, income contingent­ repayment loans for the provision at Level 3 and Level 4 were introduced in 2013 to 2014. These were known as 24+ Advanced Learner Loans (ALL). The process for ALL followed a similar model to Student Loans in Higher Education.

The report details the outcomes of Difference-in-Difference analysis using historical Individual Learner Record (ILR) data, which compared the results for Level 3 and Level 4 qualifications for learners aged 24 and over with a comparison group of learners aged 19 to 23 years on the same courses who were unaffected by the introduction of loans.

Fear of debt was a key barrier to take up of ALL, although take up was 65% of eligible learners chose to take out a loan and 44% stated that without access to a loan, they would not have studied at all. The authors also comment that a loans-based environment results in more career-focussed learning, although this was not reflected in changes to the achievement rate.

The impact of ALL was associated with a marked drop in volumes of learners and learning aims studied on eligible courses and also led to a very small fall in the achievement rate.

No cost analysis provided in the report.

7.38 Study 38

Department for Education, Estimating the impact of traineeships, 2019, accessed 24 September 2019

Traineeships are an education and training programme, in England, that provide young people aged 16 to 24 with an intensive period of work experience and work preparation training, plus offering support in improving their English and maths. This occupational training intervention is delivered by FE providers. Traineeships are primarily intended for young people who: are not currently in employment and have little work experience, but who are focused on work and the prospect of it; are aged 16 to 24 (25 with an Education, Health and Care plan or Learning Difficulty Assessment) and qualified below level 3; are believed by providers and employers to have a reasonable chance of being ready for employment, or an apprenticeship, within 6 months of engaging in a Traineeship.

The evaluation estimates the impact of Traineeships on the intended outcomes of progression to an apprenticeship, further learning or employment based on the first year of the programme. The evaluation used propensity score matching (PSM); the impact is estimated by comparing the outcomes observed for those participating in Traineeships (also known as the treatment group) against an estimate of their expected outcomes if they had not participated in the programme who have similar characteristics (the counterfactual).

The PSM estimates indicate that for 16-18-year olds, Traineeships increased the likelihood of being in any positive destination (apprenticeship, further learning or employment) 12 months after starting the programme. There was also a significant impact on attaining any positive outcome within 12 months. For 19-23-year olds, there was also a significant positive impact on both being in a positive destination 12 months after starting the Traineeship and on attaining any positive outcome within 12 months.

Authors recommend that a coordinated local response is required to identify, support and meet the needs of young people who are NEET or ‘at risk’ of disengagement.

No cost analysis provided in the report.

7.39 Study 39

MDRC, More guidance, better results? Three-Year Effects of an Enhance Student Services Program at Two Community Colleges, 2009, accessed 24 September

Among students who enrol in community colleges in the US, with the intention of earning a credential or to transfer to a 4-year institution, only 51% achieve this goal within 6 years. This research studied learners aged 18 to 34 years, who also had financial hardship. Two community colleges in Ohio provided enhanced student services and a small stipend to low­ income students via the Opening Doors programme.

Students in the Opening Doors programme were assigned to a counsellor, with whom they were expected to meet at least 2 times per semester for 2 semesters to discuss academic progress and resolve any issues that might affect their schooling. Each counsellor worked with far fewer students than the regular college counsellors, which allowed for more frequent, intensive contact. Participating students were also eligible for a $150 stipend for 2 semesters.

While the intervention resulted in increased registration rates, academic benefits were not maintained. Programme group students registered for at least one course during the second semester at a higher rate than control group students and earned an average of half a credit more during the semester. The registration impact is likely primarily the effect of Opening Doors services provided during the first semester; the programme did not substantially affect outcomes during the first semester. After students in the Opening Doors programme received their 2 semesters of enhanced counselling services, the programme continued to have a positive effect on registration rates in the semester that followed. The programme did not, however, meaningfully affect academic outcomes in subsequent semesters. The programme did not significantly increase the average number of credits that students earned after the counselling programme ended, or over the study’s 3-year follow-up period.

No cost analyses provided in the report.

7.40 Study 40

Policy Studies Institute University of Westminster, Impact of adult basic skills pathfinder extension activities: Stage 2 of the evaluation, 2003, accessed 24 September 2019

Extension activities launched by DfE in 2001 within a wider campaign to improve the level of basic skills. Two main aims of this evaluation were to provide estimate of the net impact of Extension activities relative to traditional adult basic skills provisions, and to estimate relative impacts of the different Pathfinder courses. The learners lacked basic skills with problems with reading, writing and communication; many had left school without any qualifications. First outcome of interest was learning (percentage of learners finishing the course, percentage receiving a qualification, the proportion starting or planning a new course in the following year). An intermediate outcome was a self-efficacy score measuring self­ confidence and there were also labour market outcomes, which included the percentage in employment between the start of the course and the beginning of the Stage 2 follow-up interviews. The mean age of participants was 36 years.

Analysis used PSM methodology. Impact estimates achieved by comparing outcomes of the Extension participants with a matched sample of participants in the Control group. Estimates were supplemented with some descriptive evidence where impact estimate unfeasible.

Results were mixed. Generally, differences between the different types of Extension were more pronounced than between Extension and traditional courses. Being on an Extension course raised completion rate by 14% points; while participants on Extension courses were 9% less likely to have started a new course, those who did commence new courses were more likely to enrol for more than one new course. The Extension group and the comparison group could not be differentiated according the qualifications received. Regarding labour market outcomes, learners on Extension courses were 9% more likely to be in employment at the time of the Stage 2 follow-up interview.

No cost analyses provided in the report.

7.41 Study 41

Department for Education and Skills, Evaluation of impact of skills for life learning: Report on Sweep 2, 2006, accessed 24 September 2019

Skills for Life programme designed to improve literacy, numeracy and communication skills of adults and young people aged 16 to 17 years, who had left full-time education. The programme provides free literacy, numeracy, and ESOL training free of charge to those without literacy or numeracy qualifications at Level 2 (approximately GCSEs of grades A* to C/Level 9 to Level 4). This report did not evaluate the impact of ESOL courses.

The evaluation used a longitudinal survey of participants on literacy and numeracy courses (Skills for Life learners) and a matched group with low or no literacy or numeracy qualifications. Participants were first interviewed in 2002/2003, with a second and third wave of interviews taking place one and 2 years later. The analysis was based on the first 2 waves of the survey. Comparison of benefits attributed to their course and the impact analysis suggested that Skills for Life learners tended to overestimate the benefits. Qualification gain appeared to be largely related to personal characteristics, for example older learners, those with children and those who believed they had a literacy problem were less likely to achieve qualifications. Much of the report is dedicated to the perceived benefits of participants, rather than statistical analyses.

No cost analyses provided in the report.

7.42 Study 42

MDRC, Guiding developmental math students to campus services, 2010, accessed 24 September 2019

Report provides results from the Beacon Mentoring programme in Texas, USA. The intervention targeted students enrolled in lower level maths courses, who had a high degree of failure. The intervention provided a means for participating students to access a high level of support from an individual on campus. Mentors were college employees who worked with the faculty to identify struggling students and offer help early on in their courses.

RCT. The programme led to a statistically significant decrease in the likelihood of students in mentored classes withdrawing from their main maths course before the end of the semester and robust evidence that the programme increased the use of the student success centre. Whilst there was no evidence overall that students in mentored classes were likely to earn a higher score in the final exam, there were statistically significant effects between high-risk and low-risk students. Among high-risk students, the programme group was significantly less likely to withdraw from the maths class and earned a significantly higher percentage of credits per attempt. For the low-risk group there was a statistically significant impact on percentage of days absent, but no other statistically significant effects. Part-time students were less likely to withdraw from and more likely to pass the maths class, earned more credits and, in the developmental maths classes, scored higher on the final exam.

No cost analyses provided in the report.

7.43 Study 43

Irby, Terry Rand others, The Joliet Junior College Center for Adult Basic Education and Literacy’s ‘Families about Success’: Intergenerational Programming that Works, 1992, accessed 24 September 2019

The Joliet Junior College Centre for Adult Basic Education and Literacy (CABEL) provided adult basic education and support services for adults whose skills in reading and maths were below 12th grade level. Support provided included Adult Basic Education (ABE), GED, ESOL and job and life skills. In 1992 the college provided the ‘Families about Success’ (FAS) intervention to provide parents and children with a positive, interactive educational experience; FAS enrolled low-income parents and their children (aged 3 to 10 years), who had been identified as being at risk. The intervention focused on adult basic skills/education, children’s basic literacy skills, and parenting skills.

‘Formal and informal’ assessment of the intervention revealed that FAS participants showed a higher average gain in reading and maths Test of ABE scores than those enrolled in GED classes; the kindergarten skills screening tests of FAS pre-schoolers showed ‘nearly 100%’ improvement in gross, fine motor and language/cognitive skills; older children demonstrated an increase in reading skills ‘by as much as 1.5 grade levels’. Data limited by availability and age of report.

No cost analyses provided in the report.

7.44 Study 44

MDRC, GAIN: Basic Education in a Welfare-to- Work Program California’s Greater Avenues for Independence Program, 1994, accessed 24 September 2019

This report was an evaluation of the Greater Avenues for Independence (GAIN) programme in California, USA. Established in 1985, California’s GAIN Program broke new ground by emphasizing large-scale, mandatory participation in basic education (in addition to job search, training, and unpaid work experience), for welfare recipients who were considered to need it. Three types of basic education were available through GAIN: 1) ABE programmes (that is, remedial reading and mathematics classes) for those with lower skills levels (typically at or below the eighth-grade level); 2) GED preparation programmes for students whose academic skills are strong enough to allow them to study productively for the GED test; 3) ESOL instruction, which teaches people who are not proficient in English to read and speak the language. At the time of the ongoing intervention the sample consisted of over 2,500 welfare recipients. The evaluation used previously completed analyses, plus new data based on a survey of the recipients and results of a literacy test, which were administered 2 to 3 years after they joined the research sample.

GAIN resulted in increases in the number of hours in basic education and receipt of a GED certificate; in one study only, welfare recipients demonstrated significant score increases in maths problem solving and literacy tests. Educational gains were concentrated among participants who had relatively high levels of literacy, and test score gains were concentrated in the site that created an innovative adult education programme tailored to the needs of people on welfare.

No cost analyses provided in the report.

7.45 Study 45

Thuy, V.G, High-tech for effective ESUFamily Literacy Instruction, 1992, accessed 24 September 2019

Project was a pilot, which used computers and CD-ROM drives and software to implement a combined English as a Second Language and Family Literacy curriculum for limited English proficient (LEP) adults. The project examined 4 software packages to assess if they had a significant and positive impact on the attainment of English language skills by adult LEP students (refugee and immigrant adult students). Each group received 6 hours of instruction each week, for 15 weeks. The experimental group (15 participants) had 2 hours of instruction per week on the material contained in the software, the control group (15 participants) was given the same information in printed form.

RCT. Pre-tests and post-tests administered to each group and data analysis indicates that both groups did markedly better on the post test, but the experimental group ‘improved more’ than they control group.

No cost analyses provided in the report.

7.46 Study 46

Lipiec, J.K and others, Verifying applied literacy skills (VALS) in ABE Programmes, 1993, accessed 24 September 2019

This randomised control study compared the effectiveness of a traditional GED curriculum with a literacy curriculum based on applied literacy skills.

34 adult students received GED instruction that emphasised functional and workplace contexts and supplemental instruction; the 35 control group students had instruction based on a traditional GED curriculum.

No major differences between the experimental and control groups’ test performance, retention, or success in meeting their stated goals were found.

No cost analyses provided in the report.

7.47 Study 47

Wilson, A.M, The INVEST Program: A Computer-Based System for Adult Academic Upgrading: A Pilot Project, 1992, accessed 24 September 2019

Adult learners participated in the INVEST intervention. INVEST was a networked system of basic instructional software in reading, writing, maths and life skills.

RCT. A sample of 15 students, with an average age of 32.27 and a mean education level of 9.33 years, participated in an 11-week course using INVEST – a networked system of basic instructional software offering lessons in reading, writing, mathematics, and life skills. Two groups were used for comparison, the first comparison group had a younger mean age and had completed an 18-week course, and one with an older mean age that had completed a 44-week course. Based on standardised tests given before and after participation, the attendance for the INVEST sample was 92.6% and mean number of hours on the computer was 151.77. Positive gains were made by the study group in all areas of reading and maths, and for maths there were gains of more than 1.5 years in the 11-week period. The gains in reading were not statistically significant but were in a positive direction and paralleled the relative gains noted in one of the 2 comparison groups.

No cost analyses provided.

7.48 Study 48

Weiss, B.D and others, Literacy education as a treatment for depression in patients with limited literacy and depression: A randomised controlled trial, 2006, accessed 24 September 2019

Essential skills provision for post-18-year-old adults with mental health issues (depression) and low literacy levels. Literacy levels were based on the Rapid Estimate of Adult Literacy in Medicine (REALM). The aim of the evaluation was to test if the provision of literary education, along with standard treatment for depression, would result in higher wellbeing than a comparison group treated with standard depression therapy only.

Depression severity assessed with PHQ-9 scores at baseline and follow-up evaluation (up to one year after study enrolment) with comparisons between the intervention group and control group. Whilst the median PHQ-9 scores were similar at baseline, the improvement was significantly larger in the intervention group.

No cost analyses provided.

7.49 Study 49

Lipman, H.L and Boyd, E.H, Social support and education for single mothers: a randomized controlled trial of a community-based programme, 2005, accessed 24 September 2019

Canadian study with low-skilled lone parent mothers (mean age of 32 years). The intervention was to assess the effect of a community-based programme of social support and education groups on maternal wellbeing and parenting.

Outcome measures were self-reported mood, self-esteem, social support and parenting ability.

In the short-term after the evaluation, when compared to mothers in the control group, mothers in the intervention group had improved scores for mood and self-esteem only. However, growth curve analysis of programme effects over the follow-up period showed improvements in all 4 outcomes with no significant differences between the intervention and control groups.

No cost analyses provided.

7.50 Study 50

Center for the Analysis of Post-Secondary Readiness, Making it through: Interim Findings on developmental students’ progress to college math with the Dana Center mathematics pathways, 2018, accessed 24 September 2019

This paper discusses community college reforms for post-16 learners with low skills in maths. The intervention was multiple semesters of developmental maths. In the DCMP model, specifically, colleges were encouraged to develop 3 maths pathways: a statistics pathway for students in the social sciences and health professions, a quantitative reasoning pathway for those majoring in the humanities and liberal arts, and a path to calculus for science, technology, engineering, and maths (STEM) students. The pathways started with an accelerated and revised developmental maths course, which allowed students with developmental needs and who traditionally required 2 or more semesters’ worth of courses to complete their requirements in one semester. In addition, the Dana Center developed revised curricula for the DCMP developmental and college-level maths courses. These curricula focused more heavily on statistics and quantitative reasoning content relative to algebra, and instruction was framed around more student-centred and active learning approaches with material presented in the context of real life.

In the RCT, the programme group students made improvements in both enrolling and in passing college-level maths during their second and third semesters, including reaching a critical college milestone. At the time of the report, the intervention courses had not affected the persistence rates of students. Full results expected in 2019 (not yet available).

No cost analysis provided; authors comment that in the final report they will analyse the cost effectiveness of the DCMP model in concert with its impacts on student success

7.51 Study 51

Office of Research, Evaluation and Program Support, Starting to succeed: The impact of CUNY start on academic momentum, 2018, accessed 24 September 2019

Paper presents findings from an evaluation of the CUNY Start programme, a pre­ matriculation developmental education intervention for associate degree-seeking students. Students were post-16 and lacked basic academic skills. CUNY Start students defer matriculation for one semester to receive intensive preparation in college reading, writing, and/or maths.

The analysis used PSM. CUNY Start students were more likely to take and pass ‘gateway’ maths and English courses in their first year; the intervention had the largest effect on students with the broadest remedial needs; the CUNY Start students gained similar grades to their peers in English gateway courses, but slightly lower in maths gateway courses.

No cost analyses provided.

7.52 Study 52

Bettinger, E.P and others, The role of application assistance and information in college decisions: Results from the H&R Block Fafsa Experiment, 2012, accessed 24 September 2019

This paper reports on an RCT, which was a financial ‘nudge’ intervention for low income individuals aged between 15 and 30 years. The intervention was personal assistance for applications with Federal Student Aid.

Assistance with applications for financial support led to increased engagement in college courses.

The authors conclude that the costs over 2 years of college for dependents were approximately $8,750 and approximately $14, 150 for independents (independent adults often referred to as non-traditional students). Returns to college among those who enrolled as a result of the treatment would have to be at least as large as this to consider the programme cost-effective.

7.53 Study 53

Brooks, G. and others, Randomised controlled trial of incentives to improve attendance at adult literacy classes, 2008

RCT with a population of adults with poor literacy, aged 16 to 60+ years. Almost all had left full-time education at the earliest opportunity and the majority of the rest left education shortly after that. 50% had no educational qualifications. The intervention was a financial payment of £5 for each class attended.

The main outcome of interest was class attendance (number of sessions attended); the secondary outcome was post-test (literacy) scores. There was a statistically significant reduction of approximately 1.5 sessions attended by the intervention group, compared with the control group. There was no statistically significant difference between the reading scores of the intervention group and the control group.

Authors suggest reasons for counter-intuitive findings are in line with earlier work (for example, Frey and Jegen, 2001}, which discuss crowding out effects, whereby external interventions are seen as controlling and ‘crowd out’ intrinsic motivations.

No cost analyses provided.

7.54 Study 54

Quigley, B.A and Uhland, R.L, Retaining adult learners in the first three critical weeks, 2000

RCT of ‘adult’ at risk learners and adults who have dropped out of adult basic education (ABE) or a literacy programme. The aim of the intervention was to examine whether firstly, at risk adults would stay in ABE programmes longer than the control group if they were highly supported by a counsellor/teacher working together (team approach); or, secondly, whether at risk adults would stay longer than the control group if they were in existing small classes with mainstream learners (small group approach); or, thirdly, whether at risk adults would stay longer than the control group when one-to-one trained tutors were used for enhanced instruction.

The results showed that 60% of the small group approach completed 3 months or more, compared with a 40% completion rate for the team approach. The tutoring approach had a completion rate of 20% for 3 months or more. Each of the group treatments proved more successful than the no treatment control group, where no member was retained for 3 months.

No cost analyses provided.

7.55 Study 55

Batchelder, J.S and Rachal, J.R, Efficacy of a computer assisted programme in prison, 2000

‘Random assignment’ trial with prisoners who had not had a high school education, and who were aged 19 to 53 years (mean of 30.5). The intervention was a 4-week educational programme with computer assisted instruction (CAI), to examine the efficacy of using CAI to raise achievement scores on the Comprehensive Adult Student Assessment System (CASAS) maths and reading tests.

Achievement scores of treatment group were not significantly higher than the control group after 4 weeks.

No cost analyses provided.

7.56 Study 56

Drikx, J.M and Crawford, M, Sage Publishers, Teaching reading through teaching science, 1993

RCT. The population was drawn from ‘adult’ prisoners, who participated in this US intervention, which used the natural world as a context for teaching reading to prisoners with low reading skills.

The experimental group showed small gains relative to the control group, but the only significant differences were in attendance and engagement.

No cost analyses provided.

7.57 Study 57

Wathington, H.D and others, The difference a cohort makes: understanding developmental learning in community colleges, 2011

The educational intervention was the use of learning communities for underprepared students aged 16 to 25. This was a qualitative study looking at classroom climate and academic support networks. Qualitative data gathered from community colleges in the US that participated in random assignment evaluation between learning community and non­ learning community experiences. Data collected via interviews/focus groups with learning community and non-learning community students and faculty members.

Findings were that those in the learning communities had increased engagement.

No cost analyses provided.

7.58 Study 58

Rich, Rand Shepherd, M.J, Teaching test comprehension to poor adult learners, 1993

A randomised control study. The intervention was essential skills provision for poor readers in the US, aged between 17 and 66 years. The study investigated the effectiveness of a ‘Reciprocal Teaching’ (text comprehension) programme to improve the comprehension of poor adult readers.

The experimental group showed improved literacy and test comprehension on study specific tests.

No cost analyses provided.

7.59 Study 59

Nicol, M.M and Anderson, A, A Computer-Assisted instruction vs. teacher directed teaching of numeracy in adults, 2000

A UK study examining the effects of a computer-assisted instruction (CAI) intervention on basic numeracy skills for adults with learning disabilities. Participants were aged from 22 to 47 years. The study compared CAI with teacher-led instruction with a population of adult learners with learning disabilities and supplemented traditional learning of numeracy with computer-assisted instruction to determine its educational value.

There were improvements for the intervention group on basic number diagnostic and screening tests (specific tests used for populations with learning disabilities). They found that students with mild and moderate learning disabilities faired just as well with CAI as with conventional instruction. This showed that educational technology in the form of CAI does not negatively affect learning. They identified key contributors when using CAI as individualized instruction sequences, learner response control, quick and individualized feedback, and motivation-enhancing game-like activities. Most notably, students were found to deliver quicker and more accurate answers due to automaticity from the extended practice of basic mathematics skills.

No cost analyses provided.

7.60 Study 60

Hayslip, B. and others, The impact of mental aerobics training on older adults,

2016

US randomised control study, which examined the used of mental aerobics (MA) in a residential community setting. The intervention included different types of activities based on maths, logic and verbal problems.

Authors concluded that MA can be an effective intervention in improving older adults affectivity, everyday task efficacy, and self-rated health. Subsequent provision of MA training to controls minimised the performance differences between the treatment and control conditions, as well as moderating the effect of level of education.

No cost analyses provided.

7.61 Study 61

US Department of Education Mathematica Policy Research Reports, Addressing literacy needs at work: Implementation and Impact of Workplace Literacy Programmes, 1998

National Workplace Literacy Program (US), a federal programme designed to emphasise contextualised, job-specific instruction in basic skills for adults who lack basic skills; no age range of participants given.

Five sites with different types of provision – for example, ESOL learning, literacy and maths.

RCTs in each study site. The treatment group could enrol in workplace learning immediately, while the control group had to wait until the next course cycle began.

Workplace literacy can improve literacy skills, interest in further learning, engagement in literacy tasks at home and self-rated literacy skills of learners. There was also a positive impact on teamwork skills and communication skills at work.

A greater number of instructional hours was associated with greater programme effectiveness, with 43 being the highest average number of hours.

Other factors that emerged as important for effective learning included: partnership working between employer representatives and other agencies, reimbursement of travel costs, employer participation and support, and instructors’ qualifications.

No cost analyses provided.

7.62 Study 62

Ministry of Housing, Communities and Local Government, Measuring the impact of community- based English language provision - findings from a randomised controlled trial, 2018, accessed 24 September 2019

Community-based English Language intervention; an 11-week course of 6 hours of provision per week, designed to improve functional English proficiency for individuals with an ESOL need, which was derived from the provider’s existing Talk English programme.

Randomised controlled design with a sample of 527 participants, using a waiting list for the control group. Participants were recruited from communities with very low levels of functional English (ESOL).

Findings revealed a strong and sizeable difference between treatment and control group in terms of both change in proficiency and overall proficiency at the end of the trail across all measures of speaking, listening, reading and writing.

Also found a significant impact of intervention on social integration measures, including confidence interacting with services and social interactions.

Higher educational attainment was a significant predictor of improvement in proficiency.

Outcome measures were obtained by a series of English language proficiency tests which were developed by the English-Speaking Board (ESB) and administered by ESB assessors and CBEL tutors.

No cost information in the report

7.63 Study 63

SRI Education, Evaluating Digital Learning for Adult Basic Literacy and Numeracy, 2017, accessed 24 September 2019

14 different Adult Basic Education programmes that used 5 different ICT products to facilitate the learning of basic skills (US).

A treatment group of students who used an ICT product was compared with a matched control group of students who did not. National standardised assessments were administered in each programme site. The Test of Adult Basic Education was used to measure the impact of the use of digital learning methods on learners.

The study used a PSM matched control, quasi-experimental approach. The mixed results showed positive impacts of ICT use for some sites and outcome measures, but negative impacts for others.

Students and instructors reported positive experiences of using the ICT products and reported positive effects, such as feeling more confident to use online resources for self­ study at home.

However, 1 in 5 students did not enjoy using ICT and preferred learning directly with instructors. Authors suggest that particularly for those with the lowest skills, blended and hybrid models of learning with instructors delivering 50% or more of instruction would possibly be more effective.

No cost information provided in the report.

8. About the Commission

The Social Mobility Commission is an independent advisory non-departmental public body established under the Life Chances Act 2010 as modified by the Welfare Reform and Work Act 2016. It has a duty to assess progress in improving social mobility in the UK and to promote social mobility in England.

The Commission board comprises:

• Dame Martina Milburn (Chair)
• Alastair da Costa, Chair of Capital City College Group
• Farrah Storr, Editor-in-chief, Elle
• Harvey Matthewson, Aviation Activity Officer at Aerobility and Volunteer
• Jessica Oghenegweke, Broadcast and Digital Coordinator at The Roundhouse
• Jody Walker, Senior Vice President at TJX Europe (TK Maxx and Home Sense in the UK)
• Liz Williams, Group Director of Digital Society at BT
• Pippa Dunn, Founder of Broody, helping entrepreneurs and start ups
• Saeed Atcha, Chief Executive Officer of Youth Leads UK
• Sam Friedman, Associate Professor in Sociology at London School of Economics
• Sammy Wright, Vice Principal of Southmoor Academy, Sunderland
• Sandra Wallace, Joint Managing Director Europe at DLA Piper
• Steven Cooper, Chief Executive Officer C.Hoare and Co

8.1 Learning and Work Institute

Learning and Work Institute is an independent policy, research and development organisation dedicated to lifelong learning, full employment and inclusion. We research what works, develop new ways of thinking and implement new approaches. Working with partners, we transform people’s experiences of learning and employment. What we do benefits individuals, families, communities and the wider economy.

1. In this evidence review, programmes are interventions that are designed to improve outcomes for students. ↩

2. Social Mobility Commission, Investing in What Works in Further Education and Adult Learning: Policy Brief, 2019 ↩

3. Social Mobility Commission, State of the Nation Report, 2018-19, 2019 accessed 25 October 2019 ↩

4. Department for Education, Learners and Apprentices Survey 2018: Research Report, 2018, accessed 25 October 2019 ↩

5. Government Office for Science, What are the wider benefits of learning across the life course? 2017 accessed 25 October 2019; Social Mobility Commission, State of the Nation Report, 2018-19, 2019 accessed 25 October 2019  ↩

6. The Work Foundation, The hourglass and escalator, 2011, accessed 20 September 2019; ONS, Earnings by qualification, 2011, accessed 20 September 2019 ↩

7. Department for Education, Further education and skills: March 2019, 2019, accessed 20 September 2019 ↩

8. Learning and Work Institute, Adult participation in learning survey, 2019, accessed 20 September 2019 ↩

9. Department for Education, Adult Participation in Learning Survey 2017, 2018, accessed 21 October 2019; Government Office for Science, Foresight Review into the Future of Skills and Lifelong Learning, 2017, accessed 21 October 2019 ↩

10. Social Mobility Commission, The adult skills gap: is falling investment in UK adults stalling social mobility, 2019, accessed 21 October 2019 ↩

11. ONS, Learning and Work Institute analysis of Annual Population Survey, 2018. 30.2% of adults with a NVQ level 4+ had received job-related training in the last 13 weeks, compared to 19.6% of adults with a level 3 qualification or below. ↩

12. Disadvantaged students are defined as students who were eligible for Free School Meals (FSM) at any point between age 11 and age 15; Department for Education, Post-16 education: outcomes for disadvantaged students, 2018, accessed 20 September 2019 ↩

13. Department for Education, National Achievement Rate Tables, March 2019, accessed 20 September 2019 ↩

14. Department for Education, Post-16 education: outcomes for disadvantaged students, 2018, accessed 20 September 2019 ↩

15. FE Week, Levy budget bust; Government agency warns of imminent apprenticeship over-spend, 2019, accessed 21 October 2019 ↩

16. IFS, 2019 annual report on education spending in England, 2019, accessed 20 September 2019 ↩

17. Department for Education, News Story: Chancellor announces $400 million investment for 16-19 year olds’ education, 2019, accessed 25 October 2019 ↩

18. This number excludes any duplicates, for example interim reports of the same intervention where the final report is included. ↩

19. What Works Centre for Economic Growth, Guide to scoring methods using the Maryland Scientific Methods Scale, accessed 24 September 2019 ↩

20. Centre for Vocational Education Research, Post 16 remedial policies: a literature review, 2017 ↩

21. Department for Education and Skills, Evaluation of impact of skills for life learning: Report on Sweep 2, 2006, accessed 24 September 2019 ↩

22. Department for Education, Improving engagement and attainment in maths and English courses: insights from behavioural research, 2018, accessed 24 September 2019; Hughes D and others (2016), Adult Education: Too important to be left to chance Research report for the All Part Parliamentary group for Adult Education (APPG) - Inquiry into Adult Education (accessed 25 September 2019) ↩

23. Department for Education, Improving engagement and attainment in maths and English courses: insights from behavioural research, 2018, accessed 24 September 2019 ↩

24. CfBT, Raising maths attainment through enhanced pedagogy and communication: Results from a ‘teacher-level’ randomised controlled trial, 2013, accessed 24 September 2019 ↩

25. Policy Studies Institute University of Westminster, Impact of adult basic skills pathfinder extension activities: Stage 2 of the evaluation, 2003, accessed 24 September 2019 ↩

26. MDRC, Staying on Course: Three-Year results of the National Guard Youth ChalleNGe Evaluation, 2011, accessed 24 September 2019; MDRC, Enhancing GED Instruction to Prepare Students for College and Careers, 2013, accessed 24 September 2019; MDRC, GAIN: Basic Education in a Welfare-to- Work Program California’s Greater Avenues for Independence Program, 1994, accessed 24 September 2019 ↩

27. MDRC, Doubling Graduation Rates: Three-year effects of CUNY’s Accelerated Study in Associate Programmes (ASAP) for Developmental Education Students, 2015, accessed 24 September 2019 ↩

28. Economic Mobility Corporation, Escalating Gains: The Elements of Project QUEST’s Success, 2018, accessed 24 September 2019 ↩

29. MDRC, A Random Assignment Evaluation of Learning Communities at Kingsborough Community College: Seven Years Later, 2014, accessed 24 September 2019 ↩

30. MDRC, Serving community college students on probation: 4-year findings from Chaffey College’s Opening Doors Program, 2011, accessed 24 September 2019 ↩

31. MDRC, Moving Forward: Early findings from the Performance-Based Scholarship Demonstration in Arizona, 2013, accessed 24 September 2019; MDRC, Mapping Success: Performance-Based Scholarships, Student Services, and Developmental Math, 2014, accessed 24 September 2019; MDRC, Serving community college students on probation: 4-year findings from Chaffey College’s Opening Doors Program, 2011, accessed 24 September 2019; MDRC, Bringing CUNY accelerated study in associate programmes (ASAP) to Ohio, 2016, accessed 24 September 2019; NPCR, Leaming Communities for Students in Developmental English Impact Studies at Merced College and The Community College of Baltimore County, 2012, accessed 24 September 2019; Irby, Terry Rand others, The Joliet Junior College Center for Adult Basic Education and Literacy’s ‘Families about Success’: Intergenerational Programming that Works, 1992, accessed 24 September 2019; Community College Research Center, New evidence of success for Community College remedial English students: Tracking the Outcomes of Students in the Accelerated Learning Program (ALP), 2012, accessed 24 September 2019 ↩

32. US Department of Education Mathematica Policy Research Reports, Addressing literacy needs at work: Implementation and Impact of Workplace Literacy Programmes, 1998 ↩

33. Ministry of Housing, Communities and Local Government, Measuring the impact of community-based English language provision – findings from a randomised controlled trial, 2018, accessed 24 September 2019 ↩

34. MDRC, Becoming College-Ready: Early findings from a City University of New York (CUNY) start Evaluation, 2018, accessed 24 September 2019 ↩

35. Washington, DC: Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services, Washington State’s Integrated Basic Education and Skills Training(/­ BEST) Program in Three Colleges: Implementation and Early Impact Report (PACE), 2018, accessed 24 September 2019; Community College Research Center, New evidence of success for Community College remedial English students: Tracking the Outcomes of Students in the Accelerated Learning Program (ALP), 2012, accessed 24 September 2019; NCPR, Bridging the gap: An impact study of eight developmental summer bridge programmes in Texas, 2012, accessed 24 September 2019 ↩

36. Center for the Analysis of Post-Secondary Readiness, Making it through: Interim Findings on developmental students’ progress to college math with the Dana Center mathematics pathways, 2018, accessed 24 September 2019 ↩

37. MDRC, A Randomized Controlled Trial of a Modularized, Computer-Assisted, Self-Paced Approach to Developmental Math, 2018, accessed 24 September 2019 ↩

38. MDRC, Serving community college students on probation: 4-year findings from Chaffey College’s Opening Doors Program, 2011, accessed 24 September 2019; MDRC, More guidance, better results? Three-Year Effects of an Enhance Student Services Program at Two Community Colleges, 2009, accessed 24 September ↩

39. Washington, DC: Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services, Workforce Development Council of Seattle-King County Health Careers for all Program: Implementation and Early Impact Report (PACE}, 2017, accessed 24 September 2019; MDRC, Doubling Graduation Rates: Three-year effects of CUNY’s Accelerated Study in Associate Programmes (ASAP) for Developmental Education Students, 2015, accessed 24 September 2019; MDRC, Learning Communities for Students in Developmental Math Impact Studies at Queensborough and Houston Community, 2011, accessed 24 September 2019; Bettinger, E.P and others, The role of application assistance and information in college decisions: Results from the H&R Block Fafsa Experiment, 2012, accessed 24 September 2019 ↩

40. MDRC, Paying for Persistence: Early Results of a Louisiana Scholarship Program for Low-Income Parents Attending Community College, 2006, accessed 24 September 2019 ↩

41. Policy Studies Institute University of Westminster, Impact of adult basic skills pathfinder extension activities: Stage 2 of the evaluation, 2003, accessed 24 September 2019 ↩

42. MDRC, More guidance, better results? Three-Year Effects of an Enhance Student Services Program at Two Community Colleges, 2009, accessed 24 September ↩

43. Department for Education, Improving engagement and attainment in maths and English courses: insights from behavioural research, 2018, accessed 24 September 2019 ↩

44. Brooks, G. and others (2008) Randomised control trial of incentives to improve attendance to adult literacy classes Oxford review of education 34(5) pages 493-504. ↩

45. MDRC, Mapping Success: Performance-Based Scholarships, Student Services, and Developmental Math, 2014, accessed 24 September 2019 ↩

46. Center for the Analysis of Post-Secondary Readiness, Making it through: Interim Findings on developmental students’ progress to college math with the Dana Center mathematics pathways, 2018, accessed 24 September 2019 ↩

47. MDRC, Four-Year effects on Degree Receipt and Employment Outcomes from a Performance-Based Scholarship Program in Ohio, 2015, accessed 24 September 2019 ↩

48. MDRC, Enhancing GED Instruction to Prepare Students for College and Careers, 2013, accessed 24 September 2019; MDRC, Bringing CUNY accelerated study in associate programmes (ASAP) to Ohio, 2016, accessed 24 September 2019 ↩

49. MDRC, Keeping Students on Course: An impact study of a student success course at Guildford Technical Community College, 2012, accessed 24 September 2019 ↩

50. Ministry of Housing, Communities and Local Government, Measuring the impact of community- based English language provision – findings from a randomised controlled trial, 2018, accessed 24 September 2019 ↩

51. Department for Education, Improving engagement and attainment in maths and English courses: insights from behavioural research, 2018, accessed 24 September 2019 ↩

52. Washington, DC: Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services, Bridging the Opportunity Divide for Low-Income Youth: Implementation and Early Impacts of the Year Up Program (PACE), 2018, accessed 24 September 2019 ↩

53. Washington, DC: Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services, lnstituto de/ Progreso Latino’s Carreras en Salud Program: Implementation and Early Impact Report (PACE), 2018, accessed 24 September 2019 ↩

54. MDRC, Improving Basic Skills: The Effects of Adult Education in Welfare to Work Programmes, 2002, accessed 24 September 2019 ↩

55. Economic Mobility Corporation, Escalating Gains: The Elements of Project QUEST’s Success, 2018, accessed 24 September 2019 ↩

56. US Department of Education Mathematica Policy Research Reports, Addressing literacy needs at work: Implementation and Impact of Workplace Literacy Programmes, 1998 ↩

57. MDRC, Improving Basic Skills: The Effects of Adult Education in Welfare to Work Programmes, 2002, accessed 24 September 2019 ↩

58. Department for Education and Skills, Evaluation of impact of skills for life learning: Report on Sweep 2, 2006, accessed 24 September 2019 ↩

59. NCPR, The impact of postsecondary remediation using a regression discontinuity approach: Addressing endogenous sorting and noncompliance 2008, accessed 24 September 2019 ↩

60. MDRC, Doubling Graduation Rates: Three-year effects of CUNY’s Accelerated Study in Associate Programmes (ASAP) for Developmental Education Students, 2015, accessed 24 September 2019; MDRC, Bringing CUNY accelerated study in associate programmes (ASAP) to Ohio, 2016, accessed 24 September 2019; MDRC, Becoming College-Ready: Early findings from a City University of New York (CUNY) start Evaluation, 2018, accessed 24 September 2019 ↩

61. Washington, DC: Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services, Valley Initiative for Development and Advancement: Implementation and Early Impact Report (PACE), 2017, accessed 24 September 2019; Washington, DC: Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services, lnstituto de/ Progreso Latino’s Carreras en Salud Program: Implementation and Early Impact Report (PACE), 2018, accessed 24 September 2019; Economic Mobility Corporation, Escalating Gains: The Elements of Project QUEST’s Success, 2018, accessed 24 September 2019 ↩

62. MDRC, More guidance, better results? Three-Year Effects of an Enhance Student Services Program at Two Community Colleges, 2009, accessed 24 September ↩

63. Center for the Analysis of Post-Secondary Readiness, Making it through: Interim Findings on developmental students’ progress to college math with the Dana Center mathematics pathways, 2018, accessed 24 September 2019 ↩

64. CfBT, Raising maths attainment through enhanced pedagogy and communication: Results from a ‘teacher-level’ randomised controlled trial, 2013, accessed 24 September 2019; Rich, Rand Shepherd, M.J, Teaching test comprehension to poor adult learners, 1993 ↩

65. This does not include those that moved into an apprenticeship. Moving into an apprenticeship was a separate outcome measure. ↩

66. Department for Education, Estimating the impact of traineeships, 2019, accessed 24 September 2019 ↩

67. American Economic Corporation, Does Job Corps Work? Impact Findings from the National Job Corps Study, 2008, accessed 24 September 2019; Washington, DC: Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services, Pima Community College Pathways to Healthcare Program: Implementation and Early Impact Report (PACE), 2017, accessed 24 September 2019; Washington, DC: Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services, Washington State’s Integrated Basic Education and Skills Training (I-BEST) Program in Three Colleges: Implementation and Early Impact Report (PACE), 2018, accessed 24 September 2019 ↩

68. Washington, DC: Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services Washington, Des Moines Area Community College Workforce Training Academy Connect Program: Implementation and Early Impact Report (PACE), 2018, accessed 24 September 2019; Washington, DC: Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services Madison Area Technical College Patient Care Pathway Program: Implementation and Early Impact Report (PACE), 2018, accessed 24 September 2019; Washington, DC: Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services, Workforce Development Council of Seattle-King County Health Careers for all Program: Implementation and Early Impact Report (PACE), 2017, accessed 24 September 2019 ↩

69. MDRC, Enhancing GED Instruction to Prepare Students for College and Careers, 2013, accessed 24 September 2019 ↩

70. Department for Education, Improving engagement and attainment in maths and English courses: insights from behavioural research, 2018, accessed 24 September 2019 ↩

71. Bettinger, E.P and others, The role of application assistance and information in college decisions: Results from the H&R Block Fafsa Experiment, 2012, accessed 24 September 2019 ↩

72. MDRC, Keeping Students on Course: An impact study of a student success course at Guildford Technical Community College, 2012, accessed 24 September 2019; MDRC, Serving community college students on probation: 4-year findings from Chaffey College’s Opening Doors Program, 2011, accessed 24 September 2019 ↩

73. SRI Education, Evaluating Digital Learning for Adult Basic Literacy and Numeracy, 2017, accessed 24 September 2019 ↩

74. Nicol, M.M and Anderson, A, A Computer-Assisted instruction vs. teacher directed teaching of numeracy in adults, 2000. ↩

75. MDRC, A Randomized Controlled Trial of a Modularized, Computer-Assisted, Self-Paced Approach to Developmental Math, 2018, accessed 24 September 2019; Nicol, M.M and Anderson, A, A Computer-Assisted instruction vs. teacher directed teaching of numeracy in adults, 2000 ↩

76. Thuy, V.G, High-tech for effective ESUFamily Literacy Instruction, 1992, accessed 24 September 2019; Wilson, A.M, The INVEST Program: A Computer-Based System for Adult Academic Upgrading: A Pilot Project, 1992, accessed 24 September 2019 ↩

77. NCPR, Bridging the gap: An impact study of eight developmental summer bridge programmes in Texas, 2012, accessed 24 September 2019 ↩

78. Community College Research Center, New evidence of success for Community College remedial English students: Tracking the Outcomes of Students in the Accelerated Learning Program (ALP), 2012, accessed 24 September 2019 ↩

79. MDRC, Moving Forward: Early findings from the Performance-Based Scholarship Demonstration in Arizona, 2013, accessed 24 September 2019 ↩

80. Department for Education, Improving engagement and attainment in maths and English courses: insights from behavioural research, 2018, accessed 24 September 2019 ↩

81. MDRC, Four-Year effects on Degree Receipt and Employment Outcomes from a Performance-Based Scholarship Program in Ohio, 2015, accessed 24 September 2019 ↩

82. Department for Education, The 16-19 bursary fund: Impact evaluation, 2015, accessed 24 September 2019 ↩

83. MDRC, Incremental Disbursements of Student Financial Aid: Final Report on Aid Like A Paycheck, 2019, accessed 24 September 2019 ↩

84. MDRC, Guiding developmental math students to campus services, 2010, accessed 24 September 2019 ↩

85. Quigley, B.A and Uhland, R.L, Retaining adult learners in the first 3 critical weeks, 2000 ↩

86. MDRC, A Random Assignment Evaluation of Learning Communities at Kingsborough Community College: Seven Years Later, 2014, accessed 24 September 2019 ↩

87. NPCR, Learning Communities for Students in Developmental English Impact Studies at Merced College and The Community College of Baltimore County, 2012, accessed 24 September 2019; MDRC, Learning Communities for Students in Developmental Math Impact Studies at Queensborough and Houston Community, 2011, accessed 24 September 2019 ↩

88. Economic Mobility Corporation, Escalating Gains: The Elements of Project QUEST’s Success, 2018, accessed 24 September 2019 ↩

89. Quigley, B.A and Uhland, R.L, Retaining adult learners in the first 3 critical weeks, 2000 ↩

90. Department for Education and Skills, Evaluation of impact of skills for life learning: Report on Sweep 2, 2006, accessed 24 September 2019 ↩

91. Department for Education, Estimating the impact of traineeships, 2019, accessed 24 September 2019 ↩

92. MDRC, Staying on Course: Three-Year results of the National Guard Youth ChalleNGe Evaluation, 2011, accessed 24 September 2019 ↩

93. Department for Education, Improving engagement and attainment in maths and English courses: insights from behavioural research, 2018, accessed 24 September 2019; Office of Research, Evaluation and Program Support, Starting to succeed: The impact of CUNY start on academic momentum, 2018, accessed 24 September 2019 ↩

94. MDRC, Guiding developmental math students to campus services, 2010, accessed 24 September 2019 ↩

95. MDRC, GAIN: Basic Education in a Welfare-to- Work Program California’s Greater Avenues for Independence Program, 1994, accessed 24 September 2019; MDRC, A Random Assignment Evaluation of Learning Communities at Kingsborough Community College: Seven Years Later, 2014, accessed 24 September 2019 ↩

96. Department for Education and Skills, Evaluation of impact of skills for life learning: Report on Sweep 2, 2006, accessed 24 September 2019 ↩

97. MDRC, Mapping Success: Performance-Based Scholarships, Student Services, and Developmental Math, 2014, accessed 24 September 2019 ↩

98. Department for Education, The 16- 19 bursary fund: Impact evaluation, 2015, accessed 24 September 2019 ↩

99. Office of Research, Evaluation and Program Support, Starting to succeed: The impact of CUNY start on academic momentum, 2018, accessed 24 September 2019 ↩

100. MDRC, Doubling Graduation Rates: Three-year effects of CUNY’s Accelerated Study in Associate Programmes (ASAP) for Developmental Education Students, 2015, accessed 24 September 2019 ↩

101. MDRC, Mapping Success: Performance-Based Scholarships, Student Services, and Developmental Math, 2014, accessed 24 September 2019 ↩

102. American Economic Corporation, Does Job Corps Work? Impact Findings from the National Job Corps Study, 2008, accessed 24 September 2019 ↩

103. MDRC, Incremental Disbursements of Student Financial Aid: Final Report on Aid Like A Paycheck, 2019, accessed 24 September 2019 ↩

104. Department for Education, The 16-  19 bursary fund: Impact evaluation, 2015, accessed 24 September 2019 ↩

105. NCPR, Bridging the gap: An impact study of eight developmental summer bridge programmes in Texas, 2012, accessed 24 September 2019 ↩

106. Department for Education, Improving engagement and attainment in maths and English courses: insights from behavioural research, 2018, accessed 24 September 2019 ↩

107. NPCR, Learning Communities for Students in Developmental English Impact Studies at Merced College and The Community College of Baltimore County, 2012, accessed 24 September 2019 ↩

108. Washington, DC: Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services PACE Cross-Program Implementation and Impact Study Findings, 2019, accessed 24 September 2019 ↩

109. Washington, DC: Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services, Valley Initiative for Development and Advancement: Implementation and Early Impact Report (PACE), 2017, accessed 24 September 2019 ↩

110. Department for Education, The Youth Contract for 16-17-year olds not in education, employment or training evaluation: Research report, 2014, accessed 24 September 2019 ↩

111. Department for Education, Improving engagement and attainment in maths and English courses: insights from behavioural research, 2018, accessed 24 September 2019 ↩

112. MDRC, A Randomized Controlled Trial of a Modularized, Computer-Assisted, Self-Paced Approach to Developmental Math, 2018, accessed 24 September 2019 ↩

113. MDRC, Enhancing GED Instruction to Prepare Students for College and Careers, 2013, accessed 24 September 2019 ↩

114. MDRC, Moving Forward: Early findings from the Performance-Based Scholarship Demonstration in Arizona, 2013, accessed 24 September 2019; MDRC, Doubling Graduation Rates: Three-year effects of CUNY’s Accelerated Study in Associate Programmes (ASAP) for Developmental Education Students, 2015, accessed 24 September 2019 ↩

115. MDRC, Doubling Graduation Rates: Three-year effects of CUNY’s Accelerated Study in Associate Programmes (ASAP) for Developmental Education Students, 2015, accessed 24 September 2019 ↩

116. MDRC, More guidance, better results? Three-Year Effects of an Enhance Student Services Program at Two Community Colleges, 2009, accessed 24 September 2019 ↩

117. Washington, DC: Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services PACE Cross-Program Implementation and Impact Study Findings, 2019, accessed 24 September 2019 ↩

118. MDRC, GAIN: Basic Education in a Welfare-to- Work Program California’s Greater Avenues for Independence Program, 1994, accessed 24 September 2019 ↩

119. SRI Education, Evaluating Digital Learning for Adult Basic Literacy and Numeracy, 2017, accessed 24 September 2019 ↩

120. US Department of Education Mathematica Policy Research Reports, Addressing literacy needs at work: Implementation and Impact of Workplace Literacy Programmes, 1998 ↩

121. Elizabeth Zachry Rutschow and Emily Schneider, Unlocking the Gate: What We Know About Improving Developmental Education (New York: MDRC, 2011) ↩

122. Washington, DC: Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services, lnstituto de/ Progreso Latino’s Carreras en Sa/ud Program: Implementation and Early Impact Report (PACE), 2018, accessed 24 September 2019 ↩

123. MDRC, Keeping Students on Course: An impact study of a student success course at Guildford Technical Community College, 2012, accessed 24 September 2019 ↩

124. Quigley, B.A and Uhland, R.L, Retaining adult learners in the first 3 critical weeks, 2000 ↩

125. MDRC, Enhancing GED Instruction to Prepare Students for College and Careers, 2013, accessed 24 September 2019 ↩

126. Drikx, J.M and Crawford, M, Sage Publishers, Teaching reading through teaching science, 1993 ↩

127. MDRC, More guidance, better results? Three-Year Effects of an Enhance Student Services Program at Two Community Colleges, 2009, accessed 24 September 2019 ↩

128. MDRC, Keeping Students on Course: An impact study of a student success course at Guildford Technical Community College, 2012, accessed 24 September 2019; Weiss, B.D and others, Literacy education as a treatment for depression in patients with limited literacy and depression: A randomised controlled trial, 2006, accessed 24 September 2019; Hayslip, B. and others, The impact of mental aerobics training on older adults, 2016  ↩

129. Ministry of Housing, Communities and Local Government, Measuring the impact of community-based English language provision – findings from a randomised controlled trial, 2018, accessed 24 September 2019 ↩

130. Social Mobility Commission Investing in What Works in Further Education and Adult Learning: Policy Brief, 2019 ↩