Research and analysis

Summary: Youth Employment Initiative – Impact Evaluation

Published 2 March 2022

Research Summary Template

Authors: Ian Atkinson and Matthew Cutmore (Ecorys UK)

Background

This summary presents the findings of an impact evaluation of the Youth Employment Initiative (YEI), undertaken by Ecorys. The YEI represents part of the European Commission’s (EC’s) policy response to the social and economic challenges stemming from the financial crisis of 2007-2008, and is implemented in England as part of the European Social Fund (ESF). The YEI impact evaluation commenced in April 2017 and ran until late 2019. It follows a previous process evaluation published by the Department for Work and Pensions (DWP) in 2017. The impact evaluation focuses on examining the effectiveness, efficiency and impact of the YEI.

Methodology

A mixed-method approach was adopted to evaluate the effectiveness, efficiency and impact of the YEI. The findings presented below are drawn from:

• a desk-based review of YEI and related documentation;
• secondary analysis of data, including YEI Management Information (MI), results from the separate ESF and YEI Leavers Survey commissioned by DWP, and additional statistical data from the Office for National Statistics (ONS);
• interviews with high-level stakeholders from the ESF Managing Authority (MA);
• primary research across 10 delivery areas, including interviews with YEI provider staff, European Structural and Investment Funds (ESIF) sub-committee representatives, and YEI participants; and,
• a counterfactual impact evaluation (CIE) strand, comparing outcomes for YEI participants with a comparison group.

The evaluation findings also include a cost-benefit analysis (CBA), undertaken by DWP analysts based on the results of the impact evaluation strand.

Key Findings

Effectiveness

To date, the YEI has proved generally effective in terms of targeting participants, the delivery models developed to implement provision, and the delivery of provision itself. The programme is broadly on target in terms of anticipated engagement numbers and in respect of the anticipated gender split of participants. Key factors supporting effectiveness in targeting include: effective use of local data and intelligence; use of delivery partners’ existing networks; focusing extensively on outreach activity; developing partnerships with Jobcentre Plus to encourage referrals; and engaging in co-location with other services.

The delivery models developed also appear to be effective. Notable aspects to this include the use of ‘key worker’ roles to provide consistency and an overview of individuals’ support needs, the development of partnerships and referral routes to offer a wide-ranging support offer, and the establishment of effective governance procedures. The available evidence also suggests a positive picture of the effectiveness of YEI delivery itself in terms of the provision on offer and its implementation. The range of support available, and the extent it is tailored to individual needs, are key factors promoting effectiveness.

Several aspects of the YEI support offer consistently emerged as being effective and important. These included: ‘wraparound’ support designed to address individuals’ personal and often deep-seated challenges and barriers, commonly facilitated through a key worker role; short, sharp interventions to address a small employment need or gap in a young person’s CV; English and Mathematics provision, a lack of competence and qualifications in respect of which was seen as a key barrier to finding work; training linked to employment route-ways; and community-based activities and volunteering, such elements being key in reducing isolation and increasing confidence as part of moving towards employment.

While the majority of YEI delivery and provision can be assessed as effective, in particular contexts or for particular participants some elements appeared to be less so. For example, more structured provision in a classroom setting was cited as discouraging the engagement of young people in some cases, potentially due to prior negative experiences at school.

The positive overall impression of effectiveness was also apparent when considering the quality of the employment and training offers participants received. The YEI Leavers Survey provided the main evidence for this assessment. Nine in ten respondents accessing traineeships felt that their traineeship would improve their chances of getting a job. In terms of job offers, around half of the 45 per cent of respondents reporting that they were in work six-months after leaving the YEI were on a permanent employment contract and a further 14 per cent on a contract lasting 12 months or more. This suggests the majority of jobs gained by YEI leavers were fairly stable and long term. Finally, almost two-thirds of YEI leavers receiving job offers between starting on the programme and six months after leaving rated the quality of offers received as either ‘very good’ (27 per cent) or ‘good’ (35 per cent). Less than one in ten rated their offers as either ‘poor’ or very poor’.

Efficiency

The available evidence suggests that organisations delivering the YEI are concerned with ensuring efficiency and hence, ultimately, offering value-for-money. Common examples given by provider representatives of how they sought to ensure efficient delivery included: close control of staffing numbers; ensuring appropriate caseloads; focusing on staff progression, retention and training to reduce recruitment costs; reducing transaction costs between delivery partners; and reducing overheads where possible. Other examples included reducing costs through re-use and recycling, using account holder discounts when purchasing provision and courses, and avoiding overlaps or duplication with other provision available locally.

While the general impression was one of efficient delivery, there are limitations to assessing the efficiency of specific types of YEI provision and activities. There is a lack of data to facilitate such assessment, with providers not generally identifying costs and throughput of individual activities (in terms of numbers participating). In addition, it was evident that the inter-related nature of many YEI activities naturally makes such data gathering and assessment problematic.

Accepting these limitations, work around personal development, such as building confidence, was seen as intensive though ultimately cost-effective when balanced against the positive effects this was often seen as having for young people. Short courses leading to required qualifications were also cited from this perspective, in that a relatively small outlay could fill a gap in a CV and unlock opportunities.

The case study research was used to identify aspects that promoted or hindered efficiency. A focus on unit costs in developing provision was widely cited as a factor supporting efficiency. This was noted by provider representatives as resulting from DWP as the ESF MA requesting details of such costs in the application process. Effective partnership working and governance also evidently plays a key role in enhancing efficiency, whether through ensuring that the particular skills and organisational capacities can be efficiently deployed, or that performance can be managed by the lead partner to promote efficiency across delivery partnerships.

Impact

Evidence shows that the YEI is having a positive impact on the employment prospects of participants. Compared to a similar comparison group constructed from administrative data, results from the CIE show that, on average, YEI participants were in employment for an additional 56 days in the twelve months following support. Effects on likelihood of claiming benefits are less clear, with there being no statistically significant effect on the likelihood of claiming Jobseekers’ Allowance (JSA) or Employment and Support Allowance (ESA) following YEI support. As explained in the main report, it was not possible to provide estimates on the likelihood of claiming Universal Credit (UC) due to data limitations in the context of UC roll-out during the YEI’s delivery period.

At the overall programme level, a CBA based on the results of the CIE and conducted by DWP analysts estimates that while the programme has a social return in the range of £1.50 to £1.55 per £1 spent, the fiscal return is in the range of £0.13 to £0.17 per £1 spent (‘Fiscal return’ refers to an assessment of costs and benefits as measured from the point of view of the Exchequer, including changes in benefit and tax credit payments, taxes received and programme operational costs. ‘Social return’ refers to an assessment of costs and benefits, such as changes in income, stemming from the impacts of employment programmes on different individuals and groups in society). It is acknowledged, however, that these are likely to be underestimates with the true value for money being higher.

The positive estimates of employment impacts generated by the CIE are mirrored by the evidence available from the MI data and ESF Leavers Survey. While the former shows that just under a third of participants are recorded as entering employment immediately on leaving the programme, on the basis of respondents’ self-reporting the Leavers Survey indicates that just under one in two are in employment six months after leaving. Taking account of the YEI client group, for whom the evidence indicates labour market disadvantage and multiple barriers to work are common, the mutually reinforcing evidence around employment impacts from the CIE, MI and survey data can be considered very positive.

The MI data also enables an assessment by gender and disadvantaged status, using the definitions of disadvantage given in the ESF guidance. The data shows little variation in employment outcomes by gender. However, positive employment outcomes are considerably lower for participants recorded as having an additional labour market disadvantage (in addition to being NEET), compared to those for whom a disadvantage is not recorded. Amongst those with a disadvantage, just over a quarter (26 per cent) were recorded as entering employment on leaving the YEI, with the equivalent figure for those with no disadvantage being just over a third (37 per cent).

Evidence of education and training impacts is also available from the MI data and the Leavers Survey. As at September 2019, the MI showed that just under one in five YEI leavers moved into education or training, with just under one in ten gaining a qualification on leaving. Similar evidence is available from the YEI Leavers Survey, with 16 per cent of respondents reporting that they were in education or training at the six-month point after leaving. When adding respondents in training or education six months after leaving to those reporting that they were in work at this point (45 per cent), the survey evidence presents a very positive impression of just under two-thirds of participants achieving positive destinations in the period after support. For a programme seeking to reduce numbers of young people not in employment, education or training (NEET), this is further evidence of the key aims of the YEI being met to a considerable extent.

As well as the ‘harder’ impacts evident from the YEI in terms of entry into work or learning, evidence suggests that the programme frequently has a wide range of beneficial ‘softer’ impacts, such as enhancing self-confidence. Consistent proportions of around four-fifths of respondents to the Leavers Survey, for example, reported that the YEI had helped with a range of such outcomes, including communication skills, self-confidence around work, ability to do things independently, motivation to find a job or do more training, and team working. Similarly, improved confidence, aspirations and motivation emerged as particularly strong and consistent themes in participants’ discussions of the impact of support during case study interviews.

A range of new or enhanced skills resulting from engagement with the YEI were also frequently cited by participants, including improved leadership skills, employability skills, skills around budgeting and managing finances, enterprise skills, and subject-based skills, including those related to Mathematics and English. The case study evidence also indicated that such softer outcomes are central to the achievement of broader YEI impacts, for example around reducing barriers to, and entering, work and learning. As such, they help explain, and reinforce, the positive impacts discussed above around levels of re-entry to employment and learning amongst the NEET young people supported through the YEI.

In addition to impacts on YEI participants, evidence indicates that the programme is having a number of broader impacts. The clearest and most consistent involve beneficial impacts for provider organisations. These include improved delivery systems and structures gained from insights from working with partners, enhanced ability in terms of delivering ESF-funded provision, and the development of new links and relationships with agencies and organisations working to support young people. Evidence around impacts on employers and local communities was more anecdotal and piecemeal, though still positive in cases where concrete examples were offered.

Finally, it was also evident that the experience of designing and delivering the YEI has generated broader ‘learning’ impacts. Typically, these were described in terms of helping to inform or crystallise policy insights, and/or as providing lessons for employability and skills initiatives targeted at young people. Specific examples included insights on the importance and frequent necessity of addressing mental health in the context of employability support, the need to focus more on young people further from the labour market as youth employment rates improve, and the potential gap that will be left following the end of the YEI, along with the implication that similar provision is likely to continue to be required in future.

Conclusion

Evidence across the different evaluation strands presents a positive impression of the effectiveness, efficiency and impacts generated by the YEI. This demonstrates that interventions of this type, specifically focused on targeting and supporting NEET young people, can be effective and lead to a range of positive outcomes. The support on offer is clearly welcomed by young people, with the flexibility and wraparound elements being central to participants’ engagement with the YEI and the positive results they gain from it.

Looking at the theory of change developed to structure the assessment of the programme, it is apparent that the majority of intended outcomes can be assessed as being achieved or exceeded. In particular, results in relation to the core aim of the programme, around supporting young people to (re-)enter employment, education or training, hence contributing to a reduction in NEET levels, can be viewed as very positive. In addition, evidence indicates that the YEI had a positive effect on many of the broader, intermediate outcomes likely to have contributed to these ultimate impacts, particularly in terms of enhanced confidence and interpersonal skills, alongside reduced labour market barriers. The relative success of the programme is also notable in light of the nature of the YEI client group, suggesting learning can usefully be drawn for future initiatives aimed at reducing NEET levels.