Research and analysis

Levelling Up Fund: Impact evaluation scoping study - executive summary

Published 19 June 2023

Executive Summary

Ipsos UK was commissioned by the Department for Levelling Up, Communities and Housing (DLUHC) and the Department for Transport (DfT) to undertake a feasibility study for an impact and value for money evaluation of the Levelling Up Fund (LUF). The principal focus of the project was on quantitative approaches to impact evaluation.

Headline findings

In terms of the key findings of this review:

A relatively robust quantitative impact evaluation of LUF is feasible.

• The evaluation could be achieved cost-effectively by using existing sources of administrative and secondary data. These sources will provide relatively comprehensive evidence of the impacts of LUF on economic welfare, and less extensive coverage of anticipated social impacts and effects in land and property markets.

• Comparatively robust inferences can be developed by exploiting the competitive nature of the funding allocation process. Areas associated with (a) declined but shortlisted applications and (b) declined applications sharing similar Levelling Up Priority Index scores could both provide defensible comparison groups. Statistical matching methods may also be helpful in ensuring that comparisons are only made between areas sharing similar features.

• An evaluation could be implemented with well-established econometric methods that would explore how impacts vary with distance from the locations benefitting from LUF funding. These will help identify any important offsetting effects (e.g. displacement) as well discriminate between benefits at the local and the national level (critical for any cost-benefit analysis of the programme).

• The credibility of findings will be improved by controlling for the effect of both historic and parallel regeneration programmes. This can be achieved if the specific location and timing of projects funded is known. To facilitate this, it is recommended that DLUHC compiles granular data on the location of projects where this is achievable (though it is recognised that this may not be achievable in all cases).

• There are also a variety of plausible options for valuing the social and economic benefits of the LUF as part of a value for money analysis.

Organisation of an evaluation

In terms of the potential structure of an evaluation:

• Programme level evaluation: A programme level evaluation would focus on those objectives that are common across the LUF portfolio. This would involve a relatively narrow focus on local economic growth and the quality of life of residents.

• Thematic evaluation: LUF projects can broadly be thought of as belonging to one of four ‘thematic types’ - interventions to unlock development, improve connectivity, strengthen visitor economies, or improve quality of life for residents. A programme level evaluation could be accompanied by a thematic evaluation exploring the relative effectiveness of these types of intervention.

• Geographical intervention: Funded LUF projects have a high level of spatial dispersion and while it may be possible to explore the effects of the programme in individual regions (or types of area – e.g. low, medium, and high productivity LAs), analysis at lower geographical levels will be constrained by sample sizes.

• Project level evaluation: A robust statistical impact evaluation will only be feasible by pooling projects of similar types to create a portfolio. This could be complemented by targeted project level impact evaluations (e.g. for large scale, novel, or contentious projects). Given the diversity of interventions, evaluation designs will need to be bespoke to each project, with implementation typically requiring primary research with local stakeholders, businesses, property agents and residents.

Evidence gaps and limitations

• Coverage of outcomes: There are some important potential social impacts that will not be possible to establish in an evaluation of LUF without additional evidence. Most significantly, there is no source of information that could be used to reliably establish the effects of the programme on the quality of the built environment or civic pride/pride in place outcomes. This could be addressed by commissioning a large-scale survey, and the trade-offs between cost and added value would need careful consideration. Scale economies could be achieved by co-ordinating across other evaluations with similar evidence needs (e.g. Towns Fund, Shared Prosperity Fund).

• Commercial datasets: There are a variety of commercial datasets that could be used to explore the vitality of local economies although these also usually involve significant costs. Again, scale economies could potentially be achieved with co-ordination with parallel studies.

• Detailed thematic analysis: It will be difficult to establish the effects of some types of intervention owing to small sample sizes. Sample size constraints are likely to be problematic for interventions seeking to improve connectivity between areas and interventions aiming to improve the quality of life for residents, as they represent a diverse portfolio of projects sharing little in terms of common intermediate results. A detailed understanding of how and why these interventions work (or not) could be achieved by pooling these projects with similar projects funded through other programmes, an approach that could be beneficially considered as part of a parallel study.

Timescales

• Timeframes: The LUF is funding capital investments that will not be complete until 2024/25, and their benefits may not be apparent for some time after that. Pursuing an impact evaluation is not advised until 2027/28 (and as some impacts may not be visible for the long-term, a longer-term impact evaluation could potentially be considered in 2031/32).

• Process evaluation: Stakeholders highlighted that there is a need for evidence to feed into the Spending Review expected in 2023/24. There may be benefits in undertaking a review of projects to understand progress made and explore the likelihood that they will produce their intended effects in the future (i.e. a process evaluation).