Policy paper

Freeports Programme monitoring and evaluation strategy

Published 6 May 2022

This was published under the 2019 to 2022 Johnson Conservative government

Applies to England

Executive summary

The purpose of this document is to set out the monitoring and evaluation strategy for the UK Freeports Programme.

The UK Freeports Programme

Freeports are special areas within the UK’s borders where different economic regulations apply. Freeports in England are centred around one or more air, rail, or seaport, but can extend up to 45km beyond the port(s), or further where there is a clear economic justification. The English Freeports model includes a comprehensive package of measures, comprising tax reliefs, customs, business rates retention, planning, regeneration, innovation, and trade and investment support. The first English Freeports became operational in autumn 2021, and full roll-out is expected throughout 2022.

The monitoring and evaluation of the Freeports Programme

The purpose of the Monitoring and Evaluation (M&E) of the Programme is to provide comprehensive findings to assess the effectiveness and impacts of Freeports as a new policy. The M&E will provide accountability to Parliament and the public for the implementation and overall impact of the policy. Importantly, the M&E Programme will also enable learning and capacity building as the Programme is rolled out, providing early findings to improve the delivery of different initiatives. Initially envisaged to run for 5 years, the M&E Programme will provide an initial assessment of the impacts of the Freeports Programme, which may be extended in the future to cover long-term impacts.

This Freeports M&E strategy sets out our high-level approach to the M&E of the Freeports Programme, setting out the rationale behind the evaluation, our proposed approach to the main M&E activities and the eventual outputs it will produce. It is informed by a more detailed M&E framework, which at the time of publication is under development.

Approach to monitoring and evaluation

Monitoring and evaluation are significant activities at the centre of understanding what works and why. Whilst monitoring can demonstrate progress on meeting certain goals and can guide necessary adjustments in response, evaluation establishes whether overall objectives have been met and the extent to which change has occurred as a direct response to an intervention. A full Theory of Change for the Freeports Programme has been prepared to help identify inputs, outputs, outcomes and impacts; and the causal links between those.

As a starting point for the monitoring and evaluation strategy, and based on the Theory of Change, we have identified key research questions and detailed questions / causal contributions to define the scope for monitoring and evaluation activities. These will seek to test specific causal links through a mixed-method evaluation, combining theory-based and quasi-experimental models. Work is on-going in defining the specific data indicators and sources, as well as methods to test each causal contribution and research question. In line with evaluation best practice, the Theory of Change will continue to be developed over the lifetime of the evaluation as new evidence is developed.

Supporting ongoing learning

The M&E Programme will also include a Learning and Engagement Strategy. This will enable us to learn from this new policy model for future policy development and refine the Freeports Programme as it is being implemented, supporting Freeports to adapt.

1. Introduction

This M&E Strategy has been designed to enable the assessment of how and the extent to which Freeports will contribute to pre-identified high-level objectives.

The Freeports Programme

The Freeports Programme was announced by the UK government in January 2020. Freeports are a flagship government programme that is expected to play an important part in the UK’s post Covid economic recovery and contribute to realising the levelling up plan, bringing jobs, investment and prosperity to some of our most deprived communities across the 4 nations of the UK with targeted and effective support.

Since the Freeports consultation was launched in February 2020, the UK government has been working with ports, businesses, local authorities, and wider stakeholders to design a brand-new, bespoke Freeport model for the UK, with a comprehensive package of measures designed to boost regeneration and levelling up; trade and investment; and innovation.

Location of Freeports

English Freeport locations were selected through a fair, open, and transparent competitive process, which ran from November 2020 to February 2021. In March 2021, it was announced that, subject to completing the necessary authorisation processes, the following locations would become Freeports:

1. East Midlands Airport

2. Felixstowe and Harwich including the Port of Felixstowe and Harwich International Port

3. Humber including Port of Immingham

4. Liverpool City Region including the Port of Liverpool

5. Plymouth and South Devon including the Port of Plymouth

6. Solent including the ports of Southampton, and Portsmouth International

7. Thames including the ports at London Gateway and Tilbury

8. Teesside including Teesside International Airport, the Port of Middlesbrough and the Port of Hartlepool

Since the March 2021 announcement, the UK government has been working with the 8 English Freeports to develop Full Business Cases. At the time of publication, 2 Freeports are fully operational and 7 Freeports have designated tax sites. We expect all Freeports to be fully operational by summer 2022. More information can be found on the Freeports guidance page.

The M&E strategy presented in this document concentrates on the Freeports Programme in England. We will work with devolved governments to explore how M&E may be expanded to Freeports in Scotland, Wales, and Northern Ireland in the future.

Role of monitoring and evaluation

As a new government policy, the M&E of the Freeports Programme is required to assess the effectiveness of this policy and provide ongoing learning as the programme is delivered. M&E are significant activities at the centre of understanding what works and why. Whilst monitoring can demonstrate progress on meeting certain goals and can guide necessary adjustments in response, evaluation establishes whether overall objectives have been met and the extent to which change has occurred as a direct response to an intervention.

This strategy

This strategy aims to inform stakeholders and the public on the overall approach to Freeports’ M&E. It sets out at a high-level the methodology for how these activities will be conducted and reported and pointing to the envisaged outputs.

The report has been informed by the business cases developed by the Freeports, by a rapid review of the literature on the impacts of Freeports, and by ongoing engagement with the Freeports and stakeholders.

The rest of this report is structured as follows:

• section 2 presents the Theory of Change underpinning the evaluation
• section 3 presents the M&E plan
• section 4 presents the approach to data gathering
• section 5 presents the approach to ongoing learning
• section 6 presents an overview of next steps

2. Theory of Change

The Theory of Change is an important pillar of the M&E strategy. It sets out what the Freeports Programme is expected to provide, including outputs, outcomes, and impacts, as well as the causal links between them and the associated assumptions.

The Freeports opportunity

The Freeports Programme is aiming to address specific challenges and opportunities identified across 3 main areas:

• trade and investment – With UK total trade showing a decreasing trend as a share of GDP over recent years, there is an opportunity to increase trade as well as in promoting Foreign Direct Investment (FDI), which is associated with increased productivity
• innovation and productivity – Research and Development (R&D) spending in the UK falls below the OECD average as a percentage of GDP. Furthermore, R&D and innovative activities are concentrated in a small number of major firms and parts of the country (for example the ‘golden triangle’, the Greater Southeast). Freeports are an opportunity to catalyse increased business R&D spend and innovation to create new clusters of economic activity, focusing on sectors with potential to grow in areas that have underperformed economically in recent years
• regeneration and levelling up – Freeports are located in some of the most deprived UK communities, including areas suffering from post-industrial decline and some that may have been disproportionately impacted in the early days of COVID-19. There is an opportunity to address regional inequality by creating new jobs, raising wages and productivity and transforming the local economy of Freeport areas, aligning with the missions set out in the Levelling Up White Paper

Objectives of the Programme

In the context of the challenges set out above, the UK government has designed a bespoke, world-leading UK Freeport model aiming to achieve 3 objectives:

1. Establish Freeports as national hubs for global trade and investment by focusing on delivering a diverse number of investment projects within the Freeport regions, make trade processes more efficient, maximise developments in production and acquire specialist expertise to secure Freeports’ position within supply chains.

2. Create hotbeds for innovation by focusing on private and public sector investment in research and development; by being dynamic environments that bring innovators together to collaborate in new ways; and by offering spaces to develop and trial new ideas and technologies. This will create new markets for UK products and services and drive productivity improvements, bringing jobs and investment to Freeport regions.

3. Promote regeneration through the creation of high-skilled jobs in ports linked to the areas around them, ensuring sustainable economic growth and regeneration for communities that need it most. Local economies will grow as tax measures drive private investment, carefully considered planning reforms facilitate construction and infrastructure is upgraded in Freeports.

To meet these objectives, measures include a mix of policy levers from lower tariffs, enhanced capital allowances, business rate and stamp duty reliefs to simplified planning, infrastructure investment and innovation support.

The Levelling up White Paper and how it relates to this M&E Strategy

The Levelling Up White Paper presents an actionable long-term plan for levelling up the UK, addressing geographical inequalities and providing opportunities for economic growth across the country. The Freeports Programme, as one of the White Paper’s flagship programmes, aims to contribute to the government’s levelling up plans by boosting economic growth and regeneration in some of the UK’s most deprived communities. More specifically, it offers an opportunity to harness private sector investment to unlock employment and bring economic dynamism and innovation to vulnerable local economies.

The White Paper’s long-term plan to level up the UK acts on 4 missions or focus areas:

1. Productivity: boosting productivity, pay, jobs and living standards by growing the private sector, enhancing transport infrastructure, digital connectivity, and R&D.

2. Quality of life: spreading opportunities and improved public services, improved education, skills, health, and well-being.

3. Pride in place: restoring sense of community, local pride, securing paths to housing ownership, and reducing crime.

4. Leadership: empowering local leaders and communities, devolution of power, and more resilient institutions.

These focus areas underpin the Freeports Programme and directly translate into the programme’s objectives and policy levers. The focus areas will also provide a framework for the research topics in the Freeports M&E (as explained in section 3). Moreover, the Levelling Up White Paper provides the latest theoretical and empirical evidence base for the Freeports Programme and M&E framework. The White Paper proposes a framework of 6 capitals that capture main drivers of economic and social outcomes: physical, intangible, human, financial, social, and institutional capital. According to this framework, deprived areas where one or more of these capitals are lacking can get caught in vicious cycles that lead to persistently worse outcomes in all 4 focus areas outlined above.

The current state of play in terms of metrics to measure spatial disparities across the 6 capitals is presented in the White Paper’s Technical Annex. The Freeports’ M&E framework will aim to align its indicators with relevant metrics among those set out in the Technical Annex to track the programme’s performance in levelling up outcomes over time.

Theory of Change

Developing a Theory of Change (ToC) is the first step in understanding the intervention and how it is expected to achieve the expected outcomes. A ToC captures the theory of how the intervention is expected to work (setting out all the steps expected to be involved in achieving the desired outcomes), the assumptions made, the quality and strength of the evidence supporting them, and wider contextual factors. As evaluation evidence becomes available, the ToC will continue to be developed and refined.

Approach to Theory of Change

The M&E of the Freeports Programme will seek to test to what extent the objectives set out above will be achieved and why. To this end, an initial ToC has been developed to understand how the main components of the programme can lead to the desired objectives and the associated causal mechanisms. The ToC will evolve as the M&E progresses and evidence is collected and analysed.

The steps in developing a ToC are set out below. The first steps involve producing a logic model, the theory explaining the causal links in the model and associated assumptions (detailed in Appendix A).

The logic model is structured according to a bottom-up sequence. It starts with the main inputs at the base, with each subsequent level presenting a causal chain of events that lead to the expected impacts of the Freeports Programme. The model also provides an estimated timeline for the changes expected to occur at each level of the logic model. It should be noted however that the timescale provided is indicative. In practice, changes in each Freeport might come about at different speeds. Similarly, outputs and outcomes might materialise at different time scales than those currently assumed.

Once an initial logic model is developed, a suite of causal contributions or hypotheses should be agreed which set out what the evaluation will test in more detail, clearly defining the scope for the M&E. These claims should be informed by a review of the literature and the research questions. The ToC should be tested and refined as the evaluation progresses.

The key steps in developing a ToC are:

1. Finalise the logic model – set out what is meant to happen.

2. Develop the ToC – set out how and why the logic model works, including key assumptions and risks.

3. Develop a suite of hypotheses to test – this will be centred on the causal drivers of outcomes/impacts.

4. Test whether the ToC holds.

A logic model for the Freeports Programme

A logic model was developed to visually represent the ToC for the Freeports Programme. The development of the logic model has been led by the Department for Levelling Up, Housing and Communities (DLUHC), with input from other government departments, the Freeports Hub (a consortium of firms led by PA Consulting providing technical support and specialist advice to the Freeports), and the Freeports M&E provider (a consortium of 6 firms led by Arup, also including Technopolis, Cambridge Econometrics, Grant Thornton, PCLP and the Institute for Fiscal Studies). The model presents a more developed version of the initial logic model included in the Freeports Bidding Prospectus. The model will continue to be refined as the evaluation progresses.

The logic model can be found here.

The main inputs and activities to be delivered under the Freeports Programme are found at the bottom of the diagram. The Freeports Programme involves putting into place multiple levers to help achieve the final policy outcomes. The inputs and activities in the model relate to 6 policy areas: tax, customs, trade and investment promotion, planning, regeneration and infrastructure investments, and innovation. In addition to these, an aligned programme of investments and interventions by local and national government will likely contribute to some outputs, outcomes and impacts expected from the Freeports Programme, for example interventions funded by other government programmes and funds such as the Levelling Up Fund, the Towns Fund or net zero cluster investments.

The first level in the model presents outputs in the short-term resulting from the completion of activities. Increased capacity in terms of trade, investment, infrastructure, and innovation is expected across the Freeports, as well as an increase in the volume of commercially viable land. A combination of different activities and inputs can play a role in bringing about outputs set in level 1.

The second level sets out early outcomes linked to outputs delivered in level 1. These short-to-medium term changes should reflect initial agglomeration, increased R&D investment, innovation, decarbonisation, and socio-economic levelling up effects that are crucial to the achievement of the Freeports policy objectives. The desired impacts of the programme are explained in level 3. Impacts in level 3 are aligned with the programme’s objectives and include 3 main impacts:

• increasing trade and economic activity
• enable increased competitiveness, sustainability and productivity across Freeport areas
• job creation and increased wages in deprived areas surrounding Freeport areas

Finally, the fourth level presents medium to long-term impacts to be observed as a result of the intervention. The 3 impacts shown in the previous level represent 3 pathways to generate positive effects on the sustainable and long-term growth, productivity and levelling up of local economies in sub-regions where the different Freeports are based. The long-term levelling up impacts to be observed will target the 4 focus areas identified in the Levelling Up White Paper: productivity, quality of life, pride in place and leadership.

It is worth noting that different Freeports are likely to follow different timescales in the roll-out of their interventions, therefore affecting when different outcomes and impacts will be delivered. Additionally, other factors and initiatives could contribute to achieving these outcomes and should therefore be considered to enable the M&E to isolate the impacts of the Freeports Programme.

Summary of economic theory supporting the logic model

The main theoretical insights from the economic geography literature that are particularly relevant to the Freeports initiative are:

• conceptualisation of regional economies as complex adaptive systems that evolve over time according to both internal feedback processes and in response to external stimuli
• self-reinforcing agglomeration externalities, in particular theory around the mechanisms of localisation effects
• the concept of related variety, that identifies an optimal level of industrial specialisation that is neither fully specialised nor fully diversified, but instead maximises the possibility of positive synergy between related industrial sectors

Bringing these 3 concepts together into a single narrative, we can understand the logic behind the Freeports policy as a set of direct and leveraged policy interventions designed to catalyse a reinforcing feedback loop, with the ultimate intention of stimulating the emergence of a related variety of industries to evolve separately in each of the Freeports.

These policy interventions are designed to bring together a range of factors intended to increase the attractiveness of the Freeport sites and surrounding areas (primarily) to firms and organisations involved in targeted industrial sectors, and in doing so increase the sector-specific economic mass at that location. Should this immediate effect be achieved, economic theory suggests that this increase in economic mass should then stimulate the 3 primary mechanisms of localisation agglomeration effects, namely:

• the development of specialised supply chains and support structures
• the co-evolution of a local specialised labour force to match employment opportunity
• the evolution of a specialised knowledge space and innovation ecosystem

Through the accumulation of economies of scope and scale, improved labour matching, knowledge spill overs, and other positive externalities, these 3 mechanisms are thought to provide for increased levels of total factor productivity, labour productivity, resource efficiency, and returns on capital.

Specific targeted policy interventions designed to assist in each of the 3 mechanisms are generally thought to be helpful in ensuring the strength of these crucial secondary impacts, which may take several years to come to completion. The feedback mechanism by which successful sectoral clusters then grow, either internally or by attracting new participants, thus further increasing economic mass and scope for further specialisation and scale effects, is what provides this phenomenon with long-term transformational effects on local economies.

The exact combination of industrial sectors and sub-sectors that develop is likely to be different at each Freeport. However, localisation externalities are likely to be most strongly felt by industrial sectors that interact most directly with port activity; these include logistics, distribution, warehousing, transport, manufacturing, energy, and other offshore activities.

Assumptions underpinning links in the logic model

The underlying assumptions of the logic model explain the conditions required to bring about changes from one step of the model to the next. The list of assumptions was developed through cross-government collaboration with input from the Freeports Hub and Freeports M&E provider, drawing on a rapid review of the literature.

The assumptions have been developed to capture aspects related to the individuals and organisations who will take up the intervention, the early changes originating from inputs and activities, as well as benefits and potential unintended effects that come about because of the intervention. A review of the evidence supporting these assumptions has been conducted to determine a level of confidence that the assumption will hold true. This included a rapid review of the literature on the impacts of the different levers that form part of the Freeports Programme, a review of the Freeports’ business cases being produced, and government policy. The proposed assumptions will be put to test using an evidence review and theory-based evaluation approach. The final list of assumptions is presented in Appendix A.

3. Monitoring and evaluation of the Freeports Programme

Monitoring and evaluation

The M&E of the Freeports initiative aims to be exemplary in identifying the impacts of the Freeports Programme as a whole, shedding light on how changes were achieved (what interventions worked best and the factors contributing to this), as well as enabling ongoing learning and accountability. While the M&E approach concentrates on assessing impacts at a programme level, it needs to consider how to deal with the unique characteristics of individual Freeports and how and why outcomes may differ across different Freeports. The evaluation also aims to produce insights that can help inform key stakeholders (including individual Freeports) in improving policymaking and delivery.

The indicative timeline for the M&E programme is set out below, including which elements of the ToC we will be able to report on in each year:

• year 1 (completed August 2022): develop ToC and M&E framework, gather baseline information. By the end of year one, we would expect to start assessing how Freeports are being delivered (inputs and activities of the ToC)
• years 2-4 (2023-5): updates on performance and outputs, preliminary impact assessments, and annual reviews to identify lessons learnt. By the end of year 4, we would approximately expect to look at and assess level 2 of the ToC
• year 5 and beyond (2026): final evaluation of this M&E Programme. By the end of year 5 of the current M&E Programme, we would approximately expect to look at and assess level 3 of the ToC. It is worth noting that a longer-term evaluation will be needed to fully assess long-term impacts expected from the Freeports Programme
• M&E support will also be provided to the Freeports throughout this process

This ongoing process will feed into the learning programme and enable ‘near-live’ enhancements to policy implementation and evaluation approaches. These will be reported to Parliament each year, by 30 November, in the Annual Update Report, which will provide an update on the delivery of the Freeports Programme as a whole.

The M&E Programme’s main tasks in year one include:

• revising and developing an evidence-based ToC
• developing a M&E Framework
• agreeing data indicators, baseline data and a data dashboard
• engaging with Freeports and wider stakeholders to support the M&E Programme
• sharing learning with government stakeholders, Freeports stakeholders, regulatory bodies and wider public policy and business community to inform, improve and learn from the M&E programme. This will feed into developing the Freeports programme, provide evidence to inform future UK government policy development and contribute to public policy debates around Freeports

Evaluation principles

A good M&E framework needs to be useful, credible, comprehensive and proportionate to the size and wider characteristics of the programme being evaluated (as recommended in the HM Treasury Magenta Book).

Four core types of M&E activities

The M&E of the Freeports Programme will include monitoring, a process evaluation, an impact evaluation, and a value for money evaluation, all at the programme level rather than at the individual Freeport level. These are described below:

• on-going monitoring – the main objective of on-going monitoring is to check whether the implementation of the Freeports Programme is going to plan and to identify any barriers preventing the delivery of Freeports’ business cases as planned
• process evaluation – an analysis of how the Freeports Programme is being implemented and how that is contributing to the impacts of the programme; what factors are helping Freeports to improve the delivery of business cases; whether the programme design is working, and what is working more or less well and why
• impact evaluation – an objective test of what changes have occurred, the scale of those changes and an assessment of the extent to which they can be attributed to the Freeports Programme. This is usually investigated through theory-based, experimental, and / or quasi-experimental approaches
• value for money evaluation – a comparison of the benefits and costs of the intervention

It should be noted that Freeports are also able to conduct local evaluations at their own direction. DLUHC will work with Freeports in these instances with the aim to align data collection methods and metrics, to ensure consistency across the evaluations.

Multiple evaluation approaches

An evaluation framework drawing on different evaluation approaches is being developed, which includes designing the impact, process, and value for money evaluation of the Freeports Programme. The evaluation framework combines theory-based and quasi-experimental models. This approach allows us to maximise what can be learned about a complex programme, in which traditional econometric evaluation methods may not always be feasible or address important questions.

Freeports operate in a complex environment, with a range of other infrastructure and local economic development support measures (for example, the Levelling Up Fund) in operation. Freeports will operate in very different contexts and circumstances, further adding to the intervention’s complexity. This is one of the most significant challenges to the evaluation. Having a well thought out theory behind each impact and clearly understanding causal links will be crucial to the impact evaluation.

Addressing data challenges

The M&E will need to address the methodological and data challenges posed by a large and complex policy programme while remaining feasible to implement in light of available data. It will also need to address significant evaluation challenges which are typically encountered in evaluating area-based interventions, such as measuring displacement at different geographical levels. Finally, M&E will also provide recommendations about how data can continue to be collected and evaluated for a longer period. More information on data is provided in section 4. These recommendations will be provided in each annual report.

Key evaluation questions and research topics

As a complex and high-profile policy expected to draw on significant public funding and to provide wide-ranging economic benefits, the Freeports Programme will necessitate clear research questions that define the scope of the evaluation. It is recommended that the list of key research questions is kept relatively short, underpinned by more detailed causal contributions to test.

In this context, a set of high-level research questions have been developed for each of the 4 M&E activities. They were developed while preparing the ToC and refined with stakeholders with the purpose of defining the broad scope for M&E activities. These will be reviewed on an ongoing basis as the M&E progresses. Note that some questions will inform different types of activities simultaneously, for example informing both the monitoring and process evaluation activities. Impact evaluation questions overlap significantly with value for money questions as impacts need to be assessed and quantified to feed into a value for money assessment. The high-level key research questions are set out below, broken down further into more specific questions:

How is the Freeports Programme being implemented and how is that impacting on the delivery of the programme?

• What resources are being devoted to the Freeports Programme? (Process / Impact/ Value for money)
• How is the Freeports programme helping to enhance local capacity? (Process)
• How is the Freeports programme enabling ongoing learning by Freeports and helping to improve the delivery of the programme? (Process)
• What is the impact of the Freeports Programme on governance, partnerships and stakeholder behaviour and how is that contributing to the impact of the programme? (Process)
• What are Freeports delivering and how does this compare to what they expected to deliver in their business cases and why? Are there any factors preventing Freeports from delivering what they planned? (Monitoring / Process / Impact)

To what extent and in what ways is the Freeports Programme achieving its objectives? What is driving the impacts of Freeports?

• How much investment are Freeports attracting once set up and what is driving that? (Impact / Value for money)
• What is the impact of Freeports on employment and training and what is driving that? (Impact / Value for money)
• What is the impact of Freeports on productivity and wages and what is driving that? (Impact / Value for money)
• What is the impact of Freeports on business activity including R&D spend and innovation, and what is driving that? (Impact / Value for money)
• What is the impact of Freeports on trade and exports growth and what is driving that? (Impact / Value for money)
• What is the impact of Freeports on transport and what is driving that? (Impact / Value for money)
• How are Freeports supporting the net zero agenda and what is driving that? (Impact / Value for money)
• What is the overall impact of Freeports on economic growth in the wider Freeport areas and the UK accounting for additionality and displacement, and what combination of factors is driving that? (Impact / Value for money)
• What is the overall impact of Freeports on Levelling Up and what is driving that? (Impact / Value for money)

What are the lessons of the Freeports Programme for future policy, private and public sector stakeholders, and local communities?

• What is the likely impact of the time-limited tax incentives and other policy levers on Freeports’ long-term growth and economic success? (Impact / Value for money)
• Are there any unintended medium and long-term outcomes from Freeports including impacts on security and illicit activity? (Impact / Value for money)

What is the value for money of the Freeports Programme?

• What are the likely benefits, costs and overall value for money of the Freeports Programme? (Value for money)

Developing detailed questions and causal contributions

These high-level research questions are then underpinned by a set of more detailed questions whose causal contributions aim to be tested during the M&E. These will seek to test specific causal links, including how performance varies across different Freeports. The causal contributions are largely based on the assumptions underpinning the ToC, which explain what needs to happen for the outcomes and impacts to be achieved.

For example, the research question “How much investment are Freeports attracting once set up and what is driving that?” can be broken down into more detailed questions and causal contributions to be tested as follows:

• trade and investment promotion activities ensure that Freeport benefits are understood by businesses
• Freeport incentives are significant enough for businesses to choose to locate/invest in Freeports
• lower costs and regulatory support encourage firms to move into the Freeport as anchor tenants
• the presence of an anchor tenant encourages suppliers and collaborators to locate to the Freeport in order to increase business with them, leading to clustering effects

Key research topics

Related to the evaluation questions, 10 specific research topics that are of particular interest for the evaluation have been identified. These research topics are aligned with the programme’s objectives and also aim to explore its contribution to the Levelling Up White Paper’s 4 focus areas (discussed in section 2). They have informed the list of causal contributions to test. These are:

• trade and investment – new investment in Freeports and surrounding sub-regions, as well as increases in trade through designated Freeports
• tax benefits – regeneration of underdeveloped tax sites and additional economic activity brought by tax reliefs
• customs benefits – efficiency and facilitation of trade processes, maximised production and shipping driven by customs benefits and border innovations
• planning and regulatory impacts – planning environment for developers, as well as regulatory flexibilities to minimise barriers to innovation
• levelling up and economic growth – sustainable economic growth and regeneration brought to deprived or vulnerable areas around Freeports. The definition of Levelling Up is consistent with that set out in the Levelling Up White Paper
• displacement – extent to which the increase in economic activity generated in Freeport areas represents new economic activity as opposed to economic activity that would have occurred elsewhere. Also, to consider mitigation activities against displacement of local economic activity from deprived areas
• clusters, innovation, and productivity – support to clusters of sectors present and growing in the local economy, promotion of innovation focused activity in Freeport locations and economic specialisation in activities high in GVA
• Freeport security and illicit activity – security of customs and tax sites against illicit activity (e.g., organised criminal activity, money laundering, smuggling and illegal immigration)
• net zero – Freeports’ ambitions to support decarbonisation and to reach net zero emissions by 2050 or earlier
• institutional framework and partnerships – governance and capacity of Freeports to support effective decision-making, as well as stakeholder coordination and cooperation to enable the Programme’s successful delivery
• employment and skills – changes in number of jobs, level of skills and average wages in deprived areas in Freeports and surrounding sub-regions

Approach to evaluation

Once the framework is established, the M&E Programme will provide ongoing monitoring of the Freeports’ implementation and undertake a process, impact, and value for money (VfM) evaluation, through interim and final evaluations. The M&E activities will aim at answering all research questions in the most comprehensive way possible.

Below we provide a overview of each of the 4 Freeports’ M&E components.

On-going monitoring strategy

The monitoring of Freeports will assess to what extent the plans set out in the Freeports’ business cases are being delivered, for example whether seed funding is spent as planned. This will draw heavily on data gathered by the Freeports themselves, complemented by wider data sources (the next section on data provides more information on this).

On-going monitoring will enable us to observe early outputs and outcomes from the programme as well as to learn from how the implementation is being delivered, feeding into the development of the programme. This is important as some aspects of the programme are still being defined and will continue to be shaped over the coming years.

The monitoring of Freeports will feed into an Annual Review conducted by DLUHC as part of the Freeports Performance Management and Assurance Process.

Process evaluation

The process evaluation will focus on how Freeports are being delivered. This will include different phases of the delivery process: application for Freeport status, business case development phase, implementation, and operational phases.

The process evaluation will also focus on on-going learning and how to improve the delivery of Freeports to achieve the objectives set out in each business case, for example, lessons around governance and partnerships, which vary significantly across the 8 English Freeports. To assess this, the process evaluation is likely to draw heavily on interviews with different stakeholders across the Freeports, complemented by quantitative data. The timing of the process evaluation will be significant in enabling learning related to the implementation of the programme. More information on the learning programme is provided in section 5 of this document.

Impact evaluation

The impact evaluation of the Freeports will focus on assessing a wide range of outputs, outcomes and impacts using a mixed-methods approach, combining a theory-based approach with quasi-experimental methods. The impact evaluation will also assess if the Freeports Programme is on track to achieving its objectives. Below we provide a short explanation of each of these concepts.

Theory-based evaluation

Theory-based impact evaluations aim at drawing conclusions about an intervention’s impact through rigorous testing of whether the causal chains leading to the desired change are supported by sufficiently strong evidence, so that alternative explanations can be ruled out. Testing can involve different types of techniques for information gathering (such as surveys) and data analysis, including econometric analysis.

Theory-based evaluation is explicitly concerned with both the extent of the change and why change occurs; tries to link inputs to outcomes, and how the intervention is affected by wider contexts. HMT’s Magenta Book highlights the usefulness of theory-based approaches for complex programmes and/or programmes operating in a complex environment. In these instances, the variety of external (non-intervention) factors that influence beneficiaries creates difficulties in determining and isolating the specific effect of a given intervention.

Based on a rapid review of the literature, initial interviews with the Freeports and a review of the Freeports’ business cases, we have developed an initial long list of questions or causal contributions to assess the causal contributions reflected in the ToC. An initial review of indicators and data sources is also being progressed to identify main data sources and potential data gaps.

An important part of the impact evaluation will include the use of contribution analysis. This is a step-by-step process used to examine if an intervention has contributed to an observed outcome by exploring a range of evidence for the ToC. Contribution analysis also helps us understand how different interventions may contribute to the same outcome.

Contribution analysis involves drawing on all evidence collected to determine the validity of the theoretical causal mechanisms in the ToC. Contribution analysis therefore helps to confirm or revise a ToC. The contribution analysis may find that there are instead alternative explanations for the effects seen, suggesting lower levels of programme additionality.

Quasi-experimental approaches

For selected impacts, the theory-based investigation will be complemented with quasi-experimental approaches, designed to test key causal contributions quantitatively. Selected impacts will include those highlighted in the logic model including impacts of the Freeports Programme on trade, productivity, employment and levelling up, assuming data is available to quantify these. Note that these will be assessed at a programme level as opposed to individually for each Freeport.

Experimental and quasi-experimental approaches infer the impact of an intervention through statistical comparison to a group or time period unaffected by the intervention (known as ‘the control group’). This unaffected group acts as a proxy for what would have happened to the affected group in the absence of the policy. Control groups could include competitor firms, places around the Freeport, comparable ‘non-Freeport’ ports, and even international comparators, though these may have limited value given differing regulatory environments.

Methods like this involve the use of a counterfactual. A counterfactual is the scenario that represents what would have happened in the absence of the intervention. In the case of the Freeports Programme, a perfect counterfactual would be how the 8 successful English Freeport areas and other likely affected areas would have evolved in the absence of the policy being implemented. As this is not possible, quasi-experimental approaches are required to assess the impact of the intervention.

The following quasi-experimental econometric approaches are being explored as part of the development of the M&E activities.

Difference-in-differences panel fixed effects

Difference-in-differences (DiD) models can estimate the effect of an intervention by comparing the average change in an indicator before and after the intervention took place to a similar group which did not undergo the treatment (this is called the counterfactual or comparison group). These models will be used to assess the impacts in the logic model such as increases in employment, productivity, and trade.

For Freeports, we can compare the areas which have been granted Freeport status to ports, airports and other centres of economic activity which were not granted Freeport status.

This method requires panel data, which is a data set for the same areas across time. Different data sets that provide firm-level data are currently being explored including ONS data and data sets from other sources. It is also possible to include a number of other indicators as controls to ensure that the effect of the Freeports Programme is isolated from other confounding factors.

Challenges with this approach include:

• it requires detailed data collected on a consistent basis over time
• it requires an appropriate choice of comparator group Risks and limitations include:
• this approach is suitable to assess impacts quantitatively but needs to be complemented with other approaches to fully assess causal links
• such models require large amounts of data at a very high quality – in its absence the models can become imprecise, or in case some of its underlying assumptions are not met could make the estimation impossible

Synthetic control groups

Synthetic control methods create a comparison group through a weighted combination of multiple areas which did not receive an intervention.

Weights are estimated to minimise the difference in the predicted outcome variables and predictors between treatment area and control group before the intervention.

For example, a Freeport may not be directly comparable to a non-Freeport as it has a unique combination of, for example, industrial base, education, income, trade composition. A synthetic control group solves this problem by creating a weighted average of several non-Freeports which together mimic the Freeport, thereby providing a way of projecting trajectories for variables of interest in a hypothetical scenario without Freeport designation. In turn this synthetic counterfactual enables the estimation of selected impacts for any given Freeport.

Challenges with this approach include:

• it requires detailed data collected on a consistent basis over time
• the method relies on finding counterfactual areas which are similar in every possible way to the affected areas. This may be challenging for the largest ports in the country

Risks and limitations include:

• this approach is suitable to assess impacts quantitatively but needs to be complemented with other approaches to fully assess causal links
• in case its underlying assumption around parallel trends is not met, there are no obvious alternative method specifications (unlike for panel DiD methods)

Displacement

A challenge for the impact evaluation will be to assess the additionality of impacts in the context of displacement. Displacement refers to the transfer of economic activity from wider local areas, especially other economically disadvantaged areas. This will involve careful considerations around the geography of impacts, deadweight, and leakages. In this context, it will be especially important to understand effects of displacement of local economic activity from deprived areas. Alongside that, the assessment of displacement will seek to understand other economy-wide displacement patterns, some of which may be part of clustering dynamics.

To deal with displacement, the first step of the evaluation approach will be to assess the boundary of the impact to find areas which are likely to be positively affected by the creation of Freeports, but also identify areas which may be negatively affected. This is likely to be in line with the Freeports’ “travel to work areas”, which will be defined based on travel time and other economic factors. These areas will be separately included in the regressions to allow the positive or negative impacts on them to be quantified as part of the assessment of additionality of impacts.

Displacement will also be analysed through macroeconomic modelling exercises. Results from the evaluation at a port level can be input into macroeconomic models to identify the wider economic impacts on the UK and local economies, controlling for displacement effects. This can provide a macroeconomic view and insights on additionality at a regional and UK level.

Value for money evaluation

The value for money evaluation will compare the costs and benefits of the Freeports Programme. This will be undertaken in the final stages of the evaluation when important impacts have been quantified. The approach to the value for money will inform the approach to the impact evaluation by setting out which impacts should be quantified to assess benefits, so that both the impact and value for money evaluations are aligned.

To assess the costs and benefits of the programme, the outputs, outcomes and impacts quantified in the impact evaluation will be monetised and compared against the costs of the inputs and activities associated with the programme. The evaluation will identify which benefits and costs should be specifically included within the value for money assessment based on the levers (and associated costs) and impacts in the Logic Model with consideration of the value for money assessments of each Freeport included in the Freeports’ business cases. Benefits and costs will be monetised and summarised in a value for money assessment in line with Green Book guidance. Where there is uncertainty around certain impacts, a range of value for money metrics will be provided.

The following challenges, risks and limitations associated with the value for money assessment have been identified.

Challenges identified for this evaluation include:

• the costs will be difficult to quantify, as delivering the programme is likely to involve resources devoted by different government departments which might be challenging to monitor
• there is no standard Green Book methodology that can be applied directly to the Freeports Programme, therefore the value for money assessment method will require careful consideration

Risks and limitations include:

• the impact evaluation may not give enough quantitative data to be able to do a comprehensive value for money evaluation
• benefits and costs will continue to change over time therefore the value for money assessment is likely to change over time

4. Data gathering approach

The M&E for the Freeports Programme requires the gathering and analysis of a wide range of data sources including both quantitative and qualitative, across different geographies that will be used to support a wide range of research methods. We have identified some principles for data collection and analysis.

Approach to data gathering

The M&E of the Freeports Programme will require a wide range of data sources and indicators that are consistent across all Freeports. This will include both quantitative and qualitative data. The theory-based approach will enable us to combine different sources of information using different methods of analysis at different levels (for example, programme-wide or looking at specific policy levers/actors) to assess impacts as well as the causal links in the logic model.

The M&E Programme will follow a 2-pronged approach to data gathering focusing on 2 types of data:

1. Bespoke data: small scale data sets that cover specific outcomes or impacts. These are likely to be collected and reported based on bespoke templates created for the purpose of this M&E and will include a mixture of quantitative and qualitative data (for example, surveys, in-depth interviews, case study evidence). Data collected by the Freeports is included in this category and will be used for both monitoring and evaluation purposes.

2. Contextual data: larger data sets that are not focused on specific areas and are collected at a national level on a consistent basis. This may include data from organisations like the Office for National Statistics (ONS) or different government departments and is likely to include granular data at a business level to enable econometric analysis to be undertaken.

These 2 types of data will be collected, managed, and processed in different ways, but will be combined to provide comprehensive findings across the different research questions identified. For example, on the topic of displacement, contextual data from the ONS will be analysed to assess the relative economic performance of the Freeport areas vs. non-Freeport areas in combination with surveys and interviews with businesses across the Freeports to understand if they have displaced their own activities from other locations.

Regarding quantitative data, the following principles for data gathering and reporting have been identified:

• reusability – focus on data sets that are in standard formats and updated regularly, collected using standard templates
• consistency – data will be gathered on a consistent basis and for similar timescales
• granular – aim to collect data at the lowest feasible level of granularity possible
• quality – assess quality of the data before including it in the indicators or reporting
• objectivity – observed data over modelled data is favoured; modelled indicators (such as forecasts based on other indicators) will be used as a last resort
• proportionality and practicality – data will be obtained from secondary sources as much as possible reducing the burden on Freeport stakeholders regarding data collection

Data source selection

Data related to some monitoring indicators is expected to be gathered by the Freeports themselves using standard metrics. The M&E Provider will be responsible for gathering the rest of the data, focusing on outcome and impact data, engaging with relevant government departments to discuss the right data sources and level of analysis required. The M&E provider will also assess data gaps and identify where qualitative data is required to better understand different impacts.

As we develop the set of indicators, some data required is expected to be sensitive or restricted in terms of access and publication. We will therefore need to weigh the quality, coverage, and applicability of each data source against the processes and restrictions that will need to be managed to access it.

Geography considerations

When gathering data related to the impacts of Freeports on the wider area in which they sit, we will need to consider the geography of impacts and define study areas in which we consider the Freeport to be impacting. We have defined 2 study areas for the purpose of the M&E programme:

1. Firstly, we think of the immediate impact area which is where we will expect the Freeport to directly impact the economy.

2. Secondly, we will identify a secondary impact area where we will collect and process wider contextual data linked to local authorities’ boundaries. This area will be defined in a similar way to travel to work areas, that is, as the area acting as the main labour market for the Freeport.

As part of our data gathering and analysis, we may also look at the areas around ports that do not have Freeport status (including seaports and airports) and at clusters of economic activity for each Freeport to develop a comparator group to assess additionality of impacts.

Geographic impacts will be particularly important in the context of displacement. This will be assessed based on the secondary impact area in comparison to the rest of the UK.

Challenges

Challenges with data collection are likely to be significant and include time lags in the data, lack of spatial granularity, and data gaps. We will work closely with the ONS, cross-government experts and the Freeports to mitigate these challenges. Where data gaps cannot be filled with existing indicators and data sources, other sources of information such as surveys and interviews will be considered.

5. Impactful learning programme

The learning programme will help the UK government, the Freeports, and the wider public learn from and engage with the M&E activities. This is where the impact of the evaluation begins to be realised, as the M&E activities and outputs are translated into tangible learning for stakeholders.

Learning programme

The learning programme is a significant part of the M&E activities, closely linked in particular to the process evaluation. Stakeholders have a lot to learn from the M&E programme, including whether the programme is working as intended, whether the Freeports Programme is compliant with policy, or whether it is achieving the intended goals (and to what extent). This can have important implications for how the programme is run in the present, for how it adapts in the future, and for future policies and programmes.

The learning programme is defined by 3 principles:

1. First, identify and engage stakeholders at the outset to understand how they will use the evaluation.

2. Second, guide and adapt the M&E process based on how stakeholders will use the evaluation.

3. Third, analyse, interpret, and share the research findings in a way that will be best used by stakeholders and have an effect on current and future programmes.

By identifying how stakeholders will use the M&E learning early on, the learning approach helps inform the M&E strategy from the outset and how the M&E programme evolves over time.

Identifying stakeholders and how they will use the evaluation

We have identified 4 stakeholder groups, with many subgroups and individual stakeholder organisations within each group. These groups are:

• Freeports, the impact area and delivery partners – such as Freeports themselves, local communities, and local data collectors
• governmental stakeholders
• Parliamentary stakeholders
• wider societal interests – such as members of the public outside of Freeport impact areas, public policy and research bodies

By engaging with these user groups, the aim is to encourage them to take up a stake in the M&E programme and use its outputs.

Through early engagement, we have identified 5 broad aims of the M&E programme that need to be supported through ongoing learning and engagement with the 4 stakeholder groups. These aims fall into 2 groups: reporting and accountability and using evidence to inform decision-making.

Reporting and accountability, in which learning outcomes will:

1. Inform government of the Freeports Programme’s progress against policy objectives.

2. Support public accountability for the Freeports programme.

Using evidence to inform decision making, in which learning outcomes will:

3. Use the evidence from the ongoing M&E to improve the current Freeports Programme.

4. Use the evidence from the ongoing M&E to inform future policymaking in DLUHC and UK government more broadly.

5. Use the evidence from the ongoing M&E programme to inform broader public policy debate.

Informing the M&E development and Programme

To achieve the aims of the M&E programme, we will engage with each stakeholder group to ensure the programme meets their needs. A range of tools and approaches will be used in the learning programme. We may provide external-facing reports, group-based workshops, or one-to-one engagement, depending on the best method for sharing what we are learning from the M&E.

Our engagement to date has begun to identify stakeholders’ unique and shared M&E needs, which are summarised below as the activities the learning programme will undertake to meet those needs.

Priority activities to support Freeports, the impact area and delivery partners will focus on:

• supporting quality data collection and data management for M&E
• ensuring the M&E process adds value to the development of the Freeports

Priority activities to support government stakeholders will focus on:

• supporting the UK government to ensure the impact of Freeports can be effectively monitored and evaluated
• reporting on whether the Freeport programme is delivering as planned
• providing the evidence base for adapting and improving the current policy and programme
• providing an evidence base for future policy making

Priority activities to support Parliamentary stakeholders will focus on:

• supporting Parliamentary scrutiny by contributing M&E evidence to the UK government’s annual report to Parliament on Freeports
• keeping Parliament informed of the progress of the Freeports Programme using evidence from the M&E

Priority activities to support wider societal interests will focus on:

• disseminating information on policy rationale, implementation, and performance of the Freeports that will enable the public to hold the UK government to account for the policy outcomes
• contributing to the wider public policy debate

Interpreting research and disseminating findings

As the data is analysed and findings are interpreted, we will consult with the relevant stakeholders to understand how they want to engage with the analysis. Sometimes, data and analysis will be most valuable to our stakeholders so they can interrogate and investigate themselves. In other instances, synthesised intelligence will be more helpful, providing our insights so stakeholders can take decisions. The final evaluation report and outputs will be used to engage with the M&E programme’s findings. Wherever possible, we will tailor the outputs to stakeholders’ needs. This could include, but is not limited to, providing data dashboards, detailed methodological reports, scrutiny reports, or contributing to public policy debates. We will learn from the Freeports programme to ensure it is accountable and it informs future policymaking.

This comprehensive learning programme will work with a breadth of public and private stakeholders to provide them with the resources they need to ensure the Freeports Programme is delivering against its aims, can adapt as required, and can inform future policies and programmes.

6. Next steps for the evaluation

Over the new few months, the main focus of the M&E Programme will be defining the M&E framework in detail, including data indicators and data analysis methods required, in close collaboration with Freeports and stakeholders. Once confirmed, work will start on gathering relevant baseline data to inform the evaluation. To support data collection activities, guidance and templates will be developed for Freeports to be able to gather data in a consistent way. In addition to this, the learning programme will be confirmed, and significant learning activities will start.

Appendix A

The underlying assumptions of the logic model explain the conditions required to bring about changes from one step of the model to the next. Note that these are causal contributions to be tested in the M&E and therefore guide the M&E framework. An initial level of confidence has been allocated to each assumption based on a literature review to be reviewed as the M&E progresses. The subsection below details these assumptions and at which outcome level we expect these to materialise.

Assumptions supporting causal step from activities and inputs to level 1 outputs

1. Prospective Freeports have the underlying capacity to utilise UK government funding and support; and interest to devote their own resources – Medium confidence

2. The implementation of the Freeports Programme will not lead to unintended outcomes related to security and illicit activity – Medium confidence

3. UK government has sufficient resourcing, capacity, and ministerial prioritisation to deliver the Freeports Programme – High confidence

4. Tax benefits and other Freeport incentives lead to an increase in commercially viable land in Freeport areas (rather than land that is already viable) – Medium confidence

5. Incidence of tax incentives will largely fall on economic agents whose incentives are aligned with the Freeport objectives – Medium confidence

6. There is sufficient shared vision and cohesive partnership among port operators, local authorities, and others to reach ‘go live’ – High confidence

7. Freeports’ economic impact areas are aligned with targeted local communities and existing travel to work patterns – Medium confidence

8. No planning or resourcing barriers to infrastructure development – High confidence

Assumptions supporting causal step from level 1 outputs to level 2 outcomes

9. Freeports select target sectors with sufficient growth and innovation potential and build on their existing strengths to compete in these target sectors – High confidence

10. Benefits are understood by businesses, and incentives are significant enough for businesses to choose to locate/invest in Freeports – Medium confidence

11. Freeports secure anchor tenants that contribute to policy objectives and additional businesses are attracted (multiplier effects) that do the same – Medium confidence

12. The local upskilled workforce is attractive to businesses in the Freeport area so that these do not have to rely on commuters from outside the local economic area – Medium confidence

13. Businesses in Freeports increase levels of investment in the area (both R&D capital) relative to position before – Medium confidence

14. Freeport status attracts business to stay in the long-term – Medium confidence

15. Firms locate and/or make new investments in Freeport areas only because of Freeport status (meaning no pre-existing plans to operate and/or make new investment in the area) – Medium confidence

16. Training and support activities are predominantly targeted at disadvantaged citizens from deprived areas local to the Freeport – Medium confidence

Assumptions supporting causal step from level 2 outcomes to level 3 impacts

17. Training and support to citizens from deprived areas local to the Freeport address inequalities in access to good quality / industrial cluster jobs – Medium confidence

18. Increased foreign investment at Freeport locations generates positive investment and productivity spill-overs in the local economic area – Medium confidence

19. New jobs created are secured by local upskilled people living within the local economic area – Medium confidence

20. New jobs created are high quality jobs with salaries, or upskilling, that increase per capita income in the Freeport zone – Low confidence

21. R&D spend by Freeport businesses leads to increased firm productivity in the Freeport and region – Medium confidence

22. Firms operating in Freeports engage in domestic and international trade of products and services – Medium confidence

23. National transport infrastructure has the capacity for increased traffic that will result from successful Freeports – Medium confidence

24. Freeports’ transport infrastructure is appropriate for the type/size of freight – Medium confidence

25. Freeports are sufficiently well-connected to their hinterlands – Medium confidence

26. Businesses in Freeports increase levels of economic activity in green growth sectors – Medium confidence

27. Haulage services / drivers supply will overcome current external shocks and grow in response to increased demand from successful ports – Low confidence

Assumptions supporting causal step from level 3 impacts to level 4 impacts

28. Ports and businesses in Freeports operate in a low-carbon, green, sustainable way – Medium confidence

29. Displacement of economic activity from Freeports’ local areas or other non-Freeport low-productivity areas in the UK can be minimised – Low confidence

30. Sub-regions requiring levelling up do not experience any sudden economic downturns that outweigh Freeport economic benefits – Medium confidence

31. Competition between Freeport areas does not prevent all Freeport local economies to grow, with global demand levels and differences in growth pathways contributing to reducing displacement – Low confidence

32. Changes in house and rent prices (or new housing developments) indirectly resulting from successful ports will not force out inhabitants of deprived surrounding areas – Medium confidence