Research and analysis

Developing the monitoring and evaluation framework of the Investment Transformation Programme: first phase report (HTML executive summary)

Published 20 December 2023

Read the full ‘developing the monitoring and evaluation framework of the Investment Transformation Programme: first phase report’.

Background and context

This report summarises key findings from in depth research and validation carried out by Frontier Economics to develop a monitoring and evaluation framework for the Department for Business and Trade’s (DBT) Investment Transformation Programme (ITP). This research and validation proposed a series of conclusions and recommendations for DBT on how to develop their framework. This report highlights the key conclusions and recommendations from this project.

The ITP was launched in 2022 to transform DBT into “the world’s most capable and respected Investment Promotion Agency”, and deliver high-value inward investment in support of the government’s economic ambitions including science and technology, net zero and levelling up.

Specifically, the ITP has the overarching aim of enhancing DBT’s investment promotion services through organisational, operational, and service redesign across the department’s investment network to deliver improved FDI outcomes. The programme focuses on transforming 3 key areas:

• service offer for investors by differentiating the level of service provided to focus more resource on high value investment opportunities, serving others proportionately to attract investment, create jobs and greater prosperity
• standard ways of working for DBT Staff by improving capacity, capability, and resilience to make better use of expertise, available technology, and access the intelligence and tools needed to attract and maintain investment
• delivery of strategic priorities for the department and wider government by being better placed to focus on delivering the government’s strategic objectives and the department’s priorities

These transformations are intended to increase the efficiency of investment promotion activities by allocating departmental and HMG resources in a way the is calibrated to the profile of investments and investors. This, in turn, is expected to increase the effectiveness of investment promotion activities, in the sense of enhancing the value of foreign investment and thus benefits in terms of economic growth and delivery of government priorities.

The ITP consists of a number of different workstreams. This research was commissioned to focus upon 6 streams:

• top tier investment
• future investment support services (FISS) - insourcing
• FISS - outsourcing
• aligning relationship management
• building pipeline development
• investor digital support services (IDSS)

There are 3 additional work streams beyond the scope of this work, namely:

• UK Investment Handbook
• regions and nations
• people, culture and skills

This paper outlines the monitoring and evaluation framework that we recommend for the ITP. Specifically, we focus on 3 different types of evaluation as outlined in the Magenta Book:

• a process evaluation, which seeks to establish whether the ITP is functioning as intended. It assesses what parts of the ITP have worked as expected, which ones have not and why, and what improvements to implement
• an impact evaluation, which seeks to establish the contribution the ITP has made to the overall impacts sought by investment promotion that is to what extent do we see more of the desirable impacts compared to a counterfactual case in which there was no ITP
• a value for money (VfM) analysis, which builds on impact evaluation, and seeks to establish how the costs associated with the ITP compare to the monetary value of benefits associated with the impacts attributed to the ITP

Approach

The framework we recommend uses a theory-based approach, that, in turn, supports a contribution analysis to evaluation. A contribution analysis is an approach to evaluating interventions that maps out causal pathways from different aspects of the intervention to outcomes and impacts and seeks to establish how far observed outcomes and impacts are indeed attributable to the intervention in question.

The starting point of a theory-based approach is a theory of change, which identifies the problem the intervention is intended to solve, and how the intervention is expected to solve it. In the context of investment promotion, the underlying issue is the presence of market failures that mean, in the absence of government intervention, the extent and nature of foreign investment in the UK will be inadequate from the perspective of society as a whole. These market failures may be due to externalities, coordination difficulties, imperfect and/or incomplete information, and capital constraints, and may be connected to broader and institutional and policy shortcomings. It is these market failures that investment promotion is intended to remedy; and it is the effectiveness of investment promotion in remedying these market failures that the ITP is expected to enhance.

As part of the development of the framework, Frontier Economics have been commissioned to develop a logic model for the programme. This logic models links inputs and activities to expected outputs, outcomes and impacts. We have also developed logic models for each of the 6 workstreams. Though the workstream logic models are not included in this summary report explicitly, they are depicted through the overarching programme level logic model, with the workstream models used as a basis for that overarching logic model. These models were developed on the basis of prior work done by DBT, and through an extensive process of consultation with stakeholders across 9 interactive workshops.

The development of the framework also included developing a system model to depict the ITP as a system. The system model is based on the programme level logic model and demonstrates the interrelationships between each of the workstreams and how they interact to achieve the intended outcomes of the ITP. The system model is not included in this summary report given its complex nature and its representation through the programme level logic model.

Process evaluation

The process evaluation for the ITP is different to the way in which a process evaluation is normally undertaken in the context of “standard” interventions. This is because of the particular nature of the ITP, which aims to enhance the efficiency and effectiveness of a set of other interventions, namely those that come under the heading of investment promotion. Thus, the question of whether the ITP is working as intended (that is the basic question in process evaluations) will need to take into account whether and how far investment promotion is attaining its intended objectives. This means there is a closer interdependency between process evaluation and impact evaluation than would normally be the case.

The process evaluation is especially important as in the short run it can provide information about whether the transformation envisioned by the ITP is on track. Whether it is on track can be measured through lead/ prospect generation and lead/ prospect conversion-to-project metrics, which are readily observable data. These in turn could be mapped to ITP components that the systems modelling shows to be particularly influential in affecting these metrics. Key roles are played by FISS-Insourcing, and its interactions with Aligning Relationship Management and Building Pipeline Development. Evaluation efforts would therefore focus on these workstream outcomes and their underlying outputs. A similar observation applies to digital support services.

Impact evaluation

Impact evaluations are based on a counterfactual analysis. They seek to identify to what extent changes in impact variables of interest are attributable to an intervention, by comparing observed impacts to those that would have occurred absent of the intervention. The main challenges therefore lie in measuring the difference in relation to the counterfactual, and attributing this to the ITP. The attribution question is particularly challenging in the context of the ITP, given that what is in question is how far changes in economic and investment impacts are affected by the ITP via its effects on investment promotion. This will require a mixed methods approach based on a combination of qualitative and quantitative methods.

We propose a 2-tier approach to impact evaluation, taking into account the time horizons over which data become available (reflecting in turn either the time over which impacts become observable and/ or the time required for data gathering).

In the near term, this approach seeks to harness, as far as possible, information already available to DBT, or that could be gathered or inferred reasonably quickly. The key variables of interest are: how far the ITP has influenced the lead generation/ prospects/ projects chain, and how far there have been changes to the extent to which investment promotion enhances the prospects of investment impacts. Both steps are likely to require triangulating between quantitative and qualitative information sources.

Value for money analysis

VfM measures the relationship between the benefits associated with the impacts of an intervention, and the resource costs associated with it. Benefit-to-cost ratios (BCRs) express this relationship. A ratio of 2 or more is usually considered to be consistent with high value for money, under the guidelines used by HMG departments such as the Department for Transport. A ratio in excess of one is needed to demonstrate a net benefit to society.

The VfM relies on the impact evaluation. By this we mean that (i) like the impact evaluation it is a counterfactual exercise and (ii) it relies on the findings of the impact evaluation, and specifically on the relationship between workstream outcomes and the metrics surrounding lead generation and conversion that then drive investment impacts. The VfM requires that impacts be translated into monetary measures. This is readily done for some measures (for example changes to Gross Value Added), but requires more bespoke research efforts in other cases, such as environmental benefits or distributional impacts. In addition, it also relies on information about the incremental costs of the ITP that is the costs that could have been avoided had the ITP not been implemented.

As observed in the section on impact evaluation, the relationships between the ITP workstreams and the lead generation and conversion metrics pose various measurement challenges, and will likely require a mix of qualitative and quantitative methods.

As with the impact evaluation, we can distinguish between shorter- and longer-term options for the impact evaluation. This largely reflects the approach recommended for the impact evaluation, in that the shorter-term options draw on available information, while the longer-term options involve a greater effort in data and research. In the short run, efforts would focus on verifying how far the assumptions underpinning the existing BCR calculation hold. This involves checking projections regarding cost savings and the likelihood of landing higher impact projects, and in light of this making any necessary revisions to the BCR.

Longer run options reflect the recommendations made in relation to the impact evaluation, notably the need to develop greater rigour around the responsiveness of investment impacts to investment promotion, and the responsiveness of investment promotion to the ITP. There is also a substantial research agenda in relation to developing monetary measures of non-market benefits.