Research and analysis

Local Digital programme evaluation scoping study: executive summary

Published 10 April 2024

Applies to England

Executive summary

The Department for Levelling Up, Housing and Communities (DLUHC) commissioned a consortium comprising PUBLIC, Socitm, Daintta and Perspective Economics to evaluate the Local Digital programme. The Local Digital programme aims to improve the cyber resilience of local government, and to use digital transformation to help councils deliver improved services and increase their productivity. This scoping study outlines the approach that is proposed for the process, impact and value-for-money (or economic) evaluations.

The primary focus of the study was to scope a robust and feasible methodology for understanding the impacts of the Local Digital programme. The evaluation covers the interventions that have historically comprised the programme, some of which began in 2018. However, the consortium will support the monitoring of the programme as it evolves throughout 2024.

Headline findings

The key findings of this study are as follows:

• the Local Digital programme has been comprised of separate interventions across five main workstreams. Workstreams were added at different times, starting in 2018, and the programme composition in scope of the evaluation was agreed in 2022. Each workstream has its own targets, and they can be broadly categorised as ‘digital and cyber transformation’ projects. However, they do not share a single, clearly defined overarching objective to which each workstream directly contributes
• a high-level vision for digital and cyber transformation is developed to unify the multiple workstreams around a common ambition. This vision describes a ‘new paradigm’ of individual, organisational and system readiness to make the best use of digital services and ensure councils and their residents are protected from cyber-attacks
• there are limited baseline data that illustrate the state before the interventions were implemented. The nature of many of the interventions (especially those focused on upskilling and small-scale, heterogeneous digital projects) complicates the task of identifying systematic impacts at the council or sector levels. These factors restrict the ability to use quantitative impact evaluation approaches
• a novel qualitative approach is proposed. This involves the analysis of case studies of the experiences of different types of councils that participated in Local Digital. This report outlines a provisional high-level typology of councils participating in the programme along two broad dimensions – capability and vision – that characterise councils’ position towards the programme. This approach will help to understand what interventions worked for different categories of councils in terms of supporting the new paradigm of digital and cyber transformation in the sector
• where possible, the qualitative approaches will be supplemented by limited quantitative impact analysis at the workstream level. Some surveys captured relevant information before certain workstreams began, enabling analysis aimed at isolating causal effects, such as difference-in-differences analysis, in some instances. In particular, the ‘Mitigating Malware and Ransomware’ survey, completed by over 90% of councils before the programme’s cyber interventions began, offers an opportunity to capture changes in councils’ cyber risk profile and the resulting economic impacts

Due to the reasons highlighted above and discussed throughout the report, a programme impact evaluation which relies mainly on qualitative methods is feasible.

The rest of this summary provides an overview of the programme, the key methodological proposals, and the limitations of the evaluation approach.

Programme overview

The Local Digital programme comprises a set of interventions that has expanded over time – and it continues to evolve today. The first workstream – the Local Digital Fund – was established in 2018. A package of cyber support measures was added in 2020, a more formalised training series in 2021, and a local government digital transformation pilot project – Future Councils – in 2022.

These interventions are covered by 5 workstreams in the scope of this evaluation:

• the Local Digital Fund workstream funded collaborative projects that explore the use of data and technology to tackle challenges across service areas. Councils applied for funding jointly with at least two other councils, to ensure that such challenges are common across the sector and that the solutions can be applied in multiple councils. In the latest round of funding, 17 councils participated
• the Future Councils workstream explores ways in which local authorities and DLUHC can identify the common blockers and enablers of councils adopting modern and resilient practices, systems, and processes. Eight councils participated in the Future Councils pilot in 2023
• the Cyber Support workstream is concerned with improving the cyber resilience of councils and their readiness to defend themselves against cyber threats. The associated funding offer was deliberately awarded to councils according to their self-reported preparedness to prevent or recover from malware and ransomware threats. Over 150 councils started the Cyber Support intervention with high-risk remediation actions to address
• the Cyber Assessment Framework workstream helps councils to assess their own cyber resilience by adapting the National Cyber Security Centre’s national Cyber Assessment Framework. It aims to establish the Cyber Assessment Framework as a routine part of councils managing their cyber risks
• the Training workstream supports councils and local authorities in improving staff’s skills in digital transformation and the associated ways of working (e.g. agile)

The programme was established following the Local Digital Declaration (2018), a statement co-created by DLUHC, the Government Digital Service, and local government sector representatives, which outlined a common ambition for local services of the future. This ambition is intentionally high-level to capture the range of digital and cyber transformation activities that could contribute to realising it. Moreover, the explicit and later addition of cyber workstreams created two distinct intervention strands – digital and cyber – which have had complementary but distinct objectives. Therefore, the evaluation needed to define the common ambition shared across the workstreams more concretely.

Defining the overarching ambition that Local Digital interventions contribute towards

The first part of the study defines a view of digital transformation that the programme contributes towards through all its separate interventions.

Discussions at the end of 2023 with the Local Digital delivery team, the evaluation’s Expert Advisory Group (which consists of sector representatives, economists, academics and central government officials) and other scoping activities highlighted the importance of considering the Local Digital programme as one part of a wider transitional movement. This transition aims to create the conditions for councils to be prepared to:

a) harness the opportunities of digital services and ways of working

b) withstand the threats presented by the reliance on digital services and other aspects of the modern digital environment

The evaluation settles on the following principles of digital and cyber transformation that together comprise the ‘new paradigm’ as an overarching ambition that unites the Local Digital workstreams:

• system readiness – having the technological infrastructure required to support large-scale transformation
• individual readiness – the preparedness of people to adopt new methods and adapt to changing digital landscapes
• organisational readiness – an alignment of organisational values and structures to support digital transformation goals

The evaluation will investigate the extent to which Local Digital’s interventions contribute towards these three principles, and thus whether the programme can be said to have supported the transition to the new paradigm.

The theories of change

A series of causal mechanisms are outlined for the programme and each workstream. These causal mechanisms describe how each intervention is expected to contribute to their intended outcomes. The programme-level theory of change is summarised here; please refer to the study for the detailed workstream-level theories of change.

The programme theory of change begins with the inputs (ultimately DLUHC staff and funding) and activities. The activities include the actions carried out to deliver each of the workstreams plus those of supporting roles within DLUHC, such as communications and policy work.

The outputs expected from the programme (generated by the workstreams) broadly fall into three groups. First, learning outputs will be produced, such as the direct provision of upskilling of councils (via the Training, Future Councils and Local Digital Fund workstreams), and research that uncovers digital and cyber challenges, opportunities, and requirements across the sector (Future Councils and Local Digital Fund). Second, councils are expected to apply this upskilling in the collaborative development of digital products (Future Councils and Local Digital Fund). Third are cyber improvement outputs that DLUHC will develop in collaboration with councils, such as baselining and improving upon cyber resilience, and developing a consistent way for councils to measure their own cyber resilience (Cyber Support and Cyber Assessment Framework).

For programme outcomes, the study outlines nine outcome areas that encompass the anticipated range of improvements to councils’ digital services and cyber security. Each outcome area is expected to be realised by one or more workstream and, if realised, to reflect the sector’s movement towards different dimensions of the new paradigm.

Given the iterative nature of the programme, it was important for the study to comprehensively scrutinise the causal mechanisms linking inputs and activities to outputs and outcomes. This exercise identifies potential synergies, interdependencies, and frictions between the workstreams, as well as places where impact might be limited by intervention design or confounded by other events. This part of the study therefore discusses the assumptions behind such mechanisms and identifies considerations for the impact evaluation in developing a view of the changes that resulted from the programme.

The approach to evaluating the programme

The evaluation will consist of three strands: a process evaluation to understand how the programme was delivered; an impact evaluation to identify what changed because of it; and an economic evaluation to monetise and compare the programme’s costs and benefits to understand its value for money.

The process evaluation will aim to answer its research questions through analysis of qualitative and administrative data. Interviews with delivery staff and participating council staff will provide insights into how workstreams were operationalised. Management information data will show which councils received support/funding, participated in activities and for what purpose. The participant characteristics component of this evaluation will be crucial in refining and confirming a typology of councils that this study proposes to distinguish between councils of different initial digital and cyber postures (explained below).

The impact evaluation will happen on two levels, taking a novel approach to address the data and policy design limitations. First, workstream-level analysis will identify the extent to which the nine outcome areas were realised, and use contribution analysis to confirm and refine the theories of change. This insight will inform the second level: a novel programme-level approach involving the analysis of case studies of the experiences of the different types of participating councils. The next subsection explains these key methodological proposals, which have resulted from discussions with the team and the Expert Advisory Group throughout the scoping phase of work.

The economic evaluation will aim to monetise wherever feasible the benefits of the workstreams’ outputs, as captured by the impact evaluation. These will then be compared to the workstreams’ costs to give a view of each workstream’s value for money. For the Cyber Support workstream, which benefits from good baseline data, a cyber risk and cost model will be constructed to monetise the changes that result from addressing shortfalls in councils’ cyber-attack defences and remediation plans.

The council typology and case study approach 

First, the evaluation will develop a rounded, contextual description of the participating councils from information gathered during the process evaluation. This would show which types of councils participated and may highlight important motivational factors that could uncover the direction of causality. It would also support the second aspect: developing a typology.

Second, a typology will be developed to distinguish between the participating councils on relevant dimensions. Given the limited opportunities to identify causal impacts quantitatively, the evaluation needs to remain cognisant of the possibility that councils will interact with the programme not at random, but because of their prior and heterogeneous motivational and capability attributes. The scoping study proposes an initial typology that aligns with this and identifies two key dimensions:

• buy-in to the programme’s vision, which captures the extent to which an organisation shares the ambition for digital transformation, as revealed in its actions, statements and responses to the evaluation’s primary research activities
• capability to engage with the programme, which covers resourcing, the related constraints, and an organisation’s ability to reconfigure resources to adapt to new situations

The typology will frame a series of case studies to demonstrate the impacts that were experienced in each of the participating council types. This step brings together the impacts identified by the impact evaluation activities carried out for the individual workstreams. It plans to show what types of organisations can engage effectively with the interventions and which interventions worked well. This approach will also identify the extent of common factors that may hinder the realisation of the new paradigm.

The aim is that this approach will enable the evaluation to make recommendations on how the future of the programme could accelerate the organisations already meeting the identified principles of digital transformation, and what the programme could do to support struggling organisations. 

The evaluation tools used across the programme evaluation

The case study approach will be informed by the impacts identified in the workstream impact evaluations. A series of data sources and methods (‘tools’) will be used, including:

• surveys, to gauge shorter-term before-and-after impacts, such as the Digital and Cyber Maturity Surveys (for Local Digital Fund and Future Councils) and the Mitigating Malware and Ransomware survey (for Cyber Support)
• interviews with participating councils and council staff, to gain insight into how the interventions were implemented and the extent and nature of perceived impacts
• data requests directed to councils and publicly available data, for example on the nature of council-procured software and metrics that may help with monetising impacts
• quantitative models, such as a cyber risk and impact model and some limited econometric analysis enabled by the before-and-after surveys

The impact evaluation will rely upon qualitative analysis of interviews and surveys to test whether the assumed causal mechanisms, proposed by the theories of change, explain the outcomes observed (this is called ‘contribution analysis’).

The workstream-specific evaluation methods

Other specific methods will be employed for certain workstreams where data or other factors permit the use of more robust evaluation approaches. These are outlined below.

For the Local Digital Fund and Future Councils workstream impact evaluations, a limited set of before-and-after survey rounds will support the interviews and qualitative analysis of council outputs. These ‘Digital and Cyber Maturity Surveys’ collect a range of attributes and metrics that capture councils’ digital and cyber posture. A first survey round, released in Summer 2023, baselined the Future Councils participants before the pilot, and the current Round 6 Local Digital Fund councils before they began their projects. Two repeated survey rounds are planned to occur during the evaluation period to capture changes in high-level indicators. Several non-participating councils, identified through statistical matching, were also surveyed to generate a comparison group. We expect we’ll be able to show this through difference-in-differences analysis of the high-level indicators.

Evaluation of the direct grants provided to councils by the Local Digital Fund will be supported by analysis of alternative potential sources of funding for councils’ digital projects. This will provide insight into:

• whether Local Digital Fund funding was appropriately aligned to existing funding
• the risk of Local Digital Fund duplicating existing funding
• whether alternative funding sources confound the impact analysis

This analysis of other initiatives that may influence councils’ digital transformation supports the consideration of the counterfactual.

The Cyber Support impact evaluation will be strengthened by a cyber risk and cost model informed in part by the Mitigating Malware and Ransomware survey. The survey captured a subset of factors related to the cyber risk profile of more than 90% of councils in 2020-21. As Cyber Support was designed to address this subset of factors, the survey provides a baseline from which changes can be measured. We therefore propose to develop a model for the evaluation to estimate the extent to which Cyber Support reduced the risk of malware and ransomware attacks and the associated costs. The model’s inputs would reflect the action points identified in councils’ Cyber Treatment Plans (an output of Cyber Support) enabling us to calibrate the model to capture the changes that occurred in each participating council.

Evaluation limitations

The scoped evaluation approach outlined in brief above reflects what is considered feasible to achieve within the constraints of the programme design and available evidence. Various stakeholders have provided input, including the Expert Advisory Group, the programme team and delivery partners, the Analysis and Data Directorate at DLUHC and councils themselves. But the approach is limited in important ways that are outlined below. 

Relying on non-experimental methods, the proposed approach is unable to provide a robust quantitative assessment of causality. This limits the ability to fully test the causal mechanisms outlined in the theories of change. However, the novel qualitative approach will show what causal mechanisms were perceived to have supported digital transformation for diverse types of participating organisations, and whether alternative factors could be expected to have contributed to outcomes. This insight will continue to be useful for designing interventions as the programme evolves throughout 2024.

Another limitation is that there is restricted scope for monetising the full range of costs and benefits of the interventions (both intended and unintended). This is driven by the nature of expected impacts and the methods of identifying them. The main exception to this is the detailed risk and impact model used for Cyber Support. Consequently, the economic evaluation is expected to be supported using switching values and value-for-money categories, consistent with the DLUHC appraisal guide. The case study approach will provide space for qualitative assessments of any adverse unintended impacts identified through the primary research.

Restricted sample sizes will affect both the quantitative and qualitative analysis, which in turn constrains the statistical power of the analysis. We aim to mitigate this by matching Local Digital participating councils in the Digital and Cyber Maturity Surveys with four comparison councils each, but varying data quality and some survey attrition are expected.

Timeline

• January to June 2024: The preliminary process and impact reporting stage. This involves the development of the council typology, data collection and analysis, and construction of the cyber risk and cost model
• July 2024 to March 2025. The economic evaluation and final reporting stage. This will include findings from the process, impact, and economic evaluations. A final report is anticipated around April 2025