Research and analysis

Evidence review: Digitalising welfare services

Published 7 October 2024

Abbreviations

AI

artificial intelligence

AR

augmented reality

DWP

Department for Work and Pensions

EU

European Union

ICT

information communications technology

IoT

internet of things

IT

information technology

NAV

Nye arbeids- og velferdsetaten (Norwegian Labour and Welfare Administration)

OECD

Organisation for Economic Co-operation and Development

PSI

Public Services International

QSR

quick scoping review

REA

rapid evidence assessment

VR

virtual reality

Glossary

The term digitalisation is often used interchangeably with the terms digitisation and digital transformation in different contexts.^{[footnote 1]} For this reason, it is important to provide working definitions and explain how these terms differ:

• Digitisation is defined as “the action to convert analogue or non-computerised information into digital information”.^{[footnote 2]} An example of digitisation is the creation of digital versions (sometimes referred to as ‘soft copies’) of printed out documents (sometimes referred to as ‘hard copies’).
• Digitalisation goes beyond digitisation to describe how IT or digital technologies can be used to change existing processes.^{[footnote 3]} Examples of this include the creation of new online or mobile communication channels to connect customers with firms or the use of artificial intelligence (AI) in digital service delivery. Digitalisation allows firms to use digital technologies to optimise their business processes through more efficient coordination, and to increase customer value by enhancing user experiences.^{[footnote 4]}
• Digital transformation describes an organisation-wide change that leads to the development of new business models that may expand the focus of the organisation by generating new approaches to creating and capturing value.^{[footnote 5]}

While all three terms are interconnected, this study is focused on digitalisation as relevant to the DWP (although we do include some evidence on digitisation where appropriate). This means that the study is not concerned with the simple process of converting analogue to digital data and processes (which has to large extent already been done by the DWP). Instead, this study is focused on the manner and extent to which the use of digital technologies can change the way that the DWP delivers its services to customers. The digitalisation process represents an opportunity to save on costs and – through rethinking processes – enhance customer experiences.^{[footnote 6] [footnote 7]}

Other terms, which we are using in the report, are also defined in this section:

Artificial intelligence (AI):

The capability of a machine to imitate intelligent human behaviour.^{[footnote 8]}

Blockchain:

A digital database containing information (such as records of financial transactions) that can be simultaneously used and shared within a large decentralised, publicly accessible network.^{[footnote 9]}

Cloud computing:

The practice of storing regularly used computer data on multiple servers that can be accessed through the internet.^{[footnote 10]}

Data mining:

The practice of searching through large amounts of computerised data to find useful patterns or trends.^{[footnote 11]}

Digital sensors:

Devices which automate the collection, processing and analysis of data (e.g. from citizens and devices) to translate (parts of) processes into digital information.^{[footnote 12]}

Internet of things (IoT):

The networking capability that allows information to be sent to and received from objects and devices (such as fixtures and kitchen appliances) using the internet.^{[footnote 13]}

Interactive:

involving the actions or input of a user.^{[footnote 14]}

Interoperability:

The ability of computer systems or software to exchange and make use of information. Can also be described as the ability of a system to work with or use the parts or equipment of another system.^{[footnote 15]}

Machine learning:

The process by which a computer is able to improve its own performance (as in analysing image files) by continuously incorporating new data into an existing statistical model.^{[footnote 16]}

Robotic process automation:

Software technology that makes it easy to build, deploy and manage software robots that emulate human actions interacting with digital systems and software.^{[footnote 17]}

Acknowledgements

We would like to thank the project team at the Department for Work and Pensions (DWP) for their support throughout this study. In particular, we are grateful to Eleanor Doyle and Lucy Allen of the Business Strategy Directorate. 

This report represents the views of the authors. Any remaining inaccuracies are our own.

The authors

Immaculate Motsi-Omoijiade, RAND Europe 

Pamina Smith, RAND Europe 

Axelle Devaux, RAND Europe 

Joanna Hofman, RAND Europe 

Dominic Yiangou, RAND Europe 

Isabel Flanagan, RAND Europe 

Madeline Nightingale, RAND Europe

Executive summary

The Department for Work and Pensions (DWP) is the largest public service department in the UK. It administers the State Pension and a range of working-age, disability and ill-health benefits to around 20 million claimants and customers, and many of these services are moving to being delivered online. DWP commissioned RAND Europe to review and supplement the evidence base around the impacts of digitalisation experienced by other private and public sector organisations. The evidence collected through this review can be used to inform strategic and operational decisions around the design of DWP digital services. 

The review addresses a number of research questions, grouped into four areas of interest: 1) the impact that online provision of services has on costs and savings; 2) the impacts of digitalising services on customer experience; 3) the wider societal impacts of shifting services online, even if these are more difficult to quantify; and 4) lessons learned from the digitalisation process experienced as a result of the COVID-19 pandemic. 

To respond to these questions, RAND Europe undertook a quick scoping review (QSR) complemented with additional manual searches and interviews with stakeholders in the UK and other countries who have significant experience in digitalising services. 

In relation to costs and savings associated with the digitalisation of welfare services, we found that: 

• Despite interest in the use of advanced digital technologies in service delivery, digitalisation is occurring across a narrow set of public services and is often limited to simple transactional tasks rather than the delivery of more-complex services. 
• It is difficult to gather accurate estimates of the costs involved and savings generated due to the digitalisation of services. This review suggests that organisations could use a service-by-service approach to measure the financial and economic impacts of digitalisation. 
• Digitalisation can offer staff-related cost reductions and savings. However, the extent to which these savings can be realised depends on other factors, particularly costs related to staff training and support. 
• Digitalisation can result in reduced costs and increased savings in service delivery. However, these gains may be offset by increased demand spurred by digitalisation. 
• The process of digitalisation is prone to technical difficulties, and digital channel failure (failure to achieve expected, pre-defined outcomes) is associated with unforeseen costs. 
• This review notes that having multiple means for service access and delivery is a way to make service delivery more cost efficient.

Limited interoperability (the ability to exchange information across computer systems or software) and fragmentation of information were seen as obstacles to the digitalisation of welfare services. 

In relation to customer experience, we identified different strategies that encourage customers to use digital channels and principles that facilitate the take-up of online services: 

• guaranteeing customers that non-digital options are available, launching marketing or educational campaigns, and creating engagement teams were found to be successful 
• designing a digital service with a high level of adoption and continued use requires careful attention to 
  • preferences and abilities among and within population segments, including preferences regarding privacy concerns and accountability 
  • aesthetic experience 
  • usefulness and ease of use 
  • context: if digital services increase burden for consumers or if they replace services that require urgent or very personal or emotional attention, they will fail to replace in-person services 

In relation to societal effects of digitalisation processes and notable effects of the COVID-19 pandemic, we found that: 

• Digitalisation processes may lead to more inequality. Evidence points out that particular attention should be given to ensuring that vulnerable populations, especially those at risk of digital exclusion, are protected from any negative effects related to accessing (digital) services or the internet – and its supporting technology more broadly. Evidence shows that this can be done in a number of ways, e.g. through establishing public Wi-Fi or other initiatives, both online and offline. 
• The digitalisation accelerated by the COVID-19 pandemic not only increased the type, quality and uptake of digital services, but also appears to have ensured the continued use of digital services in the future.

1. Introduction

1.1. Context

Information and communications technologies (ICT) are increasingly used to transform the public sector.^{[footnote 18]} The COVID-19 pandemic further accelerated digitalisation by several years,^{[footnote 19]} and it also boosted the digitalisation of government services. More than 80% of government services across 36 European countries are available online, and 6% of these services are delivered proactively (which means that no action is needed from customers to demand these services, as governments already assume that they are needed).^{[footnote 20]} 

In 2012, the UK Government Digital Strategy outlined plans for the government to become digital by default, meaning that digital services would be available to all those who can and choose to use them, while those who cannot are not excluded.^{[footnote 21]} Subsequent developments centred on transforming government services and making them more efficient, such as through scaling successful solutions^{[footnote 22]} and, most recently, through using data to drive efficiency and remove barriers to data interoperability.^{[footnote 23]} 

The UK Department for Work and Pensions (DWP) is the biggest public service department. It administers the State Pension and a range of working-age, disability and ill-health benefits to around 20 million claimants and customers. Many of these services are already moving to online provision, including, most notably, Universal Credit (also referred to as UC).^{[footnote 24]} The Job Entry Targeted Support programme^{[footnote 25]} was provided almost entirely online due to the COVID-19 pandemic.

1.2. Research objectives and questions

DWP commissioned RAND Europe to review and supplement the evidence base around the impacts of moving services online, through analysing the experiences of other public and private sector organisations. The evidence collected through this review can therefore inform strategic and operational decisions around the design of DWP digital services. 

The review addresses a number of research questions, grouped into four areas of interest: 1) the impact of online provision of services on costs and savings; 2) the impacts of digitalising services on customer experience; 3) the wider societal impacts of shifting services online, even if these are more difficult to quantify; and 4) lessons learned from the COVID-19 pandemic. 

To respond to these questions, RAND Europe undertook a literature review known as a quick scoping review, complemented with additional searches and interviews with a small number of stakeholders in the UK and abroad who have significant experience in the digitalisation of services. 

The research was guided by the following 9 research questions: 

1. How can we best measure the financial, economic and social impacts of digitalising services? 
2. What is the cost trade-off between digital investment and savings on staff headcount? 
3. Are there successful examples of sharing or movement of data between different IT platforms, and what are the costs and benefits of this approach? 
4. Does including a digital channel enable more people to claim a benefit or service? 
5. Does a successful channel mix vary at different points of the customer journey, and how does this vary for groups with different protected characteristics? 
6. Learning from accelerated digitalisation due to COVID-19, are there any consistent emerging and sustainable ‘wins’ which could be considered for implementation in DWP services? 
7. How have organisations managed to shift customers onto online channels, and are customers who moved online during COVID-19 continuing to use online channels? 
8. What are the human and cost implications from customers having to use a digital channel where using another channel would mean a substantially improved outcome? 
9. What are the costs and customer experience levels of a successful channel mix, and how did organisations develop and establish a mix that worked for their customers?

Table 1 summarises how the structure of the report corresponds to the research questions.

Table 1. Mapping of research questions to report sections

Research questions	Corresponding report sections
1. How can we best measure the financial, economic and social impacts of digitalising services?	2.1. Digitalisation activities that organisations invest in to achieve savings. 2.2. Measuring the financial, economic and social costs and savings of digitalising services.
2. What is the cost trade-off between digital investment and savings on staff headcount?	2.3. Potential costs and savings related to staff and service delivery.
3. Does including a digital channel enable more people to claim a benefit or service?	2.1. Digitalisation activities that organisations invest in to achieve savings. 2.3. Potential costs and savings related to staff and service delivery.
4. What are the costs and customer experience levels of a successful channel mix, and how did organisations develop and establish a mix that worked for their customers?	2.4. Costs and savings implications of failed digital channels, digital channel mixing and interoperability. 3.1. Strategies used to shift customers onto digital channels.
5. Does a successful channel mix vary at different points of the customer journey, and how does this vary for groups with different protected characteristics?	3.2. Factors affecting the customer experience of digital services.
6. What are the human and cost implications from customers having to use a digital channel where using another channel would mean a substantially improved outcome?	2.4. Costs and savings implications of failed digital channels, digital channel mixing and interoperability. 4. Social impact of digitalisation and learnings from COVID-19.
7. Are there successful examples of sharing or movement of data between different IT platforms, and what are the costs and benefits of this approach?	2.4. Costs and savings implications of failed digital channels, digital channel mixing and interoperability. 4 Social impact of digitalisation and learnings from COVID-19.
8. Learning from accelerated digitalisation due to COVID-19, are there any consistent emerging and sustainable ‘wins’ which could be considered for implementation in DWP services?	4. Social impact of digitalisation and learnings from COVID-19.
9. How have organisations managed to shift customers onto online channels, and are customers who moved online during COVID-19 continuing to use online channels?	3.1. Strategies used to shift customers onto digital channels. 4. Social impact of digitalisation and learnings from COVID-19.

1.3 Research methodology 

To address the research questions above, the study used a QSR of the literature complemented by stakeholder interviews. 

QSR was chosen over more systematic approaches to evaluating evidence (i.e. systematic review, rapid evidence assessment) because of the broad scope of the research and the fact that systematically appraising the evidence was not a key consideration.^{[footnote 26]} The choice of QSR was also informed by pragmatic considerations about the timeline of the research, which took place between January and March 2022. 

The literature search was limited to sources published between 2018 and 2022, and the geographical scope was limited to Organisation for Economic Co-operation and Development (OECD) countries and non-OECD countries with distinctive examples of digitalisation (e.g. Singapore). The chosen time frame reflected our initial practical limitation of examining only 50 sources; a longer timespan would have increased the number of articles to analyse beyond our capacity for this study. Keyword searches in Google Scholar and Web of Science yielded 300 sources. We screened the titles and/or abstracts for relevance and ultimately selected 63 of these 300 sources for more detailed review. We excluded 19 of these 63 sources due to insufficient relevance or lack of access, and 44 sources proceeded to the analysis stage.

While it takes a structured approach, a QSR does not follow the same level of rigour as a systematic review or rapid evidence assessment. It is possible that certain relevant sources were missed, particularly those published before 2018, given the time frame selected for the study. The focus on English-language sources may, likewise, have resulted in certain findings being excluded. 

We looked for evidence about the impact of digitalising welfare services in contexts similar to the ones in which DWP operates. However, the evidence presented in the literature covered digitalisation of services in wider areas, e.g. digitalisation of healthcare services. This report presents examples and evidence that were most relevant to the DWP context and which could inform digitalisation of welfare services beyond the healthcare context. 

To supplement data obtained through the scoping review, we had planned to conduct up to 30 stakeholder interviews with public authorities undertaking similar services in the UK or in other countries (focusing on the local authority level in the UK) and with private organisations (in the UK and beyond) familiar with the digitalisation of services in their respective organisations. The objective of these interviews was to better understand the context in which digitalisation happens in organisations (aims and motivations, barriers, drivers) and to illustrate findings from the QSR.

We reached out to more than 50 organisations (via email and phone) and were able to secure 6 interviews (including one featuring a group of four people) with local authorities in the UK and public authorities in other countries in Europe. Our requests for participation in the research did not generate interest among private sector organisations, which might be explained by the lack of incentive for these organisations to share information about their experience in this area. The implications of these limitations for the quality of the evidence base for this research are likely limited, given that our research relied primarily on the QSR and that our expectations for the interviews were to be able to further explore what was discussed in the literature and to illustrate findings from the literature. 

The study draws on a small number of in-depth interviews; different or additional findings might have emerged if the pool of interviewees had been larger. This limitation should be kept in mind when using the findings from this research and for understanding their implications for digitalisation of services, for the DWP and for any other organisation contemplating digitalisation of their services. 

Given that evidence gaps remained with this approach, we agreed with DWP that we would fill in remaining significant evidence gaps with additional literature searches in Google Scholar and snowballed from existing sources (i.e., identified other relevant sources from the citation list). These searches were not subject to the same time and geographical limitations as those conducted initially,^{[footnote 27]} but sources which satisfied them were ultimately preferred. 

Further details about the methodology are provided in Appendix A.

1.4 Structure of the report 

Drawing on evidence collected from the QSR and stakeholder interviews, Chapter 2 summarises evidence related to costs and savings and Chapter 3 considers impacts on customer experience. Chapter 4 focuses on the societal implications of digitalisation and any lessons learned from the COVID-19 experience. Finally, Chapter 5 summarises the main findings of this review and draws key implications from the research. Appendices A and B present the protocols for the QSR searches and for the interviews, respectively.

2. Costs and Savings

Having a clear view of the potential costs and savings of digitalising services is vital to the successful implementation of any digitalisation project. This chapter outlines which digital activities organisations are investing in to achieve savings (section 2.1). This will be followed by considering how the financial and economic costs and savings of digitalising service are measured (section 2.2). Costs and savings related to staff and to service delivery will each be discussed in section 2.3. This will be followed by an overview of the costs and savings associated with digital channels, particularly the costs and savings implications of failed digital channels, digital channel mixing and interoperability (section 2.4). Key considerations for the DWP and other organisations to take into account when thinking about the costs and savings associated with the digitalisation of welfare services are highlighted throughout and are summarised in section 2.5.

2.1 Digitalisation activities that organisations invest in to achieve savings

Digitalisation of the public sector and of welfare services has been seen as a way to increase the efficiency and cost-effectiveness of service provision in light of new pressure on services and increased financial constraints. As explained by Larson et al., “the contention is that by using technology in welfare services, it can help secure the continued economic stability of the welfare state”.^{[footnote 28]} In addition to the pressure on services from societal challenges, such as an ageing population, the public sector has to contend with increased demand associated with the rise in the cost of living, as well as, most recently, the COVID-19 pandemic.^{[footnote 29]} Using technology in public service delivery improves how service users communicate with providers because these technologies are information-processing tools.^{[footnote 30]} In general, digitalisation is expected to result in the improvement of public sector service delivery, including increased internal efficiency, better information sharing, better-informed decision making and innovation.^{[footnote 31]} These service and process improvements may result in cost savings, which are achieved by allocating resources more efficiently.^{[footnote 32]} In summary, as explained by Løberg, both researchers and governments often consider digital service provision to be more efficient than traditional service provision,^{[footnote 33]} with these efficiency gains resulting in increased savings and reduced costs. 

Regarding the specific type of activities that organisations are investing in to achieve savings, Ranerup and Henriksen’s study describes how e-government^{[footnote 34]} activities can be divided into a ‘first wave’ – mainly focused on streamlined e-service and the horizontal integration of data (combining similar data types) and vertical integration of data (merging different data types) and a ‘second wave’ – where the focus is shifted to automating processes, such as decision making in which a computer program or ‘robot’ acts as the case manager for decisions.^{[footnote 35]} This can also be understood as corresponding with a shift from the simple digitisation of activities to an emphasis on digitalisation. In practice, organisations working towards introducing ‘second wave’ service provision have focused on activities such as the handling of applications for social assistance and the offering of economic support using partly automated application processes.^{[footnote 36]} 

Our findings show a stronger emphasis on ‘second wave’ forms of digitalisation where organisations are investing in their services to achieve savings. For example, in highlighting the categories of digital technologies being used by public sector organisations in the EU, Eurofound’s study focused on the automation of work, the digitalisation of processes and the coordination of service provision using digital platforms.^{[footnote 37]} Here, automation of work is described as focusing on: 

• the replacement of human labour input by machine input for some types of tasks using algorithmic control of machinery and digital sensors. These types of tasks include those related to routine, repetitive administrative tasks (such as sending reminders and facilitating payment) and to customer support (for example, through the use of ‘chatbots’) 
• the digitalisation of processes focused on the use of sensors to translate (parts of) processes into digital information, including through the use of technologies such as the internet of things (IoT), virtual reality (VR) and augmented reality (AR) 
• the use of platforms for the bringing together service users and providers; in some instances, platforms may use technologies such as blockchain^{[footnote 38]} and cloud computing^{[footnote 39]} 

Out of each of these ‘second wave’ technologies, the automation of services is cited as the most common trend in public sector digitalisation.^{[footnote 40]} As explained by Ranerup and Henriksen, “in the public sector, civil servants and clients find themselves in an environment where automation and robot technology can be expected to make dramatic changes”.^{[footnote 41]} An example of this is the changes to social service delivery in Trelleborg, Sweden (Box 1), which provides a useful illustration of how financial aid can be delivered using automated decision making.^{[footnote 42]} 

Trelleborg Municipality introduced RPA in 2016 which, by 2017 handled 70% of applications for social assistance benefits, made 41% of decisions and processed payments.^{[footnote 43]} This use of RPA resulted in qualitatively self-reported increased accountability, decreased costs, and enhanced efficiency within the Trelleborg Municipality.^{[footnote 44]}

Box 1 Digitalisation of municipal social service delivery in Trelleborg (Sweden) ^{[footnote 45]}, ^{[footnote 46]}

Trelleborg is a city of 43,000 inhabitants located in southern Sweden. Like other municipalities in Sweden, Trelleborg offers a wide range of welfare services around childcare and education, and it also processes applications for financial aid. Trelleborg introduced fully automated decision making in relation to financial aid applications in 2016. 

The type of technology used in Trelleborg is robotic process automation (RPA), which means that the process (and, as part of it, decision making) is handled by a robot. 

One year after introducing it, Trelleborg’s RPA handled most (70%) applications for social assistance benefits, made 41% of decisions and processed all related payments using this automated decision making. 

The municipality reported that the use of RPA resulted in increased accountability, decreased costs and enhanced efficiency of the services. 

The use of RPA in Trelleborg also resulted in negative experiences for staff in relation to trusting a machine to make decisions in areas where staff value their professional judgement about priorities and circumstances. Staff also reported issues related to ensuring transparency in decision making and data protection. 

In terms of costs, the municipality reported a reduction of the cost of social assistance. However, in this instance, the Swedish Labour Market Agency spent a further 600,000 Swedish crowns (£49,128) per year after the launch of the RPA system, which suggests that ongoing maintenance, repair and improvement costs should be taken into account while calculating cost reductions.

2.1 Digitalisation activities that organisations invest in to achieve savings (continued)

This echoes findings by Deloitte which, based on responses from more than 400 individuals across various industries globally, reported how shifting mundane, labour-intensive, repetitive tasks from humans to robots through RPA resulted in 92% improved compliance, 86% improved productivity, 90% improved quality and 59% cost reduction.^{[footnote 47]} However, the implementation of RPA technologies does incur costs, with 25-30% of the total costs assigned to licensing costs (which in 2021 ranged between £11,500 and £38,000 for a single ‘bot’, or unit) and the remaining 75% of the costs consisting of yearly RPA license renewal fees; training or hiring of staff; consulting costs for implementation; infrastructure set-up; third-party integrations; cost of complementary software (e.g. process mining and process discovery software); and costly RPA repair, improvement and modification cycles.^{[footnote 48]} These cost categories are likely to differ at various stages and time horizons in the RPA implementation process. 

In addition to these cost considerations, it should be noted that RPA roll-out is not a cure-all solution. Despite some reported initial gains, a further study by Ernst and Young (EY) indicated that 30-50% of all RPA initiatives have failed, in part due to coding errors^{[footnote 49]} or cybersecurity breaches.^{[footnote 50]} An example of an RPA technical failure recorded by EY is an instance where “a telecom company deployed bots for managing its complaints-handling process. However, coding errors led to many grievances being diverted to an incorrect queue, resulting in a backlog of complaints”.^{[footnote 51]} To increase the likelihood of success, RPA implementation strategies should consider 1) the need for the right upfront design (which can significantly reduce RPA maintenance and support costs); 2) cross-functional collaboration between service provision and IT functions and; 3) the need to improve and optimise processes before automating.^{[footnote 52]}

In addition to these possible technical challenges, the use of RPA in a welfare service setting has further social considerations that need to be kept in mind. In Trelleborg Municipality, while the use of this technology led to reported efficiency gains, it also resulted in negative experiences for staff. These negative experiences were reported in the areas of 

• exercising professional knowledge where decisions requiring professional judgement became automated 
• safeguarding service user trust through data protection (not publicly disclosing certain categories of information) and the perceived lack of transparency 
•  cost reduction and the desirability of sharing the design costs of the model with other local governments^{[footnote 53]} 

These negative experiences have been attributed to an observed tension in value relationships (conflict between different values, such as professionalism, efficiency, service and engagement), namely between automated decision making used to make routine decisions so staff can prioritise more-complex tasks versus lack of professional discretion in the final decisions.^{[footnote 54]} Tensions were also observed in the use of RPA having resulted in a shift from helping citizens by facilitating payments of welfare benefits, to encouraging them to find work.^{[footnote 55]} These tensions highlight how automation also has negative effects, resulting in the need to assess efficiency gains against any indirect negative consequences for staff and service users. 

Another example of the automation of services is provided by Lindgren et al. who describe how “new opportunities for digitalisation of public service provision associated with data mining, machine learning, sensor technology, and service automation have been discussed with great optimism.”^{[footnote 56]} This is because these emerging digital technologies may “fulfil the primary goals for digital government which include improving efficiency and service quality by reducing service lead times and offering seamless service provision across organizations.”^{[footnote 57]} In Slovenia, for example, introducing a new information management system (known as IS CSD2) significantly reduced abuses of the welfare system through activities such as supporting data aggregation, decision making, standardised display of documents and the automatic calculation of social transfers, which enabled the correct payment of transfers.^{[footnote 58]}

Despite progress in introducing new digital technologies in service delivery, in practice, these new technologies are not widely used and therefore the evidence on their implications for service delivery is still evolving. For example, despite the promises of the use of ‘welfare technologies’ to promote digitalisation in Scandinavian countries, only a few of these welfare technologies are being offered by local municipalities and care homes.^{[footnote 59]} Similarly, Cepparulo and Zanfei noted that research into e-government focuses on a narrow set of public services: 

1. services generating income for the government (such as taxation and customs)
2. registration services (such as ownership, birth and marriage)
3. permits and licences (including building permits, passports and diplomas)^{[footnote 60]}

An example of this narrow use of digitisation was provided by one interviewee, who explained how the sending of digital emails (about 217 million per year) instead of paper letters resulted in estimated savings of €200 to €400 million for their public service since the introduction of email correspondence.^{[footnote 61]} This observed focus on a narrow set of public services is supported by Špaček and others’ study on the digitalisation of the core administrative services in a selection of Central European countries. Their findings show that, overall, digitalisation is relatively uncommon and, in most cases, is focused on 10 types of service areas: obtaining new IDs and travel documents; registering a new address; obtaining or changing a driving license; registering a car; solving a waste-disposal issue; paying local taxes and fees; paying for local transport; making submissions to local administration (complaints, petitions etc.); participating in local decision making; and applicating for childcare.^{[footnote 62]} This finding on the types of services covered by public sector digitalisation was supported by one interviewee, who mentioned transactions done online, including payments, as a key digitalisation initiative within a local authority context (see Box 2),^{[footnote 63]} echoing the findings that digitalisation is currently occurring primarily in transactional public services.

Box 2 Digitalisation at county council level (UK)

This county council covers over 1 million residents, and with the natural progression of digitalisation, the county council seeks to continually implement digital change to an array of different services. 

The main aspects of digitalisation in this county council are related to work, public reporting, development applications, active travel and road maintenance. Licensing for services have become an integral digital innovation for the county council, with manual handling of paperwork and license processes having been removed and digitalised. 

For many years, thousands of services were applied for physically, by fax. In 2014, this became a digital process. As of May 2021, 100% of these applications became digital, with users providing positive feedback on the efficiency, ease and time savings as a result of this new process. In addition, this saved a great deal of the county council effort, removing the need for administrative teams to process the applications and enabling them to redirect their effort elsewhere. 

In order to balance user-friendliness, offline versions of the county council services remain open to residents who are unable to complete applications online. The offline services also evolve alongside online versions. 

While there are no fully measurable cost savings the county council can report on, there have been assumed savings through better use of time and increased process efficiency. Alongside this, high customer satisfaction rates have led the county council to conclude that their digital transition is an ongoing success.

2.1 Digitalisation activities that organisations invest in to achieve savings (continued)

Overall, organisations’ customer-related digitalisation activities can be described as falling into three categories:  

• transactions (e.g. registering for elections, reporting a problem, paying a bill) 
• interactions (e.g. obtaining advice, public consultations, petitioning) 
• information provision (e.g. swim times, leaflets, web pages)^{[footnote 64]} 

All three of these categories are fulfilled using a combination of traditional and digital communication channels.^{[footnote 65]}

This focus on transactions indicates that less attention has been paid to more-complex interactions and information provision in public sector digitalisation initiatives. While it focuses only on Hungary, Romania and the Czech Republic, the study by Špaček and others is relevant to the DWP’s context. First, it provides a useful overview of the types of activities public sector organisations are investing in to achieve savings, by showing the 10 main digitalisation categories of core administrative services, as highlighted above. These focus on similar ‘macro-categories’ to those mentioned by Cepparulo and Zanfei^{[footnote 66]} and by one interviewee.^{[footnote 67]} This suggests that the digitalisation of more-complex services, which go beyond simple transactions, such as those in welfare provision, is currently still limited. Second, the study by Špaček et al. found that the digitalisation of core administrative services for citizens may be determined by the national approach to e-government policy, the level and readiness of legislation for digitalisation and the way the service delivery is organised (centralised, decentralised or mixed).^{[footnote 68]} This need for a national approach and the need for state-supplied infrastructure is further discussed below. 

Considerations for the DWP or other stakeholders in similar positions:  These findings indicate that there is a disconnect between interest in the use of more advanced digital technologies (such as AI and cloud computing), on the one hand, and the reality of the types of services that are being digitised and digitalised (which are still largely transactional), on the other hand. Organisations could consider taking a measured and incremental approach. They could first focus on ensuring adequate digitisation, and then work towards transactional digitalisation and, finally, more interactional digitalisation. Taking this incremental approach means organisations can consider the use of technologies such as RPA at appropriate times – technologies that, while providing considerable efficiency gains, have important cost and service delivery implications.

2.2 Measuring the financial, economic and social costs and savings of digitalising services

2.2.1 Measuring financial and economic costs 

The ability to measure the financial and economic impact of digitalisation is key to understanding the costs and savings associated with digitalisation. Various methods of measuring the costs and savings of digitalising services were found in the reviewed literature. The first of these is system-level analysis, which uses a whole-systems approach^{[footnote 69]} to predict the value of digitalisation. A study that illustrates how this form of analysis can be used to assess the financial impact of digitalising a service was conducted by Turner et al..^{[footnote 70]} The authors conducted a system-level analysis to predict cost-effectiveness (comparing the relative costs and outcomes of different courses of action) of a shift to online services in the testing for sexually transmitted infections (STI) in London.^{[footnote 71]} Despite this study being specific to a healthcare context, there are lessons relevant to quantitatively assessing the costs and savings associated with a shift to digital service provision. 

This study showed that measuring the financial and economic impact of digitalisation requires routinely collected, anonymised and retrospective data, such as the number of digital services users, demographics of these users, and types of services being accessed.^{[footnote 72]} In this study, this was done for both digital (online) and non-digital (in-person) services at two data points – before the shift to the digital service (which was the study’s baseline) and after the introduction of the digital service. This approach was taken to comparatively assess, track and monitor the financial effects of introducing digital services. A key data point for Turner et al.’ study was the cost per service. This was calculated using data on the primary tariff (meaning the cost of delivering that care on its own), as well as an additional tariff (meaning the cost of delivering that care alongside another, more expensive activity).^{[footnote 73]} The second consideration relates to the application of a whole-systems approach.^{[footnote 74]} In this study, the authors used the database (of routinely collected, anonymised and retrospective data, as explained above) to evaluate the pattern of service use across two inner London boroughs (Lambeth and Southwark). It considered the differences and changes in costs per service between digital and in-person services, as well as other factors that impacted access to the services. In addition, to compare the service in other urban versus rural environments, the study collated summary data from three other areas that use the same online service as examples of different urban and rural areas. They coded the types of services being accessed online and in person and made use of scenario analyses to model how changes in different variables affected the cost per service in both the digital and the in-person environment. The study projected different costs between urban and rural areas due to different levels of STI, different barriers to service use and variations in clinic budgets (including the applicability (or not) of London tariffs) across the country. The estimated cost per diagnosis online for southeast London for quarter 1 of 2017 was £732, compared with a figure of £545 for ‘rural with hub town’ areas.^{[footnote 75]} This suggests that a system-level analysis can be used to consider how costs and savings might vary in different contexts. 

The costs and savings implications of digitalisation can also be considered using cost-effectiveness analysis. For example, a study by Aspvall et al.^{[footnote 76]} assessed the cost-effectiveness of internet-delivered cognitive behavioural therapy compared with in-person cognitive behavioural therapy for children and adolescents in Sweden. The methodology of the study included collecting cost data at different stages of treatment (including before treatment) and then comparing differences in costs and health outcomes between the internet-delivered and in-person groups from the perspectives of healthcare professionals, the health care sector and wider society.^{[footnote 77]} In the case of the cost-effectiveness of internet-delivered versus in-person cognitive behavioural therapy, while the shift to digital services provision resulted in a lowering of cost, it also resulted in a decrease in the effectiveness of the intervention directed at the target care group.^{[footnote 78]} This is linked to concerns expressed in the literature about cost reductions linked to digitalisation possibly resulting in a reduction of the quality of service provision.^{[footnote 79]} 

Measuring costs and savings can also be done using a software development and design science approach. This method is seen as useful in instances where the rapid design and implementation of digital platforms and services is needed. To illustrate, this approach was taken by Lapão, et al., who took a design science approach to assess the implementation of digital monitoring services during the COVID-19 pandemic for patients with chronic diseases in Portugal.^{[footnote 80]} The design science approach uses Scrum, an agile project management method that collects feedback from end users at the end of each iteration of the software design, and prioritises new features to reduce risk and extract maximum value.^{[footnote 81]} This methodology provides a viable means of assessing the financial and economic costs and savings of digitalisation by allowing for the testing of software for value based on pre-defined goals. The key point to consider from this methodology is the need to consider the link between 1) the proposed technology or software and 2) the pre-defined goals and desired outcomes when assessing the costs and savings associated with digitalisation.  

The last methodology used to measure costs and savings of digitalisation identified in the literature is case study methodology. The ‘Digital Efficiency Report’ published by the UK government’s Central Digital and Data Office in 2012 assessed the savings made from the digitisation of transactional services offered by the central government using case studies.^{[footnote 82]} This report defined transactional services as services that involve an exchange of money, goods, services, permissions, licences or information between the government and a service user (from paying car tax to applying for a passport) resulting in a change to a government system.^{[footnote 83]} This analysis used case study methodology in which a sample of 17 services was selected. Each of these services was categorised using 4 significant factors of savings potential – volume of individual transactions per year, service function (e.g. requesting a benefit or grant), customer type and current level of digital take-up – with data about the impact of each being collected to arrive at final costs and savings estimates. Estimates on the number of digital transactions by channel, as well as unit cost ratios by channel and by department, were based on data acquired from published departmental and agency accounts.^{[footnote 84]} This method showed the resulting projected total annual savings (fiscal and cost recovery) of digitisation within the UK public service. Significantly, this analysis shows that it is effective to isolate and then aggregate costs by service function when considering the total costs and savings of shifting to digital services. Also significant is the focus of the ‘Digital Efficiency Report’ on digitisation rather than on digitalisation, highlighting how cost estimates of the shift from analogue to digital are more easily arrived at than estimating the cost and savings implications of digital technology in the mode of service delivery. This finding was reiterated by one interviewee, who highlighted the difficulty calculating costs and savings because digitisation and digitalisation activities have happened over many years.^{[footnote 85]}

These challenges in measuring the costs and savings of digitalisation are well documented.^{[footnote 86]} For example, while an early (2010) study measuring the financial impact of ICT investment in Slovenia’s tax system showed that ICT expenditure is higher than cost savings for tax administration and taxpayers (despite showing several non-financial benefits), these values were based solely on rough estimates.^{[footnote 87]} These challenges and constraints in the measurement of the costs of digitalisation are made worse because it is difficult to decide which metrics should be monitored (often with control budgets^{[footnote 88]}). Kotarba argues that more work needs to be done to help find the most appropriate data points to assess digital performance over time.^{[footnote 89]} This challenge in measuring the costs and savings of digitalisation has resulted in evidence gaps in the research around this subject. For example, as noted by Cepparulo and Zanfei, initial studies examining the costs of digitalisation have so far focused on a relatively narrow set of electronic service provision, while they have devoted much less attention to quantitative analysis of services that respond more directly to users’ needs, including e-health and e-procurement.^{[footnote 90]} This implies a limited understanding of the existing and potential impact of digital technology on providing frequently used services affecting the everyday life of individuals, households and companies.^{[footnote 91]} The overall effect of this is limited actionable evidence on the financial effects of digitalisation on the public sector. As explained by Wright, while digitalisation in the public sector has involved large levels of public investment, it has yielded few tangible results so far.^{[footnote 92]} The lack of evidence of the benefits of digitalisation is a barrier to the wider adoption of new digital technologies.^{[footnote 93]} Finally, these measurement challenges are more pronounced (but also most necessary to overcome) in the welfare context.^{[footnote 94]} Here, it has been noted that measuring and quantifying costs and savings and the focus on “efficiency, predictability, calculability and control over uncertainty poses a risk of watering down the core values of welfare technology”, with implications on various groups of customers.^{[footnote 95]}

Considerations for the DWP or other stakeholders in similar positions: These findings highlight how accurately assessing costs (including costs per service) might prove challenging in the context of DWP’s work given the complex nature of the services provided. Some costs and savings may be difficult to quantify, especially given the interconnected nature of some of the services. DWP will likely have to use proxies for some of the necessary data points and could consider focusing on a service-by-service approach to measurement. This may help to get an accurate financial and economic assessment of digitalising a range of services. Finally, the link between service costs and service quality must be kept in mind, particularly when serving vulnerable customers.

2.2.2 Measuring social costs

In addition to capturing financial and economic costs and savings of digitalisation, attention must also be paid to measuring the social costs of digitalisation. As highlighted by the United Nations Department of Economic and Social Affairs (UN/DESA), enabling affordable access to the internet for everyone and investing in digital skills is needed to ensure that no one is left behind, particularly in the era of COVID-19-accelerated digitalisation of essential public services.^{[footnote 96]} Factors such as location, income, age, sex, ethnicity and disability status are significant predictors of access to ICTs and the internet, and therefore need to be taken into account when assessing the social costs of digitalisation. The urban versus rural gap, where the percentage of households with access to the internet at home in urban areas (72%) is almost twice that in rural areas (38%) is particularly prevalent, in developed and less developed countries alike.^{[footnote 97]} 

Similar discrepancies are seen in relation to older people across all regions – for example, in the United States, 27% of individuals aged 65 years and over and in the UK, 46% of individuals aged 75 years and over do not use the internet^{[footnote 98]} – and in relation to people with disabilities, who face inequalities and additional barriers in accessing the internet.^{[footnote 99]} Lack of engagement with digital services and other factors, including affordability and accessibility of ICT devices, programmes and websites, are discussed in more detail in Chapter 3. It is important to factor these social costs into the overall cost and savings assessment of the digitalisation process. As explained by one interviewee, in line with the UK’s levelling up agenda, equality and access to digital infrastructure are fundamental for people to participate in modern life.^{[footnote 100]} However, according to the interviewee, issues of affordability and skills are not being adequately addressed at the national level (with social housing particularly lagging behind in digital infrastructure), and there is no adequately coherent national plan or responsibility for this.^{[footnote 101]} The UN/DESA report highlights the role of national and local governments in ensuring a framework for reducing the digital divide and increasing digital inclusion through considering access, affordability, skills and awareness.^{[footnote 102]} This means that these categories of social costs need to be considered at the broader, national level rather than at the level of individual organisations, because these social costs have to do with underlying national digital infrastructure and broader social policy. 

Despite these challenges, some local authorities in the UK are making considerable strides in promoting digital inclusion, for example Connecting Cambridgeshire, led by the Cambridgeshire and Peterborough Combined Authority (see Box 3). 

Connecting Cambridgeshire, in partnership with organisations including the European Agricultural Fund for Rural Development and the Department for Digital Culture Media and Sport (also referred to as DCMS), seeks to improve “digital connectivity to drive economic growth, help businesses and communities thrive, and make it easier to access public services”.^{[footnote 103]} Connecting Cambridgeshire’s superfast broadband roll-out has brought high-speed internet access to more than 98% of homes and businesses,^{[footnote 104]} showing how local authorities can play a role in taking onboard some of the social costs of digitalisation, particularly through leveraging strategic partnerships.

Box 3 Digitalisation in Cambridgeshire (UK)

Connecting Cambridgeshire is a programme hosted by Cambridgeshire County Council, which works with local authorities, government bodies and external organisations to improve Cambridgeshire and Peterborough’s digital infrastructure for businesses, communities and public services. 

A range of services on the Connecting Cambridgeshire network have become digitalised. Some of these are resident parking permits, library services, fines, and elderly and vulnerable citizen support. These digital services sit alongside non-digital alternatives to increase user-friendliness and to account for services which require in-person care. 

The decision to digitise was reached during the COVID-19 pandemic, when the UK government began to incentivise digitisation – the biggest drive being cost savings. With added pressure for councils to reduce costs, digitisation became a priority to Cambridgeshire County Council. 

Rolling out these systems required standard practices of soft-testing, disaster recovery testing and standard testing. This testing had to be extensive, as the nature of the service is vital (Connecting Cambridgeshire had to rely on creating very good digital systems). 

The implementation of digital services by Connecting Cambridgeshire has led to a reduction in staffing, which aids in cost savings; however, this has caused added pressure within local authorities, as different skill sets are now required to oversee digital services. Finding staff with the necessary skillset is challenging, as already low local authority budgets can act as a barrier to employ digitally skilled workers and re-train existing staff. 

Associated with digitalisation is the danger of technological exclusion. The county council has tried to rectify this issue by helping to support citizens’ digital skills, but the issue remains prominent. 

The Connecting Cambridgeshire programme recognises that, in the future, it may be more effective to re-engineer processes instead of simply replacing offline systems with a digital format – which may be clunky and less user-friendly – however, further opportunities to do so would require a larger upfront investment in system implementation and transformation.

2.2.2 Measuring social costs (continued)

Considerations for the DWP or other stakeholders in similar positions: Organisations such as the DWP should keep in mind the extent and nature of the social costs associated with the digitalisation of public services and factor these into their overall costs and savings analyses. In addition, organisations such as the DWP could seek to foster horizontal and vertical partnerships with local authorities and other national authorities to advocate for, shape and arrive at a unified, national approach to digital inclusion in order to better account for and meet the social costs (including those to do with affordability and access) related to the digitalisation process.

2.3 Potential costs and savings related to staff and service delivery

The need to assess the costs and savings associated with the digitalisation of welfare services is well documented.^{[footnote 105]} While these costs and savings can be assessed across different dimensions, this study particularly focused on assessing the costs and savings related to staff and to service delivery. Considering the cost implications of these two dimensions is essential in a public sector department such as the DWP, which provides people-led (staff) and people-oriented (customers) services.

2.3.1 Staff-related costs and savings

Our findings show that identifying the staff-related costs and savings associated with digitalisation is complex and multi-faceted. On the one hand, digitalisation is seen as a solution to labour shortages,^{[footnote 106]} and on the other hand, digitalisation is cited as one of the causes of staff job losses.^{[footnote 107]} The reviewed literature reiterates the need for staff training and staff buy-in to achieve any cost savings or benefits associated with the digitalisation of welfare services. Overall, there is evidence that digitalisation can lead to a reduced staff headcount (and related costs). However, this should be weighed against other staff-related considerations (such as training costs) and against the impact on the wider economy, particularly where unemployment due to digitalisation might increase costs for the taxpayer (and, indeed, the DWP). 

Most evidence on the staff-related costs and savings of digitalisation is focused on the need for staff training and support to realise the savings of digitalisation. As explained by Tomičić Furjan et al., any potential staff-related savings of digitalisation might fail to be realised without adequate training and buy-in from staff, because employees challenge the use of any new technology if they lack the “time, competencies, [or] motivation” to adapt or if they are concerned about “being replaced”.^{[footnote 108]} These fears are heightened by concerns that new technologies are being used for worker surveillance and performance monitoring, as well as increasing working time and extending job tasks.^{[footnote 109]} To address possible negative attitudes towards technologies, it is suggested that employers should: 

• explain the technologies used 
• provide training and should allocate time for learning 
• develop the ability to use the technologies and facilities 
• foster support for the use of new technologies^{[footnote 110]} 

Specifically, staff buy-in can be supported by involving staff in the design and potential implementation of new digital platforms, and to provide feedback on the functionality requirements.^{[footnote 111] [footnote 112]} There are some financial implications to ensuring these aspects of digitalisation are put in place. As explained by one interviewee, the costs of setting up devices, skills and training in the use of technology amounted to £5 million for a local authority since the start of the pandemic, in March 2020, with additional one-off costs incurred through the creation of an engagement team hired to up-skill the communities in digital skills and improve confidence to support the use of public services online.^{[footnote 113]} 

In general, staff training costs vary depending on the industry, training needs, job role, mode of training (online or in person) and format (professional qualifications, apprenticeships etc.). A 2018 survey of 180 human resources professionals from various industries across the UK showed that, on average, employers invested around £42 billion in training per year, with an average spend of £1,530 per employee.^{[footnote 114]} However, given that the average employer spends about £3,000 and 27.5 days to hire a new worker^{[footnote 115]} – almost double the average training costs – training rather than new hiring could be the more cost-effective option. Similarly, there is evidence that digitalising services can contribute to reductions in staff-related costs. In the ‘Digital Efficiency Report’, staff costs related to reductions in employee numbers were projected to account for 78% of the total savings of digitisation.^{[footnote 116]} Further work citing staff savings due to digitalisation includes Raina et al., who, in a study on telemedicine,^{[footnote 117]} found that higher use of the digital service (in this case patient attendance) resulted in more efficient use of staff time and resources. This was because prior to the introduction of the digital service, 53% of scheduled visits were either cancelled or were ‘no-shows’. After the introduction of the digital service, there was a reduction of the ‘no-show’ rate by nearly half (to 29%), ultimately leading to a reduction in costs and increased efficiency in the use of staff time when offering the service.^{[footnote 118]}

In addition to saving on staff costs, digitalisation is seen as one of the ways to address labour shortages in the social services sector. An example of this is the use of smart assistants to increase the capacity of the personal and social care workforce.^{[footnote 119]} However, introducing digital services, such as smart assistants, led to redundancies among public sector staff.^{[footnote 120]} A report commissioned by Public Services International (PSI) found that “cost-cutting driven digitalisation tends to replace and slash public service jobs.”^{[footnote 121]} While the evidence in the literature highlights these findings related to staff costs, the causal link between digitalisation and staff redundancies is not always obvious. As explained by one interviewee, “it is hard to attribute reductions in staff headcounts to digitalisation – since it began in 2010 – council has reduced its headcount by 3,000 posts … there is an impact on headcount, but it is part of a much broader reduction in headcount over last 10 years for other reasons too.”^{[footnote 122]} 

Considerations for the DWP or other stakeholders in similar positions: These findings indicate that while they are digitalising their services and assessing the overall staff-related costs and savings, organisations should factor in the costs of technical support and training, as well as the costs of putting in place other mechanisms to ensure that all staff can effectively use the technology. Organisations will also need to ensure that there is buy-in in the digitalisation process at the staff level and to put in place mitigation strategies to address any staff concerns in order to fully realise possible staff-related cost savings associated with digitalisation. While digitalising, organisations should also consider their current staffing position in terms of staffing requirements (surplus or deficit), as well as the types of services that various members of staff are responsible for (in terms of the extent to which they can and should be digitalised). In parallel, they should consider the costs and savings associated with digitalisation, as these factors are inextricably linked.

2.3.2 Service-related costs and savings

In addition to having implications for staff-related costs and savings, digitalisation has financial and economic impacts related to the scope and nature of service delivery. 

We found that digitalisation can result in reduced costs and increased savings in service delivery, particularly in the healthcare context (although this view is contentious and under ongoing debate and review)^{[footnote 123] [footnote 124]}. For example, introducing telemedicine indirectly resulted in a reduction of hospital operational costs by decreasing the rate of hospitalisation from 5.7 to 2.2 days annually per patient through providing an alternative channel to access preventive medical and health consultations from general practitioners.^{[footnote 125]} Similarly, Heoponiemi et al.’ study on online healthcare showed that ICT in health care is perceived to decrease costs and improve patient outcomes by “transforming healthcare to being more proactive, preventive, and person-centred instead of being reactive and hospital-centred.”^{[footnote 126]} Timeliness and efficiency were the most reported positive aspects of virtual care solutions, leading to system savings in the healthcare context.^{[footnote 127]} 

Similar savings have been noted in the social care context. In the UK, several local authorities found that digitalisation in the form of technology-enabled care services can cut care costs and increase the efficiency of care services.^{[footnote 128]} For example, in East Sussex, a telecare programme showed an approximate cost-savings value of £32 per client per week, for an estimated annual preventive savings of £589,000.^{[footnote 129]} An additional example showing cost savings from digitalisation in a social care context is from the previously mentioned Trelleborg Municipality study, where digitalisation and automated decision making reportedly resulted in a reduction in the cost of social assistance.^{[footnote 130]} Similarly, in neighbouring Norway, the use of welfare technology was found to reduce pressure on healthcare services by decreasing consultations, home nursing services and admissions to hospital.^{[footnote 131]}

Additionally, it has been noted that, in some instances, digitalisation may lead to increased demand in the service that offsets the cost reductions associated with the shift to digital service delivery. This is illustrated by Turner et al., who found that moving to digital may lower the unit costs of a service (making it more cost-effective), however, in this particular instance, this also increased demand for that service. Their study found that with online STI testing in inner London, the total annual cost of the service increased from £2.87m (2014) to £3.09m (2016) even though there were decreases across the average cost per unit (from £66 to £61) and the average cost per diagnosis (from £660 to £644); the increase was the result of an increase in demand.^{[footnote 132]} An increase in service demand as a result of digitalisation was also seen in a study examining how frontline workers in the Norwegian Labour and Welfare Administration (NAV) perceive digitalisation (specifically, electronic communication with clients). In terms of efficiency, staff found that while electronic communication saves them time, it also “makes them more available to clients”.^{[footnote 133]} NAV uses an electronic communication platform, Modia, as its main channel for service provision. By providing frontline workers and clients with an online messaging function, Modia changes service provision in job-oriented counselling by providing the client with direct access to their frontline worker. In addition to using features similar to an online chat, the frontline workers use Modia as an electronic inbox and answer messages when available, increasing both access and demand.^{[footnote 134]} This spur in the demand for services as a result of digitalisation means that there is a potential “resource trade-off between efficient services and available services”^{[footnote 135]} that must be considered. The final service-provision-related costs to consider are costs related to procurement and the dependency on external providers without building in-house capacity and, when digitalisation projects are financed with private investment and public–private partnerships (PPPs), cost calculations are “often unrealistic due to regular underestimation of indirect and recurring costs.”^{[footnote 136]}

Considerations for the DWP or other stakeholders in similar positions: In order to realise the potential service-provision-related cost reductions of digitalisation, organisations should consider the trade-offs between cost savings and increased demand for services brought about by digitalisation. They should also factor in the interaction between customer-centric approaches and costs when assessing the cost and savings impacts of digitalisations related to service delivery.

2.4 Costs and savings implications of failed digital channels, digital channel mixing and interoperability

2.4.1 Costs associated with the failure of digital channels

Digital channel failure occurs when the digital means of service delivery is unsuccessful in achieving its expected, pre-defined outcomes. The process of digitalisation is vulnerable to technical difficulties, particularly at the initial roll-out stage, which results in additional unforeseen costs, as has been previously discussed. There are numerous examples of failed digital channels across different industries, including in social services. 

In Finland, the Virtu.fi telecare service experienced problems in delivering the technology and programmes due to issues with internet connection, while in Norway, technical issues were experienced in relation to digital welfare technologies for children with disabilities and their families, including set-up, coordination and synchronisation between devices.^{[footnote 137]} Austria also experienced technical difficulties that negatively affected the acceptance of care robots by service users.^{[footnote 138]} 

These examples are consistent with reported digital channel failures in other sectors and industries, where inadequate change management, not hiring (or commissioning) the right personnel, the lack of clear goals and the prevalence of a ‘fail fast’ attitude to digital transformation are cited as some of the factors leading to digital channel failure.^{[footnote 139]} The use of technology for technology’s sake (without a clear set of goals) in particular should be avoided by welfare organisations. To avoid such failure, the regulatory environment around digital technologies must be kept in mind. For example, the use of technology, such as video communication software, in a highly regulated industry should focus not only on how a software tool such as Zoom or WebEx can improve employee communication, but also on the compliance implications of the new software.^{[footnote 140]} Commercial entities can take a ‘fail fast’ approach to digitalisation (exploring digital innovation by trying a variety of digitalisation endeavours in rapid succession and moving on to the next until they find one that works).^{[footnote 141]} Digitalisation in public sector organisations, however, particularly those in the welfare sector, requires careful analysis and cannot be accomplished overnight, because digital channel failures in this sector have repercussions beyond the financial bottom line. Therefore, public sector organisations digitalising their services should consider avoiding the ‘fail fast’ approach and instead “double down on their initiatives if they fail the first time and focus on doing it ‘bigger and better’” for the sake of the overall public good.^{[footnote 142]}

The failure of digital channels also comes at a financial cost, with studies showing that in 2021 1) the number of failed, scaled-back or delayed projects was very high, at 79% and 2) companies spent $5.5 million (on average) on failed projects over the course of the year.^{[footnote 143] [footnote 144]} 

A related cost that needs to be factored in with digitalisation relates to ongoing maintenance.^{[footnote 145]} As explained by Raso, while the harms (or ‘glitches’) that arise when new technologies are introduced happen in moments, implementation is an ongoing process requiring “attention not only to how digital tools are introduced, but also to how they are maintained.”^{[footnote 146]} These additional costs are seen in system improvements as well as system maintenance. For example, in the above-mentioned instance of Trelleborg, the Swedish Labour Market Agency spent a further 600,000 Swedish crowns (£49,128) in 2018 to improve the RPA system launched in 2017, highlighting the need to factor in ongoing maintenance, repair and improvement costs. Ongoing maintenance costs werealso reported as an important aspect of costs in the Danish context (see Box 4).^{[footnote 147]}

Box 4 Digitalisation in Denmark

For 20 years, the Danish government has been pursuing a digital strategy involving a large proportion of corporate and regional co-operation. Denmark has digitised national services, and all layers of government now use the same digital system, which requires a great deal of coordination. Danes who are technologically disadvantaged have the option to opt out of digital service provision and instead rely on older, physical processes, such as physical mail, as an alternative to email. 

Digital systems in Denmark can be accessed through citizen portals using an electronic identity (ID) to log on and apply to a range of 2,000 services. These systems are made with ease of use in mind, which is essential to transitioning more than 5 million Danes to digital services. However, user-friendliness and group representation vo still remain difficult issues. In an attempt to rectify these issues, the Danish government established digital training sessions for older people, people with a disability, those experiencing homeless and immigrants. Moreover, although offering digital services is compulsory e by law, the option to opt out without proof remains feasible.  

The impacts of savings can be seen as being dependent on the scale of uptake of digital services in Denmark. For instance, a digital ‘Corona-passport’ app implemented during the COVID-19 pandemic had to consider people who did not have access to a mobile phone, people who could not receive vaccinations, and tourists. In order to assess cost savings, there needs to be sufficient uptake. This can, be seen through Denmark’s savings from digital mail: 217 million emails are sent per year, and the equivalent in regular mail would have cost an estimated £20 to £40 million. 

A related cost that needs to be factored in with digitalisation relates to ongoing maintenance. 

The subsequent effect of the COVID-19 pandemic led the Danish government to realise that most of their government services can be run remotely (e.g. from home) without issue. During the pandemic, digital systems took the brunt of the citizen services load, and various sectors, including hospitals, were able to function efficiently with an already established digital infrastructure. Overall, the level of disruption to government services caused by the pandemic was low.  

One of Denmark’s strengths in establishing and implementing digital services is a key focus on the end user, who is often overlooked by other governments.

2.4.1 Costs associated with the failure of digital channels (continued)

Various strategies to avoid the failure of digital channels have been proposed. These include: 

• shifting from crisis mode to maintenance mode, particularly in the aftermath of the initial COVID-19 outbreak, and avoiding reverting back to sub-optimal, previously established ways of working  
• continuing to invest in IT staff and infrastructure 
• maintaining customer relations and investing in a Chief Data Officer to oversee the digitalisation process 
• resisting change for change’s sake by considering whether or not there is a need to change the mode of service provision;  
• aligning digitalisation with overall strategic goals;  
• ensuring continuous monitoring and evaluation to track and measure the effects of digitalisation initiatives^{[footnote 148]} 

Resisting change for change’s sake echoes concerns about avoiding technology for technology’s sake without clear strategic objectives. These and other proposed strategies to avoid the failure of digital channels can also prove costly, and therefore the cost of implementing these strategies needs to be compared with the costs of resolving digital channel failure. 

It must also be kept in mind that resisting digitalisation can be just as costly as failed digital channels.^{[footnote 149]} The notion of technical debt (or ‘tech debt’), which is the cost of additional rework caused by choosing a non-digital solution now that is easier to implement, instead of investing in digitalisation,^{[footnote 150]} should also be kept in mind. Adequate change management – with a focus on training, support, testing, communication and building an adequate strategic framework – is proposed as the main solution to avoid failed digital channels, while taking advantage of the cost-savings solutions provided by digitalisation.^{[footnote 151]}

Considerations for the DWP or other stakeholders in similar positions: The prevalence and likelihood of digital channel failure suggests that, while they are digitalising, organisations should include digital channel failure in their overall risk assessment and put in place mitigation plans for this possibility in their digitalisation implementation strategy. In addition to ensuring that the systems deployed are sound and robust, organisations should factor in the need to put in place monitoring, maintenance and repair processes in order to avoid the costs associated with failed digital channels. The DWP should further consider the underlying costs of delaying digitalisation and the implementation considerations raised by the notion of technical debt.

2.4.2 Digital channel mixing costs and savings

Putting in place multiple means for service access and delivery (channel mixing) is seen as a way to make services more cost efficient. For example, the Norwegian Labour and Welfare Administration (known as NAV) introduced new channels (including electronic communication, self-service solutions and call centre communication) as part of its cost-reduction strategy.^{[footnote 152]} Putting in place alternative communication channels (each used for different types of requests, such as benefits or counselling) allowed NAV to allocate more time to job-oriented counselling.^{[footnote 153]} More specifically, the aims of NAV’s channel strategy were to make workforce-oriented follow-up more efficient, to reduce costs, and to concentrate resources on NAV’s primary objective: to activate unemployed citizens and impose a demand for the citizen to find work.^{[footnote 154]} The channel strategy was based on routing service users away from resource-demanding face-to-face meetings and towards digital channels, which are less resource demanding for case management, essentially offering analogue services as digital solutions.^{[footnote 155]} A study examining how NAV service users with mental health challenges and co-occurring disorders experience the digitalisation of NAV’s services showed both positive and negative results, highlighting “the need for more attention to whether technical solutions hinder or support individual service users in NAV, especially those with a need for close follow-up”.^{[footnote 156]} 

A similar example comes from the Australian Department of Skills, Education and Employment, which in 2022 moved to digital servicing and profiling customers in their New Employment Services Model.^{[footnote 157]} The service offers the following channels: 

• Digital First – for job seekers who are job ready 
• Digital Plus – for job seekers requiring some additional support, such as help gaining employability skills or a qualification (predominantly online via the digital service, but may offer face-to-face training and other targeted support) 
• Enhanced Services – for job seekers with multiple or significant barriers to work (face-to-face servicing from a provider and two tiers of service, based on vocational and non-vocational barriers and capability to undertake intensive activities)^{[footnote 158]} 

While the NAV example shows how channel mixing can be used as a cost-saving strategy, there is evidence that channel mixing results in increased demand in service delivery. In the study of STI testing in London,^{[footnote 159]} the costs of the channel mix were higher due to increased demand for the new digital service (which offset the reduction in costs per service unit) combined with ongoing use of the non-digital component of the service. In this instance, although the average costs of the service (per individual diagnosis) decreased from £660 to £644, there were increases in the total annual cost of STI testing, from £2.87m (in 2014) to £3.09m (in 2016) because clinics continued to offer in-person testing activity costing £1,953,652.^{[footnote 160]} 

A final cost factor to consider in channel mixing is potential costs and savings for customers. Studies have found that introducing digital channels reduces the cost of travelling to reach the service,^{[footnote 161]} with the availability of alternate means to access services (e.g. either in person or online) leaving customers with various cost-savings options, including those related to travel.^{[footnote 162]} Any costs and savings incurred by the customer, including savings related to childcare costs and not needing to take time off work, affect their choice to access the service (or not).^{[footnote 163]}

In the UK context, in March 2017, the government published its digital strategy policy paper, which had a significant impact on the move to channel mixing by stating that, in the future, where possible and necessary, private and public sector organisations should be digital by default.^{[footnote 164]} With being digital by default in mind, the following strategy to optimise channel mix strategies (as part of a four-part strategy) was proposed: 

• define channel choice based on the understanding of a targeted market and ways to promote and share information with intended customers (hypothetical examples included that Baby Boomers likely prefer face-to-face conversations, Generation X tends to favour email, Generation Y likes a combination of email and text and Generation Z is highly adept at using social media) 
• be inclusive and cater for those without access or willingness to use digital services 
• improve customer communication by examining which channel is best for the service and then making the experience of using the service rewarding for customers 
• promote channel shift to encourage customers to interact with or access services via channels they would not naturally choose^{[footnote 165]} 

Further insight that is more specific to a public sector context can be gained by considering the Warwick District Council (WDC)’s channel strategy. This strategy is based on a combination of more traditional channels, such as face to face and telephone, and newer channels, such as social media and mobile web access, and it provides principles and guidelines for channel mixing, including: 

• move transactions and information to the web wherever possible 
• continue face-to-face services 
• investigate use of social media where valuable for customers 
• encourage customers to use self-service^{[footnote 166]} 

These principles are based on a generic model for the effectiveness of the major channels (no contact, web self-service, automated phone, phone, email or letter, and face to face), as illustrated in Figure 1, which shows how the cost of delivery typically gets cheaper for the organisation as it moves up the triangle.^{[footnote 167]} However, for some types of contact, a greater level of human contact is required, particularly for interactions that require a level of reassurance.^{[footnote 168]} Figure 1 was developed by the Allerdale Borough Council as part of their digital transformation initiative and is based on “adapting the learning of other organisations to help illustrate graphically the effectiveness of the different communication channels”.^{[footnote 169]}

Figure 1: Channel effectiveness

A triangle, with a channel on each layer. 'No contact' is at the top, followed by 'web self service', 'automated phone', 'phone', 'email/letter' and 'face to face' at the bottom. There is an arrow either side. One is labelled 'less cost' and points towards the top of the triangle. The other is labelled 'more reassurance' and points towards the base of the triangle.

Source: Allerdale Borough Council^{[footnote 170]}

Considerations for the DWP or other stakeholders in similar positions: While digitalising, organisations should keep in mind that individual circumstances, personal preferences and other factors make it difficult to calculate the rate of service use and, ultimately, of service costs. That being said, there is evidence to suggest that arriving at the optimal channel mix can bring cost savings for organisations. Key lessons from proposed channel mixing strategies include understanding customers and tailoring the target channel to their needs and preferences; providing a variety of channel options; and encouraging customers to, wherever possible, use self-service. Finally, organisations should also keep in mind the fact that more costly channels (such as face to face) also provide greater reassurance to customers than less costly channels (such as web self-service).

2.4.3 Interoperability costs and savings

Interoperability, or the ability of computer systems or software to exchange and make use of information, is a key concern in social service delivery.^{[footnote 171]} The fragmentation of information (e.g. having several databases of service users that exist in separate silos and don’t ‘talk’ to each other) is an obstacle to the digitalisation of social services.^{[footnote 172]} This is because the sharing of information across the public sector is essential in order to “shorten lead times, secure transparency, and ensure that the correct care is given to the right citizen.”^{[footnote 173]} As explained by Gil-Garcia et al., information-sharing projects are becoming increasingly important in both public and private organisations because of expected benefits, such as “better services, operational savings, and increased program effectiveness.”^{[footnote 174]} 

Digital technologies can themselves help reduce this fragmentation, with digital welfare seen as a means for the public sector to become more interconnected. As explained by one interviewee, integrating and standardising of data were put in place to drive cost savings and improvements in services, because more systems can interact in a standardised format.^{[footnote 175]} This is in line with a study by Eurofound that recommended the establishment of specific institutions responsible for the digital transformation process in order to increase interconnectivity and reduce fragmentation of social services.^{[footnote 176]}

Similar to challenges in measuring costs, literature on the costs of interoperability is scarce. The challenges of quantifying interoperability savings are illustrated by Baker’s study on the benefits of interoperability, which provides estimates of savings to be gained by increased healthcare information exchange and interoperability (also referred to as HIEI).^{[footnote 177]} Here it was noted that “there would be some savings from improvements in information sharing, though just how large those savings might be has been less clear”.^{[footnote 178]}

Finally, successful interoperability can only be achieved through broader coordination and standardisation within DWP and across public services. In Norway, it has been recognised that “municipalities, county authorities and central government agencies must be able to collaborate in order to develop user-centric, seamless and efficient digital services”.^{[footnote 179]} This has resulted in strategies to build a common ecosystem (see One Digital Public Sector Digital Strategy) based in part on: 

• common data sources, such as master data and basic data registers 
• common architectures, such as reference information, standards, guides and frameworks 
• standard access for multiple users 
• standard business models and contracts 
• harmonised financing and payment models

One Digital Public Sector Digital Strategy (Norway)

Primary goal:

• Simpler everyday life for citizens and the business sector

Secondary goals:

• Better services
• Efficient public use of resources
• Increased value creation

Goals for 2025:

• The public sector shall be digitalised in a transparent, inclusive and trustworthy way
• More tasks shall be performed digitally, and as seamless digital services
• All citizens, businesses and voluntary organisation that have the ability to do so, shall communicate digitally with the public sector
• The public sector shall exploit the potential of sharing and using data to create user-friendly services, and to promote value creation in the business sector
• Local and central government agencies shall develop their services based on a common digital ecosystem for collaboration
• Local and central government agencies shall realise benefits from digitalisation in a systematic manner

Focus areas:

• Seamless services
• Coordination across administrative levels and sectors
• Digitalisation-friendly regulations
• Data sharing
• Common national solutions in an ecosystem
• Cooperation with the private sector

Source: Norwegian Ministry of Local Government and Modernisation^{[footnote 180]}

Considerations for the DWP or other stakeholders in similar positions: While the ability to quantify the costs and savings of interoperability is limited, the lack of interoperability, fragmentation and limited data sharing appears to be an impediment to successful digitalisation. Therefore, organisations digitalising their services should aim to champion government actions to put in place strategies to coordinate and increase interconnectivity within and across all government departments in order to increase interoperability and data sharing – and with it, increase cost savings.

2.5 Summary of key considerations related to the costs and savings of digitalisation

Key considerations for the DWP to take into account when thinking about the costs and savings associated with the digitalisation of welfare services are summarised in Table 2.

Table 2. Key considerations for the DWP or other stakeholders

Considerations	Relevant Research Questions (RQs)	Key considerations for the DWP or other stakeholders
Digitalisation activities that organisations invest in to achieve savings	How can we best measure the financial, economic and social impacts of digitalising services? (RQ1), Are there successful examples of sharing or movement of data between different IT platforms, and what are the costs and benefits of this approach? (RQ3)	Our findings indicate that there is a disconnect between interest in the use of more advanced digital technologies (such as AI and cloud computing), on the one hand, and the reality of the types of services that are being digitised and digitalised (which are still largely transactional), on the other hand. Organisations could consider taking a measured and incremental approach. They could first focus on ensuring adequate digitisation, and then work towards transactional digitalisation and, finally, more interactional digitalisation. Taking this incremental approach means organisations can consider the use of technologies such as RPA at appropriate times – technologies that, while providing considerable efficiency gains, have important cost and service delivery implications.
Measuring the financial, economic and social cost of digitalising services	RQ1	Our findings highlight how accurately assessing costs (including costs per service) might prove challenging in the context of DWP’s work given the complex nature of the services provided. Some costs and savings may be difficult to quantify, especially given the interconnected nature of some of the services. DWP will likely have to use proxies for some of the necessary data points and could consider focusing on a service-by-service approach to measurement. This may help to get an accurate financial and economic assessment of digitalising a range of services. Finally, the link between service costs and service quality must be kept in mind, particularly when serving vulnerable customers.
Measuring the financial, economic and social cost of digitalising services	RQ1	Organisations such as the DWP should keep in mind the extent and nature of the social costs associated with the digitalisation of public services and factor these into their overall costs and savings analyses. In addition, organisations such as the DWP could seek to foster horizontal and vertical partnerships with local authorities and other national authorities to advocate for, shape and arrive at a unified, national approach to digital inclusion in order to better account for and meet the social costs (including those to do with affordability and access) related to the digitalisation process.
Potential costs and savings related to staff and service delivery	What is the cost trade-off between digital investment and savings on staff headcount? (RQ2), RQ3	Our findings indicate that while they are digitalising their services and assessing the overall staff-related costs and savings, organisations should factor in the costs of technical support and training, as well as the costs of putting in place other mechanisms to ensure that all staff can effectively use the technology. Organisations will also need to ensure that there is buy-in in the digitalisation process at the staff level and to put in place mitigation strategies to address any staff concerns in order to fully realise possible staff-related cost savings associated with digitalisation. While digitalising, organisations should also consider their current staffing position in terms of staffing requirements (surplus or deficit), as well as the types of services that various members of staff are responsible for (in terms of the extent to which they can and should be digitalised). In parallel, they should consider the costs and savings associated with digitalisation, as these factors are inextricably linked.
Potential costs and savings related to staff and service delivery	RQ2, RQ3	In order to realise the potential service-provision-related cost reductions of digitalisation, organisations should consider the trade-offs between cost savings and increased demand for services brought about by digitalisation. They should also factor in the interaction between customer-centric approaches and costs when assessing the cost and savings impacts of digitalisations related to service delivery.
Costs and savings implications of failed digital channels, digital channel mixing and interoperability	Does including a digital channel enable more people to claim a benefit or service? (RQ4), Learning from accelerated digitalisation due to COVID-19, are there any consistent emerging and sustainable ‘wins’ which could be considered for implementation in DWP services? (RQ6), How have organisations managed to shift customers onto online channels, and are customers who moved online during COVID-19 continuing to use online channels? (RQ7)	The prevalence and likelihood of digital channel failure suggests that, while they are digitalising, organisations should include digital channel failure in their overall risk assessment and put in place mitigation plans for this possibility in their digitalisation implementation strategy. In addition to ensuring that the systems deployed are sound and robust, organisations should factor in the need to put in place monitoring, maintenance and repair processes in order to avoid the costs associated with failed digital channels. The DWP should further consider the underlying costs of delaying digitalisation and the implementation considerations raised by the notion of technical debt.
Costs and savings implications of failed digital channels, digital channel mixing and interoperability	RQ4, RQ6, RQ7	While digitalising, organisations should keep in mind that individual circumstances, personal preferences and other factors make it difficult to calculate the rate of service use and, ultimately, of service costs. That being said, there is evidence to suggest that arriving at the optimal channel mix can bring cost savings for organisations. Key lessons from proposed channel mixing strategies include understanding customers and tailoring the target channel to their needs and preferences; providing a variety of channel options; and encouraging customers to, wherever possible, use self-service. Finally, organisations should also keep in mind the fact that more costly channels (such as face to face) also provide greater reassurance to customers than less costly channels (such as web self-service).
Costs and savings implications of failed digital channels, digital channel mixing and interoperability	RQ4, RQ6, RQ7	While the ability to quantify the costs and savings of interoperability is limited, the lack of interoperability, fragmentation and limited data sharing appears to be an impediment to successful digitalisation. Therefore, organisations digitalising their services should aim to champion government actions to put in place strategies to coordinate and increase interconnectivity within and across all government departments in order to increase interoperability and data sharing – and with it, increase cost savings.

3. Customer experience

This chapter presents evidence of strategies that organisations have used to shift customers onto digital channels and of the varying impacts of digital services on the customer, which influence adoption of the service as well as the quality of the service itself. The chapter focuses first on factors affecting customer engagement with and use of digital services and then moves on to consider the implications of digitalisation for service quality and customer experience. Key considerations for the DWP in relation to customer experience are highlighted throughout and summarised in section 3.3.

3.1 Strategies used to shift customers onto digital channels

Organisations have shifted customers onto digital channels through various means, including guaranteeing the availability of non-digital options. A county council in the UK undertook a marketing campaign to inform residents of the availability of digital services, which resulted in 78% of residents choosing to use digital services and produced overall positive customer feedback as of 2022.^{[footnote 181] [footnote 182]} Another English local authority opted to only inform residents through their official website of the availability of new digital services in addition to the regular, non-digital option, and also created a customer hotline to reassure residents of the continuing availability of a non-digital option. Their efforts resulted in only 15% of residents opting to use the call centre as of 2022.^{[footnote 183]} To moderate any possible accessibility issues, another council created an engagement and skills team to educate vulnerable customers on the proper and safe use of digital services when they fully digitalised their services in response to the COVID-19 pandemic.^{[footnote 184]} Another local council in England divided their service channels into ‘universal’ (e.g. registration, library access) and ‘targeted’ (e.g. elderly care, support for children with learning difficulties), prioritising digitising the universal services and liaising with users for feedback on how to modify the channel mix.^{[footnote 185]} 

As described by one interviewee (from a European country), government officials engaged in a large educational campaign for the general population and offered specialised courses for vulnerable groups in order to help them transition to a digitalised system for service provision.^{[footnote 186]} The government also ensured the ability for citizens to opt out of digitised services and for citizens to receive help at municipality centres for filling in forms. They also minimised user errors through allowing the pre-filling of forms in digital services.^{[footnote 187]} 

Considerations for the DWP or other stakeholders in similar positions: There are multiple methods for shifting customers successfully onto digital channels, including guaranteeing customers the availability of non-digital options, launching marketing or educational campaigns and creating engagement teams.

3.2 Factors affecting the customer experience of digital services

The literature highlighted seven factors that affect the customer experience of using digital services: accessibility, trust, aesthetic experience, usefulness, ease of use, interactivity, and the context in which the service is used.

3.2.1 Accessibility

For customers who face travel-related stress and costs, the move to digital can increase access to services^{[footnote 188]}; however, organisations can also increase access by tailoring their digital services to the needs of particular groups. This process may result in a decision to add or increase a non-digital alternative to a service. For example, according to a study conducted in England, homeless people prefer smartphones or desktop computers over laptops for internet access, reflecting the challenges they face.^{[footnote 189]} This population, however, generally have limited access to computers and the internet, so locating job-related information on the internet might not be a suitable arrangement for this group. In cases where digital services are provided, homeless people would likely benefit from a hybrid digital approach, that is, digital services supplemented with non-digital services.^{[footnote 190]} 

Organisations considering digitising their services should also recognise the diversity  of their customer base, which implies that different needs exist within each customer group. Multiple sources confirmed that using broad categories to group users is not sufficient for tailoring digital services, as each category represents numerous different sub-groups with distinctive needs. Divisions that commonly appear across many groups include: generational divides, not only between ‘young’ and ‘old’ population segments, but within those groups as well (e.g. ‘youngest-old’ to ‘oldest-old’)^{[footnote 191]}; level of ICT literacy or technology use abilities^{[footnote 192]}; health status (frail versus healthy, relevant for treatment of the elderly)^{[footnote 193]}; educational background^{[footnote 194]}; economic status^{[footnote 195]}; access to digital equipment and sufficient Wi-Fi capacity within a household^{[footnote 196]}; caring responsibilities^{[footnote 197]}; the quality and reliability of the internet system surrounding the services (i.e. vulnerability to internet outages, network slowdowns)^{[footnote 198]}; and access to reliable broadband, which especially affects rural populations.^{[footnote 199]} Homeless people^{[footnote 200]} and the elderly^{[footnote 201]} have been identified as groups with distinctive needs and preferences, along with immigrant populations (e.g. short-term economic migrants, permanent immigrants, refugees or asylum seekers).^{[footnote 202]} 

As seen previously with larger customer groups, exploring the different circumstances of these cross-cutting segments can indicate how non-digital alternatives to a service might need to be introduced or enlarged to cater to their needs or preferences. Studies have shown how lower socio-economic status and lower levels of education link to lower perceived health and economic benefits of online services, associated with access issues, a lack of digital skills and low use of such services.^{[footnote 203]} Service users from a lower socio-economic and educational background may benefit from an in-person component to complement digital services, or from tailored support for using digital systems. 

For this reason, special digital training sessions and systems were made available in one European country for vulnerable populations, including the elderly, people with a disability (some of whom may prefer digital services with read-aloud features), immigrants and people experiencing homelessness.^{[footnote 204]} Similarly, a local council in England established an engagement team to help vulnerable populations understand their digital services after they had launched.^{[footnote 205]} However, a study observed that older adults who are “healthy, self-reliant, and internet-skilled” do not require an additional tool – in this case, an online community platform for finding care support – since they can arrange care for themselves to a sufficient degree.^{[footnote 206]} Although digital literacy has been recognised as increasing uptake of digital services, in certain contexts this may not occur. In a study on virtual primary care, participants with higher e-health literacy consistently showed a lower willingness to use virtual care across all technologies evaluated in the study.^{[footnote 207]} The study authors hypothesise that those with higher e-health literacy have a better understanding of the risks involved in virtual care and potentially face a higher rate of digital fatigue.^{[footnote 208]} 

Considerations for the DWP or other stakeholders in similar positions: When designing digital services, care must be taken to not only understand the particular requirements for population segments (e.g. the elderly, people experiencing homelessness), but also inspect the variations within each segment according to major cross-cutting divisions (e.g. education level, economic status). The degree to which digital welfare services will need to be complemented or fully replaced with a non-digital alternative or assistance will depend on the outcome of these analyses.

3.2.2 Trust

Trust in the context of digital services, or ‘e-trust’, has been defined as comprising four aspects related to service provision: “integrity, benevolence, ability, and predictability”.^{[footnote 209]} Several factors can affect the degree to which users perceive these aspects as present in a particular service. A study on the contact tracing app developed by the UK government in response to the COVID-19 pandemic emphasised the need for sufficient accountability. Since citizens perceived the UK government’s general attempts to provide accountability during the pandemic as ‘insufficient’ and ‘unsatisfactory’, the involvement of service users in the design of services – which was intended to help better tailor digital service to the needs of citizens – failed.^{[footnote 210]} The study qualified this conclusion by acknowledging how the unique context of the COVID-19 pandemic may limit the findings to this particular setting, ^{[footnote 211]} but the study also demonstrates how trust in the general government can affect trust in a public digital service. 

The appropriate handling of privacy concerns through sufficient communication can also improve public trust in a service. The barrier to participation in a digital service is high when sensitive data are involved, if customers have concerns related to the potential misuse of data or find that they have little influence over managing the data.^{[footnote 212]} One study described how, due to privacy concerns, some refugees voiced concerns about using Facebook, a platform that many community organisations use for communication purposes.^{[footnote 213]} A study on Google’s failure to establish an augmented reality extension to its digital platform through the ‘Google Glass’ smart glasses noted that disclosing the rights and forms of participation of actors in the digital platform innovation process (e.g. platform owners, software developers, the general public) is ‘essential’ to avoid provoking controversies. This study also emphasised the importance of transparency about the risks and benefits of the technology. The authors view the absence of both factors as contributing partially to the failure of the smart glasses technology, because privacy concerns (especially related to facial recognition software) and technology overload (among other issues) were not adequately addressed by Google.^{[footnote 214]} Several interviewees also mentioned that privacy is a critical factor to take into consideration when implementing digital services, due to concerns over the use of AI,^{[footnote 215]} identity theft^{[footnote 216]} and other crimes.^{[footnote 217]} One interviewee said that a careful balance is necessary between a need to protect against inappropriate data sharing and a need to ensure adequate ease of use for users in practice.^{[footnote 218]}

Considerations for the DWP or other stakeholders in similar positions: To ensure a sufficient level of trust in the digital service, the organisation in charge of service delivery must take into account the privacy concerns of the targeted users, transparently communicate their rights and ensure the existence of adequate mechanisms of accountability related to the service itself.

3.2.3 Aesthetic experience, usefulness, ease of use and interactivity

Through a literature review on the effect of website design on consumer experience, Zhang et al. identified four factors as having an effect in addition to trust: aesthetic experience, usefulness, ease of use, and interactivity.^{[footnote 219] [footnote 220]} This section summarises these briefly. We acknowledge the existence of wider literature on these topics.^{[footnote 221]}  

Aesthetic experience refers to “the visual effect … of interface design”.^{[footnote 222]} A study measuring the customer experience on official city tourism online platforms found that aesthetics can have a positive effect on the emotional experience of users, due to the “halo effect of projecting an attractive image”, which in turn increases their willingness to travel to the destination, thereby fulfilling the objective of the service.^{[footnote 223]} The only other factor classified as positively influencing user emotional experience was trust.^{[footnote 224]} In Zhang et al.’ study, both aesthetic experience and trust were more important than usefulness, perceived ease of use and interactivity.^{[footnote 225]} 

Usefulness and ease of use were identified as factors by a theory known as the ‘technology acceptance model’, which outlines how users accept information systems.^{[footnote 226]} The model defined usefulness as “the degree to which a person believes that using a particular system would enhance his or her job performance” and ease of use as “the degree to which a person believes that using a particular system would be free of effort”.^{[footnote 227]} A study by Chung et al. evaluating the official website of the Korean Tourism Organisation concluded that usefulness of the website had a positive impact on potential tourists’ satisfaction with the website itself.^{[footnote 228]} The review noted that there is less data available for demonstrating a positive effect of ease of use on website satisfaction, but it cited two articles as providing evidence in other contexts, one of which is a study involving a survey of residents of a major Brazilian city that attributed the effective use of government websites in Brazil to ease of use, as well as usefulness and trustworthiness.^{[footnote 229]} The authors additionally noted that usefulness was only an important factor at the adoption stage, but that ease of use became more significant at moderate and high levels of use, after the user had already developed an opinion about usefulness.^{[footnote 230]} An interviewee also reflected how the biggest issue their government faced when first implementing digital services was ensuring sufficient user-friendliness, or ease of use, across various customer groups.^{[footnote 231]} 

The review by Chung et al. looked for but did not find evidence about the extent to which the interaction that occurs between humans and computer or networked systems affects customer experience.^{[footnote 232]} Zhang et al. did not provide any specific examples of the influence of interactivity on customer experience, but they noted that the experiment conducted in the study did not find that interactivity had a significant effect on customer experience.^{[footnote 233]} The authors acknowledged, however, that this conclusion would benefit from further testing.^{[footnote 234]} 

Considerations for the DWP or other stakeholders in similar positions: When designing digital services, particular attention should be paid to ensuring a sufficient level of aesthetic experience in order to increase the likelihood that users will have a positive emotional reaction and thereby continue to use the service. Usefulness and ease of use appear to also help both first adoption and continued use of the service.

3.2.4 Context

Certain contexts appear to benefit from the use of digital services more than others. Digital channels can be helpful wherever the organisation can ensure a reduced performance risk – that is, a lower risk that “loss or danger may occur” through the use of machines than through the use of humans.^{[footnote 235]} Making purchases, for example, can be quicker and simpler online, whereas in person, the action tends to be more time consuming.^{[footnote 236]} An interviewee from a local council mentioned that staff had to spend more time chasing service users for payments prior to digitalisation in comparison with after digitalisation.^{[footnote 237]} Similarly, digital systems resulted in a reduction in user error: Incomplete or inaccurate applications have decreased due, in part, to requiring the user to fill in certain information in order to proceed with the application submission.^{[footnote 238]} 

In the healthcare context, apps can provide valuable data to relatives or friends of care recipients to help them manage and administer care more efficiently.^{[footnote 239]} Online platforms are effective whenever they can support existing offline services to “stimulate local participation and feelings of connectedness”,^{[footnote 240]} for example, and also whenever there is a desire to eliminate stress and costs related to travel can decrease the number of ‘no-shows’ for one-to-one online appointments.^{[footnote 241]} 

However, digital services do not work as well in other contexts. Digital channels are not as useful when they place too much of a burden on users. In the healthcare context, if users are expected to take an active role in maintaining their own health due to introducing a digital service, this aspect increases the risk of user drop-out.^{[footnote 242]} Digital services can also be less effective if the customer is in an urgent, stressful situation. It has been noted that self-service digital channels for provision of advice are best avoided in the initial stages of experiencing homelessness, since in this situation people may experience “feelings of urgency and emotional distress”, which causes them to seek in-person advice.^{[footnote 243]} In the healthcare context, telemedicine cannot serve acutely ill patients requiring immediate attention. Telemedicine is more appropriate for handling chronic conditions, by providing regular follow-up on patient conditions through telemonitoring, for example.^{[footnote 244]} 

Similarly, digital services appear to be less effective in settings where personal or emotional connections are important. In the healthcare context, robots have been criticised as providing inferior care and contact, and human interactions have been described as a better way to “release anxiety” and allow people to be “reassured and relieved of tensions”.^{[footnote 245]} Conveying emotional support is more difficult to accomplish through a screen: “Sitting silently for a few moments with a patient can be helpful if the clinician is sitting in the same room. However, silence on the video monitor may not convey empathy equally”.^{[footnote 246]} Missing non-verbal cues on camera can also cause patients to undervalue appointments and be less “comfortable in explaining their well-being.”^{[footnote 247]} In a patient survey on satisfaction with telemedicine services, 60% indicated a stronger personal connection when receiving medical attention in person compared with virtually.^{[footnote 248]}

Considerations for the DWP or other stakeholders in similar positions: When designing digital services, sufficient attention must be paid to ensure the digital service is designed for an appropriate context. Digital services should be designed in such a way as to avoid imposing an undue burden on the service user, and consideration should be given to the suitability of digital channels in the context of urgent, acute or distressing customer experiences.

3.3 Summary of key considerations related to customers’ experience

Key considerations for the DWP to take into account with regards to customers’ experience of digital services are summarised in Table 3.

Table 3. Key considerations for the DWP or other stakeholders (research questions relating to customer experience)

Considerations	Relevant RQ(s)	Key considerations for the DWP/other stakeholders
Strategies used to shift customers onto digital channels	Does including a digital channel enable more people to claim a benefit or service? (RQ4), What are the costs and customer experience levels of a successful channel mix, and how did organisations develop and establish a mix that worked for their customers? (RQ9)	There are multiple methods for shifting customers successfully onto digital channels, including guaranteeing customers the availability of non-digital options, launching marketing or educational campaigns and creating engagement teams.
Factors affecting the customer experience of digital services	Does a successful channel mix vary at different points of the customer journey, and how does this vary for groups with different protected characteristics? (RQ5)	When designing digital services, care must be taken to not only understand the particular requirements for population segments (e.g. the elderly, people experiencing homelessness), but also inspect the variations within each segment according to major cross-cutting divisions (e.g. education level, economic status). The degree to which digital welfare services will need to be complemented or fully replaced with a non-digital alternative or assistance will depend on the outcome of these analyses.
Factors affecting the customer experience of digital services	RQ5	To ensure a sufficient level of trust in the digital service, the organisation in charge of service delivery must take into account the privacy concerns of the targeted users, transparently communicate their rights and ensure the existence of adequate mechanisms of accountability related to the service itself
Factors affecting the customer experience of digital services	RQ5	When designing digital services, particular attention should be paid to ensuring a sufficient level of aesthetic experience in order to increase the likelihood that users will have a positive emotional reaction and thereby continue to use the service. Usefulness and ease of use appear to also help both first adoption and continued use of the service.
Factors affecting the customer experience of digital services	RQ5	When designing digital services, sufficient attention must be paid to ensure the digital service is designed for an appropriate context. Digital services should be designed in such a way as to avoid imposing an undue burden on the service user, and consideration should be given to the suitability of digital channels in the context of urgent, acute or distressing customer experiences.

4. Social impact of digitalisation and learnings from COVID-19

This chapter covers evidence of any social effects of the digitalisation process, as well as any lessons from the digitalisation experience during the COVID-19 pandemic.

4.1 The impact of digitalisation on inequality

If the transition to a digital channel is not managed correctly, the change can lead to significant errors related to the service itself and can contribute to decreased equity among users through contributing to debt, mental and physical health issues and social exclusion, among other effects.^{[footnote 249]} In one study focused on the digitalisation of primary health care, older people became less engaged in the service, particularly those with low digital health literacy, and equity was the factor that was cited least often as being positively impacted by digitalisation in the study.^{[footnote 250]} An interviewee from a local council described how these challenges are particularly pertinent to universal (rather than targeted) services which should ensure the same level of service for all, including those who lack the necessary skills and access.^{[footnote 251]} Their local council established public Wi-Fi to help ease the divide.^{[footnote 252]}

The effects of the COVID-19 pandemic demonstrated how digitalisation can contribute to an increase in inequity through a decrease in access to both in-person and digital services for vulnerable populations. Access to in-person services, as well as the internet in public libraries and community spaces, became impossible during lockdown, and new forms of access limitations emerged, such as insufficient bandwidth and limits on the number of connected devices, due to all household members using the same internet connection simultaneously, driven by lockdown requirements.^{[footnote 253]} An interviewee from a local council also described how the pandemic “massively” increased the number of those in vulnerable categories (e.g. the homeless), an inequality that,, they believe is not being sufficiently addressed at the national level in terms of the digital divide. The interviewee cited the example of the technology available in social housing lagging behind that in residences available on the private rental market.^{[footnote 254]} Managing this divide in practice can also prove challenging. Existing mechanisms to help those with low digital health literacy might not be reaching those in need; according to a poll conducted by the Centre for Ageing Better, between November 2020 and January 2021, only 1 in 8 adults between 50 and 70 years old^{[footnote 255]} and on an income of less than £25,000 were aware of the existence of digital support organisations for helping those who are digitally challenged access services online.^{[footnote 256]} Online calls between welfare practitioners and service users can also introduce a power dynamic: “When holding a Zoom meeting with a service user, [one practitioner] felt particularly conscious of the fact that they were holding the call from their comfortable house, while the service user was showing them that her cupboards were empty as she described the difficulty of her situation.”^{[footnote 257]}

Considerations for the DWP or other stakeholders in similar positions: Inequity remains the main societal effect of digitalisation processes discussed in the reviewed literature, and these issues became more pronounced in the context of the COVID-19 pandemic. While digitalising their services, organisations should ensure that vulnerable populations are spared from negative effects (including accessibility issues related to either the service or the internet and its supporting technology).

4.2 Learnings from COVID-19

The pandemic has allowed for new types of or improved services to be provided

A benefit noted of the COVID-19 pandemic was the ability to broaden or improve the quality of the types of services that organisations can provide. One study observed that organisations were offering new types of workshops and webinars during the pandemic to “increase connections during a difficult time”, as well as due to a decreased concern about attendance.^{[footnote 258]} An English local authority noted that the ability to link NHS data with welfare data, a linking introduced as part of governmental measures taken during the pandemic, helped to provide tailored welfare services to vulnerable groups. NHS data has subsequently been removed from the local council’s database, in 2021, and this action impeded the council’s ability to easily identify vulnerable people without energy during Storm Arwen.^{[footnote 259]} Although this example of increased data sharing could be interpreted as a ‘best practice’ case study, its success is ultimately dependent on whether measures to address any privacy concerns related to identifying service users and potential subsequent discrimination were sufficiently implemented by the local council managing the data.^{[footnote 260]}

There is some evidence that customers who moved to online channels during COVID-19 will continue to use them 

European adoption of digital technologies appears to have increased dramatically during the pandemic, from 81% to 95% of consumers, according to a survey conducted by McKinsey.^{[footnote 261]} An English local council indicated that one service they provide now has 100% online use.^{[footnote 262]} Another local council in England emphasised how the elderly population are more connected than ever before, having been forced to adapt in order to access essential services.^{[footnote 263]} According to the McKinsey survey, more than 70% of European consumers state that they expect to continue to use digital services overall with the same frequency (or even more frequently) post-pandemic, but the figure is lower for digital services made available by the public sector (64%). The lower figure for the public sector might be due to decreased customer satisfaction with public sector digital services during the pandemic, as 79% of users who reported that they were ‘dissatisfied’ complained of slow, poor-looking or difficult-to-use sites or apps, compared with 41% who reported dissatisfaction with the travel industry and 46% who reported dissatisfaction with the grocery industry.^{[footnote 264]} 

Considerations for the DWP or other stakeholders in similar positions: Evidence indicates not only that the pandemic accelerated the creation and uptake of digital services, some of which would not have been created otherwise, but also that most users will continue to use digital services with the same frequency in the future.

4.3 Summary of key considerations related to the social impact of digitalisation

Key considerations for the DWP to take into account with regards to the social impact of digitalisation and learnings from COVID-19 are summarised in Table 4, below.

Table 4. Key considerations for the DWP or other stakeholders (research questions relating to o the social impact of digitalisation and learnings from COVID-19)

Topic area	Relevant RQ(s)	Key considerations for the DWP/other stakeholders
Social impact of digitalisation and learnings from COVID-19	RQ6, RQ7, RQ8, RQ9	Inequity remains the main societal effect of digitalisation processes discussed in the reviewed literature, and these issues became more pronounced in the context of the COVID-19 pandemic. While digitalising their services, organisations should ensure that vulnerable populations are spared from negative effects (including accessibility issues related to either the service or the internet and its supporting technology).
Social impact of digitalisation and learnings from COVID-19	RQ6, RQ7, RQ8, RQ9	Evidence indicates not only that the pandemic accelerated the creation and uptake of digital services, some of which would not have been created otherwise, but also that most users will continue to use digital services with the same frequency in the future.

5. Conclusion

This review was commissioned to inform strategic and operational decisions around the design of DWP digital services. RAND Europe reviewed evidence around the impacts of digitalising social services, with a focus on costs and social impacts. 

Limitations of the evidence generated from digitalisation experiences so far and the extent to which this body of evidence can inform digitalisation strategies 

As part of this review, we found that digitalisation is occurring predominantly across a rather narrow set of public services and is focused mainly on simple transactions, rather than on the delivery of complex services. In addition, our findings highlight how producing accurate costs and savings estimates for the digitalisation of services is complicated, and they suggest that organisations could use a service-by-service approach to measure the financial and economic impact of digitalisation. 

We anticipated that the COVID-19 pandemic–accelerated digitalisation would increase the evidence base about the impact of digitalisation. We found relevant learning points related to organisations’ experiences while digitalising in reaction to the COVID-19 pandemic – in terms of increasing the number and uptake of new digital services and the long-term, continued use of digital services in the future. This is promising evidence, since the digitalisation experiences, if researched, could inform future digitalisation strategies. However, evidence generated from COVID-19-related digitalisation experiences is still limited, and it would merit being revisited in the years to come, especially in relation to long-term impact on target populations. 

Overall, we found that the evidence base that those in charge of welfare policy (in the UK and other countries) could use to inform the design and definition of any future digitalisation strategy is rather thin. This also means that there is an opportunity for digitalising organisations to contribute to building the evidence base in this area. This would suggest that a solid monitoring and evaluation plan is needed as part of the digitalisation strategy. 

Lessons learned from digitalisation experiences – limited opportunities for cost savings in some contexts 

Regarding the potential costs and savings related to staff, our research has found evidence of staff-related cost reductions and savings from digitalisation. However, the extent to which these savings can be realised is dependent on the need to factor in other considerations, particularly costs related to staff training and support. Our research has also found evidence of reduced costs and increased savings in service delivery from digitalisation. However, in some instances, the increased demand spurred by digitalisation offsets the cost reductions associated with the shift to digital service delivery. Evidence was also mixed on the costs and savings implications of failed digital channels, digital channel mixing and interoperability. This means that, while there are opportunities for the DWP (and those in charge of welfare policy in other countries) to save some costs by digitalising services, cost reduction should not be the only motivation for digitalisation, especially because wider, external factors can also affect costs. 

Lessons learned from digitalisation experiences – risks related to social impact 

In relation to the social impact of digitalisation, the evidence we found highlighted more areas of risks than of opportunities for those in charge of welfare policy in the UK and other countries. While digitalisation can present opportunities in relation to increasing uptake and building a larger customer base, a major social effect of digitalisation processes mentioned in the literature is the risk for increased inequity. While digitalising, DWP should ensure that vulnerable populations at the precarious end of the digital divide are not further excluded from the services that they were using prior to digitalisation (e.g. due to issues related to accessing either digital services or the internet and its supporting technology). 

Lessons learned from digitalisation experiences – customer experience 

Other organisations’ experiences with shifting customers onto digital channels – and the varying impacts of digital services on the customer  – indicate that digitalisation influences adoption of the service and the quality of the service. We found that there are multiple methods for transitioning customers successfully onto digital channels, including reassuring customers of the availability of non-digital options, as well as marketing and targeted engagement strategies. While these lessons learned are not necessarily directly applicable to the DWP context, they could inform digitalisation strategies, bearing in mind suggestions for designing the strategy and monitoring the effects of digitalisation on customers. 

In relation to customer experience, the evidence we found that is relevant to the DWP context relates to accessibility, trust, aesthetic experience and the context in which the service is used. Designing a digital service with a high level of adoption and continued use first requires careful attention to variations in preference and ability among and within population segments, including preferences on trust. Particular attention should also be paid to ensuring that services are visually appealing, as well as useful and easy to use, to facilitate both adoption and continued use of the service. Ultimately, however, the digital service must be designed so that it is appropriate for the context. If digital services are constructed in a manner that increases user burden or if they replace services that require urgent or very personal attention, they will not deliver the quality of service necessary to survive as substitutes for in-person services.

Appendix A. Methodology of the quick scoping review

This document outlines the databases, search terms, criteria and procedures that we used in the QSR. 

Objectives of this quick scoping review 

The objective of this quick scoping review (QSR) is to gather data from ‘grey’ (meaning unpublished) and academic literature to answer the following research questions: 

1. How can we best measure the financial, economic and social impacts of digitalising services? 
2. What is the cost trade-off between digital investment and savings on staff headcount? 
3. Are there successful examples of sharing or movement of data between different IT platforms, and what are the costs and benefits of this approach? 
4. Does including a digital channel enable more people to claim a benefit or service? 
5. Does a successful channel mix vary at different points of the customer journey, and how does this vary for groups with different protected characteristics? 
6. Learning from accelerated digitalisation due to COVID-19, are there any consistent emerging and sustainable ‘wins’ which could be considered for implementation in DWP services? 
7. How have organisations managed to shift customers onto online channels, and are customers who moved online during COVID-19 continuing to use online channels? 
8. What are the human and cost implications from customers having to use a digital channel where using another channel would mean a substantially improved outcome? 
9. What are the costs and customer experience levels of a successful channel mix, and how did organisations develop and establish a mix that worked for their customers? 

Steps followed as part of this quick scoping review 

The following steps were taken in the study’s QSR: 

Step 1: Identify and refine databases, search terms, and inclusion and exclusion criteria 

We carried out 5 searches in Web of Science and 5 in Google Scholar, reviewing only the first 30 results in each search, to identify relevant academic literature. 

We used Boolean search strings to ensure the relevance of results (see Error! Reference source not found.), drawing upon the search strings suggested in our proposal and refined in consultation with the DWP. Piloting of the search terms suggested that the most effective approach was to have a main set of search terms relating to digitalisation and to run this search independently and in combination with other, more specific search terms.

Table 5 Search terms

Used in Google Scholar	Main search terms relating to digitalisation (Search 1):	(“digital services” OR “online services” OR “online platform” OR “digital platform”) (“UK” OR “United Kingdom” OR “England” OR “OECD” OR “Singapore”)
Used in Google Scholar	Search relating to COVID-19 (Search 2)	(“digital services” OR “online services” OR “online platform” OR “digital platform”) (“covid” OR “corona) (“UK” OR “United Kingdom” OR “England” OR “OECD” OR “Singapore”)
Used in Google Scholar	Search relating to welfare services (Search 3):	(“digital services” OR “online services” OR “online platform” OR “digital platform”) (“welfare” OR “benefits” OR “unemployment” OR “pension”) (“UK” OR “United Kingdom” OR “England” OR “OECD” OR “Singapore”)
Used in Google Scholar	Search relating to costs and savings (Search 4)	(“digital services” OR “online services” OR “online platform” OR “digital platform”) (“cost savings”) (“UK” OR “United Kingdom” OR “England” OR “OECD” OR “Singapore”)
Used in Google Scholar	Search relating to social impact (Search 5)	(“digital services” OR “online services” OR “online platform” OR “digital platform”) (“customer” “social impact” “customer experience”) (“UK” OR “United Kingdom” OR “England” OR “OECD” OR “Singapore”)
Used in Google Scholar	Terms narrowing the scope (included in previous searches):	“UK” OR “United Kingdom” OR “England” “OECD” OR “Singapore”
Used in Web of Science	Main search terms relating to digitalisation (Search 6):	(“digital services” OR “online services” OR “online platform” OR “digital platform”) AND (“UK” OR “United Kingdom” OR “England” OR “OECD” OR “Singapore”)
Used in Web of Science	Search relating to COVID-19 (Search 7)	(“digital services” OR “online services” OR “online platform” OR “digital platform”) AND (“covid” OR “corona) AND (“UK” OR “United Kingdom” OR “England” OR “OECD” OR “Singapore”)
Used in Web of Science	Search relating to welfare services (Search 8)	(“digital services” OR “online services” OR “online platform” OR “digital platform”) AND (“welfare” OR “benefits” OR “unemployment” OR “pension”) AND (“UK” OR “United Kingdom” OR “England” OR “OECD” OR “Singapore”)
Used in Web of Science	Search relating to costs and savings (Search 9)	(“digital services” OR “online services” OR “online platform” OR “digital platform”) AND (“cost*” OR “savings”) AND (“UK” OR “United Kingdom” OR “England” OR “OECD” OR “Singapore”)
Used in Web of Science	Search relating to social impact (Search 10)	(“digital services” OR “online services” OR “online platform” OR “digital platform”) AND (“customer” OR “social impact” OR “customer experience”) AND (“UK” OR “United Kingdom” OR “England” OR “OECD” OR “Singapore”)
Used in Web of Science	Terms narrowing the scope (included in the previous searches):	(“UK” OR “United Kingdom” OR “England” OR “OECD” OR “Singapore”)

Step 2: Carry out the search 

We conducted the full search using the above parameters and collected all sources.

Step 3: Remove duplicates, screen and select sources for the review 

We removed duplicates that resulted from searches in multiple databases and then screened the title and abstract of each source against inclusion and exclusion criteria to confirm whether the source would be selected for full-text review (Table 6). A total of 44 sources proceeded to the extraction stage for closer analysis.

Table 6 Inclusion and exclusion criteria

Included in the quick scoping review	Excluded from the quick scoping review
Articles published in and after 2018	Articles published before 2018
Articles published in English	Articles published in languages other than English
Articles published that relate to the UK, to other OECD countries, and to distinctive digitalisation exemplars from non-OECD countries (e.g. Singapore)	Articles relating to non-OECD countries and non-distinctive digitalisation exemplars from non-OECD countries (e.g. Singapore)

Box 5 Search terms

For searches 1 to 5 (Google Scholar): 

• total: 30 (results) x 5 (searches) = 150 
• duplicates: 13 
• new total: 137 
• exclude: 117 
• include: 20 

For searches 6 to 10 (Web of Science):  

• total: 30 (results) x 5 (searches) = 150 
• duplicates: 7 
• new total: 143 
• exclude: 97 
• include: 45 
• new total: 65 

We found 2 sets of duplicates across the searches, leaving the final total number of articles to review as 63. 

Following the additional exclusion of 19 sources due to insufficient relevance or lack of access, 44 sources proceeded to the extraction stage for closer analysis.

Step 4: Review the full texts 

We reviewed the full text of 44 sources identified from steps 1 to 3 and extracted information relevant to the research questions. To structure each review, we developed a data extraction tool to record information from the reviewed sources.

Step 5: Fill in remaining evidence gaps 

To fill in any remaining gaps in the evidence, we conducted additional literature searches in Google Scholar and snowballed from existing sources. These searches were not subject to the same time and geographical limitations as those conducted initially, but sources which satisfied them were ultimately preferred.

Appendix B. Interviewees and interview protocol

This appendix provides details of the interviewees of for the study and of the interview protocol and topic guide.

Table 7 Interviewees

Interview code	Type of organisation
Interview 1	Public Sector (EU)
Interview 2	Local Authority (UK)
Interview 3	Local Authority (UK)
Interview 4	Local Authority (UK)
Interview 5	Public Sector (EU)
Interview 6-A	Local Authority (UK)
Interview 6-B	Local Authority (UK)
Interview 6-C	Local Authority (UK)
Interview 6-D	Local Authority (UK)

Interview protocol:

Section A: Introduction/background

Could you please tell me about your organisation and your role? 

• What does your organisation do? What services do you offer? 
• In what territories do you operate? 
• Who are your users/customers? 
• How many users/customers do you have? 
• What is your role?

The discussion today will focus on the process of digitalising services. What do you understand by the term digitalisation? 

For the purposes of this study, digitalisation “describes how IT or digital technologies can be used to alter existing business processes” (e.g. the creation of new online or mobile communication channels).^{[footnote 265]}  The term digitalisation describes “the action of converting analogue information into digital information” (e.g. the use of digital forms in ordering processes, the use of digital surveys).

To what extent and how do these terms (either or both) apply to your own organisation’s experience?

Section B: Overview of the organisation’s digitalisation experience and process

We are interested in understanding more about your organisation’s experience of digitalising services. 

Please could you start by telling me about the services digitalised by your organisation. 

• Which services (complexity, channel mix?) 
• For which customers? 

When and how was a decision reached to digitalise these services? 

• Who was involved in the decision making?

What prompted you to digitalise these services? What were your key motivations or ambitions? 

• How important were…?
  • Potential cost savings/financial impact
  • Customer expectations and preferences
  • Organisational culture
  • Digital services offered by competitor organisations

Do the digital services offered by your organisation sit alongside other (non-digital) services and if so, how? 

• Why did you decide on this channel mix? 
• Why did you decide not to digitalise all services? 
• Does the channel mix vary for different groups of customers and if so, why? 
• To what extent can customers decide which channel to use and why?

Please can you talk me through the process of shifting customers onto digital services in this case. 

• How did you establish which customers would move and when?
  • Type/group of customers (e.g. demographics, geography etc.)
  • Stage of customer journey (new customer, point of renewal etc.) 
• How did the process work? 
  • To what extent was it gradual? 
• What problems or challenges did you experience? 
  • How did you overcome these?

Were digital services tested or piloted prior to roll-out and if so, how? 

• With whom?
• How?
• What did you learn from this exercise?
• What (if anything) did you do differently as a result?

What issues or teething problems (if any) did you experience with the initial roll-out of digital services? 

• How did you address these issues?
• What (if anything) did you do differently?
  • What impact has this had?

Section C: Impact of digitalising services

Thank you for telling us a bit more about the process of digitalising services in your organisation. We will now ask a few questions to understand the impact of digitalising services on your organisation on cost, customers and process efficiency. 

What has been the financial impact of digitalising services for your organisation? 

• What are the key costs and savings? 
• What are the trade-offs? 

What impact has digitalising services had on staffing in your organisation? 

• Number of staff/headcount 
• Transitions to more-complex/higher-priority work 
• Use of machines/AI/automation

What impact has digitalising services had on customers/users? 

• Level of demand for services/number of customers
• Profile of customers
• Accessibility
  • For which groups is this an issue and why?
• Customer experience/satisfaction
  • How is this affected by the channel mix?
  • For which groups is this an issue and why?
  • What might be done to engage or retain these customers and improve customer satisfaction?
• The rate of user error/failure

What impact has digitalising services had on the quality of services provided by your organisation, including the efficiency of service provision? 

• Impact on failure rate, e.g. rejected claims, appeals, overpayments, complaints, progress chasing
• What might be done to improve the quality of digital services and efficiency of digital service delivery?

What impact has digitalising services had on processes for data sharing in your organisation?

• The extent of data sharing
  • Internal
  • External/with partners
• The efficiency and effectiveness of data sharing
• Can you give any examples of successful (including secure) data sharing?

Have you measured or quantified the impacts of digitising services in any of these categories? 

• Costs and savings
  • Staff headcount
  • System costs
  • Estate costs
  • IT maintenance costs
  • Channel mixing costs and savings
• Social impact
  • Customer experience
  • Speed/accuracy of transactions/interactions
• Failure demand/error
• If so, how? What did you learn?

Are you able to share any specific data or information about costs, savings and other outcomes associated with digitalising services in your organisation? 

What have you learned from the digitalisation process? 

• Issues/problems/mistakes

What would you do differently next time? 

• Aims/ambitions
• Process and approach

Section D: Impact of COVID-19

Did your organisation introduce or accelerate digitalisation processes in response to the COVID-19 pandemic and if so, how?

• What were the key changes?
• Which customers were affected?
• What has been the impact or outcomes for your organisation and your customers?
• To what extent will these changes be retained long-term (post-COVID)?

What did you learn from digitalising services in the context of the COVID-19 pandemic?

• To what extent will this change your approach long-term (post-COVID) and why?

Conclusion

Is there anything else that you would like to add that we haven’t mentioned so far?

1. Larsson, A and Teigland, R (2020) ‘An introduction to digital welfare: a way forward?’, in Larsson, A and Teigland, R (eds), Digital transformation and public services: societal impacts in Sweden and beyond, London: Routledge DOI:10.4324/9780429319297-1 ↩

2. Verhoef, P, Broekhuizen, T, Bart, Y and others (2021) ‘Digital transformation: a multidisciplinary reflection and research agenda’, Journal of Business Research, volume 122, pages 889-901 https://www.sciencedirect.com/science/article/pii/S0148296319305478, 891 ↩

3. Li, F, Nucciarelli, A, Roden, S and Graham, G (2016) ‘How smart cities transform operations models: a new research agenda for operations management in the digital economy’, Production Planning & Control, 27 (6), pages 514-528 ↩

4. Pagani, M and Pardo, C (2017) ‘The impact of digital technology on relationships in a business network’, Industrial Marketing Management, volume 67, pages 185-192 ↩

5. Verhoef and others (2021) ↩

6. Ibid ↩

7. Schiffhauer, B and Seelmeyer, U (2021) ‘Responsible digital transformation of social welfare organizations’, in Ifenthaler, D, Hofhues, S, Egloffstein, M and Helbig, C (eds), Digital transformation of learning organizations, Cham: Springer https://doi.org/10.1007/978-3-030-55878-9_8 ↩

8. Merriam-Webster Dictionary (2022) https://www.merriam-webster.com/dictionary/artificial%20intelligence ↩

9. Merriam-Webster Dictionary (2022) https://www.merriam-webster.com/dictionary/blockchain ↩

10. Merriam-Webster Dictionary (2022) https://www.merriam-webster.com/dictionary/cloud%20computing ↩

11. Merriam-Webster Dictionary (2022) https://www.merriam-webster.com/dictionary/data%20mining ↩

12. Star Sensors (2021) ‘Digital sensors’ (viewed on 16 May 2022) https://thestarsensors.com/a-comprehensive-guide-of-digital-sensors-applications-and-uses/ ↩

13. Merriam-Webster Dictionary (2022) https://www.merriam-webster.com/dictionary/Internet%20of%20Things ↩

14. Merriam-Webster Dictionary (2022) https://www.merriam-webster.com/dictionary/interactivity ↩

15. Merriam-Webster Dictionary (2022) https://www.merriam-webster.com/dictionary/interoperability ↩

16. Merriam-Webster Dictionary (2022) https://www.merriam-webster.com/dictionary/machine%20learning ↩

17. UiPath (2022) ‘Robotic process automation (RPA)’ https://www.uipath.com/rpa/robotic-process-automation ↩

18. Capgemini, IDC, RAND Europe, Sogeti, DTi (2017) ‘Digitizing public services in Europe: putting ambition into action’ (viewed on 23 June 2022) https://www.capgemini.com/wp-content/uploads/2017/07/Digitizing_Public_Services_in_Europe__Putting_Ambition_into_Action.pdf ↩

19. https://www.mckinsey.com/business-functions/strategy-and-corporate-finance/our-insights/how-covid-19-has-pushed-companies-over-the-technology-tipping-point-and-transformed-business-forever  (viewed on 23 June 2022) ↩

20. European countries include all EU member states, as well as associated countries.  European Commission (2021) ‘eGovernment benchmark 2021: entering a new digital government era’ (viewed on 4 April 2022) https://op.europa.eu/en/publication-detail/-/publication/6441f9b7-4376-11ec-89db-01aa75ed71a1/language-en/format-PDF/source-search ↩

21. Cabinet Office (2012) ‘Government Digital Strategy’, London: Cabinet Office (viewed on 4 April 2022) https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/296336/Government_Digital_Stratetegy_-_November_2012.pdf ↩

22. https://www.gov.uk/government/publications/the-government-technology-innovation-strategy (viewed on 23 June 2022) ↩

23. Ibid ↩

24. https://www.gov.uk/universal-credit (viewed on 23 June 2022) ↩

25. https://www.gov.uk/government/news/jets-job-scheme-relaunching-100-000-careers (viewed on 23 June 2022) ↩

26. Collins, AM, Coughlin, D, Miller, J and Kirk, S (2015) ‘The production of quick scoping reviews and rapid evidence assessments: a how to guide’, London: Government Publishing Service https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/560521/Production_of_quick_scoping_reviews_and_rapid_evidence_assessments.pdf ↩

27. The searches were not subject to the same time and geographical limitations as those conducted initially, to maximise the chances of locating critically important information. ↩

28. Larsson and Teigland (2020) ↩

29. Interview 3 – Local Authority (UK) ↩

30. Ranerup, A and Henriksen, H (2019) ‘Value positions viewed through the lens of automated decision-making: the case of social services’, Government Information Quarterly, volume 36, page 101377 ↩

31. Špaček, D, Csótó, M and Urs, N (2020) ‘Questioning the real citizen-centricity of e-government development: digitalization of G2C services in selected CEE countries’, The NISPAcee Journal of Public Administration and Policy, volume 13, issue 1, pages 213-243 ↩

32. Eurofound (2020) ‘Impact of digitalisation social services’, Luxembourg: Publications Office of the European Union ↩

33. Løberg, I (2021) ‘Efficiency through digitalization? How electronic communication between frontline workers and clients can spur a demand for services’, Government Information Quarterly, volume 38, page 101551 ↩

34. E-government (from electronic government) refers to the use of internet technology as a platform for exchanging information, providing services and transacting with citizens, businesses and other arms of government. See’ https://www.igi-global.com/dictionary/electronic-government-e-government/9385 ↩

35. Ranerup and Henriksen (2019) ↩

36. Ibid ↩

37. Eurofound (2020) ↩

38. Ibid ↩

39. Liang, J (2012) ‘Government cloud: enhancing efficiency of e-government and providing better public services’, International Joint Conference on Service Sciences, pages 261-265 ↩

40. Ranerup and Henriksen (2019) ↩

41. Ibid ↩

42. Ibid ↩

43. Ibid ↩

44. Ibid ↩

45. Ibid ↩

46. Information gathered from Trelleborg municipality website (as of 3 June 2022) https://www.trelleborg.se/ ↩

47. Deloitte (2018) ‘Global RPA Survey, Deloitte/2018 (viewed on 3 April 2022) https://www2.deloitte.com/bg/en/pages/technology/articles/deloitte-global-rpa-survey-2018.html ↩

48. Blueprint (2021) ‘Infographic: how much does RPA really cost?’ Blueprintsys, 14 September (viewed on 3 April 2022) https://www.blueprintsys.com/blog/rpa/how-much-does-robotic-process-automation-really-cost ↩

49. Rulesets are defined as ‘a table of instructions used by a controlled interface to determine what data is allowable and how the data is handled between interconnected systems’; NIST (2022) ‘Ruleset’ (viewed on 23 June 2022) https://csrc.nist.gov/glossary/term/ruleset ↩

50. Bhatt, N (2019) ‘Five design principles to help build confidence in RPA implementations’ EY/5 (viewed on 3 April 2022) https://www.ey.com/en_uk/consulting/five-design-principles-to-help-build-confidence-in-rpa-implement ↩

51. Ibid ↩

52. Blueprint (2021) ↩

53. Ranerup and Henriksen (2019) ↩

54. Ibid ↩

55. Ibid ↩

56. Lindgren, I, Madsen, C, Hofmann, S and Melin, U (2019) ‘Close encounters of the digital kind: a research agenda for the digitalization of public services’, Government Information Quarterly, volume 36, pages 427-436, 427 ↩

57. Lindgren and others (2019), 427 ↩

58. Eurofound (2020) ↩

59. Ibid ↩

60. Cepparulo, A and Zanfei, A (2021) ‘The diffusion of public eservices in European cities’, Government Information Quarterly, volume 38, page 101561 ↩

61. Interviewee 1 ↩

62. Cepparulo and Zanfei (2021) ↩

63. Interview 2 – Local Authority (UK) ↩

64. Warwick District Council (2011) ‘WDC channel strategy’ Warwick District Council, October 2011 (viewed on 4 April 2022) https://www.warwickdc.gov.uk/download/downloads/id/661/channel_strategy ↩

65. Kershaw, M (2020) ‘Channel choice vs. channel shift in the public sector’ Prodo, 1 September (viewed on 3 April 2022) https://www.prodo.com/blog/channel-choice-vs.-channel-shift-in-the-public-sector ↩

66. Cepparulo and Zanfei (2021) ↩

67. Interview 2 – Local Authority (UK) ↩

68. Špaček and others (2020) ↩

69. A whole-system approach is defined as responding to complexity through a dynamic way of working, bringing stakeholders, including communities, together to develop a shared understanding of the challenge and integrate action to bring about sustainable, long-term systems change. See Public Health England (2019) ‘A whole systems approach to obesity’, London: Public Health England ↩

70. Turner, KME, Looker, KJ, Syred, J, Zienkiewicz, A and Baraitser, P (2019) ‘Online testing for sexually transmitted infections: a whole systems approach to predicting value’, PLoS One, volume 22, issue 14(2) DOI:10.1371/journal.pone.0212420 ↩

71. Turner and others (2019) ↩

72. Ibid ↩

73. Ibid ↩

74. A whole system approach is defined as responding to complexity through a dynamic way of working, bringing stakeholders, including communities, together to develop a shared understanding of the challenge and integrate action to bring about sustainable, long-term systems change. See Public Health England (2019) ‘A whole systems approach to obesity’, London: Public Health England ↩

75. Turner and others (2019) ↩

76. Aspvall, K, Sampaio, F, Lenhard, F, Melin, K, Norlin, L, Serlachius, E and others (2021) ‘Cost-effectiveness of internet-delivered vs in-person cognitive behavioral therapy for children and adolescents with obsessive-compulsive disorder’ JAMA Network Open, volume 4, issue 7, pages e2118516-e2118516 ↩

77. Ibid ↩

78. Ibid ↩

79. Interview 2 – Local Authority (UK) ↩

80. Lapão, LV, Peyroteo, M, Maia, M, Seixas, J, Gregório, J, Da Silva, MM and others (2021) ‘Implementation of digital monitoring services during the COVID-19 pandemic for patients with chronic diseases: design science approach’, Journal of Medical Internet Research, volume 23, issue 8, page 24181 ↩

81. Schwaber, K (2004) ‘Agile project management with Scrum’, Redmond: Microsoft Press ↩

82. Central Digital and Data Office, Cabinet Office (2012) ‘Digital efficiency report’, Gov.uk, 6 November (viewed on 28 February 2022) https://www.gov.uk/government/publications/digital-efficiency-report/digital-efficiency-report ↩

83. Central Digital and Data Office, Cabinet Office (2012) ↩

84. Ibid ↩

85. Interview 3 – Local Authority (UK) ↩

86. Cepparulo and Zanfei (2021) ↩

87. Decman, M, Stare, J and Klun, M (2010) ‘E-government and cost-effectiveness: e-taxation in Slovenia’, Transylvanian Review of Administrative Sciences, volume 6, pages 48-57 ↩

88. Control budgets are a means of comparing actual income or expenditure with planned income or expenditure to identify whether or not corrective action is required. ↩

89. Kotarba, M (2017) ‘Measuring digitalization – key metrics’, Foundations of Management, volume 9, pages 123-138 ↩

90. Cepparulo and Zanfei (2021) ↩

91. Ibid ↩

92. Wright, J (2020) ‘Technology in social care: a review of the UK policy landscape’, Sheffield: University of Sheffield ↩

93. Wright (2020) ↩

94. Frennert, S (2019) ‘Lost in digitalization? Municipality employment of welfare technologies’, Disability and Rehabilitation: Assistive Technology, volume 14, issue 6, pages 635-642 ↩

95. Ibid ↩

96. United Nations Department of Economic and Social Affairs (2021) ‘Leveraging digital technologies for social inclusion’, Policy Brief No. 92 (viewed on 3 April 2022) https://www.un.org/development/desa/dpad/publication/un-desa-policy-brief-92-leveraging-digital-technologies-for-social-inclusion/ ↩

97. Ibid ↩

98. Office for National Statistics (2020) ‘Internet users: UK 2020’ (viewed on 21 April 2022) https://www.ons.gov.uk/businessindustryandtrade/itandinternetindustry/bulletins/internetusers/2020 ↩

99. United Nations Department of Economic and Social Affairs (2021) ↩

100. Interview 3 – Local Authority (UK) ↩

101. Interview 3 – Local Authority (UK) ↩

102. United Nations Department of Economic and Social Affairs (2021) ↩

103. Connecting Cambridgeshire (2022) ‘Connecting Cambridgeshire: delivering the digital infrastructure for Cambridgeshire and Peterborough’ (viewed on 3 April 2022) https://www.connectingcambridgeshire.co.uk/ ↩

104. Ibid ↩

105. Eurofound (2020) ↩

106. Voss, E and Rego, R (2019) ‘Digitalization and public services: a labour perspective’, Public Services International (viewed on 10 March 2022) https://publicservices.international/resources/publications/full-report—digitalization-and-public-services-a-labour-perspective?id=10382&lang=en ↩

107. Ibid ↩

108. Tomičić Furjan, M, Tomičić-Pupek, K and Pihir, I (2020) ‘Understanding digital transformation initiatives: case studies analysis’, Business Systems Research, volume 11, issue 1, pages 125-141, 133 ↩

109. Voss and Rego (2019) ↩

110. Pekkarinen, S, Melkas, H and Hyypiä, M (2019) ‘Elderly care and digital services: toward a sustainable sociotechnical transition’, in Toivonen, M and Saari, E (eds) Human-centered digitalization and services, Singapore: Springer ↩

111. Lapão, LV, Peyroteo, M, Maia, M, Seixas, J, Gregório, J, Da Silva, MM and others (2021) ‘Implementation of digital monitoring services during the COVID-19 pandemic for patients with chronic diseases: design science approach’, Journal of Medical Internet Research, volume 23, issue 8, page 24181 ↩

112. Anthony Jr, B (2021) ‘Implications of telehealth and digital care solutions during COVID-19 pandemic: a qualitative literature review’, Informatics for Health and Social Care, volume 46, issue 1, pages 68-83 ↩

113. Interview 4 – Local Authority (UK) ↩

114. Hardcastle (2021) ‘Cost of staff training: how much does it cost to train your employees?’, Baltic Apprenticeships (viewed on 3 April 2022) https://www.balticapprenticeships.com/blog/employers/cost-to-train-staff#:~:text=In%20the%20UK%2C%20employers%20invested,figure%20%E2%80%93%20don’t%20panic ↩

115. Ibid ↩

116. Central Digital and Data Office, Cabinet Office (2012) ↩

117. Raina, R, Nair, N, Kim Yap, H and others (2021) ‘Survey of telemedicine by pediatric nephrologists during the COVID-19 pandemic’, Kidney International Reports, volume 6, issue 9, pages 2316-2322 https://doi.org/10.1016/j.ekir.2021.06.026 ↩

118. Ibid ↩

119. Wright (2020) ↩

120. Voss and Rego (2019) ↩

121. Ibid ↩

122. Interview 6-A – Local Authority (UK) ↩

123. Naoum, P, Pavi, E and Athanasakis, K (2021) ‘Economic evaluation of digital health interventions in palliative care: a systematic review of the literature’, Frontiers in Digital Health, volume 3, page 730755 DOI:10.3389/fdgth.2021.730755 ↩

124. Rahimi K (2019) ‘Digital health and the elusive quest for cost savings’, The Lancet: Digital Health, volume 1, issue 3, pages e108-e109. ↩

125. Raina and others (2021) ↩

126. Heponiemi, T, Jormanainen, V, Leemann, L, Manderbacka, K, Aalto, AM and Hyppönen, H (2020) ‘Digital divide in perceived benefits of online health care and social welfare services: national cross-sectional survey study’, Journal of Medical Internet Research, volume 22, issue 7, page e17616 ↩

127. Neves, AL, van Dael, J, O’Brien, N and others (2021) ‘Use and impact of virtual primary care on quality and safety: the public’s perspectives during the COVID-19 pandemic’, Journal of Telemedicine and Telecare,DOI:10.1177/1357633X211066235 ↩

128. Eurofound (2020) ↩

129. Ibid ↩

130. Ranerup and Henriksen (2019) ↩

131. Eurofound (2020) ↩

132. Turner and others (2019) ↩

133. Løberg (2021), 1 ↩

134. Ibid ↩

135. Ibid,6 ↩

136. Voss and Rego (2019) ↩

137. Eurofound (2020) ↩

138. Ibid ↩

139. Roy, M (2021) ‘Why digital transformations fail: top 6 reasons’, TechTarget 22 (viewed on 3 April 2021) https://www.techtarget.com/searchcio/tip/Top-6-reasons-why-digital-transformation-failures-happen ↩

140. Ibid ↩

141. Brown, S (2019) ‘Forget ‘fail fast’. Here’s how to truly master digital innovation’, MITsloan 9 October (viewed on 4 April 2022) https://mitsloan.mit.edu/ideas-made-to-matter/forget-fail-fast-heres-how-to-truly-master-digital-innovation#:~:text=%E2%80%9CFail%20fast%E2%80%9D%20is%20a%20popular%20mantra%20at%20companies,at%20the%20MIT%20Center%20for%20Information%20Systems%20Research ↩

142. Roy (2021) ↩

143. Choudhury, S (2021) ‘Failed digital transformation projects costs businesses big’, Enterprise Talk, January 15 (viewed 2 March 2022) https://enterprisetalk.com/featured/failed-digital-transformation-projects-costs-businesses-big/ ↩

144. These findings are based on an online survey conducted in Q3 2020 by Vanson Bourne, an independent market research organization, of 450 heads of digital transformation, such as CIOs, CDOs and CTOs, in organizations with 1,000 employees or more in the USA, UK, France and Germany. See Couchbase (2020), ‘Is COVID-19 to blame for digital transformation challenges in 2020? Santa Clara, CA: Couchbase https://info.couchbase.com/rs/302-GJY-034/images/CIO_Survey_2020.pdf ↩

145. Interview 1 – Public Sector (EU) ↩

146. Raso, J (2021) ‘Implementing digitalisation in an administrative justice context’, in Hertogh, M, Kirkham, R, Thomas, R and Tomlinson, J (eds), The Oxford Handbook of Administrative Justice, Oxford: Oxford University Press, 521 ↩

147. Interview 1 – Public Sector (EU) ↩

148. Henley, M (2021) ‘Learning from digital transformation failures’ Reworked.co, 23 September (viewed on 3 April 2022) https://www.reworked.co/leadership/learning-from-digital-transformation-failures/ ↩

149. Pavlou, C (2022) ‘Digital transformation: how to ensure it won’t fail’, Workable (viewed on 3 April 2022) https://resources.workable.com/stories-and-insights/change-management-digital-transformation ↩

150. Case, K (2000) ‘How to explain tech debt in plain English’, The Enterprisers Project (viewed on 15 May 2022) https://enterprisersproject.com/article/2020/6/technical-debt-explained-plain-english ↩

151. Pavlou (2022) ↩

152. Løberg (2021) ↩

153. Ibid ↩

154. Ibid ↩

155. Fugletveit, R and Lofthus, A-M (2021) ‘From the desk to the cyborg’s faceless interaction in the Norwegian Labour and Welfare Administration’, Nordisk välfärdsforskning | Nordic Welfare Research, volume 6, issue 10, page 18261 ↩

156. Ibid, 90 ↩

157. Australian Department of Skills, Education and Employment (2022) ‘Digital servicing and profiling in the new employment services model’, Paris: OECD (viewed on 21 April 2022) https://www.oecd.org/els/emp/Drayton_Digital-Servicing_profiling-New-Employment-Services-Model.pdf ↩

158. Ibid ↩

159. Turner and others (2019) ↩

160. Ibid ↩

161. Løberg (2021) ↩

162. Eurofound (2020) ↩

163. Interview 3 – Local Authority (UK) ↩

164. Department for Digital, Culture, Media and Sport (2017) ‘UK Digital Strategy’, Gov.uk, 11 March (viewed on 4 April 2022) https://www.gov.uk/government/publications/uk-digital-strategy ↩

165. Kershaw (2020) ↩

166. Warwick District Council (2011) ↩

167. Local Government Association (2020) ‘Encouraging channel shift through digital take up’, Local.Gov.uk, January (viewed on 20 April 2022) https://www.local.gov.uk/case-studies/encouraging-channel-shift-through-digital-take ↩

168. Warwick District Council (2011) ↩

169. Local Government Association (2020) ↩

170. Local Government Association (2020) ↩

171. Eurofound (2020) ↩

172. Ibid ↩

173. Larsson and Teigland (2020) ↩

174. Gil-Garcia, J, Chengalur-Smith, I and Duchessi, P (2007) ‘Collaborative e-government: impediments and benefits of information-sharing projects in the public sector’, European Journal of Information Systems, volume 16, issue 2, pages 121-133, 121 ↩

175. Interview 1 – Public Sector (EU) ↩

176. Eurofound (2020) ↩

177. Baker, LC (2005) ‘Benefits of interoperability: a closer look at the estimates’, Health Affairs, volume 24, Supplement 1 https://www.healthaffairs.org/doi/full/10.1377/hlthaff.W5.22 ↩

178. Baker (2005) ↩

179. Norwegian Ministry of Local Government and Modernisation (2019) ‘One digital public sector – digital strategy for the public sector 2019–2025’, Oslo: Ministry of Local Government and Modernisation (viewed on 4 April 2022) https://www.regjeringen.no/en/dokumenter/one-digital-public-sector/id2653874/ ↩

180. Norwegian Ministry of Local Government and Modernisation (2019) ↩

181. 86%, 4 out of 5 stars; 64%, 5 out of 5 stars ↩

182. Interview 6D – Local Authority (UK) ↩

183. Interview 2 – Local Authority (UK) ↩

184. Interview 4 – Local Authority (UK) ↩

185. Interview 3 – Local Authority (UK) ↩

186. Interview 5 – Public Sector (EU) ↩

187. Interview 5 – Public Sector (EU) ↩

188. McMullin, C (2021) ‘Migrant integration services and coping with the digital divide: challenges and opportunities of the COVID-19 pandemic’, Voluntary Sector Review, volume 12, issue 1, pages 129-136 ↩

189. Harris, J (2020) ‘The digitization of advice and welfare benefits services: re-imagining the homeless user’, Housing studies, volume 35, issue 1, pages 143-162 ↩

190. Ibid ↩

191. Willard, S, van Rossum, E, Spreeuwenberg, M and de Witte, L (2020) ‘Perceived impact of an online community care platform for Dutch older adults on local participation, informal caregiving, and feelings of connectedness: pretest-posttest observational study’, Journal of Medical Internet Research volume 22, issue 12, page e20304 DOI:10.2196/20304 ↩

192. Anderberg, P, Barnestein-Fonseca, P, Guzman-Parra, J, Garolera, M, Quintana, M, Mayoral-Cleries, F, Lemmens, E and Sanmartin Berglund, J (2019) ‘The effects of the digital platform Support Monitoring and Reminder Technology for Mild Dementia (SMART4MD) for people with mild cognitive impairment and their informal carers: protocol for a pilot randomized controlled trial’, JMIR Research Protocols, volume 8, issue 6, page e13711 DOI:10.2196/13711 ↩

193. Willard and others (2020) ↩

194.  McMullin (2021) ↩

195.  Heponiemi and others (2020) ↩

196.  McMullin (2021) ↩

197.  Ibid ↩

198.  Wright (2020) ↩

199.  McMullin (2021) ↩

200.  Harris (2020) ↩

201.  Willard and others (2020) ↩

202.  McMullin (2021) ↩

203.  Heponiemi and others (2020) ↩

204.  Interview 1 – Public Sector (EU) ↩

205.  Interview 4 – Local Authority (UK) ↩

206.  Willard and others (2020), 9 ↩

207.  Neves and others (2021) ↩

208.  Ibid ↩

209. Gefen, D and Straub, DW (2004) ‘Consumer trust in B2C e-commerce and the importance of social presence: experiments in e-products and e-Services’, Omega, volume 32, issue 6, pages 407-424, 408, DOI:10.1016/j.omega.2004.01.006 ↩

210. Polzer, T and Goncharenko, G (2021) ‘The UK COVID-19 app: the failed co-production of a digital public service’, Financial Accountability and Management, volume 38, issue 2, pages 1-18 https://doi.org/10.1111/faam.12307 ↩

211.  Ibid ↩

212.  Ibid ↩

213.  McMullin (2021) ↩

214.  Klein, A, Sørensen, C, Sabino deFreitas, A, Drebes Pedron, C and Elaluf-Calderwood, S (2020) ‘Understanding controversies in digital platform innovation processes: the Google Glass case’ Technological Forecasting and Social Change, volume 152, page 119883 https://doi.org/10.1016/j.techfore.2019.119883 ↩

215.  Interview 5 – Public Sector (EU) ↩

216.  Interview 1 – Public Sector (EU) ↩

217.  Interview 3 – Local Authority (UK) ↩

218.  Interview 1 – Public Sector (EU) ↩

219. NB: Trust was included as a factor in Zhang and others (2018), but accessibility was not. See Zhang, H, Gordon, S, Buhalis, D and Ding, X (2018) ‘Experience value cocreation destination online platforms’, Journal of Travel Research, volume 57, issue 8, pages 1093-1107 DOI:10.1177/0047287517733557 ↩

220.  Zhang and others (2018) ↩

221.  See, for example, Rukshan, A and Baravalle, A (2011) ‘A quantitative approach to usability evaluation of web sites’, London: Advances in Computing Technology; Fernandez, A, Insfran, E and Abrahão, S (2011) ‘Usability evaluation methods for the web: a systematic mapping study’, Information and Software Technology, volume 53, issue 8, pages 789-817 ↩

222.  Zhang and others (2018), 1095 ↩

223.  Zhang and others (2018), 1102 ↩

224.  Ibid ↩

225.  Ibid ↩

226.  van der Heijden, H (2003) ‘Factors influencing the usage of websites: the case of a generic portal in the Netherlands’, Information & Management, volume 40, issue 6, pages 541-549 https://doi.org/10.1016/S0378-7206(02)00079-4; Zhang and others (2018) ↩

227.  van der Heijden (2003); Zhang and others (2018) ↩

228.  Chung, N, Lee, H, Lee, SJ and Koo, C (2015) ‘The influence of tourism website on tourists’ behavior to determine destination selection: a case study of creative economy in Korea’, Technological Forecasting and Social Change, volume 96, pages 130-143 https://doi.org/10.1016/j.techfore.2015.03.004 ↩

229.  Mota, FPB, Bellini, CGP, Souza, JM and Oliveira, TJN (2016) ‘The influence of civic mindedness, trustworthiness, usefulness, and ease of use on the use of government websites’, Revista de Administração, volume 51, issue 4, pages 344-354 ↩

230.  Ibid ↩

231.  Interview 1 – Public Sector (EU) ↩

232.  Chung and others (2015) ↩

233.  Zhang and others (2018) ↩

234.  Ibid ↩

235.  Hofacker, CF and Corsaro, D (2020) ‘Dystopia and utopia in digital services’, Journal of Marketing Management, volume 36, issue 5-6, pages 412-419, 416 DOI: 10.1080/0267257X.2020.1739454 ↩

236.  Ibid ↩

237.  Interview 2 – Local Authority (UK) ↩

238.  Interview 2 – Local Authority (UK) ↩

239.  Wright (2020) ↩

240.  Willard and others (2020), 1 ↩

241.  McMullin (2021) ↩

242.  Heponiemi and others (2020) ↩

243.  Harris (2020), 154 ↩

244.  Raina and others (2021) ↩

245.  Hofacker and Corsaro (2020), 415 ↩

246. Raina and others (2021) ↩

247.  Ibid ↩

248.  Ibid ↩

249.  Harris (2020) ↩

250.  Neves and others (2021) ↩

251.  Interview 3 – Local Authority (UK) ↩

252.  Interview 3 – Local Authority (UK) ↩

253.  McMullin (2021) ↩

254.  Interview 3 – Local Authority (UK) ↩

255.  They also had household incomes lower than £25,000. ↩

256.  https://ageing-better.org.uk/sites/default/files/2021-07/COVID-19-and-the-digital-divide.pdf ↩

257.  McMullin (2021),134 ↩

258.  Ibid,134 ↩

259.  Interview 6A – Local Authority (UK) ↩

260.  Rezaeibagha, F, Than Win, K and Susilo, W (2015) ‘A systematic literature review on security and privacy of electronic health record systems: technical perspectives’, Health Information Management Journal, volume 44, issue 3, 23-38 ↩

261.  Survey of approximately 20,000 respondents conducted from April to May 2020. 17 European countries were included (Belgium, Czech Republic, Denmark, Finland, France, Germany, Hungary, Italy, the Netherlands, Norway, Poland, Portugal, Romania, Spain, Sweden, Switzerland, and the United Kingdom). The survey asked consumers about their online activity in 10 different industries: banking, insurance, grocery, apparel, entertainment, social media, travel, telecommunications, utilities, and the public sector; Fernandez, S, Jenkins, P and Vieira, B (2020) ‘Europe’s digital migration during COVID-19: getting past the broad trends and averages’, McKinsey (viewed on 29 October 2021) mckinsey.com/business-functions/mckinsey-digital/our-insights/europes-digital-migration-during-covid-19-getting-past-the-broad-trends-and-averages ↩

262.  Interview 2 – Local Authority (UK) ↩

263.  Interview 3 – Local Authority (UK) ↩

264.  No figure is available for the insurance industry; Fernandez and others (2020) ↩

265. Verhoef and others (2021) ↩