Research and analysis

Introduction

Updated 21 March 2024

This is an overview of a programme of 16 pilot projects to support self-isolation during the COVID-19 pandemic. It provides the background to the programme, describes the approach to evaluation and the key evaluation findings, and concludes with lessons learnt from the programme.

Background

Self-isolation of people who were aware they had COVID-19 was an integral part of the response of the UK Government to the pandemic^{[footnote 1]}. From 28 September 2020 until 24 February 2022, individuals in England were under a legal duty to isolate if they tested positive for COVID-19. In addition, all close contacts of positive cases who were reached by NHS Test and Trace (NHSTT) were legally required to self-isolate up until 16 August, 2021; after that and until 24 February 2022, contacts who were not exempt (either through being fully vaccinated or because they were under 18 years old) were still legally required to self-isolate^{[footnote 2][footnote 3]}.

To improve compliance with self-isolation and engagement with the NHSTT service, in March 2021 the UK Health Security Agency (UKHSA) initiated a programme of pilot interventions for communities disproportionately impacted by COVID-19 and in areas of enduring transmission (ET), and variant of concern (VoC) outbreaks. The pilots were conducted at a modest scale, consisted of locally designed and delivered initiatives that were tailored to suit participating local authorities with the dual aim of addressing pressing local needs and generating learning about impact and implementation. The intention was to inform the evidence base of what works and encourage innovative approaches beyond central government’s field of view to be tested, evaluated and shared with local decision-makers.

The programme’s objectives were to:

• help local government and delivery agencies share and learn from what works to improve their local responses to ET and VoC outbreaks
• ensure policy was informed by evidence on approaches to increasing self-isolation and engagement with the NHSTT service
• reach a consistent standard and approach for central support of innovation in tackling COVID-19 in communities
• help the NHSTT service to focus effort on supporting the communities and localities that were most at risk of enduring transmission and/or variant of concern outbreaks

The pilots were categorised as:

1. Crowd-sourced pilots: organisations were enabled - in consultation with UKHSA - to pilot initiatives to prevent the spread of COVID-19 in an ET or VoC area using the ‘What Works’ research pack, and to evaluate the approach; there was a funding ceiling of £50,000 per pilot.
2. Commissioned pilots: on behalf of government, UKHSA invited local authorities to pitch for funding by sending in proposals for pilot interventions that might be able to influence policy as a result of new evidence. These so-called commissioned pilots varied in the amount of funding given and the length of the pilot, and the initiative for the commissioned pilots came from the pilot areas themselves.
3. Unfunded pilots: these pilots were funded locally.

To address policy questions identified as important by UKHSA and the Department of Health and Social Care (DHSC), pilots were designed to test hypotheses of barriers to engagement with NHSTT and compliance with self-isolation. These included (but were not limited to) the following:

• personal finance: the Test and Trace Support Payment scheme does not provide sufficient confidence for individuals that loss of earnings will be covered if they must self-isolate, either because £500 does not reflect the full extent of lost earnings or they cannot be certain in advance if they will be eligible
• practical and emotional support: individuals have concerns over their practical ability to self-isolate, either because they don’t think they will get support with food delivery, dog walking, for example, or because of the impact on their mental wellbeing
• living arrangements: those living in houses of multiple occupancy or in larger, multi-generational environments are unable to effectively self-isolate and are more likely to increase transmission through household contacts as they have no alternative accommodation
• communication and motivation: some people do not see self-isolation as being important or necessary and are unaware of the support available if they have to self-isolate
• speed of contact tracing: some contacts do not find out that they are required to self-isolate until many days after their contact with someone who has tested positive. It can take several days from that contact before the infected person gets tested, receives their result and is contacted by the relevant team
• enforcement: enforcement processes have resulted in very few police actions, limiting the intended deterrent impact. However, scaling up the police role is likely to mean greater reluctance to get tested in the first place, particularly among the most disadvantaged and disproportionately affected groups

Commissioned pilots

Cheshire and Merseyside

• pilot ran between August 2021 and March 2022
• enhanced contact tracing and effective rapid support to positive cases and contacts
• improve local tracing techniques and standards
• large-scale public communications campaign
• reduced time from test to self-isolation offer

Greater Manchester       

• pilots ran in 10 local authorities between May and September 2021
• the majority of local authorities embarked on a communication campaign; scripted and non-scripted calls to contacts and cases informing them of the pilot
• person-centre social care support plans
• variety of wrap-around support including, but not limited to, alternative accommodation, medicine delivery, welfare call and/or visits, financial advice, book deliveries from local libraries

Lancashire

• series of mini-pilots across 2 ‘sprint’ periods, each lasting a few months between November 2021 and February 2022
• targeted local communications and collaboration with employers and local employment partnerships
• practical support including food parcels and prescription deliveries
• facilitated access to bespoke professional services focussed on mental wellbeing and longer-term support

London Borough of Hackney

• pilot ran between 8 July and 30 November 2021
• provided enhanced self-isolation package in high density housing areas with the provision of external accommodation
• connected cases to pathways of support offering care packages, welfare calls, food and grant support

London Borough of Newham

• pilot ran between 1 March and 30 November 2021
• provide alternative accommodation in complex multigenerational housing
• wraparounds’ (subsistence, travel to/from work)
• enhance social care support and communication strategy

Peterborough, Fenland and South Holland

• pilot ran between June and September 2021
• improve access to T&T support payments
• target employers and contract security for workers
• provision for alternative accommodation

Yorkshire and Humber

• pilot ran in 15 local authorities between November 2021 and February 2022
• targeted communication campaign through community leaders, focussed on aligning with changes to COVID-19 restrictions, information about new COVID-19 variants, self-isolation, travel guidelines and testing

Crowd-sourced pilots

Barnsley

• pilot ran between 18 October and 15 November 2021
• provide wrap around support and testing of the identified cohorts
• improved communications approach within migrant communities

Calderdale

• support online local food ordering and delivery for residents isolating or vulnerable including foodbank and voucher schemes

Gypsy, Traveller, and nomadic communities

• pilot ran from October 2021 until the end of December 2021
• connect members of Gypsy and Traveller communities with support in order to help access testing processes and engage in self-isolation procedures

Kingston

• develop a pathway for local contact tracing within Kingston Hospital (inpatients and outpatients)
• support more potentially vulnerable residents
• enhance local ‘exposure’ data for hospitalised cases
• pilot was limited to a small number of cases, and did not meet the critical mass required to merit a formal evaluation, hence no report was produced

Somerset

• pilot ran for 12 weeks from May to August 2021
• test various initiatives with migrant workers

Unfunded pilots

Bradford

• widen eligibility to TTSP

Kirklees

• income replacement, increased maximum payment, reduced evidence burden

London Borough of Havering

• pilot ran between 7 June and 18 July 2021
• upskilled testing operatives visited all positive cases within the community twice across the 10-day isolation period to complement T&T calls

Wandsworth and Lambeth

• widen eligibility to Test and Trace support payments (TTSP)

Evaluation framework

A pragmatic and proportionate evaluation framework was developed that supported an agile delivery process while aiming to maximise learning. It was envisaged that the pilots would differ in terms of the amount of support they would receive and, hence, expectations about the robustness of the evaluation approach. Crowd-sourced pilots would generally receive guidance and light-touch support; commissioned-pilots were required to have robust evaluation approach that were undertaken by externally commissioned research organisations; VoC pilots required an evaluation approach that was responsive to the rapid pace of implementation and the need to quickly generate insights.

In line with HM Treasury’s guidance on policy evaluation, the Magenta Book, there were 3 key aspects of the pilots that the evaluations aimed to address:

1. Implementation

Questions within this area focused on whether the initiatives were implemented as intended, the barriers and facilitators to doing so, and the extent of take up.

2. Experiences and mechanisms

This covered questions such as how did people find out about the initiatives (for example, through publicity or word-of-mouth) and how that influenced their attitudes and behaviour; how did they experience the initiatives, including accessing or applying for them, what difference the payments or support had on their ability to self-isolate and whether the initiatives made it more or less difficult to isolate.

3. Impact

Answering questions related to measuring the impact of the pilots involved identifying key outcomes, measuring change in the outcomes and then attributing those changes to the initiatives.

Randomised controlled trials (RCTs) are generally considered to be the gold standard in impact evaluation, but it was felt that it would be too challenging to set these up within the settings and timeframe in which the pilots were implemented. Therefore, it was envisaged that the evaluations would largely use quasi-experimental methods such as difference-in-differences analysis, synthetic control and propensity score matching.

The approaches that were used ranged from substantial quasi-experimental analysis and process and implementation research, those that primarily used using case studies and qualitative methods, to those that relied on a light touch review of admin data. One of the pilots, the London Borough of Havering, was able to set up an RCT.

An overarching theory of change was developed to capture the key elements of how the pilots were expected to work to help inform pilot development and evaluation approaches, shown in Figure 1.

Figure 1. Self-isolation pilot programme overarching theory of change

Text version of Figure 1

Step 1 consisted of 3 elements:

1. Inputs: funding, capacity.
2. Activities: increase support for self-isolation.
3. Outputs: awareness of traditional support and take-up of additional support.

These may be affected by moderating factors such as age, gender, deprivation, receipt of means-tested benefits, trust in government, caring responsibilities, employer attitudes and approach.

Step 2
Mediating mechanism: people less concerned about self-isolation. This leads to 2 types of outcome:

1. Indirect outcome: increased engagement with Test and Trace.
2. Direct outcome: increased self-isolation compliance.

It can also lead to backfire effects such as increased fraudulent claims.

Step 3
Both indirect and direct outcomes lead to 2 impacts: increased case detection and reduced transmission.

End of text version of Figure 1

Initiatives related to supporting self-isolation were anticipated to improve compliance of confirmed positive cases and their contacts who were reached by NHS Test and Trace. There were also expected to be a range of ways in which the initiatives could support engagement with NHS Test and Trace, including increasing the proportion of symptomatic cases accessing tests, the number and quality of contacts shared by positive cases and the proportion of contacts that respond to being traced.

The evaluation framework specified key outcomes of interest to UKHSA and the Government along with identifying outcome metrics and data that was available to support the evaluation of the pilots, as shown in Figure 2.

Figure 2. Self-isolation pilot outcomes and metrics

Text version of Figure 2

Step 1. Engagement with testing leads to cases responding to contact tracers and identifcation of testing rate of those in the lower three IMD deciles (metric).

Step 2. Cases responding to contact tracers leads to sharing of contacts and estimation of proportion of cases reached (metric).

Step 3. Sharing of contacts leads to contacts responding to contact tracers and estimation of proportion of cases sharing at least one contact which in turn leads to estimation of mean numb contacts shared (metrics).

Step 4. Contacts responding to contact tracers leads to contact and cases complying with self-isolation and estimation of proportion of contacts notified (metric).

Step 5. Contacts and cases complying with self-isolation leads to estimation of proportion of contacts or cases with 100% success in check-in calls and proportion of isolating cases (contacts) with no non-household contacts (metrics).

End of text version of Figure 2

The data for these metrics was drawn from the Contact Tracing and Advisory Service (CTAS) database, held by NHS Test and Trace. The advantage of using this data was that it had a high degree of geographical granularity (as the postcode of each individual tested or traced was recorded), which enabled the pilot areas to be compared to areas where pilots were not taking place. The key indicators used in the pilot evaluations were the following.

1. Engagement with testing: the indicator for engagement was the testing rate of those in the lowest three deciles according to the indicators of multiple deprivation, with the number of tests in those areas being divided by the population based on ONS census data.
2. Cases responding to contact tracers: this indicator was the proportion of individuals who tested positive (cases) who responded to contact tracers attempts to reach them for contact tracing purposes.
3. Sharing contacts: there were 2 indicators for this metric, the first was the proportion of cases who shared at least one contact when reached and the second was the average number of contacts disclosed by cases.
4. Contacts responding to contact tracers: this indicator was the proportion of contacts whose records indicated that they had been successfully reached by contract tracers.
5. Contacts and cases complying with self-isolation: There were 2 indicators for this metric. The first was the outcome of self-isolation check-in calls to positive cases on days 4, 7 and 10. A case was considered to be complying with self-isolation if they responded to check-in calls on all 3 days and also confirmed verbally that they were self-isolating each time. The second indicator was based on contacts that re-entered the system during their isolation period having tested positive for COVID-19. The proportion of these ‘isolating cases’ that had no non-household contacts gave an indication of isolation compliance among contacts.

Each pilot and evaluation was expected to review the key indicators and identify which were most appropriate for their intervention. Pilots could also identify additional outcomes (such as reducing the negative impact of self-isolation), though indicators for these outcomes were not available in the NHS Test and Trace database. This was a recognised limitation of the evaluation framework.

Findings

This section sets out the findings of the pilots. The pilots are grouped by the broad approach or type of intervention that was used.

Variations to the Test and Trace Support Payment

Three pilots trialled varying the Test and Trace Support Payment, which was put in place to help people on low incomes who would lose out financially as a result of isolating. Kirklees Local Authority reduced the evidence burden needed to prove loss of income and raised the ceiling on the amount that could be claimed up to a maximum of £1,000 but no evidence of impact was found. Bradford Local Authority widened eligibility for TTSP to include those who earned up to £500 per week (from a threshold of £350 per week). There was no increase in the number of applications, but there was a positive impact on compliance with an increase in number of tracing calls answered but a negative impact on the number of contacts shared. Wandsworth and Lambeth Local Authorities widened eligibility for TTSP to those earning up to £30,000 per year. Again there was no increase in the number of applications and no overall impact on compliance but there may have been a small, positive impact on testing rates during the first weeks of the pilot.

Practical support

A range of pilots were aimed at increasing practical support for those isolating in various ways. A large pilot in Greater Manchester which offered enhanced and tailored support reached over 45,000 in total, but no impacts were detected on engagement or compliance. A smaller pilot in the London Borough of Hackney also tested enhanced and tailored support but had a low take up of the offer (less than 30 residents), while the London Borough of Newham introduced telephone welfare checks, but no evidence of impact was found. Newham also offered hotel accommodation to those who were isolating, but less than 50 people accepted the offer. The London Borough of Havering tested the effectiveness of in-person visits to cases that were isolating and a positive impact was found, though there was no evidence of any effect on take up of TTSP.

Communication and engagement

Three larger pilots focused on improved communication and innovative approaches to engagement. Peterborough, Fenland and South Holland aimed to introduce a suite of measures, though changes in the expected population undermined the reach of the pilot. However, it did partner with a third sector organisation to better engage with low paid migrant and insecure workers and evidence of a positive impact on compliance was found. Neither a pilot in Yorkshire and Humber testing a behaviourally informed communications campaign nor a pilot in Cheshire and Merseyside that focused on an enhanced contact tracing system and improved targeting of communications found any evidence of impact.

Three additional pilots of contact tracers in hospitals in the Royal Borough of Kingston, an online shopping platform in Calderdale and increased testing of migrant workers in Somerset and Barnsley, all experienced low take-up of the services. The crowd sourced pilot in the Royal Borough of Kingston did not meet critical mass of cases to warrant a formal evaluation.

Conclusion

The programme of self-isolation support pilots consisted of locally-delivered initiatives based on contextual knowledge and tailored approaches rather than top-down one-size-fits-all type interventions that could be imposed on a national scale. In adopting the former approach, it was recognised that it might be hard for Local Authorities to deliver their pilots, and that pilots might be conducted at a modest scale and with sampling techniques that would not be conducive to generalisable findings. However, on balance it was felt that it was important to encourage innovative approaches and that the benefits of this in terms of identifying new evidence about what works outweighed the risks.

The pilots were developed and introduced during a national crisis within a continually changing environment. This meant it was difficult to implement the pilots and Local Authorities spoke about how important it was to remain flexible and be willing to adapt their interventions to changing circumstances. They also emphasised the importance of collaboration across geographic and administrative borders, and with the local voluntary, community and social enterprise sector, which was seen as knowing local communities and being trusted by them. Despite this collaboration, a key theme across the pilots was that it was often difficult to engage communities and that lower-than-expected reach was a common problem.

A number of the pilots did find evidence of positive impacts, but in general it proved difficult to improve compliance with self-isolation or engagement with NHS Test and Trace as measured by testing rates or numbers of contacts shared. For many pilots, however, these outcomes were part of a broader set of aims related to building a sense of community and solidarity and reducing some of the negative effects of self-isolation. Unfortunately, the data was not available to assess the impact of the pilots on these outcomes but given the context within which the pilots were operating, the importance of these wider goals should not be underestimated.

1. Department of Health and Social Care (2021) ‘Removing barriers to self-isolation and improving adherence’ ↩

2. Moving to step 4 of the roadmap ↩

3. Self-isolation removed for double-jabbed close contacts from 16 August (press release) ↩