Research and analysis

Preface by Professor Dame Theresa Marteau

Updated 21 March 2024

Preface

The publication of these reports is to be welcomed for some fundamental lessons they offer in evaluating policies both in and outside of emergencies.

The reports describe 16 pilot initiatives to increase low rates of self-isolation in communities most exposed to infection during COVID-19 but with the fewest financial, material and social resources needed to self-isolate when infected. Optimising rates of self-isolation was central to NHS Test and Trace’s objective to prevent transmission of SARS-CoV-2.

At the time the pilots were initiated by the Department of Health and Social Care (DHSC) in March 2021, rates of self-isolation were estimated to be between 30% and 50%, and lower amongst those on low incomes. More precise estimates were not available due to the absence of standardised measures of this key variable.

The 16 pilots varied in their focus, location and duration. Support offered included:

• help to access government payments of £500 for self-isolation
• help with shopping and dog-walking
• offers of emotional support
• access to temporary accommodation in which to self-isolate

The pilots took place across the UK, varying in duration from one to 8 months, and were all independent of each other.

This set of reports comprises a carefully conducted and well-documented evaluation. Mindful of the limitations, the low rates of self-isolation in these communities seemed largely unchanged by these 16 pilots. If this seems counter-intuitive, it is worth noting that the pilots were designed and managed locally; a trade-off for decentralised control is smaller scale interventions with lower statistical power making it harder to demonstrate any other than large effect sizes.

Evaluating interventions during emergencies is difficult but not impossible. It happens when systems for robust evaluation are the norm before the emergency, as is the case for vaccines and medicines, a norm built on regulations. For non-pharmaceutical interventions (NPI) there are clear guidelines for evaluation – including the UK Government’s Magenta Book. The Events Research Programme is an example of an NPI that was evaluated comprising a co-ordinated set of studies assessing the risk of transmission of COVID-19 from attending mass events such as football matches and festivals.

Effective policy-making requires evidence to be used and generated across the 3 stages outlined in the Magenta Book: first, in the design of a policy, which should show credible estimates of the quantified contribution of each intervention towards achieving the goal; second, in monitoring a policy once implemented so that if it is off-track, it is modified to get back on track; and third, outcome evaluation, so that evidence accumulates on the effectiveness of each policy.

Focusing on this third stage, a National Audit Office report identified barriers to better provision and use of evaluation by government departments and made a number of recommendations for improving evaluation of major government spending projects. The Evaluation Task Force, a joint unit between Cabinet Office and HM Treasury, has been set up to address these.

We – policy-makers and researchers – need to heed the fundamental lessons which these timely reports on 16 pilot studies provide and strengthen our existing systems for using and generating evidence for effective policies at national and local levels.

Such strong systems will ensure first, that interventions intended to achieve a goal – in this case increasing rates of self-isolation – could credibly do so based on existing evidence, and second, that robust evaluations of interventions become defaults both outside and inside of emergencies.

Professor Theresa Marteau

Professor Theresa Marteau DBE FMedSci is the Director of the Behaviour and Health Research Unit, University of Cambridge School of Clinical Medicine. 

Throughout the COVID-19 pandemic, she participated in the UK government’s Scientific Advisory Group for Emergencies (SAGE) and its subgroup the Scientific Pandemic Insights Group on Behaviour (SPI-B) and the Environment and Modelling Group (EMG), and was the independent chair of the Science Board for the Events Research Programme (ERP).

She was also a leading member of the Testing Initiatives Evaluation Board (TIEB), set up initially by NHS Test and Trace and subsequently an advisory governance board for UKHSA. TIEB was the group that provided oversight for the evaluation of the programme of self-isolation support pilots. This preface was written in a personal capacity.