Research and analysis

Comparison of manufacturer reported lateral flow device sensitivity with the sensitivity observed in evaluation

Updated 27 April 2023

Following a ministerial commission in August 2020, the UK Health Security Agency (UKHSA) has referred lateral flow antigen devices to UKHSA’s laboratories at Porton Down for evaluation. The purpose of the evaluation is to identify lateral flow devices (LFDs) that display performance characteristics desirable for mass population, community-based testing. The desirable performance characteristics are very high specificity and very high sensitivity against viral loads associated with infectiousness.

The LFDs that pass the evaluation are eligible for procurement and deployment into UKHSA’s testing programme.

The evaluation consists of:

• a desktop review (Phase 1)
• a futility test, to short-list devices for further evaluation (Phase 2)
• a more detailed clinical evaluation to assess the sensitivity and the specificity of the device (Phase 3)
• an ongoing programme of evaluation against variants of concern

UKHSA asks suppliers to provide a copy of the instructions for use for the device, as well as other technical and commercial information, to allow it to conduct the desktop review. The instructions for use usually include performance data, which is derived from clinical studies undertaken by the manufacturer of the device.

The purpose of this report is to compare the manufacturer reported sensitivity data, as reported in the instructions for use, with the sensitivity data observed in the Phase 3 evaluation at Porton Down.

Sensitivity observed at Porton Down

UKHSA evaluates LFDs at the laboratories at Porton Down. The evaluation of the LFDs that are included in this report occurred between August 2020 and November 2022.

Porton Down has evaluated 171 LFDs, with a further 17 under review. Of the 171 LFDs that have completed evaluation:

• 86 failed at Phase 2
• 21 failed at Phase 3 (14 due to low sensitivity and a further 7 failed for other reasons, for example, low specificity or a high rate of kit failure)
• 12 failed against variants of concern
• 52 have passed all phases of evaluation

The sensitivity data in this report relates to the 69 LFDs that have undergone Phase 3 sensitivity testing (both those that passed and those that failed).

Sixty-nine LFDs is made up of:

• the 52 devices that have passed all phases of evaluation
• 14 devices that failed Phase 3 sensitivity testing
• 3 devices that passed all phases of evaluation but subsequently failed against a variant of concern

All of the devices that Porton Down has evaluated are CE marked for professional use or self-testing manufacturers can self-certify their LFDs for professional use, but need approval from a notified body to be used for self-testing.

Porton Down use approximately 200 clinical viral transport medium (VTM) samples for the Phase 3 sensitivity testing. The number of samples used to evaluate the sensitivity of LFDs ranged from 158 to 200. The samples are taken from patients and reflect a range of cycle threshold (Ct) values. The samples are initially frozen upon receipt and then thawed, diluted, aliquoted into small single-use volumes and refrozen. The samples are thawed for testing and inactivated in AVL-T. AVL is Qiagen product 19073, to which we add Triton x-100.

SARS-CoV-2 ribonucleic acid (RNA) is detected via polymerase chain reaction (PCR) using the Roche cobas® 8800 system. VTM samples (100µl) are added to the manufacturer’s LFD buffer and testing on LFDs is performed according to the manufacturer’s instructions. The number of freeze-thaw cycles is kept to a minimum, with most samples undergoing 2 freeze-thaw cycles.The sample that is tested on the LFD is benchmarked by concurrent PCR, which ensures consistency in the number of freeze-thaw cycles.

Four panels of clinical samples have been used in the evaluation. Panels of samples were obtained in September 2020, January 2021, April 2021, and July 2021. Each of the panels reflects the variants of SARS-CoV-2 circulating at the time the panels were obtained. Porton Down sought to standardise the panels, by ensuring that the samples in each panel cover a range of Ct values. However, given the inherent variability of clinical samples, there are differences in the composition of the panels.

Using the data generated through the Phase 3 sensitivity testing, we have calculated the sensitivity for each device by dividing the number of positive results by the number of known PCR positive samples tested. Using this metric, the sensitivity of the LFDs observed in testing at Porton Down ranges from 32% to 81%.

Manufacturer reported sensitivity data

LFDs are regulated medical devices. The regulation requires manufacturers to provide instructions for use with their device and to include performance data in the instructions for use, as referenced in Annexes I of the In-Vitro Diagnostic medical device Regulation. The instructions for use usually include the following performance data for the device:

• the sensitivity (often with 95% confidence intervals)
• the specificity (often with 95% confidence intervals)
• the limit of detection

The manufacturers also include data on the cross reactivity of the device with other pathogens.

The manufacturers derive the performance data from clinical studies. We have no reason to doubt the veracity of the manufacturers’ performance data. However, the instructions of use do not include details on where the clinical studies were conducted or the range of Ct values in the samples. For example, a clinical study conducted in a hospital, with patients severely ill with SARS-CoV-2 would be expected to return a higher sensitivity for a LFD than a clinical study conducted on asymptomatic individuals in the community.

Of the 69 LFDs that have undergone sensitivity testing at Porton Down, 58 include the sensitivity of the device in the instructions for use. The instructions for use for all 58 devices state that the sensitivity is at least 80%, with 72.4% (42 out of 58) stating that the sensitivity of the device is greater than 95%. Two of the instructions for use state that the sensitivity of the device is greater than 99%.

Comparison of the performance data

In Figures 1.1 and 1.2 below, we compare the sensitivity observed in the evaluation at Porton Down with the sensitivity reported by the manufacturers in the instructions for use.

Figure 1.1 comparison of the manufacturer reported sensitivity with the sensitivity observed in the evaluation at Porton Down

Figure 1.2 comparison of manufacturer reported sensitivity with the sensitivity observed in the evaluation at Porton Down (axes changed to focus on relevant parameters)

Figures 1.1 and 1.2 show that there is no significant correlation (Spearman rank correlation r=-0.17, p= 0.18) between the sensitivity reported by the manufacturer and the sensitivity observed at Porton Down.

In Figure 1.3 below, we compare the sensitivity of the observed in the evaluation at Porton Down with the limit of detection reported in the manufacturer’s instructions for use.

Figure 1.3 comparison of the manufacturer reported limit of detection with the sensitivity observed in the evaluation at Porton Down (TCID50 is the median tissue culture infectious dose)

Figure 1.3 shows that there is no significant correlation (Spearman rank correlation r= -0.06, p= 0.70) between the limit of detection reported by the manufacturer and the sensitivity observed at Porton Down.

In Figure 1.4 below, we compare the sensitivity at 10,000 RNA copies per millilitre (copies/ml) using regression analysis observed in the evaluation at Porton Down with the limit of detection reported in the manufacturer’s instructions for use. The purpose of Figure 1.4 is to compare the limit of detection reported by the manufacturer with the sensitivity observed at Porton Down for samples with a low viral load.

Figure 1.4 comparison of the manufacturer reported limit of detection with the sensitivity at 10,000 copies/ml observed in the evaluation at Porton Down

Figure 1.4 shows that there is no significant correlation (Spearman correlation r= -0.08, p= 0.57) between the limit of detection reported by the manufacturer and the sensitivity at 10,000 RNA copies/ml observed at Porton Down.

Summary

In conclusion, for the sample of devices that underwent Phase 3 sensitivity testing at Porton Down, there is no correlation between the:

• sensitivity data reported by LFD manufacturers and the sensitivity data observed at Porton Down
• limit of detection data reported by LFD manufacturers and the sensitivity data observed at Porton Down

These conclusions support the need for UKHSA to evaluate LFDs at Porton Down to identify high performing devices, rather than relying solely on the manufacturers’ reported performance.

Acknowledgements

Tom Collinge, Product and Delivery, UKHSA

Somya Agrawal, Product and Delivery, UKHSA

Richard Vipond PhD, Diagnostics and Innovation, UKHSA

Dr Abbie Bown, Clinical Scientist, UKHSA

Professor Tim Peto, Professor of Medicine, University of Oxford

Raghavendran Kulasegaran-Shylini PhD, Public Health and Clinical Oversight, UKHSA

Matthias E. Futschik PhD, Public Health and Clinical Oversight, UKHSA

Jeanne Pouquet, Regulatory and Product Development, UKHSA

Professor Mark Wilcox, Professor of Medical Microbiology, University of Leeds; Infection Lead of Leeds NIHR Diagnostic Technologies Medical Technology and In Vitro Diagnostic Co-operative, Leeds Teaching Hospitals NHS Trust and Co-Chair of COVID-19 Technical Validation Group, UKHSA

Professor Dame Sue Hill, Chief Scientific Officer for England, Senior Responsible Officer for Genomics in the NHS

Tom Fowler PhD FFPH, Public Health and Clinical Oversight, UKHSA