Guidance

Guidance for manufacturers: diagnostic assurance with SARS-CoV-2 variants in circulation

Updated 8 August 2023

© Crown copyright 2023

This publication is licensed under the terms of the Open Government Licence v3.0 except where otherwise stated. To view this licence, visit nationalarchives.gov.uk/doc/open-government-licence/version/3 or write to the Information Policy Team, The National Archives, Kew, London TW9 4DU, or email: psi@nationalarchives.gov.uk.

Where we have identified any third party copyright information you will need to obtain permission from the copyright holders concerned.

This publication is available at https://www.gov.uk/government/publications/how-tests-and-testing-kits-for-coronavirus-covid-19-work/guidance-for-manufacturers-diagnostic-assurance-with-sars-cov-2-variants-in-circulation

Since December 2020, the MHRA have been made aware of increasing concerns relating to the performance of diagnostic tests for SARS-CoV-2 virus variants. UKHSA, formerly known Public Health England (PHE) produced the first report of new SARS-CoV-2 variants of concern (VOCs) on 14 December 2020. The MHRA has been monitoring the issue and collaborating with external partners including the UK Health and Security Agency (UKHSA) and regularly engages with suppliers and manufacturers of test products in the UK to review their post-market assurance processes for the most recently published variants in circulation in the UK.

Pathogen Diagnostic Assurance Group

A Pathogen Diagnostic Assurance Group (PDAG) has been established to provide an assurance framework for the performance of SARS-CoV-2 in-vitro diagnostic (IVD) assays. PDAG replaces the VOC assurance working group, and is a collaboration of UKHSA, MHRA, NHS England, COG-UK, EQA/NEQAS, and partners in the devolved administrations.

PDAG monitor the performance of assays through:

  • in silico monitoring (computer-based analysis)
  • laboratory test performance quality monitoring
  • wet testing (where indicated by the in silico and quality monitoring processes)

Manufacturer bimonthly in silico updates

The MHRA have reminded manufacturers of their obligations with respect to monitoring and validation of their assays when new variants and emerging mutations are identified during routine in silico analysis. Ideally, this kind of specificity (inclusivity) evaluation should be performed in wet-lab testing done empirically with real clinical samples, however, it is not always possible to obtain a complete representative collection of SARS-CoV-2 circulating strains, including all newly circulating variants. Alternatively, cultured organism or synthetic molecules may also be utilised to explore the potential impact of these variants on performance, but this could be resource and time-consuming. These issues can instead be mitigated by in silico sequence monitoring, using bioinformatics tools and publicly available sequence data available from sources such as GISAID, where new genomes are continuously uploaded into the databases, and this can be used as a first step of a full evaluation process.

Diagnostic assays for detection of SARS-CoV-2 were crucial in the management of the pandemic. The evolutionary mutation rate of SARS-CoV-2 sequence change was previously estimated from circulating mutations to be about 1 every 2 weeks or ∼1×10−3 per site per year (Duchene et al., 2020; Hill et al., 2020; Nextstrain database 2020) and such mutations may impact assay performance. If mutations are located within target assay primer/probe of a molecular assay or affect the binding affinity due to 3D structure of an antigen-antibody complex in an antigen-based assay, the resulting mismatches can lead to IVD test failure or loss of assay sensitivity.

Highly divergent lineages that have emerged over the course of a few months, are characterized by a remarkable number of sequence changes, and some of them have been designated as Variants of Concern (VOC). These mutations are characteristic of a variant and are known as lineage “defining” mutations i.e., the UK variant (Alpha), also known as VOC 202012/01, belongs to Global Initiative on Sharing Avian Influenza Data (GISAID) clade GR, Pangolin lineage B.1.1.7, and Nextstrain clade 20I/501Y.V1 and is defined by 23 mutations: 13 non-synonymous mutations, 4 deletions, and 6 synonymous mutations. One of these mutations at positions 69 and 70 of the spike protein (Δ69–70) was first identified as associated with a widely used, commercial diagnostic real-time PCR assay that led to target failure (S gene target failure: SGTF). A rise in the frequency of SGTF in the UK subsequently led to an increase in diagnostic genomic surveillance and gave rise to the requirement of the former VOC assurance working group to inform best practices in clinical care and public health throughout the pandemic and beyond.

However, within a lineage, there may be additional emerging mutations which revert some changes or accumulate new ones and may not yet have been identified as an emerging lineage, due to low proportion of these mutations within a population. As such, these mutations have not been associated with any lineages as yet and samples may be clinically scarce. These emerging mutations still require monitoring in order to assess their potential public health risks and mitigate any risks. The nature and severity of any impact will depend on the assay but can, in a worst-case scenario, lead to a false negative result.

It is therefore crucial to continuously perform in silico analysis, not only to monitor the effects of the mutations defining VOCs, but also the impact of all the mutations present in the variant genomes, including nucleotide changes that were slowly acquired through time or emerged independently and are not representative of the variants’ population.

Manufacturers/suppliers have been asked to notify the MHRA immediately of their evaluation of the performance of their devices. As needed, the MHRA will be requesting additional information/action from specific manufacturers if their devices are suspected or at risk of performance issues based on available information.

The recommendations below are intended to guide manufacturers of authorised IVD solutions provided within the United Kingdom (UK), however they are also provided to increase awareness of in silico monitoring for developers of IVD solutions. The recommendations will be subject to re-evaluation and revisions as appropriate.

Expectations for manufacturers

Due to the diversity of genomic sequences of SARS-CoV-2, a proactive approach is required in both diagnostic assay design and surveillance to monitor pathogen evolution in real-time. This will allow assays that may be affected by variants to be identified and the risk of false negative results mitigated.

Manufacturers are responsible for the safety and performance of their assays whilst they are available on the UK market.

To help fulfil this responsibility, the MHRA have reminded manufacturers to have a Post Market Surveillance Plan (PMSP) in place to continuously monitor, investigate and assess the influence of newly emerging variants of SARS-CoV-2 on assay performance.

The PMSP shall include:

a. Bimonthly* in silico checks of assay targets.

This should be conducted against databases such as GISAID (https://www.gisaid.org) or COG-UK (COVID-19 Genomics UK Consortium (cogconsortium.uk). If a manufacturer mines sequences from another suitable database, a description and justification of which database was used must be provided.

[*Please note that any indication of a variant of having a direct impact on a test must be immediately investigated].

It is recommended that the manufacturer analyses all the VOCs, variants or signals published within the latest regular updates on SARS-CoV-2 lineage prevalence and growth rate documents found at SARS-CoV-2: genome sequence prevalence and growth rate - GOV.UK (www.gov.uk).

It is required that both favourable and unfavourable data should be reported to the MHRA as assurance of either positive or negative performance.

If the performance of an assay is directly impacted by these new virus variant(s), a Field Safety Notice (FSNs) should be issued immediately to alert customers. See further guidance on effective FSNs.

b. Scientific literature and post market intelligence gathering.

Ongoing scientific literature review should be performed to aid in the monitoring of the emergence of new variants or changes in the distribution of existing variants. This will assist in better preparing manufacturers for the potential impact such variants may have on test performance and allow them to implement appropriate mitigation.

c. Outcomes of EQA schemes, when available.

If a manufacturer has participated in a SARS-CoV-2 Proficiency Testing scheme, the manufacturer must provide details of this scheme and the results from the latest round of participation.

d. Details of use of reference material (or clinical materials, when available).

The preliminary in silico analysis specifically provides information on what should be tested in the laboratory. As a second step, it is essential to confirm experimentally the effect of these mismatches on the assay.

For molecular assays the impact of any mutations identified through in silico analysis which result in mismatches within the target region that are likely to impact the performance of the IVD assay should be assessed through in vitro or ‘wet’ testing of synthetic constructs, g-blocks or a clinical isolate, containing the known mutation.

The impact of mutations identified through in silico analysis on antigen based IVD assays is not as clear as for molecular tests and therefore identification of any non-synonymous mutations in the genetic sequence should be evaluated through wet testing of synthetic constructs containing full-length antigen protein of both the wild-type or variant analysed or a clinical isolate, containing the known mismatch or amino acid change.

e. Reporting potential safety issues of any emerging, clinically significant lineages of SARS-CoV-2 on the performance of an assay to MHRA.

In line with the UK Medical Devices Regulations 2002, the MHRA considers reports relating to circulating variants to be serious public health threats, therefore significant safety issues (for example, decrease in performance) should be reported within 48 hours.

In line with this, we have developed specific guidance for manufacturers of molecular and/or antigen assays (including lateral flow tests) as outlined below.

Expectations for manufacturers of molecular assays

1. A bimonthly update report containing all the above information should be submitted to the MHRA. See further guidance for molecular IVD manufacturers:

2. Manufacturers should monitor the emergence of new variants using the available information on the GISAID website (https://www.gisaid.org/) for a variant lineage against change(s) in the primer/probe/other nucleotide target sequences used within their assays. This should be conducted every other month for a rolling window of approximately 60 days based on collection date (not submission date) within this time period, unless they are notified by the MHRA that a new variant of concern has arisen, which requires immediate attention.

3. As part of routine monitoring, if mutation(s) arise within each primer/probe/other oligonucleotide target sequence that may likely compromise the assay performance, this will need to be monitored closely to determine if these mutation(s) are resulting in a newly emerging lineage that may impact the assay.

4. Manufacturers should inform the MHRA of the outcome of the initial risk assessment of any impact on each primer/probe/other oligonucleotide target sequence and on the performance of the assay in light of the newly identified variants or mutations identified during the in silico analysis. This assessment should be further supported by including the calculation of melting temperature (Tm) with the assay primers or probes and the assay target sequence. It is recommended that all assay conditions are included in the calculation (such as ion concentration and primer excess) in order to best reflect the real-world conditions of the assay.

5. If it is observed that a variant is spreading significantly i.e., having sustained increase week on week within the UK and may impact the assay, then this should be communicated to the MHRA within 2 days as a priority.

Expectations for manufacturers of antigen assays (including antigen lateral flow tests)

While antigen-based assays do not directly target the genetic sequence, non-synonymous mutations within the associated genomic region can have adverse and unpredictable effects on assay performance. Hence, in silico analysis is still crucial for antigen based at the amino acid level as these changes can be visualised and monitored within the SARS-Cov-2 genome.

1. A bimonthly update report containing all the above information should be submitted to the MHRA. Further guidance for antigen based IVD manufacturers:

2. Manufacturers should monitor the emergence of variants using the available information on the GISAID website (https://www.gisaid.org/) for the variant lineage with change(s) in the target antigen(s) of their assays i.e. region that antibody is raised against. This should be conducted every other month for a rolling window of approximately 60 days based on collection date (not submission date) within this time period, unless they are notified by the MHRA that a new variant of concern has arisen, which requires immediate attention.

3. As part of routine monitoring, if mutation(s) effectuate amino acid changes in target antigen(s) that may compromise the assay performance, this will need to be monitored closely to determine if these mutation(s) are resulting in a newly emerging lineage that may impact the assay.

4. Manufacturers should inform the MHRA of the outcome of the initial risk assessment of any impact on the performance of the assay in light of the newly identified variants or amino acid mismatches identified during the in silico analysis. This assessment should be further supported using other predictive models such as 3D modelling of the impact of the mutation on antibody-antigen binding.

5. If it is observed that a variant is spreading significantly i.e. having sustained increase week on week within the UK and may impact the assay, then this should be communicated to the MHRA within 2 days as a priority.

6. Additionally, when evaluating if amino acid changes in target antigen(s) can still cover a variant detection, it is also important to consider all the amino acid changes that can be present in the variant population outside of the antigen binding region as the overall 3D structure of the SARS-CoV-2 target protein may have an impact on the assay. If amino acid changes are found from in silico analysis for point 2 above, they should be analysed alongside any defining mutation/amino acid change(s) for this variant.

IVD in silico Recommendations

All manufacturers of in vitro diagnostic (IVD) assays, that detect pathogens directly (such as antigen or molecular tests targeting proteins or RNA respectively), shall monitor in silico the impact of all significant variants (VOCs, variants, or signals) during routine test evaluation and ensure that the performance specified by the manufacturer during initial validation is maintained.

Manufacturers shall take into consideration all relevant areas which are hot spots for mutations at the time of test development and validation as well as the potential impact of future genetic variants. Initial test validation must take into account all possible sequence variation in the target region and post marketing in silico surveillance be tailored to the specific target.

The version information and references of all bioinformatics tools and software shall be recorded for each analysis to allow for audit, and the use of differing tools or versions between analyses is discouraged unless adequate validation and quality assurance has been performed. Manufacturers or third parties using such in-house validated and quality assured software should ensure that any code written for the purpose of in silico analysis follows best coding practices, including that the code is readable and easily maintained.

For regular monitoring of target sequences, it is recommended that short sequences are extracted from complete records that correspond to regions associated with the target nucleotide sequence in order to reduce the chances of non-specific alignment and allow the region of interest to be more accurately quality controlled. Any sequence containing unknown or ambiguous bases or amino acids within the target region should be removed from the database to prevent misidentification of mutations. More details of recommended in silico tools are described in Bioinformatics recommendations sections in respective guidance documents.

In summary, in silico analysis is to be used as a method of prioritising further investigations based on the predicted impact of sequence variation on assay performance. It is strongly recommended that further in vitro or ‘wet’ testing is performed in order to validate these predictions.

Further information

Detailed guidance on documents for molecular tests

Detailed guidance on documents for antigen-based tests

Urgent enquiries related to variants should be reported to covid19.variants@mhra.gov.uk.

Private providers of SARS-CoV-2 antigen and nucleic acid tests should contact the MHRA at device.registrations@mhra.gov.uk to ensure products are registered and monitored.

Back to top