Guidance

Software and AI as a Medical Device Change Programme - Roadmap

Updated 14 June 2023

Introduction

Software (including Artificial Intelligence (AI)) plays an essential part in health and social care. In the UK, many of these products are regulated as medical devices. Last year, the MHRA announced the Software and AI as a Medical Device Change Programme, a programme of work to ensure regulatory requirements for software and AI are clear and patients are protected. This Programme builds upon wider reforms for medical devices as a whole detailed in the Government response to consultation on the future regulation of medical devices in the United Kingdom. Set out below is further information on each work package of the Change Programme, including deliverables to meet each set of broad objectives, and further information on how the broad Change Programme will be implemented.

The Change Programme will deliver bold steps to provide a regulatory framework that provides a high degree of protection for patients and public, but also makes sure that the UK is recognised globally as a home of responsible innovation for medical device software looking towards a global market. Broadly, to achieve this aim, we will focus on ensuring that:

A.The requirements for software and AI as a medical device provide assurance that these devices are acceptably safe and function as intended, thereby protecting patients and public.

B.The requirements for manufacturers are clear, supported by both clarificatory guidance and streamlined processes that work for software, as well as bolstered with the tools to demonstrate compliance, for instance, via the designation of standards.

C.The friction is taken out of the market by working with key partners such as the National Institute for Health and Care Excellence (NICE) and NHS England to align domestically, de-duplicate, and combine requirements, ultimately providing a joined-up offer for digital health within the UK. Internationally, we will work with other regulators both bilaterally, and multilaterally through the International Medical Device Regulators Forum (IMDRF) to strengthen international convergence and consensus on software and AI products.

This programme includes eleven work packages across two workstreams. The first stream contains eight work packages to make key reforms across the software as a medical device (SaMD) lifecycle, the second stream has three work packages and considers the challenges that AI as a medical device (AIaMD) can pose over and above classically programmed software.

Our approach

Legislative change

Some (but not the majority) of the changes intended will be in the form of secondary legislation, these have been consulted upon as a part of a consultation on the future regulation of medical devices in the United Kingdom. Much of the programme builds upon legislation through guidance, for instance, by clarifying what medical device requirements mean in the context of software and AI.

Developed across government

This programme of work has been developed with support and input across government and the NHS as well as with targeted engagement with devolved administrations, trade associations, academia, and industry partners. Much of the work below will continue to be developed in collaboration with partner organisations in the Multi Agency Advisory Service, namely NICE, Care Quality Commission (CQC), and Health Research Authority (HRA).

Patient centred, supported by industry

We will also announce plans for patient/public and industry engagement to support this work programme. We recognise that the use of AIaMD raises broader societal questions and that the public voice needs to be heard in establishing how the MHRA regulates these devices to ensure safety. The new regulatory framework must also be pragmatic and attract responsible innovation to the UK. Building on the existing MHRA Patient and the Public Engagement Strategy, we will engage patients, public, and the wider sector (including healthcare and industry) throughout this change programme and most of these deliverables will be released first as drafts for wider comment and input before being published.

Inclusive innovation

The MHRA recognises that SaMD and AIaMD must perform across all populations within the intended use of the device and serve the needs of diverse communities. We will build on wider work examining health inequalities in medical device regulation, ensuring specific work on SaMD and AIaMD builds on that foundation, focusing on specific issues that AIaMD can pose over and above medical devices.

Leading regulatory innovation that drives international harmonisation

We are sensitive to the fact that regulatory innovation that departs from international consensus can create an additional burden for the market. We want to see effective products brought to patients quickly, supported by the evidence for their safe use and to facilitate safety surveillance. Accordingly, we intend to drive forward international consensus in this area and redouble our contributions to groups such as the IMDRF as well as through wider bilateral and multilateral efforts that accelerate regulatory innovation and harmonisation to minimise that burden on industry.

Tools to demonstrate conformity

We recognise that many of the deliverables below require the development of standards and tools to help manufacturers clearly demonstrate that their device meets the requirements set. We are therefore working closely with British Standards Institute (BSI) in its role as the UK national standards body, to ensure that there is a wide set of standards to help manufacturers meet the regulatory requirements. Together with BSI, we will jointly publish opportunities for existing and future standards mapped against the work packages. Additionally, we are collaborating with groups in the wider setting of standards for study reporting and assurance frameworks.

Principled but practical

We recognise that there are key elements of SaMD and AIaMD that we do not directly regulate but must not ignore. For instance, SaMD and AIaMD often process personal data and so we are working in partnership with both the Information Commissioner’s Office and the National Data Guardian to consider these elements as they relate to patients, such as data leaks and trust in products. We recognise broadly agreed ethical principles for AI in health and social care and are cognizant of those principles as well as complementary principles such as the Office of AI’s Establishing a pro-innovation approach to regulating AI when developing our regulatory framework.

Work packages

The major work packages in the Software and AI as a Medical Device Change Programme are set out below, and other work packages, for example WP 8 on Mobile Health and Apps, are nested within or spread over multiple work packages. We describe the problem that work packages seek to address, the objectives that break down that problem, and the specific deliverables that meet the ambition.

Many work packages build on the conclusions of other work packages and so we identify the key links between work packages wherever possible.

Finally, the below work packages have one primary aim: to protect patients and public whilst ensuring that we accelerate responsible innovation.

We will publish the below deliverables in a stepped manner: the first tranche (WP1-02, WP4-01, WP9-02, WP9-05, WP11-01) are currently intended to have first drafts published before the end of 2022. Further tranches shall be announced in due course.

WP 1 Qualification

Problem statement: There is currently a lack of clarity as to what qualifies as SaMD^{[footnote 1]} and software in a medical device; this clarity is required to ensure appropriate, effective and proportionate regulation of these devices.

Objectives

1. Ensure medical device regulations capture sufficient breadth of software to protect patients and public
2. Ensure there is sufficient clarity yet flexibility of qualification to effectively and proportionately regulate SaMD
3. To improve the wider regulation of digital health, through supporting and working with other regulators and processes where software does not qualify as a medical device

Deliverables

WP1-01 Regulatory Guidance

What qualifies as SaMD?

It can be difficult to establish what qualifies as SaMD or software in a medical device.

We will publish guidance that brings clarity to a range of issues that have become evident as SaMD has evolved. Over time, this guidance will seek to clarify the following distinctions with respect to software:

• SaMD versus wellbeing and lifestyle software products
• Medical device software versus IVD software
• Medical devices versus medicines and companion diagnostics
• Any requirements that might apply to ‘in house’ SaMD
• Research use only ‘exemption’ for SaMD
• Custom made devices
• Requirements for software in a kit / software system / software in procedure pack / software as a service^{[footnote 2]}
• Accessories to a medical device or IVD
• Devices with no medical function

We will also link this work with separate work on Classification of SaMD and to the UK Borderlines Manual, ensuring industry is provided with ample examples to illustrate key boundaries.

WP1-02 Regulatory Guidance

Crafting an intended purpose in the context of SaMD

The cornerstone to almost any activity required by medical device regulation is having a well-crafted and defined intended purpose. Failure to adequately define the intended purpose of SaMD will impair any subsequent efforts to comply with medical device regulation. For instance, designing a robust quality management system becomes much more difficult, ambiguity will impede the generation of sufficient clinical evidence, and the implementation of an adequate post market surveillance system will be hampered.

The MHRA will develop guidance assisting manufacturers to:

• Define the intended purpose of their medical device with adequate specificity
• Avoid common pitfalls when defining intended purposes, for instance, the issue of ‘hydra devices’, devices with many specific intended purposes but no overarching intended purpose
• Clearly articulate what populations are within scope of the intended purpose, linking this to clinical evidence
• Understand how intended purposes link to quality management systems, risk management, and other processes
• Indicate when and how intended purposes should be revisited and how to expand intended purposes

WP1-03 Regulatory Guidance

Clarifying the concept of ‘manufacturer’ for SaMD

We recognise that software can be deployed and modified in such a way that it can be uncertain what entity – if any – counts as a manufacturer. For instance, one issue is open-source code. If the code is written and made available, then subsequently modified, the entity making the modified version may take on the responsibilities of the manufacturer of this modified code if it qualifies as a medical device. We will also seek to clarify the status of those that deploy SaMD via third party websites.

The MHRA will develop and publish a policy position on the implications for the ‘manufacturer’ and associated guidance to help guide to the appropriate route to market.

WP 2 Classification

Problem statement: Currently, the Medical Device Regulations 2002 (as amended) do not classify software proportionate to the risk it might pose to patient and public safety.

Objectives

1. Ensure classification rules closely follow the risk that specific SaMD poses to patient and public safety where this is known
2. Ensure classification rules impose safety and performance requirements proportionate to the risk which SaMD applications pose
3. Ensure that classification rules provide sufficient flexibility so that the risk profile of novel devices can be addressed without needlessly restricting innovation.

Deliverables

WP2-01 Secondary Legislation

Reform of classification rules for SaMD

We will bring forth secondary legislation to implement rules more closely aligned to the IMDRF Software as a Medical Device: Possible Framework for Risk Categorization and Corresponding Considerations.

We are working with NICE on its Evidence Standards Framework to ensure this aligns with SaMD classification rules wherever possible.

WP2-02 Secondary Legislation and Process

Exploration of an ‘airlock process’ for SaMD

Some manufacturers of innovative products that meet a critical unmet clinical need may struggle to generate evidence in the premarket phase. Accordingly, this process will allow software to generate real world evidence for a limited period of time while being continuously monitored.

If we decide to proceed with this proposal, we will consult further on how to develop an approach to allow for heightened monitoring of a SaMD in the early stages of its lifetime, whilst a better understanding of the risk profile is generated.

We will undertake further work to develop and implement an ‘airlock process’ (akin to a regulatory sandbox) for SaMD. This process will be applied:

• According to set of criteria applied by MHRA and / or Approved Bodies
• Allow for heightened monitoring of the device
• Have a clear set of entry and exit criteria.

We propose that manufacturers will be able to gain access to the UK market earlier, and in a controlled manner to generate the evidence to better understand the risk profile and the final risk classification of the device.

WP2-03 Regulatory Guidance

Guidance on classification rules for SaMD

New classification rules will require robust guidance to ensure sensible and consistent interpretation.

We will publish guidance outlining key terms not defined in secondary legislation, ensuring this guidance is supported by a robust set of examples to clarify any borderline issues.

WP 3 Premarket requirements

Problem statement: Clearer premarket requirements that fit software are needed to ensure a smoother path to market for manufacturers and better protection for patients and the public.

Objectives

1. Ensure SaMD is supported by adequate data on safety, effectiveness, and quality prior to being placed on the market, ensuring these requirements are proportionate to the risk the device presents
2. Ensure that premarket requirements, especially clinical evidence and clinical investigation requirements, are clear in how they apply to and are appropriate for SaMD
3. Ensure that any premarket submission includes all necessary information to support the safe use of SaMD.

Deliverables

WP3-01 Secondary Legislation

Review of essential requirements for software

The essential requirements in the Medical Devices Directive are no longer in line with international best practice.^{[footnote 3]} An update is needed to ensure they provide robust assurance that software meets its intended purpose and is safe for patients and users.

We have already reviewed the current essential requirements, considering:

• Necessary modifications to existing essential requirements to reflect the nature of SaMD and AIaMD
• The need for additional essential requirements that relate to SaMD and AIaMD
• Ease of use changes, such as sub-dividing those essential requirements for software in particular

The results of this review are detailed in the government response to the Consultation on the future regulation of medical devices in the United Kingdom.

WP3-02 Best Practice Guidance

Best practice SaMD development and deployment

Best practice for SaMD development requires the best of both software development and medical device development.

We will work with BSI as the UK national standards body to review existing best practice development and deployment activities for SaMD as outlined by designated standard IEC 62304, amongst others. This assessment will look for areas of improvement across the lifecycle from the regulator’s perspective with particular focus on safety by design and deployment strategies.

The new MHRA guidance will not replace IEC 62304 or other key standards utilised by manufacturers of SaMD, but it will serve to highlight areas where current best practice may not meet regulatory requirements or regulatory definition of the ‘state of the art’.^{[footnote 4]} This work will be useful in seeking to align the expectations of manufacturers and the standards community with regulatory requirements and we will provide updated guidance as the state of the art advances. We will also highlight current state of the art with respect to avoiding bias and ensuring adequate representation of populations within the intended purpose.

WP3-03 Regulatory Guidance

The position of retrospective non-interventional studies

Currently it is unclear whether retrospective non-interventional performance studies for SaMD might require notification to MHRA as a clinical investigation or registration with MHRA as an IVD undergoing performance evaluation.

We will update existing guidance to indicate when these studies qualify as clinical investigations or an IVD undergoing performance evaluations, thereby clarifying what needs to be notified or registered with us.

WP3-04 Joint Regulatory Guidance

Data-driven SaMD: Research, Development, and Governance

We will work with the Health Research Authority (HRA) to produce joint regulatory guidance that highlights:

• The main points of consideration from both the perspective of governance of research as well as medical device development
• The major processes that need to be undertaken and the key points at which these processes begin.
• The key pieces of existing guidance to support all of the above

It is expected that this joint guidance will clarify how the medical device and research governance process relate to one another and how best to address both sets of requirements.

WP3-05 Regulatory Guidance

Human-centred SaMD

‘Human-centred’ here refers to the concept of ‘human-centred design’ that is ‘an approach to interactive systems that aims to make systems usable and useful by focusing on the users, their needs and requirements, and by applying human factors/ergonomics, and usability knowledge and techniques.’^{[footnote 5]}

The risk that users will systematically misinterpret the outputs of software or be otherwise adversely influenced by using the software is often not effectively mitigated against or addressed in existing evidence, leading to safety concerns.

We will produce regulatory guidance clarifying the importance of human factors, usability, ergonomic, or behavioural science evidence, linking this literature to essential requirements, and expanding upon the earlier deliverable Best practice SaMD development and deployment.

It is also anticipated that there will be a complementary programme of work to support and expand upon this initial guidance.

WP3-06 Regulatory Guidance

Registration and nomenclature for SaMD

Existing nomenclature as it applies to software can be broad and imprecise, thereby impairing effective signal detection for these devices.

We will work to advance nomenclature for software (as a medical device), towards sufficient granularity to better enable signal detection and post-market trending. Wherever possible we will work with existing bodies such as the Global Medical Device Nomenclature (GMDN) Agency to improve alignment with other regulators.

WP 4 Post Market

Problem statement: The safety signal for SaMD that MHRA receives needs to be stronger. The post market surveillance system needs to be adapted so that signals are received and not lost to noise. This is needed to enable stronger vigilance by manufacturers and to detect and mitigate the risk of patient safety incidents. Additionally, change management requirements also require review given the challenges seen in the software, including but not limited to rapid updating.

Objectives

1. Strengthen our post market surveillance system to support quicker and more comprehensive capture of adverse incidents for SaMD, enabling better detection of safety signals,
2. Consider how to utilise real world evidence to provide further assurance that SaMD functions as intended, maintains performance, and continues to provide assurance with respect to safety by supporting investigation of signals
3. Articulate clear change management requirements for SaMD

Deliverables

WP4-01 Processes

Review of adverse incident signal detection for SaMD

We have already conducted a preliminary analysis of the individual case reports identified for SaMD and the processes that analyse and flag concerns. We will:

• Consider what additional sources of data would support detection of safety signals to better identify the potential risks SaMD poses
• Consider how best to highlight the benefits of reporting adverse events via the Yellow Card Scheme
• Consider how to better curate and analyse the totality of data to generate safety signals for SaMD
• Consider revision to standard operating procedures and processes so that the MHRA recognises SaMD risks early and responds swiftly, monitoring the impact of risk mitigation measures

All of this together will ensure that we identify safety signals sooner, have the tools to distinguish between signal versus noise, and act swiftly in response to signals of concern, improving patient and public safety.

WP4-02 Regulatory Guidance

Adverse incidents in the context of use of SaMD

Currently, we receive relatively few reports of adverse incidents from manufacturers and users in relation to SaMD; investigations indicate that adverse incidents considered reportable by MHRA are often not reported. One reason is the failure to recognise the potential ‘indirect harm’ that SaMD might cause.

We will produce guidance clarifying:

• What constitutes a reportable adverse incident, emphasising ‘indirect harm’ in the context of SaMD use
• The requirements for reporting,providing an overview of different examples of reportable incidents
• Emphasise that harm may occur because of a medical decision, action taken or not taken based on the information or results provided by the device
• Indication of appropriate next steps, depending on the incident in question, for instance, how to report the incident to MHRA
• Emphasis on ensuring populations within the intended purpose are considered within the vigilance process

WP4-03 Regulatory Guidance

Change management for SaMD

Change management requirements and how they apply to SaMD must be clearer, this clarity being required to ensure these devices maintain performance over time and that patients as well as public are protected.

We will produce guidance that makes plain:

• The key principles of change management in software and how change management links to the earlier deliverable Best practice SaMD development and deployment
• How change management links to quality management systems, risk management, and other requirements
• That change should be anticipated, planned, and documented
• That there is an obligation to actively monitor and collect information to avoid degradation in performance over time
• We will distinguish between intentional change anticipated by the predetermined change control plan, intentional or unintentional change not anticipated by that plan, and changes to the intended purpose of the device.

This guidance will have a necessary link to the development of predetermined change control plans outlined below.

WP4-04 Secondary Legislation and Processes

Predetermined change control plans and change protocols

More often than not, change in SaMD is to be expected and should be anticipated but current change management procedures can be cumbersome to keep pace with such change.

We will make provision in secondary legislation for predetermined change control plans (PCCPs) for SaMD. Additionally, we will craft a procedure that:

• Details a change management process to anticipate change in software over time and detail how this process links to existing quality management and risk management procedures
• This change management process will be likely to include two primary elements: SaMD Pre-Specification and Algorithm Change Protocol
• Provide details on how to define what metrics to track and how to agree performance ‘bands’, such that change inside those bands does not have to be reported to MHRA or the Approved Body
• Outline how the change management process itself links to aspects of the product lifecycle to maximise operational effectiveness whilst minimising product risks

We will work with Approved Bodies to jointly develop these plans and procedures. Additionally, wherever possible, we will work with international regulators to develop this into a globally harmonised process.

WP4-05 Regulatory Guidance

Expansion of intended purposes of SaMD

It is common for SaMD to expand the intended medical purpose, intended population, or intended users over time. However, this expansion in the intended purpose of a device must be supported by corresponding expansion in clinical evidence, proper documentation, and reporting.

We will produce regulatory guidance linking to both the previous work package outputs on defining SaMD intended purpose and best practice development guidance, that details how expansion of SaMD intended purpose should occur supported by evidence and the proper processes that should be undertaken.

WP 5 Cyber Secure Medical Devices

Problem statement: Existing medical device regulations do not currently consider the evolving state of the art for mitigating the risks presented by cyber security vulnerabilities and the operational issues presented by legacy software medical devices and systems.

Objectives

1. Articulate how cybersecurity issues translate to SaMD issues
2. Ensure that cybersecurity is adequately reflected in SaMD requirements and in post market surveillance requirements
3. Work closely with other bodies, for instance, through the Connected Medical Device Security Steering Group to ensure SaMD cybersecurity policy capitalises on synergies

Deliverables

WP5-01 Secondary Legislation

Cybersecurity requirements for medical devices and IVDs

Increasingly, many medical devices and IVDs include software and are often networked, meaning that non-malicious or malicious interference with the device can lead to malfunction of the device, loss or tampering with personal data, damage to the device, and ultimately injury to the patient.

As outlined in the government response to the consultation on the Medical Devices legislation, we intend to develop secondary legislation to impose cybersecurity and IT requirements to guard against the above risks. This secondary legislation will:

• Align with the Connected Medical Device Security Steering Group principles
• Be consistent with and build upon complementary requirements such as the Department of Culture, Media, and Sport’s Product Security and Telecommunications Infrastructure Bill, NHS DCB (Data Coordination Board) standards, and NHS Digital Technology Assessment Criteria requirements
• Be harmonised with international best practice

WP5-02 Regulatory Guidance

Guidance elucidating cybersecurity requirements for medical device and IVDs

Future requirements in legislation will require supporting guidance to interpret and make clear what the legislative positions require.

We will produce guidance on cybersecurity and related requirements for medical devices and IVDs; this guidance will:

• Position cybersecurity issues in the context of lifecycle management processes
• Make plain that cybersecurity and related issues constitute shared responsibilities, most often between the manufacturer, systems manufacturers, and local deploying organisation
• Clarify specific requirements and responsibilities that primarily fall to the manufacturer of the device
• Clarify necessary reporting of cybersecurity vulnerabilities for medical devices and IVDs

We will develop this guidance with the Connected Medical Device Security Steering Group and other relevant stakeholders.

WP5-03 Best Practice Guidance

Management of unsupported software devices

Manufacturers may not have a plan to exit the market for their products. This fact in the context of software produces unique risks, unsupported devices that are no longer maintained by their manufacturer still being in service, despite the security and safety risk they might pose. These risks must be more effectively managed.

We will produce best practice guidance for manufacturers, system providers, and local deploying organisations on how best to manage the risks that unsupported devices might present. This guidance will link to previous guidance on cybersecurity requirements more generally and emphasise the shared responsibilities between the various actors identified.

We will work with key stakeholders, primarily through the joint NHS Digital and MHRA Connected Medical Device Security Steering Group to develop this guidance.

WP5-04 Processes

Reporting of relevant cybersecurity vulnerabilities

Currently, we receive relatively few reports of cybersecurity vulnerabilities connected to medical devices, but as these vulnerabilities can lead to serious adverse incidents, we should improve reporting procedures to enable proactive mitigation of harm.

We will work closely with industry to clarify, bolster, and make consistent reporting of cybersecurity incidents and any vulnerability identified from manufacturers. In addition, we will also make plain the requirements to report these vulnerabilities, as outlined in earlier guidance on cybersecurity requirements.

Please note that work packages 6, 7 and 8 do not have discrete deliverables assigned to them in this programme. The content of these work packages has been absorbed into wider SaMD and AIaMD deliverables outlined in this programme.

WP 9 AI RIG (AI Rigour)

Problem statement: There is a lack of clarity on how to best meet medical device requirements for products utilising artificial intelligence, to ensure these products achieve the appropriate level of safety and meet their intended purpose.

Objectives

1. Utilise existing regulatory frameworks to ensure AIaMD placed on the market is supported by robust assurance that it is safe and effective
2. Develop supplementary guidance to better ensure AIaMD placed on the market is supported by robust assurance with respect to safety and effectiveness
3. Outline technical methods to test AIaMD to ensure the device is safe and effective

Deliverables

WP9-01 Guiding Principles

Good machine learning practice for medical device development

Delivered October 2021

The basic principles of good machine learning practice (GMLP) published in October 2021 in collaboration with FDA and Health Canada, can be overlooked and it may not be obvious how these might apply to medical devices.

These 10 guiding principles are intended to lay the foundation for developing Good Machine Learning Practice that addresses the unique nature of these products. They will also help cultivate future growth in this rapidly progressing field.

The 10 guiding principles identify areas where the International Medical Device Regulators Forum (IMDRF), international standards organizations and other collaborative bodies could work to advance GMLP. Areas of collaboration include research, creating educational tools and resources, international harmonization, and consensus standards, which may help inform regulatory policies and regulatory guidelines.

WP9-02 Regulatory Guidance

Good machine learning practice for medical device development mapping

Manufacturers can find it difficult to identify how GMLP links to existing legal requirements. This document will map how the identified GMLP principles can be tied to key responsibilities outlined in the Medical Device Regulations 2002 (as amended) and relevant guidance.

WP9-03 Standards Mapping

Good machine learning practice for medical device development standards mapping

Standards are key in meeting regulatory requirements and aligning state of the art at an international level. We will work with BSI as the UK national standards body and other international partners to map relevant standards to consider when meeting the GMLP principles. This document will be complementary to the GMLP regulatory mapping work (WP9-02) in providing a current snapshot of the standards landscape as it relates to meeting the internationally agreed GMLP principles.

WP9-04 Best Practice Guidance

Best practice AIaMD development and deployment

This document will outline high-level best practice AIaMD development and deployment methods, supplementing and expanding upon the earlier deliverable Best practice SaMD development and deployment. This guidance will also outline best practice methods to assess the performance of AIaMD across the lifecycle. We expect that this guidance will draw upon related work such as medical algorithmic audits.

WP9-05 Best Practice Guidance

AIaMD for all

This guidance will clarify and expand upon GMLP 3 “Clinical Study Participants and Data Sets Are Representative of the Intended Patient Population”, going beyond the Good machine learning practice mapped guidance. Broadly, this guidance will break down bias in AIaMD into three broad challenges:

• Performance of AIaMD across populations and different real-world conditions
• Ensuring data are properly contextualised to avoid AIaMD perpetuating inequalities or leading to poorer performance in subpopulations
• Working to ensure that AIaMD meets the needs of the communities in which it is deployed in terms of verification and validation.

In addition, with respect to the first challenge, this guidance will provide a high-level framework to identify, measure, manage, and mitigate bias. We will endeavour to work with international partners to advance this work wherever possible.

WP9-06 Standards Development

Tools to identify bias

We will assist in the development of standards, frameworks, and tools to assist with the identification and measurement of bias. For example, we will work with the STANDING Together project which aims to establish standards for data inclusivity and generalisability via an international consensus process to ensure that datasets underpinning AI systems are representative and do not risk leaving underrepresented and minority groups behind through data gaps.

WP9-07 Experimental Work

Bias detection and mitigation

We will consider statistical and machine learning methods to detect, measure, and correct for bias in datasets. An approach will be developed to identify under-represented features in data and then use synthetic data to oversample the under-represented features and achieve a better overall distribution of features. This new approach will be compared to existing approaches like Synthetic Minority Oversampling TEchnique (SMOTE) and Adaptive Synthetic Sampling (AdaSyn) which are used to detect rare events, to rebalance imbalances in the data. These approaches will be tested using at least two different case studies involving risk prediction. We will ensure this experimental work is linked with associated work mentioned previously and use this experimental work to also inform regulatory policy.

WP 10 Project Glass Box (AI Interpretability)

Problem statement: Current medical device requirements do not take into account adequate consideration of human interpretability and its consequence for safety and effectiveness for AIaMD.

Objectives

1. Articulate how opacity of AIaMD translates into safety, effectiveness, or quality concerns
2. Develop guidance regarding interpretability of AIaMD to ensure that AI models are sufficiently transparent to be reproducible and testable
3. Develop guidance regarding interpretability of AIaMD to ensure that the relationship of interpretability to usability is made plain and emphasised in relation to safety and effectiveness

Deliverables

WP10-01 Best Practice Guidance

Human-centred AIaMD

This guidance will build upon Human-centred SaMD guidance, considering the further challenges that AI can pose, especially human uninterpretable AI. This guidance will emphasise that it is the performance of the human-AI team not just the performance of the model that is key to ensuring AIaMD is safe and effective.

Broadly, this document will consider:

• The importance of being transparent about the intended user population
• The importance of updating the intended user population when changes are made to the model
• A description of how uninterpretable AI (global or local) might impact the performance of the human-AI team
• Consideration of the various methodologies that might be used to consider whether any given AIaMD is human interpretable or not; this might build upon existing standards such as IEEE 7001-2021

WP10-02 Standards Development

Trustworthy AIaMD

We will support wider work on tools and a framework to assure that AI is trustworthy. We will work with a range of bodies, including the Trustworthiness Auditing for AI project but also Health Education England, the General Medical Council, and patient groups (via the engagement strategy) to ensure that AIaMD is trustworthy for healthcare professionals and patients.

WP 11 Project Ship of Theseus (AI Adaptivity)

Problem statement: Existing requirements and processes surrounding the notification and management of change need to fit and be streamlined for AIaMD.

Objectives

1. Articulate problems of fit with medical device regulation for adaptive AIaMD, distinguishing between models that are locked, batch-trained, or continuous learning on streaming data
2. Clarify how adaptive AIaMD of each type might fit within existing change management processes required by medical device regulations
3. Where appropriate, craft new guidance for adaptive AIaMD that does not fit within existing change management processes

Deliverables

WP11-01 Guiding Principles

Adaptivity and change management in AIaMD

Building on updates to managing change in SaMD and AIaMD (see WP4-04 and WP11-03 respectively), this paper will breakdown the types of change management issues currently seen into categories of product adaptivity. These will be reviewed, and the challenges seen in each case identified and linked to existing or developing regulatory approaches to address such challenges.

The broad adaptivity categories of change type are:

• Static AIaMD - these AIaMD are static and not updated, presenting the challenge of data drift, the performance degrading over time
• Batch-trained – this class of AIaMD are trained periodically in batches, akin to SaMD product updates this change would fall within existing regulatory processes. The challenge here is to clarify change management requirements and streamline processes to reduce administrative burden.
• Individualised models – these changes result models that may share challenges seen in custom made devices due to the highly personalised nature of each iteration.
• Continuous learning on streaming data – this fully adaptive model presents novel challenges in managing change over time and continuously ensuring performance is maintained as each data point is a new training point and that data are input continuously.

WP11-02 Experimental Work

Concept drift and significant/substantial change

We will explore the use of concept drift in determining when a significant or substantial change in performance has occurred. Specifically, this work looks to explore methods to detect when change, including change beyond manufacturers’ control e.g., in the product environment and/or in the intended population can be reliably detected and informative to quality and regulatory processes for managing change post-market. Data that have been highly volatile to change (e.g. data collected on COVID) and assumed to be more stable (such as longitudinal cardiovascular data) will be utilised along with a range of AI models and analysis techniques with the aim of developing a methodology to determining significant change in AIaMD.

WP11-03 Process

Pre-determined change control plans for AIaMD

Proposals to implement predetermined change control plans (PCCP) for SaMD (WP4-04) will be reviewed for AIaMD. The suitability and applicability of this approach to change management will be reviewed in the context of common challenges for AIaMD such as poor interpretability and data bias. This process review seeks to ensure that PCCPs are a beneficial tool for managing often rapid and complex change events in AIaMD. Additionally, this review will be to ensure that PCCP in AIaMD are also in alignment with best practice guidance.

1. N.B. SaMD and software in a medical device (SiMD) is shorthand for both medical device software and IVD software. ↩

2. Directive 93/42/EEC, Directive 98/79/EC. ↩

3. Directive 93/42/EEC, Annex I, 2. ↩

4. Directive 93/42/EEC, Annex I, 2. ↩

5. ISO 9241-210:2019 ↩