Guidance

Transparency for machine learning-enabled medical devices: guiding principles

Published 13 June 2024

Background

In 2021, the U.S. Food and Drug Administration (FDA), Health Canada and the United Kingdom’s Medicines and Healthcare products Regulatory Agency (MHRA) jointly identified 10 guiding principles for good machine learning practice (GMLP). GMLP supports the development of safe, effective and high-quality artificial intelligence/machine learning technologies that can learn from real-world use and, in some cases, improve device performance.

Learn more: Good machine learning practice for medical device development: Guiding principles

The FDA, Health Canada and MHRA have further identified guiding principles for transparency for machine learning-enabled medical devices (MLMDs). These principles build upon the GMLP principles, especially:

• principle 7: focus is placed on the performance of the human-AI team
• principle 9: users are provided clear, essential information

While the guiding principles presented here promote transparency for MLMDs, transparency is a good practice to consider for all medical devices.

Transparency

In this document:

• ‘transparency’ describes the degree to which appropriate information about an MLMD (including its intended use, development, performance and, when available, logic) is clearly communicated to relevant audiences
• ‘logic’ refers to information about how an output or result was reached or the basis for a decision or action
• ‘explainability’ refers to the degree to which this logic can be explained in a way that a person can understand

Logic and explainability are aspects of transparency.

Effective transparency:

• ensures that information that could impact risks and patient outcomes is communicated
• considers the information that the intended user or audience needs and the context it is used in
• uses the best media, timing and strategies for successful communication
• relies on a holistic understanding of users, environments and workflows

Human-centred design

Another important concept related to transparency is human-centered design. This is an iterative process that addresses the whole user experience and involves relevant parties throughout design and development. This approach can help:

• develop MLMDs with a high degree of transparency
• validate transparency
• ensure that users have all of the device-related information they need

Learn more about human-centered design.

The guiding principles

The guiding principles for transparency of MLMDs consider:

• who (relevant audiences)
• why (motivation)
• what (relevant information)
• where (placement of information)
• when (timing)
• how (methods used to support transparency)

Who: relevant audiences

Transparency is relevant to:

• those who use the device (such as health care professionals, patients, caregivers)
• those who receive healthcare with the device (such as patients)
• additional parties, including those who make decisions about the device to support patient outcomes (such as support staff, administrators, payers, governing bodies)

Why: motivation

Transparency is essential to patient-centered care and for the safety and effectiveness of a device.

The use of MLMDs can involve understanding complex and context-dependent information. Transparent information can be used to identify and evaluate the device’s risks and benefits and to help ensure the device is used safely and effectively.

Effective transparency that helps parties make informed decisions can help control risks. For example, a clear intended use supports an understanding of whether the device is intended to inform or replace the judgment of a healthcare provider.

Transparency helps parties detect and investigate errors or a decline in performance. It can also promote health equity, as knowledge of how a device works and how it was developed can help identify bias and assess whether a system or output is justifiable.

Upholding transparency over time can help support the maintenance and continued safety of a device.

The transparent and consistent presentation of information, including known gaps in information, can have many benefits. It builds fluency and efficiency in the use of MLMDs. It can also foster trust and confidence in the technology and encourage the adoption of beneficial technologies and access to them.

What: relevant information

The type of information that is appropriate to share will vary across the range of MLMDs. It will depend on the benefits and risks of each MLMD and the needs of their intended users.

It’s good practice to provide information that enhances understanding of the device and its intended use. A clear and accurate description of a device generally includes: 

• information about its medical purpose and function
• the diseases or conditions it’s meant to address
• intended users, use environments and target populations

It’s also good practice to explain how the device fits in the healthcare workflow. This would include a description of the intended inputs and outputs. It would also describe how the output is intended to affect healthcare decisions or actions, or the judgment of a healthcare professional.

Details about device performance, benefits and risks can help you decide how best to use the device. Information about risk management activities across its lifecycle, such as bias management strategies, can also be helpful.

It’s valuable to provide the basis of a device output or other information that explains how the MLMD reaches its output (the ‘logic’ of the model), when this information is available and easily understood. Access to this information helps to critically assess the device and its output when making patient care decisions.

Another good practice is to provide information on product development and risk management activities across the total product lifecycle. Relevant information:

• explains the underlying technology and machine learning approaches
• characterises the training and testing data
• provides summaries of clinical studies
• provides ongoing updates on model and dataset characteristics, performance monitoring and detection and investigation of issues and risks across the total product lifecycle

Clinically relevant limitations, gaps in information or specific contraindication statements are also good practice to communicate. This can include:

• known biases or failure modes
• confidence intervals associated with the provided outputs
• known gaps in the data characterisation (such as identifying any patient populations that are not well represented in training or clinical datasets and, therefore, may be at risk of bias)
• limitations in developing the model or evaluating model performance
• known circumstances or use cases where the device input will not align with the data used to develop and validate the device

It can also be helpful to provide information about how the safety and effectiveness of the device is maintained throughout the total product lifecycle. This could include:

• how to conduct local site-specific acceptance testing or validation
• plans for ongoing performance monitoring
• transparent reporting of successes and failures
• change management strategies
• proactive approaches to address vulnerabilities

Where: placement of information

Device information is available through the user interface. This interface includes all elements of the device the user interacts with (those the user sees, hears and touches). For example, training, physical controls, display elements, packaging, labeling, and alarms are part of the user interface.

A good practice is to optimise use of the software user interface so that the information it conveys is responsive to the user. This software user interface can allow information to be personalised, adaptive and reciprocal. You can address user needs with a variety of modalities, including audio, video, on-screen text, alerts, diagrams, software safeguards and document libraries.

When: timing of communication

Considering the information needs throughout each stage of the total product lifecycle can support successful transparency. Detailed device information may be needed when considering whether to acquire or implement a device, and whether and how to use it.

It may also be helpful to provide timely notifications when the device is updated or modified, or when new information about the device is discovered. It may be appropriate to provide targeted information, such as on-screen instructions or warnings:

• at a specific stage in the workflow (such as during high-risk steps)
• upon specific triggers (such as when certain input or output features are present)

How: methods to support transparency

Communicating information about MLMDs requires a holistic understanding of users, environments and workflows. You can address these by applying human-centered design principles.

Human-centered design means taking into account a human-centered perspective that:

• is iterative
• addresses the whole user experience
• involves relevant parties throughout design and development

It involves responsive and iterative design, validation, monitoring and communications. For example, information is more accessible and usable if you consider:

• providing the appropriate level of detail for the intended audience
• arranging the content in order of importance or to best support the ability of users to process the information and make informed decisions

Plain language may be appropriate in some cases for understanding and usability. In other cases, technical language may be relevant for specialised clinical users.

Summary of transparency guiding principles

Who: relevant audiences for transparency

Transparency is relevant to all parties involved in a patient’s healthcare, including those intended to:

• use or receive healthcare with the device
• make decisions about the device to support patient outcomes

Why: motivation for transparency

Transparency supports:

• safe and effective use
• patient-centred care
• identification and evaluation of risks and benefits of a device
• informed decision-making and risk management
• device maintenance and detection of errors or performance degradation
• health equity through identification of bias
• increased fluency and confidence in MLMD use, increased adoption of the technology

What: relevant information

Enabling an understanding of the MLMD includes sharing relevant information on:

• device characterisation and intended use
• how the device fits into health care workflow, including the intended impact on the judgment of a health care professional
• device performance
• device benefits and risks
• product development and risk management activities across the lifecycle
• logic of the model, when available
• device limitations, including biases, confidence intervals and data characterisation gaps
• how safety and effectiveness are maintained across the lifecycle

Where: placement of information

Maximizing the utility of the software user interface can:

• make information more responsive
• allow information to be personalized, adaptive and reciprocal
• address user needs through a variety of modalities

When: timing of communication

Timely communication can support successful transparency, such as:

• considering information needs at different stages of the total product lifecycle
• providing notifications of device updates
• providing targeted information when it’s needed in the workflow

How: methods to support transparency

Human-centred design principles can support transparency

Using these guiding principles

These guiding principles are intended as considerations when adopting and advancing good transparency practices. Continued engagement on this topic can help inform the collaborative development, implementation and iteration of good transparency practices and consensus standards in this rapidly evolving field.

Contact us

We welcome your continued feedback through the FDA public docket (FDA-2019-N-1185) at Regulations.gov and we look forward to engaging with you on these efforts.

Contact us directly:

• FDA digitalhealth@fda.hhs.gov
• Health Canada mddpolicypolitiquesdim@hc-sc.gc.ca
• MHRA software@mhra.gov.uk