Horizon Scanning Case Study: Guiding principles that can inform the development of Good Machine Learning Practice
Horizon scanning at the Medicines and Healthcare products Regulatory Agency (MHRA), and collaboration with other medical device regulators, leads to the development of guiding principles that can help inform the development of Good Machine Learning Practice.
Background
Artificial Intelligence (AI) and Machine Learning (ML) technologies make predictions from data to learn and adapt performance on a given set of tasks. These approaches have the potential to transform healthcare by deriving new and important insights from data generated every day during health care delivery. The applications of these technologies range from drug discovery, disease diagnosis, biomarker discovery, and patient monitoring.
The Issue
New breakthroughs and technologies are being announced at breakneck speed from a variety of developers with expertise in digital technology, software engineering and healthcare. The speed at which AI/ML technologies are being developed, together with the range of developers’ backgrounds and expertise, means gaining consensus in the field in terms of definitions, legal and data requirements, and regulatory guidance is challenging. These products also raise unique considerations due to their complexity and the iterative, data-driven nature of their development.
What did the MHRA do?
Through horizon scanning and analysis of operational business and interactions with both domestic and international stakeholders, we noted increasing growth in the number of AI/ML related technologies being developed and, due to the lack of consensus in the field, began discussing the topic with other regulators. A collaboration with colleagues at the U.S. Food and Drug Administration (FDA) and Health Canada resulted in the development of 10 Good Machine Learning Practice (GMLP) guiding principles that can inform the development of AI/ML enabled medical devices for a range of applications including disease prediction, diagnosis, monitoring and treatment. The aim is that these GMLP guiding principles lay the foundation for developing safe and effective products, and helps to cultivate future growth in this rapidly developing field. The 10 guiding principles identify areas where the International Medical Device Regulators Forum (IMDRF), international standards organizations and other bodies could work to advance GMLP.
Outcomes
The MHRA, together in collaboration with other medical device regulators globally, identified the need for improved consensus in the AI/ML field and thereby guiding principles that can inform the development of GMLP were agreed and published.
Russell Pearson, an MHRA AI Regulation and Policy Specialist, who was closely involved in this project said: ‘We hope that improved consensus in the field of AI/ML in healthcare will improve the speed to access and the quality of these products so bringing benefits to patients. These guiding principles are just the first step in ongoing work that the MHRA has planned in our Software and AI as a Medical Device Change Programme and we look forward to further collaborations to support international harmonization and consensus standards for this field.’
Summary of the Guiding Principles
| 1 | Multi-disciplinary expertise is leveraged throughout the total product life cycle |
| 2 | Good software engineering and security practices are implemented |
| 3 | Clinical study participants and data sets are representative of the intended patient population |
| 4 | Training data sets are independent of test sets |
| 5 | Selected reference datasets are based upon best available methods |
| 6 | Model design is tailored to the available data and reflects the intended use of the device |
| 7 | Focus is placed on the performance of the human-AI team |
| 8 | Testing demonstrates device performance during clinically relevant conditions |
| 9 | Users are provided clear, essential information |
| 10 | Deployed models are monitored for performance and re-training risks are managed |