Holistic AI: Open Source Library

Case study from Holistic AI.

Background & Description

The Holistic AI Open Source Library is a tool used to assess and improve the trustworthiness of AI systems. The Library is provided as a user-friendly Python module that can be downloaded and installed and is supported by documentation that details each method contained in the Jupyter notebooks which guide the user through example implementations. Holistic AI has also provided a number of tutorials for using the Library on our blog.

It serves as a resource for identifying and mitigating risks associated with AI systems, reducing preventable harms and supporting AI assurance efforts.

Currently, the Library has functions for data visualisation, bias metrics, and bias mitigation for both classification and regression models. Specifically, the library includes metrics for assessing bias for binary and multiclass classification, regression, clustering, and recommender systems, and provides pre-processing, in-processing, and post-processing approaches to bias mitigation through built-in functions.

The Library will continue to evolve, with new techniques to measure and mitigate bias added, as well as capabilities to address additional technical risk verticals such as explainability and robustness.

How this technique applies to the AI White Paper Regulatory Principles

More information on the AI White Paper Regulatory Principles.

Fairness

The Holistic AI Open Source Library supports a number of bias metrics and mitigation approaches derived from research in the field. These can be used to assess models for bias and mitigate them, with performance metrics providing a way to compare models and mitigations to find the fairest model while maximising accuracy.

Why we took this approach

Although there are other bias metric libraries already available, they are often not user-friendly and can be complicated. The Holistic AI Open Source Library is easy to use and is supplemented by several easy-to-follow tutorials using real-life datasets. It is important that bias mitigation tools are as widely available and accessible as possible, and the Holistic AI library is there to fulfil this need.

Benefits to the organisation using the technique

Bias can easily be assessed and mitigated in one easy-to-use dashboard. AI is increasingly being targeted by global regulation and is being used in critical contexts every day. Identifying and mitigating bias early can prevent harm before it occurs, reducing legal liability, reputational risk, and financial damage

Limitations of the approach

The library currently focuses on bias, but will soon be expanded to address other technical risk verticals.

Updates to this page

Published 19 September 2023