Advai: Robustness Assessment of a Facial Verification System Against Adversarial Attacks

Case study from Advai.

From:
Centre for Data Ethics and Innovation and Department for Science, Innovation and Technology
Published
12 December 2023
Use case:
Image recognition and video processing, Machine learning, Deep learning, Virtual agents or artificial conversational interfaces and Robotic process automation and decision management
Sector:
Manufacturing (SIC Code Section C), Energy & Utilities (SIC Code Sections D & E), Construction (SIC Code Section F), Retail (SIC Code Section G) and Transportation & Storage (SIC Code Section H)
Show 6 more
Digital & Comms (SIC Code Section J), Financial and Insurance (SIC Code Section K), Professional, Scientific & Professional Activities (SIC Code Section M), Administrative & Support Services (SIC Code Section N), Public Administration & Defence (SIC Code Section O), and Other Services (SIC Code Section S)
Principle:
Safety, security and robustness, Appropriate transparency and explainability, Fairness and Accountability and governance
Key function:
R&D, Product and service development and Risk management
AI Assurance Technique:
Data assurance, Formal verification, Performance testing, Risk Assessment and Bias Audit
Assurance Technique Approach:
Technical and Procedural

Background & Description

Advai were involved in evaluating the resilience of a facial verification system used for authentication, namely in the context of preventing image manipulation and ensuring robustness against adversarial attacks. The focus was on determining the system’s ability to detect fraudulent efforts to bypass facial verification or ‘liveness’ detection and to resist manipulations from fake imagery and feature space attacks.

How this technique applies to the AI White Paper Regulatory Principles

More information on the AI White Paper Regulatory Principles.

Safety, Security & Robustness

The assessment is crucial for validating the system’s security measures against adversarial image and feature manipulation attacks, ensuring that the system is robust and reliable.

Appropriate Transparency & Explainability

Providing transparency through a comparative rating for non-technical audiences to consider robustness and security-related metrics in their selection of a facial verification provider. Explainability was enhanced through the identification of specific features that unduly influenced a model or led to successful manipulation.

Fairness

We identified biases the model displayed against certain features like beards, glasses or sex-based features. We recommended the inclusion of diverse training data that our analysis showed the model needed for better balance. Subsequent improvements promote fairness at any business making use of the facial verification system.

Accountability & Governance

The red teaming approach and rigorous comparison with industry benchmarks holds the system to high standards of accountability. The reporting against these standards improves corporate governance.

Why we took this approach

This multifaceted attack approach was employed to uncover weaknesses that could be exploited by bad actors. Rigorous, empirical methods that exploit algorithmic traits of a system allow for a more objective analysis of the system. This surpasses the standard approach in industry, involving the testing of these systems using testing datasets (of normal unaltered images) to assign only an accuracy score. Further, the approach identifies specific components of model vulnerability thereby providing clear next steps for improving the facial verification system’s robustness.

Benefits to the organisation using the technique

  • Enhanced security through the discovery of system vulnerabilities and the implementation of training data and system mitigations.

  • Increased trust in the facial verification system from users and clients due to its measurable resistance to sophisticated attacks.

  • Insights into the model’s biases, allowing for the instalment of operational boundaries to prevent these biases, and the development of a better approach to procuring representative data.

  • Assurance for stakeholders through a demonstrated comparative advantage over other industry benchmarks.

Limitations of the approach

  • The adversarial attacks may not encompass all potential real-world scenarios, especially as attack methodologies evolve.

  • Findings may necessitate continuous re-evaluation and updating of the system’s security measures.

  • Recommendations may lead to increased complexity and cost in the system’s operation and maintenance.

Further AI Assurance Information

Updates to this page

Published 12 December 2023
Contents