Transparency data

Justice technology and artificial intelligence (AI) roundtable: 26 March 2024 (accessible version)

Updated 20 August 2024

Attendees:

Chair: Jonathan Fisher KC

Host: Fiona Rutherford – Chief Executive, JUSTICE

Anita Clifford – Barrister, Red Lion Chambers

Professor Mark Watson-Gandy – Biometrics and Forensics Ethics Group (Chair)

Sarra Fotheringham – College of Policing (Policing Standards Manager for Digital, Cyber & Data)

Caroline Dorman – CPS (Serious Economic Organised Crime & International Directorate)

Fran Begley – Ernst & Young LLP (Director, Forensic & Integrity Services)

Liz Archer – Ernst & Young LLP (Associate Director)

James Raymond – HMRC (Head of Forensics)

Ian Spencer – HMRC (Operational Lead, Cyber, Crime & Forensic Services)

Hamera Asfa Malik-Wright – SFO (Deputy Head of Policy)

Nicholas Jinks – UBS (Regional Head, Switzerland/ EMEA eDiscovery)

Home Office officials

Disclosure officers and AI

1. It was noted that the CPS need to be able to prosecute quickly and with confidence.

2. One participant felt that if technology was introduced as a tool, the job of the disclosure officer would change. The officer would need to undertake further training on how to use the technology at a basic level as well as use their disclosure knowledge to operate it correctly. Another participant suggested that further training for disclosure officers may not be necessary as they could work with a professional who understands the tool; reservations were expressed about whether there was sufficient funding to facilitate this.

3. It was suggested that there would be a higher level of confidence if only one tool was used by everyone. On the other hand, consideration would need to be given to whether this would lead to a monopoly forming.

4. It was observed that disclosure officers would need a good understanding of relevance given that the AI would learn from humans. If the operator classed every file as relevant, the AI would not know what to look for. Understanding the defence position would be key to the assessment of relevance.

Advance material management software

5. Discussion turned to the use of specific advance material management software such as, but not limited to, Axcelerate and Relativity. Participants noted that previously when using such tools, screenshots of how searches were conducted on schedules have been included to build confidence and allow the defence an opportunity to question the methodology used.

6. In terms of redaction, some such software have built in an auto-redact tool. Mistakes have been previously found in a small number of results, though problems mainly lay in Excel spreadsheets and the way names and numbers were recorded as raw data rather than the tool itself or the way it was being used. Concerns were expressed about the idea of automating redaction, as it would be difficult to justify decisions taken if they were questioned by a judge. It was noted that it would soon be possible for AI to start explaining its decisions, rather than just giving results.

Further issues

7. A number of further points were raised including the issue of hallucinations regarding generative AI, which was suggested to be the biggest problem with ChatGPT. OpenAI are aware of this issue, and so this would likely be a priority for them to fix. It was noted that AI models will continuously improve and new versions will be released. Statistical sampling and elusion testing could be used to check that they were working as intended. Whilst dip sampling is still currently necessary, it was thought that we are probably less than 5-10 years away from AI being widely used and trusted.

8. Finally, participants wondered whether data protection issues would arise if a big technology firm were to buy software used by UK law enforcement. It was noted that one technology firm is already looking to introduce their own eDiscovery tool and have already worked quite closely with such software companies. The issue of security (especially regarding government use) was flagged, as some such material management software is cloud-based. It was noted that these tools are not designed solely to assist with investigations in the criminal justice system.