Food Standards Agency: Food Hygiene Rating Scheme – AI
The aim of this tool is to help local authorities manage the hygiene inspection of food establishments. It supports local authorities to prioritise which businesses to inspect by predicting which might be at a higher risk of non-compliance with food hygiene regulations.
Tier 1 – Overview
Name
Food Hygiene Rating Scheme (FHRS) - AI
Description
1. How the tool works:
1.1 The tool is made up of a machine learning model, integrated into a web application that can be accessed by Stakeholders within the Local Authorities (LAs) in England, Wales and Northern Ireland. Use of the system is voluntary and aims to provide a standard methodology for prioritising inspections of food businesses based on their predicted food hygiene rating.
1.2 A machine learning framework called LightGBM was used to develop the FHRS AI model. This model was trained on data from three sources: internal Food Standards Agency (FSA) FHRS data, publicly available Census data from the 2011 census and open data from HERE API.
1.3 Using this data, the model is trained to predict the food hygiene rating of an establishment awaiting its first inspection, as well as predicting whether the establishment is compliant or not.
1.4 Access to the web application is only granted to stakeholders within Local Authorities who have agreed to the terms and conditions of use. Users will have access to information on establishments within their remit, for example, users from the Ealing LA will only be able to access information about establishments within the Ealing area.
1.5 This information is presented to the user in three ways:
- as a table showing the establishment, predicted compliance and the predicted rating
- as an interactive map, again showing the establishment, predicted compliance, predicted rating, and finally the type of establishment
- as a CSV file download
2.How it is incorporated into decision making process:
2.1 This will be understood in more detail post alpha testing phase. A high-level summary of how we expect this to be incorporated into the decision-making process can be found in section 3.1
3. Problem we’re aiming to solve:
3.1 A significantly greater number of registrations of new businesses have been received by local authorities compared with pre-pandemic levels, and more businesses have changed their functions, which may require a new inspection to take place.
3.2 As a result, the number of businesses classified as ‘Awaiting Inspection’ on the Food Hygiene Rating Scheme website has increased steadily since the beginning of the pandemic. This has been the key driver behind the development of the FHRS AI use case.
3.3 More broadly, it is necessary to investigate whether this new technology can help LAs to manage their resources and increase the impact of their regulatory activity.
4. Justification/Rationale:
4.1 The objective is to help local authorities become more efficient in managing the hygiene inspection workload in the post-pandemic environment of constrained resources and rapidly evolving business models.
4.2 Not using Machine Learning may reduce the ability of local authorities to make use of data and digital to tackle their backlogs or to operate more efficiently. In the long-term this unrealised benefit may threaten the integrity of the food hygiene rating scheme or reduce consumer protection because a backlog may remain, or local authority resources may be used less efficiently.
URL of the website
The link below provides details of the Food Standards Agency’s Food Hygiene Rating Scheme (FHRS). This scheme sets out to help consumers choose where to eat out or shop for food by providing each establishment with a food hygiene rating between 0 and 5. The link does not cover the use of the FHRS AI tool currently, rather it should be read to familiarise yourself with the broader FHRS scheme that will be supported by the tool.
Food Hygiene Rating Scheme - Food Standards Agency
Contact email
strategic.surveillance@foodgov.onmicrosoft.com
Tier 2 – Owner and Responsibility
1.1 Organisation/ department
Food Standards Agency (FSA)
1.2 Team
Regulatory Compliance Division (RCD)
1.3 Senior responsible owner
Julie Pierce (FSA Director of Openness, Data, Digital, Wales and Science)
1.4 Supplier or developer of the algorithmic tool
Cognizant Worldwide Limited
1.5 External supplier identifier
07195160
1.6 External supplier role
Cognizant, in collaboration with the FSA’s Strategic Surveillance Team and Regulatory Compliance Division, and numerous Local Authorities were involved in the creation of the FHRS AI system..
This involved:
- working with business and technical subject matter experts (SMEs) within the FSA to understand and define the use-case scope, approach and expected benefits and risks
- collating and preparing relevant data sources to be used for predictive modelling based on SME feedback and ensuring data is of an acceptable quality
- collaborating with SMEs to assess valuable approaches and datasets
- developing and testing a minimum viable product through iterative feedback sessions with the business owner and relevant SMEs
- presenting the minimum viable product (MVP), proposed next steps and roadmap for MVP deployment
- liaising with local authorities to test the model against human intellect and analyse the results
- facilitate future testing initiatives with local authorities
As well as a development role, Cognizant have provided Responsible AI consulting and delivery services, including the application of a parallel Responsible AI sprint to assess risk and impact, enable model explainability and assess fairness, using a variety of artefacts, processes and tools.
1.7 Terms of access to data for external supplier
Cognizant employees (and other subsequent data service providers) have access to secure FSA devices with their own login credentials. Access to any internal data is governed by contractual agreements between the FSA and the data service providers. The data is stored in a secure government cloud and access is managed by the FSA security team. In this particular case, the majority of data used to enable the system is from 2011 UK Census data and from the HERE API. This is data that can be publicly accessed by anyone free of charge. Information about an establishment’s Food Hygiene ratings are also readily available online.
Tier 2 – Description
2.1 Scope
The system has been designed for:
- Supporting local authorities by taking a data-driven approach to prioritising their inspection regime. The FHRS rating and compliance prediction from the model, along with the knowledge of the subject matter experts engaging with the system, should be used together to deploy LA resources more effectively to establishments who show traits of being at a higher risk of non-compliance, and therefore those which may have a lower food hygiene rating.
The system has not been designed for:
- Replacing the current FHRS scheme and inspection regime. This means that food hygiene rating predicted by the model should not be used in place of the rating provided via a formal inspection in any situation
- Using the predicted food hygiene rating in isolation to prioritise the inspection regime. This process should be a collaborative interface between the system and human decision maker to avoid any automated decision making
2.2 Benefit
The primary benefit the tool is expected to deliver is increased efficiency for local authorities in prioritising and inspecting food businesses, especially those within the ‘awaiting inspection’ category which are still awaiting their first inspection since registration. Currently, this process is manual, labour intensive and inconsistent across Local Authorities. Therefore, the use of a robust, data driven approach may allow for a more efficient utilisation of limited resources, by prioritising inspections of those businesses that are more likely to be non-compliant.
2.3 Alternatives considered
No non-algorithmic alternatives were considered for this use case. In terms of how the decision process is currently conducted, each local authority has a different process for triaging food establishments based on historical information and information inherent to the establishment, such as type of food, whether it is a care facility etc. This information would then be used to prioritise the inspection regime. Due to the variations across all LA’s, it would be impractical to document how each one conducts the triage process in detail but in general the following factors are looked at as part of the decision-making process.
- Catering to Priority Groups
- Handling Process and Food Type
- Food Operator History
- Premise Structure
- Reach and Distribution
2.4 Type of model
The algorithm being used as part of the FHRS AI use case at the FSA is the Light Gradient Boosting Machine (LightGBM) which is a framework for Machine Learning. The LightGBM is based on decision tree algorithms and used for ranking, classification and other machine learning tasks. As part of the use case, the model is used for classifying businesses based on the identified features as being compliant or not and further used for predicting a food hygiene rating (0-5).
2.5 Frequency of usage
Since the use case is in a MVP (Minimum Viable Product) stage, it hasn’t been released to actual end users as yet and hence the frequency of usage is something that cannot be determined at this point in time (February 2022). The Alpha pilot is currently planned for beginning in April 2022, where in the end users (at the participating Local Authorities) will have access to the FHRS AI service for use in their day-to-day workings. This section will be updated depending on the outcomes of the Alpha Pilot where we will be looking at the number of users accessing the service and time spent amongst other aspects.
2.6 Phase
Currently, the use case has been developed as a Minimum Viable Product. It is now entering a testing phase where it will be shared with a limited set of Local Authorities for system testing and feedback with multiple rounds planned. This phase can be described to be a ‘Minimum Viable Product (MVP) testing’ phase. If the FSA decides to operationalise the service, further iterations of enhancement and testing will be conducted.
2.7 Maintenance
Since the use case is in a MVP (Minimum Viable Product) stage, it has not been released to actual end users as yet and hence the maintenance schedule is something that cannot be determined at this point in time (June 2022). The Alpha pilot started at the beginning of April 2022, wherein the end users (the participating Local Authorities) have access to the FHRS AI service for use in their day-to-day workings. This section will be updated depending on the outcomes of the Alpha Pilot through which we aim to capture feedback around the following:
- Time saved by usage of service
- Usefulness of service in prioritising inspections
- Suitability for use as part of day to day work in the future
- User Interface design
- Requirement for a bi-lingual interface, especially for the Welsh LAs
- Feedback on onboarding process and guidance document
- Qualitative feedback on end to end process
2.8 System architecture
The below captures the high-level design for the key details of the service.
Tier 2 – Oversight
3.1 Process integration
Utilising the service, the Environmental Health Officers (EHOs) are provided with the AI predictions, which are supplemented with their knowledge about the businesses in the area, to prioritise inspections and update their inspection plan. Please find the attached graphical representation of of the , as envisioned for the future.
Once the service is released to Local Authorities for use as part of the Alpha pilot, we will have a better understanding of usage and how it fits into the LAs’ decision making processes.
Note: the system has been designed to provide decision support to Local Authorities. FSA has advised Local Authorities to never use this system in place of the current inspection regime or use it in isolation without further supporting information.
3.2 Provided information
The FHRS AI service is designed to share the compliance predictions for Food Establishments which are awaiting inspection. Through different consumption options available through the service (a list of predictions, a geographic prediction map or a downloadable predictions file) the end users get access to the following information
- Local Authority Code and Name the food business falls under
- Business Name, Type and Operating Post Code
- Compliance Prediction along with the predicted FHRS rating (0-5)
- Probability of a food establishment being compliant
The above information is only available to end users for the Local Authorities that they belong to based on the data security rules applied.
3.3 Human decision
The role of the FHRS AI service within the LA decision making process for prioritising business inspections has been illustrated in 3.1.
It is expected that there will be possibly multiple review points, with users being able to access only the predictions of their own Local Authority.
3.4 Required training
While there is no training provided or a pre-requisite for using the service, a user guide has been created for users of the system. This will be a single document which the end users can reference to understand how the service needs to be accessed and the different features available along with providing guidance in the form of FAQs on the recommended usage.
3.5 Appeals and review
The FHRS AI service is a voluntary service being made available for Local Authorities and is foreseen being used as a data driven approach to support the inspection process by helping them prioritise inspections of businesses awaiting inspection.
Since there will be no change to the current inspection process by introducing the model, the existing appeal and review mechanisms will remain in place. Although the model is used for prioritisation purposes, it should not impact how the establishment is assessed during the inspection and therefore any challenges to a food hygiene rating would be made using the existing FHRS appeal mechanism.
Tier 2 – Information on data
4.1 Source data name
The different internal and external sources that are processed as part of the FHRS AI model are
- FHRS Data (FSA Internal)
- Census Data (Open)
- HERE API Data (Open)
4.2 Source data
The FHRS AI dataset contains approximately 120 data descriptors. While all of these cannot be detailed, the broad groups that these belong to can be represented as follows:
- Business Category and Type (Internal Data)
- Business Name and address (Internal Data)
- Local Authority Code and Average food hygiene rating (Internal Data)
- Demographics of Area (External Data)
- Business Size (External Data)
- Socio-Economic and Census Details of Area (External Data)
- Business Operating Hours (External Data)
4.3 Source data URL
The dataset utilises data from sources both internal and external to the FSA. No URL is available for this dataset as it is not publicly available.
4.4 Data collection
The majority of data used to create the dataset comes from the Food Hygiene Rating Scheme collected by the different Local Authorities across England, Wales and Northern Ireland. The Local Authorities are responsible for conducting inspections of food establishments. Along with the FHRS system, the Census data prepared by the Office of National Statistics was used along with geographic data extracted from the HERE API.
The FHRS system database has a history of hygiene inspections along with information about all the food businesses in the country. Since the data is collected and updated by the Local Authorities, this is considered as a true record of the core data about a food business. The Census data is maintained by the ONS and is assumed to require no additional verification/validation. The HERE API has not undergone a formal vetting process.
4.5 Data sharing agreements
There were no data sharing agreements required for the development of this system.
4.6 Data access and storage
The dataset will be accessed by data scientists at the FSA and be used for generating predictions to identify likely non-compliant food businesses which can inform Local Authorities’ inspection planning. The Local Authorities will not have access to the raw data and will only handle the predictions being generated from the model. The details regarding governance are considered sensitive and are not being shared outside of the Agency for now with the details still to be finalised as an outcome of the Alpha pilot.
Tier 2 – Risk mitigation and impact assessment
5.1 Impact assessment name
The different impact assessments conducted during the development of the use case were
- Responsible AI Risk Assessment
- Stakeholder Impact Assessment
- Privacy Impact Assessment
5.2 Impact assessment description
Please find below a description of the impact assessments mentioned as part of section 5.1
-
Responsible AI Risk Assessment - Identify potential risks related to the use case, data and technology used to enable the use-case. Identification of these risks naturally leads to the consideration and documentation of potential mitigation techniques. This is conducted iteratively throughout the development of the use case, ensuring risks are continuously identified, assessed and managed.
-
Stakeholder Impact Assessment - Build confidence in the way you’ve designed and deployed your system by bringing to light unseen risks that can threaten to affect individuals and the public good. Use this assessment to demonstrate forethought and due diligence to the wider public that you and the project team have collaborated to evaluate the social impact and sustainability of your AI project.
-
Privacy Impact Assessment - Structured process to identify and minimise data protection risks of a project. They are a legal requirement under the GDPR when processing data that may lead to high risks to individuals. Generally, it is good practice to complete a privacy impact assessment when using personal data or data you think may constitute personal data.
5.3 Impact assessment date
The assessments are conducted in an iterative process throughout the design, development and delivery of the use case. While the assessments were carried out at an interval of every 10-12 weeks, the exact dates can be provided if absolutely required.
5.4 Impact assessment link
Due to the confidential nature of the information contained in these assessments, this information cannot currently be shared outside of the agency.
5.5 Risk name
The foreseen risks associated with the use of the FHRS AI service include the following
- Use of Personal Information
- Model Bias
- Inspection Bias
- Decision Automation Bias / Technological Halo Effect
5.6 Risk description
The risks listed in 5.5 are further detailed out below
- A number of businesses are registered at private home addresses. This information is required to be managed and used securely as it may constitute personal information.
- Potential bias from the model (e.g. consistently scoring establishments of a certain type much lower, less accurate predictions)
- Potential bias from inspectors seeing predicted food hygiene ratings and whether the system has classified the establishment as compliant or not. This may have an impact on how the organisation is perceived before receiving a full inspection.
- With the use of AI/ML there is a chance of decision automation bias or automation distrust bias occurring. Essentially, this refers to a user being over or under reliant on the system leading to a degradation of human-reasoning.
5.7 Risk mitigation
The risks listed in 5.5 and further elaborated in 5.6 have been carefully looked at and mitigated through
- While the private addresses are not included as part of the Alpha version of the service, a Privacy Impact Assessment has been conducted and signed off by the FSA’s Knowledge, Information Management and Security (KIMS) team
- Integration of explainability and fairness related tooling during exploration and model development. These tools will also be integrated and monitored post-alpha testing to detect and mitigate potential biases from the system once fully operational
- Continuously reflect, act and justify sessions with business and technical subject matter experts throughout the delivery of the project, along with the use of the three impact assessments outlined earlier to identify, assess and manage project risks
- Development of usage guidance for local authorities specifically outlining how the service is expected to be used. This document also clearly states how the service should not be used, for example, the model outcome must not be the only indicator used when prioritising businesses for inspection.