Guidance

Essential Shared Data Assets (ESDAs) - Guidance for departments (HTML)

Updated 26 April 2024

1. Year one objectives and supporting resources

Essential Shared Data Assets are the data assets that are critical from a cross-government perspective. Our shared ambition is for all departments to have 10 to 20 ESDAs identified, including those related to the ‘Top 75 services’, by April 2024. Once this process becomes established across government, we expect the number of ESDAs identified to grow over time. As the CDDO-led work on the Data Marketplace evolves, guidance and processes will be updated and simplified.

These supporting resources have been designed to assist you as part of the Beta phase in initiating, identifying and communicating information about ESDAs.

The supporting resources cover areas of the process. These areas include:

• planning - led by a central team on behalf of the organisation. This stage will require analysis, development of a communications plan and other supporting material to enable an agreed list of ESDAs
• assessment - the central team within the organisation will engage with all directorates to identify, assess, agree and collate a list of designated ESDAs
• submitting returns - the central team within the organisation will need to provide two returns to the Framework and Standards Team within CDDO which conforms to agreed metadata specifications
• peer review - the Frameworks and Standards Team will collate a list of returns from across government departments and will make it available to CDO Council for peer review

At the end of the Beta phase the ESDA initiative will be subject to review with a decision made as to whether it should be continued or revised according to insights from participating organisations.

1.1 Year one objectives for departments

As part of the Beta, all departments should complete an audit and assessment of their data assets and determine up to 20 ESDAs (more can be provided if departments are able to) by the end of April 2024. This must include data assets related to the delivery of any of the top 75 services listed within the Roadmap for Digital and Data.

Completion of your return of designated ESDAs means that you have:

• clear accountability, leadership and delivery plan for the audit
• designation of ESDAs and ownership in place for each ESDA
• reported and communicated the results to internal stakeholders
• reviewed the list of ESDAs in the context of cross-government strategic policy and operational priorities
• developed and maintained the minimum metadata required for each ESDA
• communicated and embedded the Data Ownership Model for those responsible for the data assets designated as ESDAs

Departments are also responsible for coordinating and assuring returns from their arm’s length bodies to the CDDO. Departments and their arm’s length bodies are aligned in departmental plans, delivery of services and shared objectives and as such it is important that they are coordinated and consistent in the identification of ESDAs. Department’s should prioritise engagement and returns from their arm’s length bodies that are likely to collect and be responsible for data to fulfil their functions.

1.2 Communicate with the Central Digital and Data Office

The Frameworks and Standards Team at the Central Digital and Data Office are available to provide advice to assist departments carry out the process of determining and reporting their ESDAs. Through the first year, departments can/should work with CDDO as follows:

• End of December 2023 - provide an initial return of their potential ESDAs with a limited number of metadata requirements
• January to February 2024 - CDDO to carry out light touch quality assurance process of initial returns and engagement with departments
• End of April 2024 - provide final return of designated ESDAs to CDDO in compliance with full metadata requirements as per the Cross-Government Metadata Exchange Model Specification
• May 2024  - CDDO to carry out light touch quality assurance process ahead of peer review via CDO Council
• June 2024 - review of ESDA initiative and data ownership model reporting to DSA Steering Board and CDO Council

Engagement with CDDO will ensure that you receive the advice and support required, as well as provide CDDO with the opportunity to understand and respond to possible issues.

1.3 Feedback on these resources

These resources support people and teams with responsibility for carrying out an audit of their data and undertaking an assessment and designation of an ESDA. The supporting resources were developed by the CDDO and were iterated following the conclusion of the alpha pilot.

The designation of ESDAs will be achieved through a self-assessment process and, where appropriate, agreement between the parties involved in the disclosure of data.

The first list will consist of data assets your organisation provides to another government department or agency to support the delivery of an essential process or purpose for which that organisation is responsible. Your organisation is responsible for the designation of ESDAs on this list.

We want to assure the list of ESDAs provided by organisations and to do that we need a second list from you. The second list consists of data acquired by your organisation from another government department or agency to deliver an essential process or purpose, which is the responsibility of your organisation. Where possible your organisation should engage with the provider organisations to ensure they have considered its designation as an ESDA. The second list requires a smaller set of metadata about these datasets your organisation is reliant upon to help ensure CDDO can undertake a review, identify any significant gaps and engage appropriately with departments and agencies for their potential designation of their data assets as ESDAs.

1.4 CDDO Support

The CDDO will support departments in a number of ways throughout the Beta phase. Firstly, it will make available a package of material for organisations to help explain and promote this initiative in plain english for senior leaders and other stakeholders likely to have an interest. The CDDO will also establish a cross-government community so officials can share experiences, identify good practices and issues which can be addressed through updated guidance. 

1.4 CDDO Support

The CDDO will support departments in a number of ways throughout the Beta phase. Firstly, it will make available a package of material for organisations to help explain and promote this initiative in plain english for senior leaders and other stakeholders likely to have an interest. The CDDO will also establish a cross-government community so officials can share experiences, identify good practices and issues which can be addressed through updated guidance. 

1.5 Leadership and accountability

You will need senior, strategic and delivery leadership and accountability to ensure ESDAs are appropriately assessed and designated. The CDO (or equivalent role) is responsible for collating the list of ESDAs for their organisation. In addition to Data Owners and Stewards, a wide range of stakeholders across your organisation, such as service owners, analysts and policymakers, will be accountable for ensuring the list of ESDAs accurately reflects the data critical for essential processes and purposes within their areas of responsibility. 

1.6 Developing maturity

The intention is for a light touch process to be carried out in the first year, with each department being asked to provide a list of 10 to 20 ESDAs. We expect central teams to use their departmental knowledge to provide returns in consultation with the responsible senior leaders and Data Owners. Once this has been embedded, we will look to departments to undertake more detailed engagement and auditing in subsequent years to build a more comprehensive list of ESDAs.

As part of the government’s commitment to developing a data marketplace by 2025, the CDDO is creating a federated data catalogue proof of concept. The data catalogue will hold metadata about ESDAs but will not hold any of the actual data assets. We aim to provide a searchable catalogue of metadata about ESDAs available to all government departments, which will gather metadata regarding data assets designated as ESDAs directly from department’s data catalogues via APIs.

2. Planning

A central team under the leadership of the CDO should take responsibility for this initiative within your organisation. If your organisation doesn’t have a CDO, the most senior official within your organisation responsible for data across the organisation will take on responsibility.

Firstly, the team should develop a delivery timetable. It is essential to factor in that the designation of ESDAs must be agreed upon by the Data Owner, who is likely to be a senior official with many competing priorities. As such, you must allocate a realistic timetable for engaging with them and obtaining their approval for a designation. If the Data Owner role does not exist within your organisation, you may need to speak with the information asset owner, who is accountable for crucial decisions about using and managing a particular asset.

The team should review the existing organisational data catalogue (or information asset register where a data catalogue does not exist) and any critical data assets or critical data elements identified. Where data-sharing arrangements exist for those data assets, the threshold for an essential process or purpose may have already been met. In this case, the data asset should be flagged as a potential ESDA but recognising that purpose limitations and conditions must be understood and adhered to for any further reuse of that data asset.

The National Situation Centre (SitCen) within the Cabinet Office maps data related to the National Security Risk Assessment for use in COBR-level responses. A network of Data Liaison Officers (DLO) has been established to act as primary points of contact in the major departments to identify crises-related data. When considering data assets for designation as an ESDA which relate to defence, national security and resilience processes or purposes the team should engage with their DLO and consider which assets are classified at Official level and suitable for designation as an ESDA.

A shortlist of potential ESDAs should be collated with the contact details of their respective accountable owners. Finally, the CDO should review the shortlist before responsible owners are contacted to agree on the designation of their respective ESDAs.

A communications strategy and supporting material should be created and ready for the next delivery stage. 

2.1 Important delivery considerations

Planning the process requires making a range of decisions and choices. Documenting your decisions and rationale for later reference and sharing with others in governance is an example of best practice.

2.2 Strategic targeting for your return

In this Beta phase, you should focus on strategically important parts of your organisation. These could be:

• a specific domain
• service, including those identified as a Top 75 service
• business unit or function fundamental to your remit, strategy or business success

You should agree on which strategically important part of the organisation will be in scope with your CDO.

Top 75 services must be in scope of the assessment. Not all data assets underpinning Top 75 services will be designated an ESDA. If attributes or data elements in a data asset stemming from a Top 75 service are acquired by another government department or agency to deliver an essential purpose or process which it is responsible for, you should designate the whole data asset as an ESDA. You should also identify any data sources from another government department or agency that your Top 75 Service is reliant upon and propose its designation as an ESDA with the Data Owner.

 A ‘whole organisation’ approach may be the best route to value for smaller or less complex organisations.

The designation of a data asset as an ESDA can be removed, such as when a service is no longer provided or where an essential purpose or process uses an alternative data source.

3. Assessment

The central team will contact each Data Owner with details of the data assets the Data Owner is responsible for. The Data Owner will also receive guidance with a request to confirm whether any of the data assets identified should be determined an ESDA.

If you’re contacted as the data asset owner of a potential ESDA, you should check and ensure you’re the appropriate accountable owner for the data asset that’s been identified.

If you have changed your role or you’re no longer the Data Owner for the particular data asset(s), you must notify the central team and provide the name of the new Data Owner.

You should work with your Data Stewards to understand and determine whether any data assets you are accountable for are shared with another department or agency to support the delivery of an essential process or purpose which is the responsibility of that organisation.

You should consider what processes or purposes could not be delivered by another organisation if they could no longer access the data you have accountability for. Data Owners should look at existing data sharing agreements in place for their data assets, as the criteria for an ESDA are likely to have been met as part of the rationale for agreeing to the share. Data Owners (through their Data Stewards where necessary) should ensure that an appropriate representative from the acquiring department agrees with the designation of the data asset as an ESDA.

A data asset should be designated as an ESDA if it includes one or more attributes or data elements which are needed for an essential purpose or process. It should be noted that operationally, multiple data sets could be derived from an ESDA to satisfy different requirements and to ensure appropriate, ethical and legal data sharing.

Similarly, where datasets may be extracted from a data asset, such as on temporal (e.g. UK births in 2021 and UK births in 2022) or geographical (e.g. births in 2023 by specific regions) lines, departments should designate the source data asset as the single ESDA.

Data with a one-time need should generally be excluded from the definition of an ESDA.  However, there may be instances (such as developing a policy paper) when a data asset will temporarily be an ESDA. In such cases, the rationale for its inclusion and subsequent change to designation should be provided to the central team and recorded for auditing purposes. In all cases, the designation of an ESDA should not be permanent and always subject to review.

3.1 Recording a designation

An essential objective of identifying cross-government critical data assets is to quantify and understand the data-sharing pressures on provider organisations. To realise this, it is necessary to determine which data has been designated as an essential shared data asset.

There are requirements for the information you need to provide on your ESDA to ensure that returns from organisations are consistent and support the discoverability and shared understanding of ESDAs across government. The CDDO, in partnership with a wide range of public sector organisations, has developed a Cross-Government Metadata Exchange Model Specification. Data Owners may delegate responsibility for providing this metadata to their Data Steward.

The central team should ensure that the organisation records designations of data assets as ESDAs. A centrally managed spreadsheet or database of ESDAs should be created and maintained. A summary list of the ESDAs for each organisation must be returned to CDDO with metadata conforming to the Cross-Government Metadata Exchange Model Specification.

If your organisation has a data catalogue, you may wish to flag within your catalogue if a data asset is an ESDA. That metadata element must include information about the acquirer and the essential shared data asset category under which the designation was agreed, amongst other elements as described in the guidance. A metadata element such as this will enable programmatic identification of assets designated as an essential shared data asset.

3.2 Disputes

Where parties do not reach an agreement or dispute the assessment result, the CDDO will moderate the process. The CDO Council will make the final decision when a resolution has not been achieved.

3.3 Enable your organisation to explore what the list means

The designation of ESDAs provides a picture of what data sharing is essential for your organisation. This is important from a risk management perspective, ensuring resources are appropriately allocated, and to support the targeting of data improvements. Organisations will need to recognise and understand any input data in the supply chain that the ESDA is reliant upon. Organisations may wish to promote the development of Data Product Specifications and establish collective ownership of it across the data supply chain. All input data (to create an ESDA) does not need to be considered ‘critical by default’; rather specific recognition that their supply is essential to create critical data. 

Your organisation is encouraged to review the list of its ESDAs as part of its corporate risk governance processes and monitor any changes to its list.

4. Submitting returns

The CDO (or equivalent role) for your organisation should review and approve the list of ESDAs collated by the central team. The central team must maintain an audit of communications with Data Owners, including any explanations provided to determine ESDAs to avoid duplicated requests and unnecessary contact. Quality assurance of returns should be carried out before submission to CDDO. A template spreadsheet is available (one for initial returns by December 2023 and another one for full returns to be provided by April 2024), which should be completed and returned to the Framework and Standards Team within the CDDO via data-standards-authority@digital.cabinet-office.gov.uk. Organisations must collate and maintain a comprehensive list of their ESDAs, but they have the discretion to not provide the recommended or optional metadata fields if they a) provide the underlying reason or b) provide with a request for it to be hidden from view and only accessed/visible to restricted users so this dispensation is not misused.

Departments are expected to provide two returns. By the end of December 2023, departments are expected to provide an initial list of ESDAs. The following metadata will be required for the potential ESDAs your organisation has identified and listed in these initial returns which should conform to the Cross-Government Metadata Exchange Model Specification:

• Title - Mandatory
• Alternative title - Recommended
• Summary - Recommended
• Description - Mandatory
• Keyword - Optional
• Theme - Recommended
• Contact Point - Mandatory
• Contact Point Email - Mandatory
• Creator - Mandatory
• Identifier - Mandatory
• Update Frequency - Mandated

This return should also include an initial list of data assets that your organisation acquires from other departments or agencies to deliver an essential process or purpose, which is the responsibility of your organisation. The following metadata is required for this list:

• Title - Mandatory
• Department (from which you receive the data) - Mandatory
• Contact Point - Mandatory
• Contact Point Email - Mandatory
• Purpose or process supported - Recommended

Departments must also identify their ALBs which are likely to hold and share data with other parts of government and engage with them to coordinate and assure returns from them to CDDO.

The CDDO will carry out some light touch quality assurance of returns and may contact departments if any potential ESDAs have been identified through returns from OGDs.

Between January and the end of April Departments should be working to provide their final returns of their designated ESDAs. This will encompass agreement of the designation of ESDAs by Data Owners and the associated metadata for the data assets they are responsible for along with ensuring they understand and accept their accountabilities as a Data Owner set out in the Data Ownership Model.

5. Peer review

Once returns have been received from all organisations within scope, the Framework and Standards Team will put together a summary list of ESDAs across government. The summary list will be provided to the members of CDO Council and peer reviewed. The peer review will provide the opportunity to assess the quality of the list and metadata provided, including completeness. The peer review may result in actions being placed on CDOs to include additional ESDAs if agreed. Insights from the peer review will help inform updates to the policy and guidance. 

6. Legal and copyright notices

© Crown copyright, 2023

Licensed under the Open Government Licence v3.0 except where otherwise stated.

Provenance:

Essential Shared Data Assets

Licence:

This work is licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.

Creative Commons, PO Box 1866, Mountain View, CA 94042, USA.

The licence allows:

Sharing, copying and redistributing the content.

Adapting - remixing, transforming, and building upon the material.

NB. For the purposes of use in government, please contact CDDO before making changes to the guidance.

Under the following terms:

• Attribution — You must give appropriate credit, provide a link to the licence, and indicate if changes were made. You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use.
• NonCommercial — You may not use the material with the primary purpose of commercial advantage or monetary compensation.
• ShareAlike —  If you remix, transform, or build upon the material, you must distribute your contributions under the same licence as the original.
• No additional restrictions — You may not apply legal terms or technological measures that legally restrict others from doing anything the licence permits.