Policy paper

Investigatory Powers (Amendment) Bill: Bulk Personal Datasets and Third Party Bulk Personal Datasets

Updated 26 April 2024

What are Bulk Personal Datasets?

The UK intelligence services (MI5, SIS and GCHQ) need to examine a variety of data from a wide range of sources to meet their statutory functions, in some circumstances this data will be retained by the intelligence services, or – if it is held by third parties – it may be examined in situ.

In 2016 the Investigatory Powers Act (IPA 2016) put BPDs on to a statutory footing for the first time. Part 7 of the IPA 2016 sets out a robust regime, including a requirement that the retention and examination of a BPD must be under a warrant issued by the Secretary of State and approved by a Judicial Commissioner (JC). This is known as the “double lock”. The BPD Code of Practice provides additional guidance and procedures governing the retention and examination of BPDs by the intelligence services.

The current definition of a BPD in the IPA 2016 is very broad and can capture a wide range of datasets. Whilst it will obviously apply to highly sensitive datasets, the current definition also captures publicly and commercially available datasets, including for example commonly used datasets that can be freely downloaded by anyone from the internet. The IPA 2016 currently applies the same level of safeguards to all BPDs, regardless of the sensitivity or public availability of their contents, or the level of intrusion associated with the intelligence services retaining and examining them.

This broad definition continues to cause operational difficulties for the intelligence services, by inhibiting their ability to use this data at the scale and pace needed to keep pace with technological advances and ahead of the UK’s adversaries. These difficulties are likely to increase with the continued exponential growth of publicly and commercially available data.

Case study: example of how BPDs are currently used

MI5 became aware of an unidentified individual whose behaviour indicated they may have been preparing themselves to conduct a terrorist attack. The individual was planning to travel to the UK and MI5 was concerned that, if the intelligence proved correct, the individual would pose a threat to UK national security and British lives could be at risk.

It was imperative that MI5 fully identified the individual before they travelled to the UK, but only partial information about the individual was available. Based on this scant detail, Investigatory Powers Act provisions on bulk personal data enabled MI5 to fully identify the individual and confirm they were of national security concern. The successful identification of the individual using bulk data allowed further intelligence to be gathered, illuminating their activities and intent, and enabling a successful disruption that mitigated the terrorist threat to the UK.

Why is change to the BPD regime needed?

The UK faces a broader and more complex range of threats compared to 2016, with the clues hidden in exponentially growing and varied data as a result of significant shifts in the way, and frequency, in which people communicate with each other. The proposed changes in the bill will improve the quality and speed of analysts’ decision-making, improving our intelligence services ability to keep the public safe in a digital age, whilst adhering to strong, proportionate safeguards and oversight that continue to strike an appropriate balance between privacy and security.

Changes to the BPD regime are needed:

1. to enable the intelligence services to keep up-to-date in tackling existing and emerging threats to our national security

2. to keep pace with significant technological developments

3. to provide greater operational agility and ability to process the ever-growing volume of data across society; and

4. to distinguish between different types of datasets to ensure that the safeguards are both robust and proportionate to the expectation of privacy based on the nature of the dataset.

1. Tackling threats to national security

• It is essential that the intelligence services have the necessary tools to help keep the public safe. Their job is becoming increasingly difficult in the face of evolving and varied threats to our national security. Their ability to access data at volume and pace is critical to ensuring that they can keep the public safe.
• As terrorism shifts away from complex plots directed from overseas and towards self-inspired ‘lone actor’ events, the intelligence services must increasingly access a wide range of publicly- and commercially- available data in order to spot individual behaviours that, when aggregated, can lead them to the threat.
• By making changes to the way in which the retention and examination of bulk personal datasets is regulated, we can enable the intelligence services to unlock the full potential of datasets where there is low or no reasonable expectation of privacy, such as publicly available data. Ultimately this will help them to detect threats and reduce the potential for missed opportunities to prevent terrorist attacks.

2. Keeping pace with technological developments

• The technologies that people are using to communicate with each other has developed significantly since the 2016 act. This alone has caused a huge increase in the volume of data across society. These communication platforms are not just used by the public, but by criminals, hostile state actors and those wishing to do us harm.
• In order to find the data relevant to serious threats to our national security, the intelligence services need to be able to process that data at a greater pace and greater scale to reflect this societal shift.
• This is not something that can be done without the help of modern technology, such as machine learning. The intelligence services are not interested in examining data that is not operationally relevant, but in finding ways to identify the specific threat in vast quantities of data.

3. Operational agility

The current act includes stringent handling requirements for BPDs. These requirements and safeguards are absolutely essential and appropriate for sensitive datasets. However, they have a detrimental and disproportionate impact on operational agility when applied to datasets that are already publicly available and have a lower expectation of privacy.

• For example, currently if an intelligence service sought to retain and examine a public telephone directory, which could be accessed and obtained online, a “double-locked” BPD warrant would be required, and the dataset would need to be handled in accordance with the safeguards that were designed for highly sensitive datasets.
• This includes holding the dataset in an environment designed to protect the UK’s most sensitive data, which limits opportunities to collaborate with partners, particularly on developing shared technical solutions or capabilities. This is disproportionate to the likely intrusion associated with the retention and examination of such datasets.

4.  Safeguards

• The measures in the bill will enable the intelligence services to distinguish between highly sensitive datasets and those in respect of which there is a low or no reasonable expectation of privacy, and to apply more proportionate safeguards to the latter. The Investigatory Powers Commissioner will continue to provide robust independent oversight of the retention and examination of all BPDs, including those to which the new provisions will apply.
• The bill would also increase the duration of a BPD warrant from six to twelve months in order to better demonstrate the necessity and proportionality of retaining and examining the data, the case for which can be made more effectively over this longer time period.

What safeguards will be in place to oversee the new BPD regime?

The new regime will set out strong, proportionate safeguards that reflect the nature of the datasets to which it is intended to apply.

In order to ensure that the new regime is appropriately applied, a system of prior judicial approval will apply, as well as stringent ex-post facto oversight through regular IPCO inspections. 

What are Third Party Bulk Personal Datasets?

Third party bulk personal datasets (3PD) are datasets that, were they retained and examined by the intelligence services, would constitute a BPD and require a warrant under Part 7 of the IPA, but which are instead held by third parties and examined by the intelligence services in situ. 3PDs can include datasets held by wider UK government bodies and commercial entities.

3PDs provide the same critical operational benefit as BPDs. This data is used by the intelligence services in multiple different ways; for example, to provide ‘building block’ intelligence, such as names of subjects of interests, details of travel, and their associates. This allows analysts to pull together an assessment on the possible meanings of the fragmentary intelligence the security and intelligence services receive.

The 3PD provisions in the bill will apply to the examination of third party bulk personal datasets by an intelligence service where that examination is not generally available (including on a commercial basis).

Why are you making changes to the 3PD regime?

Lord Anderson’s review of the IPA noted that the Investigatory Powers Commissioners Office (IPCO) conducted an ‘extensive review’ of 3PDs in 2019 and concluded that the intelligence services’ current access was compliant with Part 7, but also recommended that the government consider bringing them within IPCO’s oversight.

The government is committed to transparency and supports the position of the IPC and Lord Anderson in respect of 3PDs. The Investigatory Powers (Amendment) Bill has provided the opportunity to place UKIC examination of 3PDs onto a statutory footing.

This will ensure that 3PD examination is underpinned by robust statutory oversight and safeguards, while also ensuring that our intelligence agencies have the critical tools they need to protect national security and keep the public safe.

3PD case studies

Third party bulk personal datasets are datasets held by external partners such as other government departments or by commercial entities, which if it were retained by UKIC, would meet the criteria of a BPD. This is data that is being collected or held by these external entities for their own needs or business models, which they grant access to by others. Access supports the intelligence agencies in carrying out their core missions in protecting the UK’s national security.

Case study 1 (HMG 3PD)

For example, an intelligence service may access government held immigration related datasets to conduct checks to ensure those entering the UK do not pose a risk to national security.

Case study 2 (commercial / non HMG 3PD)

Many commercial companies acquire various datasets as part of their own business objectives and offer access to these to a variety of customers. Access to such datasets may offer the intelligence agencies different capabilities and insights in support of carrying out their statutory functions. It would not be feasible or proportionate for UKIC to acquire and retain the data themselves.

What safeguards will there be for the 3PD measures?

The regime will introduce 3PD warrants which are subject to a “double lock”. The double lock provides extra assurance that 3PD warrants will only be issued when it is in support of UKIC’s statutory functions. Introduction of a 3PD warrant which is subject to a “double lock” mirrors the existing and well-established safeguards that underpin other powers in the IPA.

The Investigatory Powers Commissioner and their office will have a role in overseeing the intelligence services use of 3PDs ensuring access is necessary, proportionate and lawful.