This article is more than 1 year old

Identifying terrorists: Let's find a value for needle in haystack

Or DO we bulk-slurp citizens' data, trawl whole %^* thing

IPB So you're looking to stop a terrorist attack. What do you do?

The choices are:

(a) build the largest haystack about all the population because you know that the needle has to be in there “somewhere”; or

(b) have the powers to look at all the relevant smaller haystacks that are around when you have inkling as to what kind of needle you are looking for.

In Article 8 Human Rights terms, does Parliament enact legislation that allows the national security agencies to collect bulk personal data when there is no prior suspicion, so these agencies can do speculative searches in the hope they get lucky?  Or do you have the traditional civil liberties view that you need a modicum of prior suspicion before you go looking?

The Home Office prefer the former; the civil liberties lobby the latter - and that is one of the key divisive issues at the heart of the Draft Investigatory Powers Bill (“DIP”) published last week.

Clearly, David Anderson QC, who has looked at the issue of bulk data collection, recognises that the civil liberties stance that requires prior suspicion must adapt; this is because Annex 9 to his report lists half a dozen cases where access to bulk communications datasets have produced results. However, this does not mean the Home Office can use this support in order to propose draft legislation that goes off the scale in the other direction.

In summary, unfettered and mass personal data collection in the name of national security and serious crime is one of DIP’s main proposals.  The Home Office, perhaps sensing a chance to enhance the surveillance powers for the organisations that reporting to it, has drafted general provisions that gives the national security agencies (and police in the context of “serious crime”) the power to obtain any personal database.

Note that a proposal based on powers to obtain bulk personal dataset avoids all the legal problems associated by the forced retention of bulk personal dataset;  in effect, instead of retention the Home Office has suggested a power for the national security agencies to “obtain bulk personal datasets on demand”. 

The bulk personal dataset provisions in DIP

First of all, a “bulk personal dataset” is flexibly defined.  Clause 150 of DIP says: 

An intelligence service obtains a bulk personal dataset if— (a) it obtains a set of information that includes personal data relating to a number of individuals, (b) the nature of the set is such that it is likely that the majority of the individuals are not, and are unlikely to become, of interest to the intelligence service in the exercise of its functions, and (c) if (after any initial examination of the contents) the intelligence service were to decide to retain the set for the purpose of the exercise of its functions, the set would be held electronically for analysis in the exercise of those functions”.

The explanatory notes state that “Examples of these datasets include the telephone directory or the electoral roll”.

For completeness: “Personal data” has the same meaning as in the Data Protection Act 1998 except that it also includes data relating to a deceased individual; in addition “data” is defined to include “any information which is not data”. (Those taking the BCS DP Practitioner exam should be able to explain why and understand that DIP powers apply to collections of personal information in manual records contrary to what other Home Office publications say).

Obtaining a bulk dataset is legitimised by warrant signed by the Secretary of State and is also approved by a Judge; a warrant can apply to a class of bulk datasets (e.g. all electoral rolls).  The key test applied by the Secretary of State is whether such data collection is “necessary” and “proportionate” in terms of operational requirements and of Article 8 of the Human Rights Act (soon to be modified by the Government); these tests are then confirmed by a senior judge.  Warrants are valid for six months but they can be renewed.

As far as I can see, the following appear to be targets for dataset acquisition (in whole or part):

  • Databases that contains basic personal data about the population (e.g. Individualised Electoral Rolls, complete telephone directories, NHS spine; Credit reference, Census records).
  • Databases that monitor movement (e.g. Passenger Name Records detailing flights; Automatic Number Plate Recognition datasets that monitor the movements of vehicles on major roads; Oyster card database in London).
  • Databases to monitor spending (e.g. banking, credit card and cash withdrawal transactions).
  • Databases in the hands of the State (e.g. Police National Computer databases, DWP, HMRC datasets and CCTV data collections).

DIP does not specifically exclude bulk personal datasets that contain Sensitive Personal Data; one thus has to assume that these can be the target of data collection too. 

Safeguards in DIP 

Assuming there is an operational requirement to obtain a bulk personal dataset, the DIP safeguards appear to be as follows: 

  • The obtaining and subsequent processing of a bulk personal dataset is subject to Home Secretary sign-off and a judicial oversight with respect to necessity and proportionality– repeated every six months if need be.
  • Bulk dataset holdings have to be reported to the relevant Secretary of State (who signed the warrant) and there will be an internal review process where senior managers, legal advisors etc will audit or review bulk dataset collections in terms of necessity and proportionality (i.e. I have to admit that this looks like “marking your own homework” to me).
  • There will be a Code of Practice which outlines an alternative regime to the Data Protection Principles; these outline data processing rules under which bulk personal datasets are to be used, accessed and disclosed.
  • A new Investigatory Powers Commissioner who will oversee compliance with this Code and can inspect documents etc to see if the Code is being applied properly.
  • Any private misuse (e.g. by a member of staff) of a bulk dataset is an offence (e.g. up to 12 months imprisonment).

In further detail, Schedule 3, paragraph 2 of DIP specifies the data processing rules in the Code; it will include provisions about:

“(a) why, how and where the data is held,

(b) who may access the data on behalf of the authority,

(c) with whom, and under what conditions, the data may be disclosed,

(d) the processing of the data for purposes otherwise than in connection with the purposes for which it was obtained or retained,

(e) the processing of the data together with other data,

(f) the processes for determining how long the data should be held and for the destruction of the data.”

Finally, the Home Office assert on the face of DIP (without evidence so far) that their proposals are “necessary” and “proportionate” mainly because, I suspect, these are the responsibilities of the Secretary of State and the senior judge as part of their warrant signing/approval procedures.

Missing (and really obvious) safeguards

The following safeguards do not feature in DIP (and remember that “the majority of the individuals are not, and are unlikely to become, of interest to the intelligence service”):

  • There is no role for Data Protection Principles; the national security agencies prefer their own version as identified above in Schedule 6, paragraph 3(2) and this has consequences that I describe in the next section.
  • The Code is not supported by penalties if there is a major transgression.
  • Onward data sharing of personal data for a purpose unconnected to national security and serious crime is permissible under the heading “the processing of the data for purposes otherwise than in connection with the purposes for which it was obtained or retained”.
  • Data matching across any combination of bulk datasets is permissible (and this should be considered in the context of onward data sharing of the product of data matching - see above).
  • There is no exclusion for some kinds of records (e.g. mass collection from medical records databases) on the face of the Bill  (any exclusion is decided by the warranting procedure).
  • Personal data can be retained for longer than would be permissible under the Fifth Data Protection Principle (explained in the next section).
  • There is no provision that considers lawful processing, fair processing, excessive processing, transfers outside the EEA or security of personal data specified in the text of Schedule 6, paragraph 3(2); hence the Intelligence Services Commissioner cannot consider them.
  • The new Intelligence Services Commissioner has no role in handling or investigating complaints from data subjects (remember the majority of data subjects are “not of interest” to the intelligence service).
  • Bulk dataset holdings are not reported to the Intelligence Services Commissioner; I think all elements associated with “marking your own homework” proposed in DIP should really be transferred to the Intelligence Services Commissioner and his staff.
  • Organisations that are required to provide bulk datasets cannot raise a formal complaint to the Intelligence Services Commissioner that the warrant provides for disproportionate data sharing (i.e. organisations should be able to ask for a review of a warrant if they have concerns over proportionality).
  • The Intelligence Services Commissioner has no role in assessing whether datasets, once accessed, have proved to be useful; he ought to be able to establish Key Performance Indicators that demonstrate that bulk access is worthwhile (with the implication that if access is not worthwhile, the warrant becomes void and the datasets destroyed).
  • There are no rights for data subjects even when a bulk personal dataset is of no further interest to the authorities. For instance, I cannot see why the product of a Subject Access request cannot be at the very least deferred until that point when a dataset is deleted (i.e. when a dataset is deleted, the product of an earlier Subject Access request from someone who is of no interest to the intelligence services can be released).

In addition, existing routes for the national security agencies to obtain bulk personal datasets are not closed.  For example, Schedule 1 of Counter-Terrorism Act 2008 which modifies the “Representation of the People (England and Wales) Regulations 2001 (S.I. 2001/341)” is not repealed.  This modification includes Regulation 108A which is entitles the “Supply of full register etc to the security services”.  All other access routes to datasets should be closed if DIP is to progress.

Data Protection Principles not welcome here

Note that the Section 28 National Security Certificate regime has to remain in place in order to ensure the Principles and rights granted by the Data Protection Act are negated; this has unfortunate consequences.

For instance, the Second Principle requires a data controller not to disclose personal data for an incompatible purpose.  So if the national security agencies wanted only to disclose or use the bulk dataset for compatible purposes, then there would be no need for the text in Schedule 3, paragraph 2(c) that allows them to decide “with whom, and under what conditions, the data may be disclosed” or paragraph 2(d) “the processing of the data for purposes otherwise than in connection with the purposes for which it was obtained or retained”.  

All you need to do is incorporate into Schedule 3 is the text of the Second Principle.  By eschewing this Principle, DIP is thereby (inadvertently one hopes) legitimising processing for incompatible purposes.

Similarly, the absence of the text of the Fifth Principle sends the message that this Principle is too restrictive for the needs of the national security agencies.  If the personal data were to be retained by the national security agencies for “no longer than is necessary”, then the wording of Schedule 3, paragraph 2(f) would be otiose.  In other words, the fact that the Fifth Principle is absent from Schedule 3,  implies the Fifth Principle is too restrictive for DIP and, by implication, the national security agencies want to retain personal data for longer than the Principle would allow.

The same argument applies to other Principles (e.g. Seventh, Eighth)

Concluding comments

Parliament should at least learn from the bulk communications data acquisition which was legitimised under section 94 of the Telecommunications Act 1984 (see last blog on Section 94); these general DIP bulk personal dataset collection powers do not have any kind of “sunset” arrangements.  This is alarming, given that future technological developments are likely to allow for the collection of more intrusive personal datasets not less.

The proposed changes to Article 8 of the Human Rights Act should be considered in the context of this Bill; there is a risk that any change to Article 8 by the Government could move the goal posts in terms of “necessity” and “proportionality” (i.e. the key warranting safeguards in DIP).  That is why the Government should publish the legal advice that shows that DIP’s bulk dataset provisions are Article 8 compliant.

As I said before, the national security agencies need to embrace the Data Protection Principles and the types of safeguards I have identified above.  If not, bulk personal dataset collection and mass surveillance are indistinguishable.


The Data Protection Principles can restore public trust in bulk data collection

Should Section 28 national security certificates exclude the Data Protection Principles? (this is the position in DIP).

Last blog on Section 94

Annex 9 of Anderson Report “A Question of Trust” (PDF)

Wider use of electoral roll

Confidentiality of Census

Bulk data sets: Home Office factsheet (pdf)

All DIP Bill publications

This story originally appeared at HAWKTALK, the blog of Amberhawk Training Ltd.

More about

More about

More about


Send us news

Other stories you might like