Exclusive The Home Office has privately conceded that its plan to store details of every internet communication may not be possible - and that it has pinned the multibillion pound project's hopes on snooping technology not yet developed.
Officials working on the Interception Modernisation Programme (IMP) made the admission last week in a meeting with the internet industry, according to correspondence seen by The Register.
Under IMP, intelligence and law enforcement agencies aim to capture information on who contacts whom, when and where from any social network, chat room, VoIP service or other, as-yet-undeveloped communications application.
An ongoing Home Office consultation, published in April, proposes that access providers store and process the terabytes of data such a system would harvest daily, at a cost to taxpayers of £2bn over 10 years. It argues authorities' capability to access communications data during investigations and intelligence gathering is being eroded by the internet.
At last week's meeting, the Internet Service Provider's Association (ISPA) however raised concerns with officials that Deep Packet Inspection (DPI) technology - the equipment it is envisaged will tap into internet connections - is currently incapable of such ubiquitous surveillance.
In a leaked report on the meeting to members it said the Home Office representatives "agreed that there was not a simple solution but reported that they had seen some impressive commercial solutions in the market already."
ISPA said IMP officials had added that they expected the new laws needed to legitimise the system would take "a number of years" to pass, and that by that time technology will "hopefully have caught up with law enforcement requirements."
The government aims to extract coherent information about online behaviour from raw internet traffic. "The Programme indicated that it is interested in a broad range of communications data," ISPA told members.
"Including those that are included in the protocol (such as which location you visited in Second Life or the people you had conversations with on Facebook)."
The admission that the Home Office plans to spend billions on a project that relies on technology that does not exist is likely to be seized upon by opponents. A cross-party group of MPs and Lords will begin an inquiry into IMP on July 1, informed by a critical briefing published this week by the London School of Economics.
Professor Peter Sommer, an author of the briefing, said: "There is an astonishing contrast between the definite assertions in the Home Office consultation document and the answers officials have given to ISPA.
"The £2bn 'high level budgetary estimate of the economic costs' now appears to be for technology which currently does not exist but which it is hoped might be available once the desired new legislation is in place. It appears that the consultation document was published without much discussion about how the ambitions might practically be realised."
The Home Office said in a statement: "Technology is evolving and new innovative forms of internet based communications are emerging. If we do not make changes now to maintain existing capabilities and look ahead to the future, the police, security and intelligence agencies will no longer be able to use this data to fight crime.
"The implementation costs of the range of options discussed in the consultation document are estimated to be up to £2bn over a ten year roll-out period. The costs of maintaining these capabilities are more than justified by the costs of failure to implement."
ISPA said the meeting was part of its work on behalf of members to develop a formal response to the Home Office consultation. It declined to comment specifically on the technical doubts raised.
Aside from the central practical question of whether IMP goals are even possible, ISPA also raised the issue of encryption at the meeting last week. Mass take up of scrambling technology - which would render a lot of internet communications data unreadable by DPI probes - had been feared by government before, officials said, but did not materialise.
The meeting also discussed plans for infrastructure owners such as BT to collect data from customers of other providers using their networks. ISPA said this raised legal concerns that some firms might be able to gain commercial intelligence about rivals. The Home Office "accepted that this was a valid concern and agreed that legislation could prevent this".
ISPA plans a forum on the legal implications of IMP on July 1, the same day MPs and Lords are to hear evidence on the project. ®