Protests from ISPs and phone providers have further delayed government plans to massively increase monitoring of phone calls, web browsing and emails, it's revealed today.
As a result of concerns over costs and technical feasibility, it is now expected that the legislation necessary to implement the £2bn surveillance programme - known in Whitehall as the Interception Modernisation Programme - will not be included in next week's Queen's Speech.
The move is likely to delay the project until long after the next election, to the frustration of law enforcement and intelligence bodies. The poll must be called before 4 June.
The Home Office today published the responses to its consultation on the plans and said it would continue to work with the communications industry.
"The consultation showed widespread recognition of the importance of communications data in protecting the public and an appreciation of the challenges which rapidly changing technology poses," said Home Office minister David Hanson.
"We will now work with communications service providers and others to develop these proposals, and aim to introduce necessary legislation as soon as possible."
The consultation proposed that ISPs and phone providers would store details of who contacts whom, when, where and how online.
It would involve inserting Deep Packet Inspection probes inside networks to intercept data from third party services such as GMail, Facebook and Skype. At present most ISPs store only basic IP session logs.
Officials originally considered storing all the new data in one central government warehouse, but by the time the consultation was published in June had rejected this idea on privacy grounds, preferring to outsource storage to industry. Configuration of the probes would be the responsibility of GCHQ, the Cheltenham-based electronic intelligence agency.
GCHQ has already begun preparing for its role collecting and mining the terabytes of data with its own separate but closely related project, Mastering the Internet.
The Home Office has consistently argued that an expanded data collection effort is essential to allow it to "maintain capability" to investigate crime.
In its submission to the consultation however, LINX, which represents more than 300 ISPs, called officials' assertions "disingenuous". "The volume of data the government now proposes [communications providers] should collect and retain will be unprecedented, as is the overall level of intrusion into the privacy of the citizenry," it wrote.
Mobile phone operators have also been critical. Giving evidence to an informal Parliamentary committee, T-Mobile's data protection chief suggested the proposals were technically impossible.
"Somehow we would be required to open up proprietary communications protocols," he said. "We're not sure that can be done."
With more time, industry representatives are likely to seek more technical detail and financial guarantees. Despite the further delays, Home Office officials are today understood to remain set on implementing IMP.
Hanson said: "Communications data is crucial to the fight against crime and in keeping people safe. It is a highly technical area and one which demands a fine balance between privacy and maintaining the capabilities of the police and security services."
The Tories, who according to opinion polls are likely to win the next election, broadly agree with the current government's line that "maintaining capability" to access communications records is essential. ®