The government plans to spend £2bn for ISPs to intercept details of their customers' emails, VoIP calls, instant messaging and social networking.
Under the proposals, mobile and fixed line operators will be required to process and link the data together to build complete profiles of every UK internet user's online activity. Police and the intelligence services would then access the profiles, which will be stored for 12 months, on a case-by-case basis.
Home Secretary Jacqui Smith said today she had "ruled out" the idea of a central government-run store of communications data on privacy grounds. Instead, the consultation on the Interception Modernisation Programme (IMP) proposes a "middle way" requiring ISPs to retain much more data than they currently do.
Most of the proposed 10-year budget for the system would be spent on deep packet inpection equipment that would allow ISPs to tap into third party communciations data carried by their networks. Authorities are worried that the growth of internet-based communications services such as Skype diminishes their ability to monitor who contacts whom, when, where and how.
"This option would resolve the problem that some communications data which may be important to public authorities will not otherwise be retained in this country," the IMP consultation document says.
"However it would not address the problem of fragmentation: as data is increasingly held by a wider range of communications service providers, it might take longer than it does at present to piece together data from different companies relating to one person or communications device."
In response to such fragmentation, the government plans to order ISPs to "not only to collect and store data but to organise it, matching third party data to their own data where it had features in common".
The plan is likely to be seen as onerous by ISPs. According to one senior industry source, many in the industry hope the next government will abandon it. "I don't know anyone in the communications sector that will be sorry if IMP gets cancelled the day after the election," the source said.
"I agree that what we're asking the industry to do is something that will put a burden on them," Smith said at today's IMP briefing for journalists. She said providers will be refunded the cost of collecting and processing the data by the government.
By tapping ISPs to collect and process communications data from across the whole internet, officials will avoid opposition to the plans from websites and other third party services. Facebook recently voiced opposition to government monitoring of its users, but when the data is intercepted in transit, it will have no choice. Intelligence and security agencies already have close relationships with ISPs.
The IMP consultation will run for 12 weeks. Details of how to respond are here. ®