The cover has been blown on an NSA program which collects data on “nearly everything a user does on the internet” even as the debate rages over the secretive US agency's mass surveillance of innocent people.
The XKeyscore program covers emails, social media activity and browsing history and is accessible to NSA analysts with little or no prior authorisation, according to a leaked presentation published by The Guardian today.
The slide deck, disclosed by NSA whistleblower Edward Snowden and published alongside an accompanying story, was released just hours before NSA director General Keith Alexander was due to deliver an eagerly anticipated keynote presentation at the Black Hat security conference in Las Vegas.
The Guardian reports that the top secret National Security Agency program allows analysts to search through a database "containing emails, online chats and the browsing histories of millions of individuals". In the leaked documents, the NSA describes XKeyscore as its "widest-reaching" internet intelligence system.
Targets data in transit
The release is arguably the most significant disclosure about the NSA's web surveillance operations since the first revelations about the spy agency's controversial PRISM web data mining program, which collects data from email, chat and VoIP. That program harvests information from users of services provided by Google, Facebook, Apple, Yahoo! and AOL, and was said to have been carried out with the indirect assistance of those companies.
While PRISM involves stored data, XKeyscore appears to involve mining through data in transit, either from the premises of a telco or through a fibre-optic tap. Leaked training materials explain how analysts fill in a simple online form before gaining access to data sorted by identifiers, such as target email addresses. Only a broad justification of the reason for a request, which is reportedly not subject to a review by any court or senior NSA personnel, is needed.
The Guardian reports that the leaked files provide substance to Snowden's claims that while working as an NSA contractor he "could "wiretap anyone, from you or your accountant, to a federal judge or even the president, if I had a personal email".
He made those claims in a video interview in early June soon after he outed himself as the source of leaks about the NSA's secret surveillance programmes.
Analysts can combine XKeyscore with data from other NSA systems to obtain "real-time" interception of a target's internet activity, said the paper.
"XKeyscore provides the technological capability, if not the legal authority, to target even US persons for extensive electronic surveillance without a warrant," said The Guardian's Glenn Greenwald.
They don't even need to know who you are to track you down
According to the slides, spooks can query the system by name, telephone number, IP address and keywords as well as email address. Just searching by email address alone will not give a target's full range of activities on the net, but a range of carefully selected queries are needed to prevent analysts being swamped with an unmanageable dump of information to sort through.
Spooks are advised to use metadata also stored in XKeyscore in order to narrow down their queries. Queries can be mixed and matched in order to try to pin down a group of suspects without even knowing targeting information, such as email addresses.
One example cited in the training document says that XKeyscore can be used to search for someone whose language is out of place in a region, or who is using encryption and "searching the web for suspicious stuff". Another example states that XKeyscore is the only system that allows analysts to directly target traffic from "VPN startups in country X” to “give me the data so I can decrypt and discover the users".
"No other system performs this on raw unselected bulk traffic," the 2008 vintage training manual (marked "Top Secret" and apparently shared only with the NSA's peers in the UK, Australia, Canada and New Zealand) explains.
XKeyscore also provides a means to index exploitable computers in a specified country, as well as a way of obtaining the email address of persons of interest using Google Earth.
One leaked document describes how the program "searches within bodies of emails, webpages and documents", including the "To, From, CC, BCC lines" and the 'Contact Us' pages on websites". XKeyscore also also allows analysts to pull together logs of the IP addresses of visitors to specified websites.
An NSA tool called DNI Presenter is used to read the content of harvested emails. The same tool enables analysts to read the content of Facebook private messages.
Content remains on the system for only three to five days, while metadata is stored for 30 days. One leaked document states: "At some sites, the amount of data we receive per day (20+ terabytes) can only be stored for as little as 24 hours."
However, NSA systems allow flagged data on Xkeyscore to be moved onto other databases such as Pinwale, where material can be stored for for up to five years.
Despite the short shelf life of data stored on XKeyscore in one month last year, the system collected at least 41 billion total records.
NSA training manuals state that 300 terrorists have been captured using intelligence from XKeyscore before 2008, a claim that will doubtless be used to justify the program and criticise its exposure.
In a statement to The Guardian, the NSA said: "NSA's activities are focused and specifically deployed against – and only against – legitimate foreign intelligence targets in response to requirements that our leaders need for information necessary to protect our nation and its interests.
"XKeyscore is used as a part of NSA's lawful foreign signals intelligence collection system.
"Allegations of widespread, unchecked analyst access to NSA collection data are simply not true. Access to XKeyscore, as well as all of NSA's analytic tools, is limited to only those personnel who require access for their assigned tasks … In addition, there are multiple technical, manual and supervisory checks and balances within the system to prevent deliberate misuse from occurring."
The NSA statement continues: "Every search by an NSA analyst is fully auditable, to ensure that they are proper and within the law.
"These types of programs allow us to collect the information that enables us to perform our missions successfully – to defend the nation and to protect US and allied troops abroad." ®