Blighty's GCHQ stashes away 50+ billion records a day on people. Just let that sink in

SIGINT and DIGINT revealed


The enormous scale of GCHQ's surveillance was revealed on Friday by newly published Snowden documents. The files note the growth in capabilities enjoyed by the UK government's snoopers since intercepting communications in bulk from 2007.

These details were revealed in a series of documents published by The Intercept including one on the "flat data store" codenamed BLACK HOLE, and a document calling itself "the one-stop shop for Cyber Defence Operations legal and policy information."

When the slide on BLACK HOLE was composed in March 2009, the flat data store held more than 1.1 trillion things which GCHQ had collected since August 2007.

The store weighed in at 217TB when uncompressed, the largest share of which was HTTP data (41 per cent), which alongside web search (19 per cent) and SMTP data (12 per cent) accounted for almost three quarters of all that it held.

Additional data covered instant messenger records, hacking logs for Computer Network Exploitation (CNE) operations, and the use of "Anonymisers."

The collection began after Section 32 of the Terrorism Act 2007 had amended RIPA to extend interception warrants.

By 2010, GCHQ stated it was logging "30bn metadata records per day. By 2012, collection had increased to 50 billion per day, and work was underway to double capacity to 100 billion."

GCHQ has since "developed new population scale analytics for multi-petabyte cluster," which allows "population scale target discovery."

In a vision document for 2013, its aim was to have created "the world's biggest SIGINT engine to run cyber operations and to enable IA, Effects and SIGINT ... [as well as] to perform CNE exfiltration, eAD, beaconry, and geo-location."

BLACK HOLE's recorded events contain only metadata, according to the "Events" page from the GCWiki, although it notes that "sometimes there are grey areas between events and content" citing how the subject of an email is generally transmitted in the header portion of the SMTP communication, despite being considered content.

Slides showing GCHQ's Content-Metadata Matrix suggest that the spooks' views of what is metadata extends to passwords, buddylists, and folders used to organize emails.

The majority of GCHQ's operational data is acquired through the agency's operational activities, whether they are interception, computer network exploitation (CNE, or aggressive hacking), or through JTRIG operations.

One new document also discloses a number of tools used to analyze the data stored in BLACK HOLE, which are complementary and provide an insight into the depth and breadth of GCHQ's surveillance practices. These tools all come under a portion of GCHQ's analysis project called BLAZING SADDLES.

It is worth noting that the word "target" here does not mean a person specified for investigation by a warrant, but merely a hypothetical identity which has had identifiers allocated to it.

  • AUTOASSOC provides information as to which Target Detection Identities (TDIs) have been seen at the same time and from the same IP addresses as other TDIs – allowing the spooks to enlarge the number of identifiers tied to a particular target.

  • HRMap provides information about host-referrer relationships, examining how internauts traverse the web, i.e., what route they have taken to a particular site, and where they proceed to.

  • INFINITE MONKEYS is a tool which targets v-bulletin software, to reveal the forum accounts of targets and additionally to target particular forum users.

  • KARMA POLICE, which we have reported on, allows the spooks to know which websites the target visited, and when/where those targets occur – all of which is additionally tied to IPs.

  • MARBLED GECKO provides information about the use of Google Earth and Google Maps, which combined with MUTANT BROTH allows the noseys to see who is looking at particular areas of the Earth.

  • MEMORY HOLE provides information on web searches made on engines such as Google's. It provides information on when, where, and from which IP addresses particular searches were made.

  • MUTANT BROTH is a tool to sift through BLACK HOLE data by TDIs, such as cookies. It allows the spooks to create a profile of any given target's online activities.

  • SAMUEL PEPYS is described as "a near real-time Internet diarisation tool. It enables powerful IP stream analysis/profiling by fusing all available traffic types in one place. It contains both unselected events and content."
  • SOCIAL ANIMAL provides information about how targets interact with other targets, and with files/pictures/video on the internet.

  • SOCIAL ANTHROPOID is a "converged comms events database" which enables the spooks to see who their targets have communicated with "via phone, internet, or using converged channels (e.g., sending emails from a phone or making voice calls over the internet)." This project is set to subsume SOCIAL ANIMAL.

  • GOLDEN AXE, which shares its name with a classic side-scrolling Sega game, is primarily for International Mobile Equipment Identity defeats – allowing the spooks to figure out whether particular mobile devices uniquely identify targets. The Register understands that some handsets may have identical IMEI, as in India.

These tools were being used in a Joint Collaboration Environment titled Innov8, which was testing large-scale analytics using both GCHQ and NSA data.

A sample search was provided, based on automatic TDIs, which showed visits to pornography site YouPorn, as well as Reuters, Facebook, Yahoo, and Google.

The Intercept noted that MUTANT BROTH's ability to identify cookies was integral to GCHQ's attack on Belgian telco Belgacom.

Cookies associated with the IPs revealed the Google, Yahoo, and LinkedIn accounts of three Belgacom engineers, whose computers were then targeted by the agency and infected with malware.

The hack, codenamed "Operation Socialist," gained access to Belgacom's Core GRX routers so the spooks could run man-in-the middle attacks against targets roaming with smartphones.


Other stories you might like

  • Stolen university credentials up for sale by Russian crooks, FBI warns
    Forget dark-web souks, thousands of these are already being traded on public bazaars

    Russian crooks are selling network credentials and virtual private network access for a "multitude" of US universities and colleges on criminal marketplaces, according to the FBI.

    According to a warning issued on Thursday, these stolen credentials sell for thousands of dollars on both dark web and public internet forums, and could lead to subsequent cyberattacks against individual employees or the schools themselves.

    "The exposure of usernames and passwords can lead to brute force credential stuffing computer network attacks, whereby attackers attempt logins across various internet sites or exploit them for subsequent cyber attacks as criminal actors take advantage of users recycling the same credentials across multiple accounts, internet sites, and services," the Feds' alert [PDF] said.

    Continue reading
  • Big Tech loves talking up privacy – while trying to kill privacy legislation
    Study claims Amazon, Apple, Google, Meta, Microsoft work to derail data rules

    Amazon, Apple, Google, Meta, and Microsoft often support privacy in public statements, but behind the scenes they've been working through some common organizations to weaken or kill privacy legislation in US states.

    That's according to a report this week from news non-profit The Markup, which said the corporations hire lobbyists from the same few groups and law firms to defang or drown state privacy bills.

    The report examined 31 states when state legislatures were considering privacy legislation and identified 445 lobbyists and lobbying firms working on behalf of Amazon, Apple, Google, Meta, and Microsoft, along with industry groups like TechNet and the State Privacy and Security Coalition.

    Continue reading
  • SEC probes Musk for not properly disclosing Twitter stake
    Meanwhile, social network's board rejects resignation of one its directors

    America's financial watchdog is investigating whether Elon Musk adequately disclosed his purchase of Twitter shares last month, just as his bid to take over the social media company hangs in the balance. 

    A letter [PDF] from the SEC addressed to the tech billionaire said he "[did] not appear" to have filed the proper form detailing his 9.2 percent stake in Twitter "required 10 days from the date of acquisition," and asked him to provide more information. Musk's shares made him one of Twitter's largest shareholders. The letter is dated April 4, and was shared this week by the regulator.

    Musk quickly moved to try and buy the whole company outright in a deal initially worth over $44 billion. Musk sold a chunk of his shares in Tesla worth $8.4 billion and bagged another $7.14 billion from investors to help finance the $21 billion he promised to put forward for the deal. The remaining $25.5 billion bill was secured via debt financing by Morgan Stanley, Bank of America, Barclays, and others. But the takeover is not going smoothly.

    Continue reading

Biting the hand that feeds IT © 1998–2022