Decoding the Chinese Super Micro super spy-chip super-scandal: What do we know – and who is telling the truth?
Who's your money on? Bloomberg's sources? Apple? Amazon? Super Micro?
Analysis Chinese government agents sneaked spy chips into Super Micro servers used by Amazon, Apple, the US government, and about 30 other organizations, giving Beijing's snoops access to highly sensitive data, according to a bombshell Bloomberg report today.
The story, which has been a year in the making and covers events it says happened three years ago, had a huge impact on the markets: the company at the center of the story, San Jose-based Super Micro, saw its share price drop by nearly 50 per cent; likewise Apple's share price dropped by just under two per cent, and Amazon's dropped by more than two per cent.
But the article has been strongly denied by the three main companies involved: Apple, Amazon, and Super Micro. Each has issued strong and seemingly unambiguous statements denying the existence and discovery of such chips or any investigation by the US intelligence services into the surveillance implants.
These statements will have gone through layers of lawyers to make sure they do not open these publicly traded corporations to lawsuits and securities fraud claims down the line. Similarly, Bloomberg employs veteran reporters and layers of editors, who check and refine stories, and has a zero tolerance for inaccuracies.
So which is true: did the Chinese government succeed in infiltrating the hardware supply chain and install spy chips in highly sensitive US systems; or did Bloomberg's journalists go too far in their assertions? We'll dig in.
First up, the key details of the exclusive. According to the report, tiny microchips that were made to look like signal conditioning couplers were added to Super Micro data center server motherboards manufactured by sub-contractors based in China.
Those spy chips were not on the original board designs, and were secretly added after factory bosses were pressured or bribed into altering the blueprints, it is claimed. The surveillance chips, we're told, contained enough memory and processing power to effectively backdoor the host systems so that outside agents could, say, meddle with the servers and exfiltrate information.
The Bloomberg article is not particularly technical, so a lot of us are having to guesstimate how the hack worked. From what we can tell, the spy chip was designed to look like an innocuous component on the motherboard with a few connector pins – just enough for power and a serial interface, perhaps. One version was sandwiched between the fiberglass layers of the PCB, it is claimed.
The spy chip could have been placed electrically between the baseboard management controller (BMC) and its SPI flash or serial EEPROM storage containing the BMC's firmware. Thus, when the BMC fetched and executed its code from this memory, the spy chip would intercept the signals and modify the bitstream to inject malicious code into the BMC processor, allowing its masters to control the BMC.
Supermicro wraps crypto-blanket around server firmware to hide it from malware injectorsREAD MORE
The BMC is a crucial component on a server motherboard. It allows administrators to remotely monitor and repair machines, typically over a network, without having to find the box in a data center, physically pull it out of the rack, fix it, and re-rack it. The BMC and its firmware can be told to power-cycle the server, reinstall or modify the host operating system, mount additional storage containing malicious code and data, access a virtual keyboard and terminal connected to the computer, and so on. If you can reach the BMC and its software, you have total control over the box.
With the BMC compromised, it is possible the alleged spies modified the controller's firmware and/or the host operating system and software to allow attackers to connect in or allow data to flow out. We've been covering BMC security issues for a while.
Here is Bloomberg's layman explanation for how that snoop-chip worked: the component "manipulated the core operating instructions that tell the server what to do as data move across a motherboard… this happened at a crucial moment, as small bits of the operating system were being stored in the board’s temporary memory en route to the server’s central processor, the CPU. The implant was placed on the board in a way that allowed it to effectively edit this information queue, injecting its own code or altering the order of the instructions the CPU was meant to follow."
There are a few things to bear in mind: one is that it should be possible to detect weird network traffic coming from the compromised machine, and another is that modifying BMC firmware on the fly to compromise the host system is non-trivial but also not impossible. Various methods are described, here.
"It is technically plausible," said infosec expert and US military veteran Jake Williams in a hastily organized web conference on Thursday morning. "If I wanted to do this, this is how I'd do it."
The BMC would be a "great place to put it," said Williams, because the controller has access to the server's main memory, allowing it to inject backdoor code into the host operating system kernel. From there, it could pull down second-stage spyware and execute it, assuming this doesn't set off any firewall rules.
A third thing to consider is this: if true, a lot of effort went into this surveillance operation. It's not the sort of thing that would be added to any Super Micro server shipping to any old company – it would be highly targeted to minimize its discovery. If you've bought Super Micro kit, it's very unlikely it has a spy chip in it, we reckon, if the report is correct. Other than Apple and Amazon, the other 30 or so organizations that used allegedly compromised Super Micro boxes included a major bank and government contractors.
A fourth thing is this: why go to the bother of smuggling another chip on the board, when a chip already due to be placed in the circuitry could be tampered with during manufacture, using bribes and pressure? Why not switch the SPI flash chip with a backdoored one – one that looks identical to a legit one? Perhaps the disguised signal coupler was the best way to go.
And a fifth thing: the chip allegedly fits on a pencil tip. That it can intercept and rewrite data on the fly from SPI flash or a serial EEPROM is not impossible. However, it has to contain enough data to replace the fetched BMC firmware code, that then alters the running operating system or otherwise implements a viable backdoor. Either the chip pictured in Bloomberg's article is incorrect and just an illustration, and the actual device is larger, or there is state-of-the-art custom semiconductor fabrication involved here.
This implant chip. Let's say it's intercepting SPI flash / serial EEPROM reads and rewriting them. Not impossible. But then it has to contain enough data to alter the BMC firmware. To then alter the host OS. To then act as a useful backdoor.— The Register (@TheRegister) October 5, 2018
What lithography is it using? 7nm? pic.twitter.com/8kbSYMuRR2
One final point: you would expect corporations like Apple and Amazon to have in place systems that detect not only unexpected network traffic, but also unexpected operating system states. It should be possible that alterations to the kernel and the stack of software above it should set off alarms during or after boot.
Bloomberg claims the chip was first noticed in 2015 in a third-party security audit of Super Micro servers that was carried out when Amazon was doing due diligence into a company called Elemental Technologies that it was thinking of acquiring. Elemental used Super Micro's servers to do super-fast video processing.
Amazon reported what it found to the authorities and, according to Bloomberg, that "sent a shudder" through the intelligence community because similar motherboards were in use "in Department of Defense data centers, the CIA’s drone operations, and the onboard networks of Navy warships."
Around the same time, Apple also found the tiny chips, according to the report, "after detecting odd network activity and firmware problems." Apple contacted the FBI and gave the agency access to the actual hardware. US intelligence agencies then tracked the hardware components backwards through the supply chain, and used their various spying programs to sift through intercepted communications, eventually ending up with a focus on four sub-contracting factories in China.
According to Bloomberg, the US intelligence agencies were then able to uncover how the seeding process worked: "Plant managers were approached by people who claimed to represent Super Micro or who held positions suggesting a connection to the government. The middlemen would request changes to the motherboards’ original designs, initially offering bribes in conjunction with their unusual requests. If that didn’t work, they threatened factory managers with inspections that could shut down their plants. Once arrangements were in place, the middlemen would organize delivery of the chips to the factories."
This explanation seemingly passes the sniff test: it fits what we know of US intelligence agencies investigative approaches, their spy programs, and how the Chinese government works when interacting with private businesses.
The report then provides various forms of circumstantial evidence that adds weight to the idea that this all happened by pointing to subsequent actions of both Apple and Amazon. Apple ditched Super Micro entirely as a supplier, over the course of just a few weeks, despite planning to put in a massive order for thousands of motherboards. And Amazon sold off its Beijing data center to its local partner, Beijing Sinnet, for $300m.
Denials starting to making sense
Both of those things happened in the right timeframe for it to be a direct result of such an investigation. But Apple claims that the decision to ditch Super Micro was over malware that had been inadvertently fetched from Super Micro's customer portal: a downloadable network interface driver had been infected with a software nasty by Chinese hackers in 2015, and accidentally installed on an internal Apple Windows-based development machine, it is claimed. Facebook also may have fetched the dodgy driver for the Super Micro boxes it had in its lab. The malware apparently attempted to spy on network traffic. There was another issue with the server motherboards' network cards: they shipped with outdated firmware that had a known security hole in it, we're told.
Amazon says its sale to Sinnet was a "transfer-of-assets agreement mandated by new China regulations for non-Chinese cloud providers to continue to operate in China," and had nothing to do with discovering any spy chips.
Up to this point, you could be forgiven for believing Bloomberg's story in its entirely and discounting Amazon, Apple and Super Micro's denials for trying to cover their backs while refusing to acknowledge understandably confidential national security investigations.
Except the denials are far more precise and concrete than typical non-denial denials. It remains very unlikely that public companies would issue outright falsehoods, even in the current political climate, due to the market and regulatory ramifications if they were found to be outright lying to investors. Usually, assessing whether a company is telling the truth comprises of carefully parsing statements and seeing what aspects of a story they don't address.
Typical giveaways are when such statements are over-the-top, using emotive but imprecise language, or when a denial is either overly specific – such that it walks past the main allegation – or is unnecessarily vague – so it sounds like a denial but actually isn't.
And there are examples of those in the various statements put out by the companies. For example, Amazon brings up in its response to Bloomberg the old canard "there are so many inaccuracies in this article as it relates to Amazon that they’re hard to count," which is a classic way of casting doubt without actually tackling the issues substantively.
It also calls the suggestion that it sold off its Beijing data center to step away from compromised servers as "absurd" – a strong, emotive word but such a decision would not be absurd at all if the story is true.
But Amazon also says: "It’s untrue that AWS knew about a supply chain compromise, an issue with malicious chips, or hardware modifications when acquiring Elemental. It’s also untrue that AWS knew about servers containing malicious chips or modifications in data centers based in China, or that AWS worked with the FBI to investigate or provide data about malicious hardware."
You can parse this. For example the key element in the first denial is "when acquiring Elemental." What timeframe does that encompass? And how do you define "AWS"? Did the security people making the decision work for AWS, or another arm of Amazon?
If Amazon wanted to outright deny the story, it could have said something like: "AWS and Amazon deny any knowledge of supply chain compromise, an issue with malicious chips, or hardware modifications with respect to Elemental or Super Micro beyond the assertions made to us by Bloomberg."
In a second denial, the wording gets a little stronger: "At no time, past or present, have we ever found any issues relating to modified hardware or malicious chips in SuperMicro motherboards in any Elemental or Amazon systems. Nor have we engaged in an investigation with the government."
This is a much harder denial to parse. It seems like a pretty straight-up denial. There is one possible parsing escape route – the use of "we" – as in "at no time have we ever found." Strictly speaking, it wasn't Amazon but the third-party security company that it asked to carry out the audit. But things are definitely growing a little thin at this point.
Amazon's denial goes on to detail other issues it had with Super Micro motherboards – the implication being that Bloomberg has got the wrong end of the stick. But other problems with the boards don't preclude the spy-chip explanation and could in fact be manifestations of the fact that third-parties are able to install whatever they want on the motherboards through such a chip.
Apple's denial is typical Apple. Reflecting its superiority complex, it mocks the news organization: "Over the course of the past year, Bloomberg has contacted us multiple times with claims, sometimes vague and sometimes elaborate, of an alleged security incident at Apple. Each time, we have conducted rigorous internal investigations based on their inquiries and each time we have found absolutely no evidence to support any of them."
It also talks about how "deeply disappointed" it is in the reporters because they were "not open to the possibility that they or their sources might be wrong or misinformed." And even suggests they may have got confused with a "previously-reported 2016 incident in which we discovered an infected driver on a single Super Micro server in one of our labs."
So far, so Apple. But it also makes a strong denial that deserves attention: "On this we can be very clear: Apple has never found malicious chips, 'hardware manipulations' or vulnerabilities purposely planted in any server. Apple never had any contact with the FBI or any other agency about such an incident. We are not aware of any investigation by the FBI, nor are our contacts in law enforcement."
Whichever way you parse that, it remains a strong denial. If it turns out the Bloomberg report is true, it would be hard to paint that sentence as anything but a lie.
It is also worth noting that neither Amazon nor Apple went for the usual "we do not discuss any national security or law enforcement issues as a matter of policy" – which is the most common tacit way of acknowledging something happened without saying what.
As for Super Micro, it denies knowing anything about any investigations – which is likely entirely true – but does not impact the story at all. No one is suggesting that Super Micro knowingly compromised its own products. The server maker ultimately "strongly refutes reports that servers it sold to customers contained malicious microchips in the motherboards of those systems."
Hm, an invite-only cybersecurity powwow?
So let's briefly take a different tack: where did the story come from, who are Bloomberg's sources, and where could it have got things wrong?
Reading into the story, it seems that the most likely start point for the entire investigation stems from a meeting in late 2015 that was organized by the Pentagon. The story describes it as "a small, invite-only meeting in McLean, Virginia" with "several dozen tech executives and investors."
The fact it was a meeting in McLean, near the CIA headquarters, rather than a more formal location, suggests it was an informal confab. And the number of people present makes it easy for someone who was there to pass on the details to reporters without being identified.
The meeting happened shortly after a cybersecurity agreement between President Barack Obama and Chinese President Xi Jinping in which China said it would no longer turn a blind eye to intellectual property theft from American companies. According to Bloomberg's sources, some in the intelligence community were concerned that China had developed more advanced ways to hack servers – and the story notes that a next-generation spy chip may be thin enough that it could be embedded between the layers of fiberglass that the other components are attached.
The core details of the story – that the US intelligence agencies carried out an investigation after it was informed by private sector about a possible spy chip on their server motherboards – can be traced to that meeting.
Bloomberg's version of the meeting says that "attendees weren’t told the name of the hardware maker involved, but it was clear to at least some in the room that it was Super Micro."
Given that tip-off, Bloomberg's reporters have been chasing the story and, as far as we can tell, hit on two other key sources – someone who claims to have seen a confidential internal report from Amazon and its third-party contractor that dug into the issue and a second person who "saw digital photos and X-ray images of the chips."
The crucial report
Bloomberg says that third-party contractor was based in Ontario, Canada. Amazon went out of its way to say that it "commissioned a single external security company to do a security assessment for us as well. That report did not identify any issues with modified chips or hardware." It repeats that point later, saying, "this was the sole external security report commissioned" and notes that Bloomberg has "refused to share any details of any purported other report with us."
Which makes you wonder: where did this alleged report come from? Who commissioned it? Who wrote it? Should we trust who claims to have seen it? The entire story may hinge on that report that Bloomberg claims exists and Amazon denies.
From that point, Bloomberg's story is built on another 14 people – that it has chosen to keep anonymous – confirming various aspects of the story. There are "six current and former senior national security officials" that it says have confirmed the "discovery of the chips and the government’s investigation."
It claims to have two people inside Amazon (AWS) that "provided extensive information on how the attack played out at Elemental" and three people inside Apple, two of whom confirmed to Bloomberg that "the company reported the incident to the FBI but kept details about what it had detected tightly held, even internally."
So we have:
- Two Amazon employees
- Three Apple employees
- Six intelligence agencies officials
- Six other people that Bloomberg says confirmed various different aspects of the story
That is clearly enough to run a story. But is it possible there was all a big misunderstanding somewhere down the line?
The McLean, Virginia briefing could easily be Pentagon officials over-playing their fears about Chinese involvement because it benefits them – the assembled tech leaders would no doubt raise their concerns privately and that would get back to the White House and intelligence services and create a sense that, despite the new agreement with China, it was essential to still keep an eye on them.
If your entire job is tracking China's espionage efforts in the tech industry, and the Obama-Xi agreement could see your budget slashed, then giving an off-the-record briefing that warned about secret chips could well ensure the funds keep flowing.
As to the reports – from both Amazon and Apple – that Bloomberg says its sources have seen. It is worth noting that Bloomberg does not claim to have seen those reports itself. How closely were its sources able to scrutinize those reports? Could they have been mistaken?
From that point, it is very possible that the other sources that Bloomberg felt were confirming its story were confirming something else: that China is trying to get into the hardware supply chain. Which is no doubt true, as US intelligence agencies have repeatedly warned in the past year, particularly with respect to mobile phones.
So it is possible that the reporters did an excellent job but ended up in the wrong place, with half a story but going down the wrong path. It is equally possible that they have got 90 per cent of the way there and Apple and Amazon are carefully using the last 10 per cent to issue careful denials.
It's worth asking one more question: what would everyone gain from misstating the truth?
Well, Bloomberg's reporters clearly have the story of a lifetime, and were driven to publish it, to the extent that it is very possible that they disregarded company denials, convinced that they were closing ranks on them over a very sensitive story.
Bloomberg reporters receive bonuses based indirectly on how much they shift markets with their reporting. This story undoubtedly did that. The publisher employs roughly 2,000 journalists, who are encouraged to work together and share information through their Bloomberg Terminals, with many layers of editing and fact checking, and it has a zero tolerance on errors: it is inconceivable that it would publish a story this huge that wasn't watertight.
Apple and Amazon may be driven to deny the story even if it is true. The yarn threatens to cause billions of dollars of potential damage to their business. It would push countless companies to look at their own hardware solutions rather than rely on them as third parties. You can see the impact of that in their two per cent share falls today. Apple and Amazon are also extremely tricky with the press, carefully spinning their way out of sticky situations with caveats in a way that makes us naturally distrust their statements.
Plus, of course, both companies would want to keep any highly confidential information and contacts with intelligence services as quiet as possible. Even if the story is true, they may be ordered to deny it as vigorously as possible by the Feds on national security grounds. But it is striking quite how vigorous those denials have been on this story. Again, whatever happened to the tried and tested PR response, "We do not comment on rumor or speculation, especially with regards to national security"?
Already out there
Plus of course the impact has already been felt.
Infosec companies are already advising companies what to do, talking about the situation as if it is already a done deal. "First of all, you are unlikely going to spot the additional component on your own. Amazon apparently was able to do so after comparing drawings of a motherboard to what was actually built," notes one post matter-of-factly, adding: "Should you stop buying Supermicro motherboards? The real question is: What are the alternatives?"
You dirty DRAC: IT bods uncover Dell server firmware security slipREAD MORE
Williams argued for "heightened vigilance" for anyone with Super Micro boards in their systems. Even if the story is true, that doesn't mean that every board will have the spy chip, he notes; it was likely a very small number of motherboards were compromised. But you could be one of them.
The only way to detect if your company's systems have been infiltrated is network monitoring. "There is zero chance this will be picked up by antivirus software," he warned.
Alan Paller, director of the SANS Institute, told The Register:
Two reasons why I'm confident that Bloomberg’s report is accurate. First, I have known both Jordan and Michael [Jordan Robertson and Michael Riley, the Bloomberg story's authors] for more than decade and their due diligence is world class. Second, the objective that this “grain of rice” chip accomplishes is the single highest priority cybersecurity objective for intelligence agencies of all major countries participating the this arena.
At the Cloudflare Internet Summit, in response to a question from The Register about Bloomberg’s report, Jeff Immelt, chairman of Athenahealth and former chairman and CEO of General Electric, said he hadn’t yet seen the claims but observed that supply chain concerns represent a huge threat to enterprises.
Immelt said he believes the government should be working with industry to present a united front in terms of cyber security. “We need I think a collective transparent review as it applies to security capabilities. And that just hasn’t happened yet,” he said.
Of course the bigger question is not really about tiny secret spy chips but overall security. There is no reason why a similar ability to hack into motherboards couldn't be included in chips expected to be on the circuit boards – and so be physically undetectable. And, of course, the majority of the world's chips are manufactured, you guessed it, in China and Taiwan. You know: the country that makes everyone's iPhones. ®
Additional reporting by Thomas Claburn and Chris Williams.