Analysis Chinese government agents sneaked spy chips into Super Micro servers used by Amazon, Apple, the US government, and about 30 other organizations, giving Beijing's snoops access to highly sensitive data, according to a bombshell Bloomberg report today.
The story, which has been a year in the making and covers events it says happened three years ago, had a huge impact on the markets: the company at the center of the story, San Jose-based Super Micro, saw its share price drop by nearly 50 per cent; likewise Apple's share price dropped by just under two per cent, and Amazon's dropped by more than two per cent.
But the article has been strongly denied by the three main companies involved: Apple, Amazon, and Super Micro. Each has issued strong and seemingly unambiguous statements denying the existence and discovery of such chips or any investigation by the US intelligence services into the surveillance implants.
These statements will have gone through layers of lawyers to make sure they do not open these publicly traded corporations to lawsuits and securities fraud claims down the line. Similarly, Bloomberg employs veteran reporters and layers of editors, who check and refine stories, and has a zero tolerance for inaccuracies.
So which is true: did the Chinese government succeed in infiltrating the hardware supply chain and install spy chips in highly sensitive US systems; or did Bloomberg's journalists go too far in their assertions? We'll dig in.
First up, the key details of the exclusive. According to the report, tiny microchips that were made to look like signal conditioning couplers were added to Super Micro data center server motherboards manufactured by sub-contractors based in China.
Those spy chips were not on the original board designs, and were secretly added after factory bosses were pressured or bribed into altering the blueprints, it is claimed. The surveillance chips, we're told, contained enough memory and processing power to effectively backdoor the host systems so that outside agents could, say, meddle with the servers and exfiltrate information.
The Bloomberg article is not particularly technical, so a lot of us are having to guesstimate how the hack worked. From what we can tell, the spy chip was designed to look like an innocuous component on the motherboard with a few connector pins – just enough for power and a serial interface, perhaps. One version was sandwiched between the fiberglass layers of the PCB, it is claimed.
The spy chip could have been placed electrically between the baseboard management controller (BMC) and its SPI flash or serial EEPROM storage containing the BMC's firmware. Thus, when the BMC fetched and executed its code from this memory, the spy chip would intercept the signals and modify the bitstream to inject malicious code into the BMC processor, allowing its masters to control the BMC.
Supermicro wraps crypto-blanket around server firmware to hide it from malware injectorsREAD MORE
The BMC is a crucial component on a server motherboard. It allows administrators to remotely monitor and repair machines, typically over a network, without having to find the box in a data center, physically pull it out of the rack, fix it, and re-rack it. The BMC and its firmware can be told to power-cycle the server, reinstall or modify the host operating system, mount additional storage containing malicious code and data, access a virtual keyboard and terminal connected to the computer, and so on. If you can reach the BMC and its software, you have total control over the box.
With the BMC compromised, it is possible the alleged spies modified the controller's firmware and/or the host operating system and software to allow attackers to connect in or allow data to flow out. We've been covering BMC security issues for a while.
Here is Bloomberg's layman explanation for how that snoop-chip worked: the component "manipulated the core operating instructions that tell the server what to do as data move across a motherboard… this happened at a crucial moment, as small bits of the operating system were being stored in the board’s temporary memory en route to the server’s central processor, the CPU. The implant was placed on the board in a way that allowed it to effectively edit this information queue, injecting its own code or altering the order of the instructions the CPU was meant to follow."
There are a few things to bear in mind: one is that it should be possible to detect weird network traffic coming from the compromised machine, and another is that modifying BMC firmware on the fly to compromise the host system is non-trivial but also not impossible. Various methods are described, here.
"It is technically plausible," said infosec expert and US military veteran Jake Williams in a hastily organized web conference on Thursday morning. "If I wanted to do this, this is how I'd do it."
The BMC would be a "great place to put it," said Williams, because the controller has access to the server's main memory, allowing it to inject backdoor code into the host operating system kernel. From there, it could pull down second-stage spyware and execute it, assuming this doesn't set off any firewall rules.
A third thing to consider is this: if true, a lot of effort went into this surveillance operation. It's not the sort of thing that would be added to any Super Micro server shipping to any old company – it would be highly targeted to minimize its discovery. If you've bought Super Micro kit, it's very unlikely it has a spy chip in it, we reckon, if the report is correct. Other than Apple and Amazon, the other 30 or so organizations that used allegedly compromised Super Micro boxes included a major bank and government contractors.
A fourth thing is this: why go to the bother of smuggling another chip on the board, when a chip already due to be placed in the circuitry could be tampered with during manufacture, using bribes and pressure? Why not switch the SPI flash chip with a backdoored one – one that looks identical to a legit one? Perhaps the disguised signal coupler was the best way to go.
And a fifth thing: the chip allegedly fits on a pencil tip. That it can intercept and rewrite data on the fly from SPI flash or a serial EEPROM is not impossible. However, it has to contain enough data to replace the fetched BMC firmware code, that then alters the running operating system or otherwise implements a viable backdoor. Either the chip pictured in Bloomberg's article is incorrect and just an illustration, and the actual device is larger, or there is state-of-the-art custom semiconductor fabrication involved here.
This implant chip. Let's say it's intercepting SPI flash / serial EEPROM reads and rewriting them. Not impossible. But then it has to contain enough data to alter the BMC firmware. To then alter the host OS. To then act as a useful backdoor.— The Register (@TheRegister) October 5, 2018
What lithography is it using? 7nm? pic.twitter.com/8kbSYMuRR2
One final point: you would expect corporations like Apple and Amazon to have in place systems that detect not only unexpected network traffic, but also unexpected operating system states. It should be possible that alterations to the kernel and the stack of software above it should set off alarms during or after boot.
Bloomberg claims the chip was first noticed in 2015 in a third-party security audit of Super Micro servers that was carried out when Amazon was doing due diligence into a company called Elemental Technologies that it was thinking of acquiring. Elemental used Super Micro's servers to do super-fast video processing.
Amazon reported what it found to the authorities and, according to Bloomberg, that "sent a shudder" through the intelligence community because similar motherboards were in use "in Department of Defense data centers, the CIA’s drone operations, and the onboard networks of Navy warships."
Around the same time, Apple also found the tiny chips, according to the report, "after detecting odd network activity and firmware problems." Apple contacted the FBI and gave the agency access to the actual hardware. US intelligence agencies then tracked the hardware components backwards through the supply chain, and used their various spying programs to sift through intercepted communications, eventually ending up with a focus on four sub-contracting factories in China.
According to Bloomberg, the US intelligence agencies were then able to uncover how the seeding process worked: "Plant managers were approached by people who claimed to represent Super Micro or who held positions suggesting a connection to the government. The middlemen would request changes to the motherboards’ original designs, initially offering bribes in conjunction with their unusual requests. If that didn’t work, they threatened factory managers with inspections that could shut down their plants. Once arrangements were in place, the middlemen would organize delivery of the chips to the factories."
This explanation seemingly passes the sniff test: it fits what we know of US intelligence agencies investigative approaches, their spy programs, and how the Chinese government works when interacting with private businesses.
The report then provides various forms of circumstantial evidence that adds weight to the idea that this all happened by pointing to subsequent actions of both Apple and Amazon. Apple ditched Super Micro entirely as a supplier, over the course of just a few weeks, despite planning to put in a massive order for thousands of motherboards. And Amazon sold off its Beijing data center to its local partner, Beijing Sinnet, for $300m.