Black Hat Next time you leave things to the last minute, remember this well.
Despite having known about the Meltdown and Spectre security vulnerabilities for roughly six months, Intel and other chip giants still hadn't warned the US government's cybersecurity nerve-center by the time The Register blew the lid off the design flaws.
Chipzilla and its semiconductor-slinging rivals had planned to tell US-CERT – Homeland Security's Computer Emergency Response Team – around January 3 that they were going to go public on January 9 with details of processor bugs that could be exploited by malware to steal sensitive information, such as passwords and crypto-keys, from PCs, Macs, smartphones, and other devices.
The chip designers had been alerted to the Meltdown and Spectre vulnerabilities months before, around June 2017, but kept everything hush-hush under a strict embargo as they worked on squashing the bugs.
On Tuesday, January 2, after piecing together snippets of Linux kernel source code changes, mailing list posts by software engineers, and clues whispered to us by industry insiders, El Reg broke the news that operating system makers were scrambling to rewrite portions of their software to mitigate what came to be known as Meltdown and Spectre.
That sparked another mad dash in the tech world, as the vendors that had planned to go public with patches, mitigations, and details of the design blunders in seven days' time were now forced to scrap their embargo – and move disclosure forward to January 3.
Woo-yay, Meltdown CPU fixes are here. Now, Spectre flaws will haunt tech industry for yearsREAD MORE
This timeline is according to industry bods appearing on a panel on Wednesday evening at this year's Black Hat USA hacking conference in Las Vegas. The speakers were Art Manion of the Software Engineering Institute's CERT Coordination Center (CERT/CC); Christopher Robinson of Red Hat; Eric Doerr of Microsoft; and Matt Linton of Google.
Their panel, titled "The True Story of Fighting Meltdown and Spectre," sought to reveal what went on behind the scenes in the months, days, and hours leading up to The Register's exclusive. And, for what it's worth, we vultures were not aware of any embargo, and did not receive any guidance from Intel's PR team despite contacting it while preparing the piece.
On January 3, after Google, Red Hat, Intel, Arm, AMD, and others spilled the beans, CERT/CC, which is sponsored by the Department of Homeland Security, published an advisory based on this now-public information, having had no heads up. A day later, US-CERT issued its formal alert, noting it too had only found out about the bugs that week, on January 3, and referenced CERT/CC's writeup.
This also explains why CERT/CC initially advised people on January 3 to replace their vulnerable processor hardware, as it had not been fully briefed on the availability and rollout of microcode patches and software mitigations. It corrected its advisory when it became clear less drastic options were available.
“The embargo holders had been planning to tell CERT a week before the embargo lifted,” Linton, whose job title at Google is chaos specialist, told The Register after the panel session. “Had they known, CERT could have advised people that patches were available but instead initially recommended those affected should replace their processors.”
Manion, a senior vulnerability analyst at CERT/CC, said somewhat jokingly that he was a little hurt not to have been told about the issue earlier. But, he said, that did at least allow him to get a decent Christmas break – a lot of kernel developers lost their festive holiday time to rewriting memory management code.
Eagle-eyed Googlers notified chip makers and designers in and around June 2017 that their speculative execution engines – used to prime processors with instructions to execute in order to run software as fast as possible – had various exploitable security shortcomings.
About a month later, the information was shared more widely within Google, and in turn, operating system developers were alerted, as workarounds would require changes to kernels. Two other teams independent of the ad giant – Werner Haas, and Thomas Prescher of Cyberus Technology, and Daniel Gruss, Moritz Lipp, Stefan Mangard, and Michael Schwarz at the Graz University of Technology – also separately stumbled upon and privately disclosed the flaws.
The whole shebang sparked a remarkable collaboration between rivals, we're told. "Months before the public learned about the challenges with speculative execution, defenders from hardware, platform, cloud, and service providers were working together around the clock building mitigations and coordinating a response to help protect the billions of users depending on their platforms," the panel explained in their session blurb. "Along the way, competitors became partners, and an unprecedented level of information was shared."
At first, every biz involved worked alone on the issue, until in the autumn of 2017 there was an unprecedented meet-up of engineers from fierce rivals. The goal: to develop better solutions to Meltdown and Spectre.
“We had the face-to-face in November,” said Doerr, general manager of the Microsoft Security Response Center. “It’s funny how rare that kind of meeting is. There was pushback at Microsoft that the legal arrangements would be hard. Honestly, I was blown away by the collaboration in the room. It was a leap of faith to trust that this was the right thing to do.”
Robinson, a Red Hat security team lead, said that given the thousands of people working on the project in secret, he was surprised the news didn’t leak earlier. But when it did, he got a call on January 3 saying the patches needed to be deployed that day.
Eight hours later, he was pushing out fixes. He told The Register that a lot of minor work still had to be done after the initial public disclosure as the disclosure date had been set for January 9.
“The attack is brilliant, it’s very creative, and it’s stunning no one found it earlier,” Robinson said. “Every year we push out 100 fixes for vulnerabilities more severe than Spectre, which we only rated as 'important' on our scale."
During the panel discussion, the situation with Lenovo was brought up: there were claims earlier that Intel alerted the Chinese PC maker – and thus, via proxy, the Chinese government – about the CPU-level flaws before it warned the US government. Given the above timeline, we can imagine our story detonating the embargo in between Chipzilla warning the computer manufacturer and Uncle Sam.
How to (slowly) steal secrets over the network from chip security holes: NetSpectre summonedREAD MORE
An audience member introduced himself as a Lenovo staffer who was briefed of Meltdown and Spectre ahead of the planned disclosure date, and he denied the Chinese government had been made aware of the issue in advance. By his reckoning, only a couple of dozen people at Lenovo knew about the issue, and all were based in the US, apart from one developer in Japan.
Linton had been quizzed by US politicians somewhat miffed at the handling of the bug disclosures, and defended the decision not to tell the American government until literally the final week. He was told the US administration could have taken some actions, such as moving sensitive virtual machines to an environment where the vulnerabilities definitely could not be exploited. Frankly, he said, he was shocked the government wasn’t already doing this already as basic operational security.
One of the key points your humble hack picked up from the panel session, apart from the CERT timing kerfuffle, was the collaboration between bitter rival corporations, that face-to-face communications are incredibly important, and is something the industry should encourage more – with one important restriction.
“It really is all about communications,” Linton said. “But the rules have to be that no one throws shade at anyone else. This nascent sense of cooperation is something we want to nurture. The one thing what will kill it is a press release in which someone claims to be better [at security] than the others in the group.” ®