Meltdown, Spectre bug patch slowdown gets real – and what you can do about it

Chip flaw fixes not so insignificant after all


Analysis Having shot itself in the foot by prioritizing processor speed over security, the chip industry's fix involves doing the same to customers.

The patches being put in place to address the Meltdown and Spectre bugs that affect most modern CPUs were supposed be airy little things of no consequence. Instead, for some unlucky people, they're anchors.

Having helped find the flaws, Google insisted the software fixes that have begun to appear "introduce minimal performance impact," and insisted the performance hit will diminish over time.

Intel said as much in its statement, claiming "any performance impacts are workload-dependent, and, for the average computer user, should not be significant and will be mitigated over time."

That may be true eventually, thanks in part to a processor feature called Processor-Context ID, or PCID. But more on that later.

Woo-yay, Meltdown CPU fixes are here. Now, Spectre flaws will haunt tech industry for years

READ MORE

At the moment, the speed consequences of patching these bugs is significant enough to elicit attention and complaints. To be clear: we here at El Reg highly recommend you install the CPU security bug patches as soon as possible. We just want folks – particularly cloud subscribers and IT admins – to be aware of the effects.

While most casual desktop users and gamers won't notice any prolonged slowdown, or any performance hit at all, people running IO or system-call intensive software, such as databases on backend servers, may notice the difference.

Red Hat has clocked the patch performance impact as ranging from one to 20 per cent.

Epic Games on Friday explained the cause of recent login and stability issues experienced by its players, noting: "All of our cloud services are affected by updates required to mitigate the Meltdown vulnerability."

The company, which relies on AWS servers, posted a screenshot of a graph depicting a spike in CPU utilization after a host was patched. The Register asked Epic to elaborate on its findings, but a spokesperson said the developer had nothing further to add at the moment.

Discussions on the mailing list for Lustre, a parallel distributed filesystem, described slowdowns ranging from 10 per cent to as high as 45 per cent for certain IO intensive applications.

"We found terrible performance on the test system with zfs+compression+lustre," wrote Arman Khalatyan of the Leibniz Institute for Astrophysics Potsdam in a memo on Monday.

On Reddit, a Monero coin miner reported a slowdown of about 45 per cent after applying the Meltdown patch. On that thread, another person cited a hash rate decrease of 10 to 15 per cent.

Quora, which relies on AWS, on Saturday said it is "facing a slowdown due to the patch applied by AWS for Intel's Meltdown and Spectre issues."

Via Twitter, Francis Wolinski, a data scientist with Paris-based Blueprint Strategy, noted that Python slowed significantly (about 37 per cent) after applying the Meltdown patch for Windows 7.

Also via Twitter, Ian Chan, director of engineering for analytics firm Branch Metrics, described CPU utilization increases of five to 20 per cent after the Meltdown patch was applied to the AWS EC2 hypervisor handling its Kafka instances.

Amazon customers have sent The Register several screenshots of CPU utilization showing spikes similar to those that have been publicly discussed. Before the weekend, Amazon confirmed the updates will ding AWS virtual-machine performance to some degree, albeit with no "meaningful performance impact for most customer workloads" expected, apparently.

AWS CPU utilization spike

Soar ... An example AWS CPU utilization spike after installing CPU flaw security patches (Click to enlarge)

These figures are in keeping with the estimates first reported by The Register, a performance hit of roughly five to 30 per cent, with the caveat that any such results are highly variable and depend on a number of factors such as the workload in question and the technology involved.

El Reg's sister site The Next Platform estimated that the amount of computing value lost to the slowdown amounts to $6 billion annually.

These delays are largely the consequence of Meltdown patches, which on Linux enforce separation between the kernel and user virtual memory address spaces through Kernel Page Table Isolation, or KPTI.

Beyond Linux, Microsoft has patched Windows Server 2008 R2, 2012 R2, and 2016, among other flavors of its operating system. Apple has also mitigated Meltdown and Spectre in iOS and macOS.

Spectre mitigations – which involve recompiling software with countermeasures such as Google's retpoline as well as microcode updates depending on the processor model – have just begun to appear. Though considered only partial fixes for a problem that will take some time to sort out, they're nonetheless expected to affect performance, too (beyond knackering some AMD PCs if you're using Windows).

PCID

If there's a bright side to all this, it's that the PCID feature in Intel's x86-64 chips since 2010 can reduce the performance hit from patching Meltdown. (If you have a 32-bit system, you're on your own.)

Remediating Meltdown – which is present in modern Intel processors – involves enforcing complete separation between user processes' virtual memory spaces and the kernel's virtual memory areas. Rather than map the kernel into the top portion of every process's virtual memory space where it remains invisible unless required to handle an interrupt or system call, the kernel is moved to a separate virtual address space and context. This fix prevents malware from exploiting the Meltdown CPU bug to read kernel memory from user mode, and is referred to as Kernel Page Table Isolation.

Switching back and forth between these contexts – from the user process context to the kernel context and back to the user process – involves reloading page tables, one set describing the user process and another describing the kernel. These tables map the process or kernel's virtual memory to physical blocks of RAM or swap space.

These context switches from user process to kernel to process not only takes time, it also flushes any cached virtual-to-physical memory translations, all in all causing a performance hit, particularly on workloads that involve a lot of IO or system calls. But with PCID, there's no need to flush the entire translation lookaside buffer (TLB) cache on every context switch as selected TLB entries can be retained in the processor.

PCID first saw Linux support in the 4.14 kernel released in November 2017, and thus it's not necessarily available by default with every Linux instance, particularly on virtual machines.

In a Google Groups post on Sunday, Gil Tene, CTO and cofounder of enterprise Java biz Azul Systems, said PCID has become critical both for security and performance on Intel's x86 platform. But he observed that it isn't present on many of the virtualized Linux instances he's looked at.

Most KVM guests – kernel-based virtual machines – don't include PCID, according to Tene, while most VMware guests do. And about half of the AWS instances he looked at don't have it.

"You REALLY want PCID in your processor," wrote Tene. "Without it, you may be running insecurely (Meltdown fixes turned off by default), or you may run so slow you'll be wishing for a security intrusion to put you out of your misery."

In other words, if you're seeing crap performance after applying these fixes, look at your kernel configuration and get PCID enabled – if the hardware feature is present in your chipset. Windows should, for what it's worth, use PCID if it's provided by the processor. ®


Other stories you might like

  • North Korea pulled in $400m in cryptocurrency heists last year – report

    Plus: FIFA 22 players lose their identity and Texas gets phony QR codes

    In brief Thieves operating for the North Korean government made off with almost $400m in digicash last year in a concerted attack to steal and launder as much currency as they could.

    A report from blockchain biz Chainalysis found that attackers were going after investment houses and currency exchanges in a bid to purloin funds and send them back to the Glorious Leader's coffers. They then use mixing software to make masses of micropayments to new wallets, before consolidating them all again into a new account and moving the funds.

    Bitcoin used to be a top target but Ether is now the most stolen currency, say the researchers, accounting for 58 per cent of the funds filched. Bitcoin accounted for just 20 per cent, a fall of more than 50 per cent since 2019 - although part of the reason might be that they are now so valuable people are taking more care with them.

    Continue reading
  • Tesla Full Self-Driving videos prompt California's DMV to rethink policy on accidents

    Plus: AI systems can identify different chess players by their moves and more

    In brief California’s Department of Motor Vehicles said it’s “revisiting” its opinion of whether Tesla’s so-called Full Self-Driving feature needs more oversight after a series of videos demonstrate how the technology can be dangerous.

    “Recent software updates, videos showing dangerous use of that technology, open investigations by the National Highway Traffic Safety Administration, and the opinions of other experts in this space,” have made the DMV think twice about Tesla, according to a letter sent to California’s Senator Lena Gonzalez (D-Long Beach), chair of the Senate’s transportation committee, and first reported by the LA Times.

    Tesla isn’t required to report the number of crashes to California’s DMV unlike other self-driving car companies like Waymo or Cruise because it operates at lower levels of autonomy and requires human supervision. But that may change after videos like drivers having to take over to avoid accidentally swerving into pedestrians crossing the road or failing to detect a truck in the middle of the road continue circulating.

    Continue reading
  • Alien life on Super-Earth can survive longer than us due to long-lasting protection from cosmic rays

    Laser experiments show their magnetic fields shielding their surfaces from radiation last longer

    Life on Super-Earths may have more time to develop and evolve, thanks to their long-lasting magnetic fields protecting them against harmful cosmic rays, according to new research published in Science.

    Space is a hazardous environment. Streams of charged particles traveling at very close to the speed of light, ejected from stars and distant galaxies, bombard planets. The intense radiation can strip atmospheres and cause oceans on planetary surfaces to dry up over time, leaving them arid and incapable of supporting habitable life. Cosmic rays, however, are deflected away from Earth, however, since it’s shielded by its magnetic field.

    Now, a team of researchers led by the Lawrence Livermore National Laboratory (LLNL) believe that Super-Earths - planets that are more massive than Earth but less than Neptune - may have magnetic fields too. Their defensive bubbles, in fact, are estimated to stay intact for longer than the one around Earth, meaning life on their surfaces will have more time to develop and survive.

    Continue reading

Biting the hand that feeds IT © 1998–2022