CPU bug patch saga: Antivirus tools caught with their hands in the Windows cookie jar

You're fondling our kernel wrong, grumbles Microsoft

Microsoft's workaround to protect Windows computers from the Intel processor security flaw dubbed Meltdown has revealed the rootkit-like nature of modern security tools.

Some anti-malware packages are incompatible with Redmond's Meltdown patch, released last week, because the tools make, according to Microsoft, “unsupported calls into Windows kernel memory,” crashing the system with a blue screen of death. In extreme cases, systems fail to boot up when antivirus packages clash with the patch.

The problem arises because the Meltdown patch involves moving the kernel into its own private virtual memory address space. Usually, operating systems such as Windows and Linux map the kernel into the top region of every user process's virtual memory space. The kernel is marked invisible to the running programs, although due to the Meltdown design oversight in Intel's modern chips, its memory can still be read by applications. This is bad because it means programs can siphon off passwords and other secrets held in protected kernel memory.

Certain antivirus products drill deep into the kernel's internals in order to keep tabs on the system and detect the presence of malware. These tools turn out to trash the computer if the kernel is moved out the way into a separate context.

In other words, Microsoft went to shift its cookies out of its jar, and caught antivirus makers with their hands stuck in the pot.

Thus, Microsoft asked anti-malware vendors to test whether or not their software is compatible with the security update, and set a specific Windows registry key to confirm all is well. Only when the key is set will the operating system allow the Meltdown workaround to be installed and activated. Therefore, if an antivirus tool does not set the key, or the user does not set the key manually for some reason, the security fix is not applied.

In fact, until this registry key is set, the user won’t be able to apply any Windows security updates – not just this month's patches, just any of them in future. This is bad news. Remember how the infamous WannaCry ransomware spread like wildfire across out-of-date Windows 7 systems last May? Now here's a load more machines not being updated that will be attacked by the next malware epidemic.

Redmond said it was working with anti-malware software vendors to resolve the issue. Compliance so far is patchy with some antivirus makers in the process of ironing out problem while others have set themselves up in opposition to Redmond’s approach.

Anti-malware vendor SentinelOne slammed Microsoft's handling of the issue, claiming “this is going to leave millions of endpoints exposed." SentinelOne is upset that "the responsibility of setting the registry key" is shifted to the AV vendor. "While our testing revealed no incompatibilities, we are unwilling to take on the risk of setting this registry key,” the security software house said.

“This is because our customers may have other software products that use unsupported/undocumented APIs that are incompatible with Microsoft’s latest patches. In such a case, our customers may experience stop errors/system instabilities caused by other products that are not compatible with Microsoft fixes,” Sentinel One staff explained in a blog post.

SentinelOne said it planned to give customers the choice of whether to set the registry key. “While some vendors in the market are taking the approach of checking for incompatible software, we do not believe that this approach can be done in a comprehensive manner,” it concluded. The security firm advised customers to test the patch with its agent and their full stack of software applications before flipping the registry key switch.

Other anti-malware vendors want punters to set the registry key themselves, presumably because it absolves the companies of blame if stuff falls apart post-patch. These include Carbon Black, Palo Alto, FireEye and Cylance. Avast, ESET and Kaspersky, among others, are up and running, meanwhile, by setting the registry key, while for others – Trend, McAfee, etc – are still testing their gear.

Kevin Beaumont, a UK-based infosec guru who has been keeping close tabs on the problem, has put together a spreadsheet on which anti-malware products are setting the registry key, and which are compatible with the Meltdown workaround.


Symantec Endpoint Protection users are advised not to apply Microsoft’s Meltdown fix just yet due to a conflict. “After applying Microsoft Windows Security Updates released on January 3rd, 2018, the Symantec Endpoint Protection (SEP) system tray icon reports there are multiple problems. No errors are reported if the SEP client UI is opened,” Symantec said.

When will Symantec’s update get released? “The UI fix is undergoing QA testing this week and will be released soon,” Symantec product manager Adam Licata told El Reg.

Another problem is that malware detection engines may not be able to set Windows registry keys. “The producer can't just 'ship an update that sets the key'," said antivirus industry veteran Vesselin Bontchev. "They'd have to modify the product, maybe substantially, to add such a capability – and that can't be done quickly."

Work on that front in underway, but it does add an additional complication to an already messy situation.

Industry pundit Graham Cluley is sympathetic about Microsoft’s handling of the potential conflict between its kernel memory redesign to counteract Meltdown, and security software. Anti-virus vendors have little choice but to comply, he said.

“Microsoft is caught between a rock and a hard place on this one,” Cluley wrote in a blog post. “The last thing they want to do is roll out an update that causes computers to crash. It's a painful decision, but if they can determine which computers don't appear to be running a ‘safe’ anti-virus program then they're probably right not to push out security updates to that PC.

“Anti-virus vendors have little choice. They will have to fix their products to fall into line, as customers won't be satisfied with being blocked from receiving Microsoft security updates."

Martijn Grooten‏, editor of industry journal Virus Bulletin and sometime security researcher, reckoned the manual key policy was workable. “I think the manual key approach is justified for vendors with customers who may have multiple AVs running on the same system (more common with next-gens; note that registry key implies all AVs are compatible),” he said.

Beaumont has put together a blog post about Microsoft Meltdown CPU security fixes and antivirus vendors here. ®

Other stories you might like

  • Despite global uncertainty, $500m hit doesn't rattle Nvidia execs
    CEO acknowledges impact of war, pandemic but says fundamentals ‘are really good’

    Nvidia is expecting a $500 million hit to its global datacenter and consumer business in the second quarter due to COVID lockdowns in China and Russia's invasion of Ukraine. Despite those and other macroeconomic concerns, executives are still optimistic about future prospects.

    "The full impact and duration of the war in Ukraine and COVID lockdowns in China is difficult to predict. However, the impact of our technology and our market opportunities remain unchanged," said Jensen Huang, Nvidia's CEO and co-founder, during the company's first-quarter earnings call.

    Those two statements might sound a little contradictory, including to some investors, particularly following the stock selloff yesterday after concerns over Russia and China prompted Nvidia to issue lower-than-expected guidance for second-quarter revenue.

    Continue reading
  • Another AI supercomputer from HPE: Champollion lands in France
    That's the second in a week following similar system in Munich also aimed at researchers

    HPE is lifting the lid on a new AI supercomputer – the second this week – aimed at building and training larger machine learning models to underpin research.

    Based at HPE's Center of Excellence in Grenoble, France, the new supercomputer is to be named Champollion after the French scholar who made advances in deciphering Egyptian hieroglyphs in the 19th century. It was built in partnership with Nvidia using AMD-based Apollo computer nodes fitted with Nvidia's A100 GPUs.

    Champollion brings together HPC and purpose-built AI technologies to train machine learning models at scale and unlock results faster, HPE said. HPE already provides HPC and AI resources from its Grenoble facilities for customers, and the broader research community to access, and said it plans to provide access to Champollion for scientists and engineers globally to accelerate testing of their AI models and research.

    Continue reading
  • Workday nearly doubles losses as waves of deals pushed back
    Figures disappoint analysts as SaaSy HR and finance application vendor navigates economic uncertainty

    HR and finance application vendor Workday's CEO, Aneel Bhusri, confirmed deal wins expected for the three-month period ending April 30 were being pushed back until later in 2022.

    The SaaS company boss was speaking as Workday recorded an operating loss of $72.8 million in its first quarter [PDF] of fiscal '23, nearly double the $38.3 million loss recorded for the same period a year earlier. Workday also saw revenue increase to $1.43 billion in the period, up 22 percent year-on-year.

    However, the company increased its revenue guidance for the full financial year. It said revenues would be between $5.537 billion and $5.557 billion, an increase of 22 percent on earlier estimates.

    Continue reading

Biting the hand that feeds IT © 1998–2022