VMware admits 'time bomb' rolled past quality control

Glum CEO says 'we're sorry'

VMware’s CEO has blamed a chunk of leftover pre-release code for a bug that yesterday prevented virtual servers around the world from powering up when the clock hit 12 August.

Despite the company carrying out quality assurance of its product, VMware failed to spot that it had released the leftover bit of time-bombed code within the ESX 3.5 and ESXi 3.5 Update 2 final versions of its hypervisors that were released about two weeks ago.

VMware boss Paul Maritz apologised for the major balls-up. He said the code caused the product license to expire. And, worse still, he admitted that the problem had been overlooked in recent patches to the product.

The code time bomb prevented virtual machines from powering on or leaving suspend mode and unable to migrate using the firm's VMotion software.

Maritz said VMware has released an “express patch” for customers unlucky enough to have already installed or upgraded to ESX or ESXi 3.5u2.

Its temporary work-around requires sys admins to turn the clock back to kick-start virtual machines and imagine that 12 August was just a bad dream.

However, as we noted yesterday, a whole swathe of customers have grumbled that this is inadequate since it’s a legal requirement for most businesses to timestamp all server transactions.

The company, which claims to have "the industry’s most popular and reliable hypervisor", also plans to spit out a full replacement for Update 2 that should be used by customers who want to perform fresh installs of ESX or ESXi.

Here are a few choice words from a very sad-looking Maritz:

We are doing everything in our power to make sure this doesn’t happen again. VMware prides itself on the quality and reliability of our products, and this incident has prompted a thorough self-examination of how we create and deliver products to our customers. We have kicked off a comprehensive, in-depth review of our QA and release processes, and will quickly make the needed changes.

I want to apologise for the disruption and difficulty this issue may have caused to our customers and our partners. Your confidence in VMware is extremely important to us, and we are committed to restoring that confidence fully and quickly.

Shares in VMware also took a hit yesterday, dipping more than five per cent on the New York Stock Exchange after news broke of the Palo Alto, California-based company's major code cock-up. ®

Similar topics

Narrower topics

Other stories you might like

  • VMware patches critical guest-to-host vulnerabilities
    Time to fix code like it's 2020

    In an advisory this week, VMware alerted users to guest-to-host vulnerabilities in the XHCI and UHCI USB controllers in its ESXi hypervisor, plus an important flaw fixed in NSX Data Center for vSphere.

    In all, five vulnerabilities were discovered in VMware's ESXi, Workstation, Cloud Foundation (ESXi), and Fusion during the Tianfu Cup 2021, a Chinese vulnerability competition, by the country's Kunlun Lab. Bugs that Kunlun discovered were disclosed privately to VMware – though last year China passed a new law ordering security researchers to reveal findings to the country's Ministry of Public Security at least two days before anyone else.

    The vendor said it hadn't seen any evidence the competition's findings had been exploited in the wild. Patches have been issued, now it's up to admins to schedule them. The vulnerabilities range from use-after-free() and double-fetch flaws that can be exploited to execute code on the host, to an old-fashioned denial of service (DoS). The full list for ESXi, Workstation, Cloud Foundation, and Fusion is:

    Continue reading
  • Microsoft patches Y2K-like bug that borked on-prem Exchange Server
    Happy New Year. Welcome back! Now apply this patch – which Microsoft warns isn't easy – if you want email to work

    Microsoft has kicked off 2022 by issuing a patch for Exchange Server 2016 and 2019, which both possessed a “latent date issue” that saw emails queued up instead of being dispatched to inboxes.

    “The problem relates to a date check failure with the change of the new year,” states a January 1st post to the Exchange Blog.

    Exchange’s malware scanning engine is the source of the problem, as Exchange checks the version of that software and then tries to write the date into a variable. But that variable’s maximum value is 2,147,483,647 and the value Exchange tries to write - 2,201,010,001, to reflect the date of January 1st, 2022, at midnight – exceeds the variable’s maximum threshold.

    Continue reading
  • Patch now? Why enterprise exploits are still partying like it's 1999
    Am I only dreaming, or is this burning an Eternal Blue?

    Feature Some vulnerabilities remain unreported for the longest time. The 12-year-old Dell SupportAssist remote code execution (RCE) flaw – which was finally unearthed earlier this year – would be one example.

    Others, however, have not only been long since reported and had patches released, but continue to pose a threat to enterprises. A joint advisory from the National Cyber Security Centre (NCSC) and the US Cybersecurity and Infrastructure Security Agency (CISA), published in late July, listed the top 30 publicly known vulnerabilities that are routinely being exploited by threat actors. Many of these are a good few years old, including one Microsoft Office RCE that was patched in 2017 but had been around since the year 2000.

    Eoin Keary, CEO and founder of Edgescan, told The Register that the oldest common vulnerability discovered in its latest quarterly vulnerability scans report (CVE-1999-0517, impacting Simple Network Management Protocol) dated back to 1999. Which raises the question, why are threat actors being allowed to party like it's, um... 1999?

    Continue reading

Biting the hand that feeds IT © 1998–2022