Irate VMware customers were left unable to power up their virtual servers this morning because of a bug that killed their systems when the clock clicked round to 12 August.
The bug was sent out to customers in ESX 3.5 update 2, VMware's latest hypervisor, which went out on 27 July. The version could have been downloaded and installed by thousands of customers since then.
Over the past 12 hours we’ve received angry reports from businesses and individuals affected by the cock-up.
VMware told El Reg it was aware of the problem. The firm’s group product marketing manager Martin Niemer said: “We are sending communication to all customers who have downloaded the software and we are aggressively working on a fix which should be within a short time frame.”
He declined to comment on how many customers would have been hit by the embarrassing date blunder. Niemer claimed that given it’s only been two weeks since ESX 3.5 update 2 was made available for download, it was unlikely that many people would have installed it in a live production environment.
But the firm’s forum suggests a different story. Since the problem first came to light, VMware's thread about the issue has been viewed more than 2,500 times.
We put that figure* to Niemer. “I cannot tell you how many customers but it only affects people who have downloaded since 27 July, so you can imagine it’s not a very big number of customers so far,” he said, before somewhat contradicting himself with this statement: “We know who they are and we’re going to contact them.”
Niemer was also unable to offer a time frame for when angry customers can expect to see a fix. “We cannot give an exact time frame but it should be within a few days... but I cannot give an exact date right now.”
We asked if the firm accepted that the bug was a major cock-up for VMware. “We’ve identified the problem and we’re working on the fix, and of course there’s going to be a post-mortem to understand what happened,” he said.
Niemer added that a work-around has been offered whereby customers should manually set the date of all ESX 3.5u2 hosts back to 10 August as a temporary fix. However, he accepted that this was not exactly a satisfactory solution for all businesses.
Some users have complained that doing this would contradict legal requirements that they must have the correct timestamps on their system.
Reg reader Duncan said VMware's FAIL represented a "fantastic bug for a company trying to embed itself into the modern computing world". While another reader, Eric, said the "time bomb" contained in the update was causing a lot of panic among businesses.
"Customers were fuming this morning having planned downtime for weeks. VMware has a lot of answering to do on this and no doubt share price will take a hit again," he said.
VMware said it will keep us up to date as to when a fix will be provided. In the meantime, anyone who has downloaded the update but not installed it is best advised to leave it gathering dust. ®
*At time of writing the thread has had more than 4,000 views.