Google cloud VMs given same IP addresses ... and down they went

Yikes! And the fix is to delete and rebuild the VM.

23 Reg comments Got Tips?

Google gave some of its cloud customers a rotten weekend by breaking a bunch of virtual machines.

Detailed in this incident report, the company first noticed problems at nearly beer o’clock on Friday afternoon, June 15th, Pacific Time – just after midnight on Saturday for European users and early Saturday morning in Asia.

The problem was described as “Google Compute Engine VM instances allocated with duplicate internal IP addresses”.

By 17:11 the company said “We believe that customers can work around the issue by launching then stopping f1 micro instances until no more duplicate IP addresses are obtained. We are awaiting confirmation that the provided workaround works for customers.”

By 20:03 the company had a better handle on the mess, telling users that “Instances that were stopped at any time between 2018-06-14 08:42 and 2018-06-15 13:40 US/Pacific may fail to start with networking. A newly allocated VM instance has the same IP address as a VM instance which was stopped within the mentioned time period.”

That advisory suggested the matter was serious because it said the next update wouldn’t land until 03:30 on Saturday the 16th.

By now Google also had a mitigation: “instances should be recreated, that is a delete (without deleting the boot disk), and a create.”

Google delivered its promised update promptly, at 03:33 with news that it was working on the problem at “Google Cloud Engine VMs that have an internal IP that is not assigned to another VM within the same project, region and network should no longer see this issue occurring, however instances where another VM is using their internal IP may fail to start with networking.”

And then after lunch on the 16th, Google declared the problem mostly fixed.

“The issue with Google Compute Engine VM instances being allocated duplicate external IP addresses has been resolved for all affected projects as of Saturday,” the company stated.

But the pain isn’t over for users, because the mitigation advice for affected VMs remains the same: “to delete (without deleting the boot disk), and recreate the affected VM instances.”

We’ve asked Google to detail the incident as we’d like to know how many VMs were impacted and more details about the cause of the matter. And also how it managed to mess up IP address management, which has not a problem of note for many years!

If the company brings us more information, we’ll update this story. ®


Keep Reading

Macs, iPhones, iPads to get encrypted DNS – how'd you like them Apples?

WWDC Cupertino idiot-tax corp is fashionably late to the party

Google rolls out pro-privacy DNS-over-HTTPS support in Chrome 83... with a handy kill switch for corporate IT

Plus: Web giant hears developers' extension concerns, ignores them

Networking boffins detect wide abuse of IPv4 addresses bought on secondary market

Suggests poorly-regulated address-marts are favorites of folks who want ‘clean’ addresses that give botnets a break

DNS this week stands for Drowning Needed Services: Design flaw in name server system can be exploited to flood machines offline

Microsoft, BIND, Google, Cloudflare, Amazon, others fix up software or offer workarounds

Talk about the fox guarding the hen house. Comcast to handle DNS-over-HTTPS for Firefox-using subscribers

Last November: These ISPs know too much! June: God bless the ISPs

Apple insists it's totally not doing that thing it wasn't accused of: We're not handing over Safari URLs to Tencent – just people's IP addresses

Cupertino in China Syndrome meltdown

Canada's .ca overlord rolls out free privacy-protecting DNS-over-HTTPS service for folks in Great White North

L’ACEI lance le Bouclier canadien dans le but de protéger gratuitement la vie privée et la sécurité des Canadiens en ligne

FYI: When Virgin Media said it leaked 'limited contact info', it meant p0rno filter requests, IP addresses, IMEIs as well as names, addresses and more

Infosec biz that found the database spill raises eyebrow at UK ISP's advisory to subscribers

Biting the hand that feeds IT © 1998–2020