Data Centre

Cloud

Google Cloud partially evaporates for hours amid power supply failure: Two US East Coast zones rattled

Networking, Kubernetes, storage, virtual machine systems hit by outage

6 Got Tips?

Google Cloud is having a wobbly Monday. Its Kubernetes platform and networking services were partially unavailable for hours today, and its virtual-machine hosting and in-memory storage systems had a limited outage.

The web giant's Cloud Networking service fell over around 0800 PT (1500 UTC) today due to a power supply failure. Connections to virtual machines in Google's us-east1-c and us-east1-d zones started failing, and the breakdown spread to other Google Cloud services, such as its Persistent Disk product.

Here's the official word from Google at time of writing, 1300 PT (2000 UTC), some five hours on from when trouble started:

We are experiencing an issue with Cloud Networking in us-east1-c and us-east1-d, beginning on Monday, 2020-06-29 07:54 US/Pacific, affecting multiple Google Cloud Services.

Services in us-east1-d have been fully restored. Services in us-east1-c are fully restored except for Persistent Disk which is partially restored. No ETA for full recovery of Persistent Disk yet. Impact is due to power failure. A more detailed analysis will be available at a later time.

Our engineering team is working on recovery of impacted services. We will provide an update by Monday, 2020-06-29 13:00 US/Pacific with current details. We apologize to all who are affected by the disruption.

Diagnosis: Some services in us-east1-c and us-east1-d are failing, customers impacted by this incident would likely experience a total unavailability of zonal services hosted in us-east1-c or us-east1-d. It is possible for customers to experience service interruption in none, one, or both zones.

Workaround: Other zones in the region are not impacted. If possible, migrating workloads would mitigate impact. If workloads are unable to be migrated, there is no workaround at this time.

And just as we were about to publish this article, Google said it expects to full resolve the Networking and storage issues within this hour:

The issue with Cloud Networking and Persistent Disk has been resolved for the majority of affected projects as of Monday, 2020-06-29 10:20 US/Pacific, and we expect full mitigation to occur for remaining projects within the hour.

Next up, Google's Kubernetes Engine in the us-east1-c zone took a hit at around 0700 PT (1400 UTC), and at time of writing nearly six hours on, is still down:

We are experiencing an issue with Google Kubernetes Engine in us-east1-c where clusters may be unavailable or unreachable beginning at Monday, 2020-06-29 07:15 US/Pacific US/Pacific. Mitigation work is currently underway by our engineering team.

Related problems with Compute Engine virtual machines, lasting about an hour, have, we're told, been cleared up:

The issue with Google Compute Engine in zones us-east1-c and us-east1-d where existing VMs may be unavailable or unreachable, and new VM creation may fail beginning on Monday, 2020-06-29 08:20 US/Pacific has been resolved for all affected users as of Monday, 2020-06-29 09:45 US/Pacific.

The impact to us-east1-d has been mitigated by Monday, 2020-06-29 08:45 US/Pacific. The impact to us-east1-c has been mitigated by Monday, 2020-06-29 09:45 US/Pacific.

Finally, Cloud Memorystore in us-east1-c and us-east1-d fell over around 0800 PT and was restored at 1130 PT. ®

Updated to add

As this article was published, Google updated its status boards to say its Kubernetes Engine outage has been resolved.

Get our Weekly newsletter
6 Comments

Keep Reading

Kick Google all you like, Mozilla tells US government, so long we keep getting our Google-bucks

In case you've forgotten: Google sends Mozilla about $400m a year

Google to 'surface' friendly banks' business loans through Pay app

To help Indian small businesses cope with COVID crunch

Nokia snuggles up with Google Cloud as it aims to switch off on-prem servers within next two years

Yep, there it is. 'Digital transformation'

Google creates secure file locker for countries where people often share smartphones

Handy service for India and Nigeria, or feature creep that constricts competition?

Google Apps Scripts debugger is buggered for devs using V8 runtime: Fix coming... in Q4

'We need this issue solved ASAP'

Google wants to listen in to whatever you get up to in hotel rooms

Disrupting clock radios and concierges with plan to let Nest Hub devices take orders for fresh towels

I won't be ignored: Google to banish caller roulette with Verified Calls

Authenticated business: 'No, there's a good reason for this call. Honest. Don't hang u-'

Google to pull plug on Play Music, its streaming service that couldn't beat Spotify, in favour of YouTube Music

Phased shutdown ends December, and subs will be shunted to other platform

Power platform envy? Google wants to 'empower non-technical employees' with new Business Apps category

AppSheet Automation to combine workflow with data-driven applications for - whisper it - citizen devs...

DuckDuckGo cries fowl after being expunged from Google's Android search preferences menu for most of Europe

Pro-privacy upstart claims EU antitrust remedy is not fit for purpose

Tech Resources

Zero trust strategies to zap ransomware peril

Join industry veteran and security pro Mike Wronski of Nutanix as he explains to Tim Phillips about zero trust strategies combined with HCI can improve your security posture, defend against threats, help prevent your business from being the next victim of ransomware.

[Report] Real Stories From Rapid7 Penetration Testers

Hoodies off. Shoes on. Step into the attacker mindset.

Top 20 Private Cloud Questions Answered

Download this asset for straight answers to your top private cloud questions.

Practical tips for Office 365 tenant-to-tenant migration

You merged. You sold a company. You’re migrating to a sovereign data centre. Whatever the reason for your Office 365 tenant-to-tenant migration, it can be one of the most important and stressful projects you will ever take on. It will be a session packed with practical help that might just save your job … and your sanity.