Google Cloud's US-East load balancers are lousy with latency
Monday morning rush job, anyone? Asking 'cos there's no ETA for a fix other than moving to another region
Google Cloud is having a nasty not-quite outage in its US-East region.
At the thoroughly indecent time of 05:56 AM on Sunday, Pacific time, Google reported an incident it called "Cloud L2 overload in us-east4 causing harm in us-east4 & nearby regions" and ascribed the cause to "an issue with Cloud Load Balancing."
Google Cloud has three such regions: South Carolina hosts east1, North Virginia is home to east4, and Ohio is the site for east5. Confusingly, Google Cloud does not have an east2 or east3 region.
At 07:56 the incident was renamed "Google Engineers are investigating high latency and elevated errors in US-East."
The issue has persisted ever since and kept that moniker, with Google listing South Carolina (us-east1) and Northern Virginia (us-east4) as the currently affected locations.
Mitigation work had commenced by 11:41 AM, but at 21:34 Google's 11th status update on the incident still reported a diagnosis of "Higher latency and possibly elevated errors on customers' global L7 Load Balancers during US peak hours (~05:00 to ~19:00)."
That won't have been a colossal issue on Sunday, US time. But as The Register writes this story the clock has just tipped past 01:00 AM on the US East Coast. As workers arrive at their desks on Monday, greater inconvenience is to be expected.
- WFH? Google Cloud's offices like a 'ghost town' before new policy
- Google: You get crypto, you get crypto, almost everyone gets email crypto!
- Taking notes from AWS, Google prepares custom Arm server chips of its own
- Big three cloud giants tighten grip as overall spending slows
Google's offered a workaround for customers – move their backends to another region. "Customers moving workloads to us-west, europe and asia may see improvements," the ads and search giant advises.
But it has no workarounds at this time "for customers who are required to host in us-east regions." Nor can searchzilla offer a time at which the incident will ebb.
"We do not have an ETA for mitigation at this point" customers are being told.
All of which adds up to an interesting choice for some Google Cloud customers as they arrive at work on Monday: wait out this incident and hope it doesn't degrade apps or annoy customers, or make a hasty move to another region. Because fast moves made under pressure always go well.
Hit the comments to let us know which option you chose. ®