Off-Prem

SaaS

Google Cloud partially evaporates for hours amid power supply failure: Two US East Coast zones rattled

Networking, Kubernetes, storage, virtual machine systems hit by outage


Google Cloud is having a wobbly Monday. Its Kubernetes platform and networking services were partially unavailable for hours today, and its virtual-machine hosting and in-memory storage systems had a limited outage.

The web giant's Cloud Networking service fell over around 0800 PT (1500 UTC) today due to a power supply failure. Connections to virtual machines in Google's us-east1-c and us-east1-d zones started failing, and the breakdown spread to other Google Cloud services, such as its Persistent Disk product.

Here's the official word from Google at time of writing, 1300 PT (2000 UTC), some five hours on from when trouble started:

We are experiencing an issue with Cloud Networking in us-east1-c and us-east1-d, beginning on Monday, 2020-06-29 07:54 US/Pacific, affecting multiple Google Cloud Services.

Services in us-east1-d have been fully restored. Services in us-east1-c are fully restored except for Persistent Disk which is partially restored. No ETA for full recovery of Persistent Disk yet. Impact is due to power failure. A more detailed analysis will be available at a later time.

Our engineering team is working on recovery of impacted services. We will provide an update by Monday, 2020-06-29 13:00 US/Pacific with current details. We apologize to all who are affected by the disruption.

Diagnosis: Some services in us-east1-c and us-east1-d are failing, customers impacted by this incident would likely experience a total unavailability of zonal services hosted in us-east1-c or us-east1-d. It is possible for customers to experience service interruption in none, one, or both zones.

Workaround: Other zones in the region are not impacted. If possible, migrating workloads would mitigate impact. If workloads are unable to be migrated, there is no workaround at this time.

And just as we were about to publish this article, Google said it expects to full resolve the Networking and storage issues within this hour:

The issue with Cloud Networking and Persistent Disk has been resolved for the majority of affected projects as of Monday, 2020-06-29 10:20 US/Pacific, and we expect full mitigation to occur for remaining projects within the hour.

Next up, Google's Kubernetes Engine in the us-east1-c zone took a hit at around 0700 PT (1400 UTC), and at time of writing nearly six hours on, is still down:

We are experiencing an issue with Google Kubernetes Engine in us-east1-c where clusters may be unavailable or unreachable beginning at Monday, 2020-06-29 07:15 US/Pacific US/Pacific. Mitigation work is currently underway by our engineering team.

Related problems with Compute Engine virtual machines, lasting about an hour, have, we're told, been cleared up:

The issue with Google Compute Engine in zones us-east1-c and us-east1-d where existing VMs may be unavailable or unreachable, and new VM creation may fail beginning on Monday, 2020-06-29 08:20 US/Pacific has been resolved for all affected users as of Monday, 2020-06-29 09:45 US/Pacific.

The impact to us-east1-d has been mitigated by Monday, 2020-06-29 08:45 US/Pacific. The impact to us-east1-c has been mitigated by Monday, 2020-06-29 09:45 US/Pacific.

Finally, Cloud Memorystore in us-east1-c and us-east1-d fell over around 0800 PT and was restored at 1130 PT. ®

Updated to add

As this article was published, Google updated its status boards to say its Kubernetes Engine outage has been resolved.

Send us news
6 Comments

Team behind delayed ERP project was aware of problems but didn't inform Surrey Council County for months

Move from SAP to Unit4 held up by comms 'issues' between supplier and authority

Problems with Surrey County Council's £30m projects to replace an ageing SAP R/3 system with a Unit4 SaaS application were known in June, but not discussed with key council committees until after September.

In April and June last year, new requirements from the HR department continued to arrive after the main software build was complete. The application supplier assured the council these changes could be accommodated within the original project timeline, but by September it became clear it wasn't going to make that December 2021 launch date, a council meeting heard late last week.

Earlier this month it was revealed the council had incurred £3.2m additional costs on the project as the go-live date was reset for April 2022.

Continue reading

IPv6 is built to be better, but that's not the route to success

Why won't you love me, sobs perennially spurned protocol

Opinion In the World of Tomorrow that's always 10 years away, Linux dominates the desktop, quantum computers control the fusion reactors, and all Android phones receive regular system updates. And the internet runs on IPv6.

This sort of talk irks IPv6 stans, mostly because it's true. They are serious-minded, far-seeing, sober engineering types who are both baffled and angry that IPv4 still rules the world in 2022. This is not how it was supposed to be.

IPv4 was designed by expert prophetic dreamers more than 40 years ago to be future-proof, but the future it actually created outstripped their dreams. IPv6 was the engineers' answer, born from a decade and a half of experience, and solving IPv4's undeniable routing, addressing, security and performance problems at the unprecedented scale it was being asked to support.

Continue reading

Pop quiz: The network team didn't make your change. The server is in a locked room. What do you do?

Not all heroes wear capes. Some are Unix admins.

Who, Me? Welcome to another entry in The Register's Who, Me? archives. Today, a reader goes full Hollywood to save the day (and fix some IP addressing).

Our story comes from Dave and takes us back to the Australia of the 1990s. It was the era of Paul Keating and John Howard and, significantly, a time of advancement in telecommunications technology.

Riding that wave was our reader, "Dave" (no, not his real name) who was working in software and infrastructure for a government agency. His team had developed an imaging system ("back when that was hard," he said modestly) that could display trademark registrations on the new-fangled Windows desktops that were popping up all over the place.

Continue reading

Apple again extends waiver for in-app purchasing requirement for online experiences

We'll know the pandemic is over when Apple starts insisting on its cut of fees for events again

If you're looking for a sign that the COVID-19 pandemic has eased and life is approaching normal, Apple has a bad omen: the fruity company has again extended viral relief to developers.

Apple has offered a smidgen of help since early 2020 by waiving App Store Review Guideline 3.1.1, which requires apps offering paid online group services to do so via in-app purchases.

By dropping that requirement, Apple reckoned it helped some businesses.

Continue reading

Myanmar's military junta seeks ban on VPNs and digital currency

People would no longer be able to rely on VPNs for their preferred communication tool, Facebook

Myanmar's military junta has floated a cyber security law that would ban the use of virtual private networks, under penalty of imprisonment and/or fines, leaving digital rights organisations concerned about the effects of further closing the country off digitally to the outside world.

The draft bill, dated January 13 is signed by Soe Thein, permanent secretary of the military's transport and communications ministry and is undergoing request for comments until January 28. Upon adoption, it will subject VPN users to between one and three years inside, and fines of up to five million Myanmar Kyats ($2,800).

The bill also bans the use of digital currency, under penalty of imprisonment for six months to a year, and the same fine used to deter VPN use.

Continue reading

Cloudflare signals big Asian push, starting in Singapore

Seeks salespeople focused on expansion – and casinos – and adds a trio of senior managers

Cloudflare has signalled significant expansion into the Asia-Pacific, Japan, and China, using Singapore as a beachhead.

The Register has spotted a raft of job ads from the net-grooming company, among them a "Head of Expansion, APJC" whose job will be to manage a clutch of expansion-oriented salespeople in pursuit of new business.

Another role, titled "Regional Major Account Executive – Gaming and Online Casino" gets the job of working with "iGaming and Online Casino related companies across APAC" to bring more bucks through Cloudflare's door.

Continue reading

Australian Prime Minister's WeChat Shanghaied by Chinese patriots

Politicians rush to blame Beijing, Tencent says the account was transferred from its original registrar and it's probing that shift

Update Australian Prime Minister Scott Morrison's WeChat account has been taken over by entities that have rebranded it "Australian Chinese new life" and used the account to offer advice on living in Australia for the nation's Chinese community.

Morrison, leader of the right-of-centre Liberal Party of Australia, has used Tencent-owned WeChat as a campaigning tool to reach Australia's sizable Chinese community – many of whom are concentrated in particular seats and are therefore considered a sought-after voting bloc.

Other members of the government have concluded the takeover of the account must be the work of Chinese authorities. Evidence for that claim, other than Beijing's ability to boss Chinese companies, has not been provided.

Continue reading

IBM finally finds a private equiteer willing to purchase Watson Health

As poker players ponder whether tech is ruining their favourite game

In-brief IBM has offloaded healthcare data and analytics assets from its Watson Health business, with private equity firm Francisco Partners hand over around $1bn for the privilege.

The takeover “is a clear next step as IBM becomes even more focused on our platform-based hybrid cloud and AI strategy,” Tom Rosamilia, senior vice president, IBM Software, told newswire Bloomberg. “IBM remains committed to Watson, our broader AI business, and to the clients and partners we support in healthcare IT.”

Launched in 2015, IBM Watson Health hasn’t been able to turn a profit despite the company spending $4bn in acquisitions to grow the business and its capabilities.

Continue reading

Desktop-deprived Linus Torvalds releases first release candidate of ‘not huge’ kernel 5.17

Life on the road increases reliance on cloudy tools instead of Emperor Penguin's preferred local tests

The first release candidate for version 5.17 of the Linux kernel has rolled off the production line – despite fears that working from a laptop might complicate matters.

Emperor Penguin Linus Torvalds is currently on the road and, when announcing the release of Linux 5.16 predicted that the version 5.17 release merge window would be “somewhat painful” due to his travels, and use of a laptop – something Torvalds said “I generally try to avoid.”

Torvalds’ laptop aversion comes from the fact that he likes to do lots of local testing on his beastly workstation powered by a 32-core AMD Ryzen Threadripper. Linus’ lappie appears not to match his desktop, so he ends up using more automated build testing in the cloud.

Continue reading

Another US president, time for another big Intel factory promise by another CEO

Let's not get too excited about this right away

Comment Intel puts on a show for its biggest manufacturing announcements, with episodes every few years using a rotating cast of CEOs and US presidents.

Intel boss Pat Gelsinger and President Joe Biden were the latest to join the series, on Friday jointly announcing the chip maker's investment of $20bn in plants near Columbus, Ohio. The fabs could be operational by 2025 and make chips down to 2nm and beyond.

"This is our first major site announcement in 40 years," Gelsinger said on on-stage later in the day with Ohio Governor Mike DeWine (R).

Continue reading

European silicon output shrinking, metal smelters closing as electricity prices quadruple, trade body warns

Probably something to tackle before those chip fabs are built, eh?

Soaring electricity prices have derailed manufacturing involving silicon and non-ferrous metals in Europe, politicians were warned this week.

Eurometaux, a European metals association, urged action [PDF] from the EU, fearing the region could experience spikes in electricity prices for the next decade if nothing is done to control the situation.

The power crisis has already curtailed production and shut down facilities in silicon and metals industries across EU nations. "After a quadrupling of electricity prices, over half of the EU’s aluminium and zinc smelters are today operating at reduced capacity or have temporarily closed, together with a significant reduction in silicon output," Eurometaux said.

Continue reading