Google Cloud partially fixes load balancer SNAFU that hit Discord, Spotify, others today

If your favorite site was acting up, this might be why


Updated Google Cloud suffered a brief outage, seemingly bringing down or disrupting a whole bunch of websites relying on its systems.

If you've had trouble accessing Snapchat, Discord, Spotify, Etsy, retailers like Home Depot, and others today, this is likely why: a fault developed in Google Cloud's networking infrastructure, resulting in websites throwing up 404 errors. Netizens found themselves unable to log into or use certain services properly.

The good news is that, by now, the IT breakdown has been resolved in that sites using Google's cloud-based load balancers should work again.

The bad news is that Google's customers can't update their load balancing configurations until the web giant gives the word, and when that will be isn't known.

The outage was acknowledged by Google at 1010 PST (1810 UTC), about 35 minutes minutes after websites apparently started going wrong, and a fix was deployed within a few minutes to stop the "page not found" errors. Since that update, though, changes by customers to their external proxy load balancers are being ignored.

According to the tech titan's status page:

Summary: Global: Experiencing Issue with Cloud networking

Description: We believe the issue with Cloud Networking is partially resolved.

Customers will be unable to apply changes to their load balancers until the issue is fully resolved.

We do not have an ETA for full resolution at this point.

We will provide an update by Tuesday, 2021-11-16 12:28 US/Pacific with current details.

Diagnosis: Customers impacted by the issue may have encountered 404 errors when accessing web pages served by the Google External Proxy Load Balancer between 09:35 and 10:10 US/Pacific.

Customer impact from 10:10 US/Pacific onward is configuration changes to External Proxy Load Balancers not taking effect.

Workaround: None at this time.

The load-balancing blunder had a knock-on effect of temporarily disrupting Google Cloud's App Engine, Cloud Functions, Apigee, and Cloud Run services.

We'll let you know when Google considers this situation fully restored. ®

Updated to add

"The issue with Cloud Networking has been resolved for all affected projects as of Tuesday, 2021-11-16 11:28 US/Pacific," Google said on its status page, noting the issue has been fully fixed.

"Customer impact from 10:10 to 11:28 US/Pacific was configuration changes to External Proxy Load Balancers not taking effect. As of 11:28 US/Pacific configuration pushes resumed."

Similar topics


Other stories you might like

  • AsmREPL: Wing your way through x86-64 assembly language

    Assemblers unite

    Ruby developer and internet japester Aaron Patterson has published a REPL for 64-bit x86 assembly language, enabling interactive coding in the lowest-level language of all.

    REPL stands for "read-evaluate-print loop", and REPLs were first seen in Lisp development environments such as Lisp Machines. They allow incremental development: programmers can write code on the fly, entering expressions or blocks of code, having them evaluated – executed – immediately, and the results printed out. This was viable because of the way Lisp blurred the lines between interpreted and compiled languages; these days, they're a standard feature of most scripting languages.

    Patterson has previously offered ground-breaking developer productivity enhancements such as an analogue terminal bell and performance-enhancing firmware for the Stack Overflow keyboard. This only has Ctrl, C, and V keys for extra-easy copy-pasting, but Patterson's firmware removes the tedious need to hold control.

    Continue reading
  • Microsoft adds Buy Now, Pay Later financing option to Edge – and everyone hates it

    There's always Use Another Browser

    As the festive season approaches, Microsoft has decided to add "Buy Now, Pay Later" financing options to its Edge browser in the US.

    The feature turned up in recent weeks, first in beta and canary before it was made available "by default" to all users of Microsoft Edge version 96.

    The Buy Now Pay Later (BNPL) option pops up at the browser level (rather than on checkout at an ecommerce site) and permits users to split any purchase between $35 and $1,000 made via Edge into four instalments spread over six weeks.

    Continue reading
  • Visiting a booby-trapped webpage could give attackers code execution privileges on HP network printers

    Patches available for 150 affected products

    Tricking users into visiting a malicious webpage could allow malicious people to compromise 150 models of HP multi-function printers, according to F-Secure researchers.

    The Finland-headquartered infosec firm said it had found "exploitable" flaws in the HP printers that allowed attackers to "seize control of vulnerable devices, steal information, and further infiltrate networks in pursuit of other objectives such as stealing or changing other data" – and, inevitably, "spreading ransomware."

    "In all likelihood, a lot of companies are using these vulnerable devices," said F-Secure researchers Alexander Bolshev and Timo Hirvonen.

    Continue reading

Biting the hand that feeds IT © 1998–2021