Google botches another cloud patch, this time messing up App Engine

Not a lot of problems, but third fluff in a few weeks is not a good look

1 Reg comments Got Tips?

Google's botched another cloud patch.

The Chocolate Factory's cloud operations have had a tough time of things this year, suffering a network mess in February and then a brownout in early March.

Over the last couple of days the company's App Engine experienced “elevated 503 errors on <1% of applications for a typical duration of 50 minutes.” Not a colossal hassle, to be sure, but as Google itself says in a incident report “We know how important high uptime and low error rates are to you and your users”.

Here's Google's explanation of the event's root cause:

“A latent bug in the App Engine quota handling code was triggered during a routine software update of the quota system. This resulted in App Engine returning over-quota errors to some applications that were not over quota. As App Engine software updates are rolled out progressively, only some applications were affected by the update before the issue was detected and remediated.”

Google says it will “... add monitoring and alerting for the quota issue that resulted in spurious 503 errors, create a new quick response protocol for handling erroneous quota responses, and will modify application quota behavior to tolerate novel quota system behavior with lower application impact.”

The new issue suggests that not all parts of Google's cloud business have adopted the new pre-patch preparation plan it pledged to adopt in the wake of the two earlier incidents mentioned above.

All of which may or may not make you more confident about using the new Google Launcher, announced on Thursday, which offers the chance to “ launch more than 120 popular open source packages that have been configured by Bitnami or Google Click to Deploy.”

The new offering doesn't look to be something that will get the likes of Pivotal's Cloud Foundry quaking in its boots (not least because it introduced simpler spawning to AWS this week), but does look a handy way to spin up instances of common development environments and applications. The Launcher's launchpad is here, if you feel like starting a countdown. ®


Biting the hand that feeds IT © 1998–2020