Google, Oracle clouds still affected by UK heatwave

Some hard disks appear to have cooked, along with bit barn designers' best laid plans

Cloud outages at Google and Oracle caused by the UK's heatwave have ended, but users have been warned some problems persist.

Google's incident report states that the "cooling related failure" was resolved at 2043 Pacific Time on Tuesday (0343 UTC).

But the ad giant's update also reports: "A small number of HDD backed Persistent Disk volumes are still experiencing impact and will exhibit IO errors."

That sounds like some drives cooked during the "cooling related failure." Hard disks are seldom rated to run at more than 50C, and as the mercury topped 40C in London yesterday it’s not hard to imagine that temperatures became so hot that mechanical disks in a densely packed machine faced an extreme environment and suffered more than solid-state components.

Google understands the frailties of spinning rust so will presumably have replicated data on multiple devices and understand how to recover the devices. If data is lost, it will dent the G-Cloud's reputation.

Oracle's most recent update is time-stamped 0357 UTC and opens with news that "data center cooling infrastructure has been restored and temperatures have returned to normal operating levels" and adds: "All services and customer resources have been restored."

But the update also states "a subset of Oracle Integration Cloud resources continue to experience impact. Engineers are actively working to mitigate those remaining service resources."

So maybe not all services have been restored.

Weather forecasts for the UK predict that parts of the nation hosting Google and Oracle datacenters will on Wednesday top out at a pleasant 27C, and may experience a thunderstorm late in the day.

The Register hopes the big clouds have remembered to close their windows after yesterday's meltdowns.

Datacenters in other parts of the world routinely cope with temperatures experienced in the UK on Tuesday. But yesterday was London's first recorded day above 40 degrees – like, ever – so datacenter designers have a viable excuse for their creations not coping. It's not a brilliant excuse, though, given cloud operators and the companies that host them pride themselves on over-engineering to ensure resilience. ®