Google Cloud's watery Parisian outage enters third week, with no end in sight
To make matters worse, other bits of the same region have wobbled
Two weeks after the unwelcome "water intrusion" inside a Parisian Google Cloud datacenter, the stricken facility remains offline – with no indication when it might resume operations.
Google Cloud's europe-west9 region took a shower on April 25. As datacenters and water don't mix, outages resulted.
It later emerged that the deluge did most damage at the europe-west9-a zone – one of three in the region.
Datacenter operator Global Switch admitted to an incident at a time that coincided with Google Cloud's woes.
"A fire incident has occurred in a room at one of the two datacenters in our Paris campus this morning," states an April 26 announcement. Two days later, an update revealed "the fire in one of the two datacenters in our Paris Campus was rapidly extinguished and was contained to one room only. The datacenter has remained live and operational with a very small proportion of its capacity affected."
- Irony alert: Major airport to be interrupted for two hours to replace UPS
- A right Royal pain in the Dallas: City IT systems crippled by ransomware
- European air traffic control confirms website 'under attack' by pro-Russia hackers
- Central UK govt awards £12M+ contract to leave Google Workspace for Microsoft 365
But Google Cloud's Europe-west9-a has been down ever since. La chocolaterie has not updated the advice it offers customers, which still reads as follows:
Our teams on the ground are assessing the physical infrastructure in the cluster. The current impacted product list has been updated. The impact for Cloud Bigtable continues in europe-west9-a. For the remaining products, impact is limited to instances located in the affected datacenter. Previously unaffected instances for these products will continue to work with no impact. There is no ETA for full recovery of affected instances in europe-west9-a at this time. We expect to see extended outages for these resources.
Why the delays? A French forum thread about the incident suggests it was more than just a fire. A post claims that the incident started when a cooling pump malfunctioned, causing a leak. That leak reached the battery room, where it sparked a fire.
Even les sapeurs-pompiers [That's French for firefighters – Ed] tread carefully around electrical fires, and this one generated a lot of heat. So much so that it was hours before attempts at extinguishment were possible and/or safe. That meant putting out the flames took hours.
The clean-up from that sort of incident would be substantial.
To make matters worse, last weekend Google Cloud's europe-west9-c zone had a significant wobble for nearly five hours. According to Google's status report, the incident meant "Google Cloud products in europe-west9-c may not be available to customers."
Paris in the Spring is supposed to be a time of delights, not the gloom of cloudy outages that bring metaphorical gray skies to IT operations. But the latter is what Google's dishing up at present.
C'est la vie. ®