OVH writes off another data centre – SBG1 – and reveals new smoking battery incident

Customers concerned over 2017 posts describing power outage, confession to sub-par design

OVH has written off a second data centre because of the March 10th fire that destroyed its SBG2 facility.

In a Saturday update, OVH founder and CTO Octave Klaba wrote: “We don’t plan to restart SBG1. Ever.” Previous guidance on SBG1 suggested only a few racks were destroyed, while others were considered recoverable. The fire has already destroyed the SBG2 data centre.

Klaba's announcement followed news of a second fire on the site.

Update March,19 9:30pm
The batteries were in unused room, not used, not connected. We didn’t have the fire, but we had lot of smoke. The situation is under control now. Everybody is safe.

We stop the operation in SBG for this night. We will take the decision tomorrow.

— Octave Klaba (@olesovhcom) March 19, 2021

Klaba has also detailed further delays to restoration of the Strasbourg site as new conduits are dug to bring power to SBG3. The CTO offered further information in a pair of Sunday tweets detailing restoration efforts.

Update March,21 10:30pm

SBG3:
pCI: 60% of VPS: UP. 100% CEPH: UP. We continue to clean up pCS data + metadata.
pCC: clean up. tomorrow we hope to restart 40%.
BM: 25% UP

SBG4:
ETA: Wed,24

SBG1:
61AD/62AD in SBG4: Fri,26/Sun,28
61E/62E in RBX/GRA: no ETA yet because clean up. pic.twitter.com/w3yYCMZHhq

— Octave Klaba (@olesovhcom) March 21, 2021

OVH customers, meanwhile, have picked up on a post to the OVH forums by Klaba from November 2017 that discussed a power outage that the CTO described as “the worst incident that can happen to us” - one which The Register referred to on the day of the fire.

Damage at OVH Strasbourg.
Click to enlarge

The post describes how the Strasbourg site initially used shipping containers and Klaba wrote that decision meant “we have not prepared the site for large scale.” He then said the company also made two design errors:

1. We did not bring the SBG site up to internal standards which provide for 2 independent 20KV electrical inlets, like all our DC sites which have several double electrical inlets. This is a significant investment of around 2 to 3 million euros per electrical arrival, but we believe that this is part of our internal standard.
2. We built the SBG2 electrical network by placing it on the SBG1 electrical network, instead of making them independent of each other, as in all our data centers. At OVH, each datacenter number means that the electrical network is independent of another datacenter. Everywhere except on the SBG site.

At the time, Klaba promised that work to remedy the problems would start ASAP.

The 2017 post was made in the wake of a power outage that took down the data centres, a scenario Klaba described as “the worst incident that can happen to us.”

The Register asked OVH weeks ago how far along it was with the power grid upgrade it referred to in the post, which was said to involve "de-installation of maritime containers" (shipping containers) and major electrical work. It has yet to respond. ®