OVH reveals it's scrubbing servers – to get smoke residue off before rebooting

Quite a few have come back online, but it takes seven hours to restore each rack

French cloud operator OVH has revealed how it is cleaning every server it thinks can be returned to service in its scorched Strasbourg data centres.

Founder and chair Octave Klaba used his Twitter account so show off some of the work being done by the company's clean-up crew.

Update March,24 6:30pm

The cleaning takes time. We have 80 people (SBG3) + 20 people (Croix). On the left, a motherboard with the smoke pollution on the CPU socket. It’s very corrosive ! If we power up, it’s dead. Same the disk. On the right, the same device 24h after cleaned up pic.twitter.com/omaOkBSUpk

— Octave Klaba (@olesovhcom) March 24, 2021

Klaba has also said that all servers should be cleaned by Tuesday, but that racking and stacking the infrastructure for some services is taking longer than anticipated.

Update 28, 10am

3/4

Full recovery of pCC/HPC is slow since we need to assemble all the services (which are in 100 racks/3 floors) for each customer. It looks like we assemble hundreds puzzles of 6-12 pieces with some pieces unavailable since some racks are cleaning up.

— Octave Klaba (@olesovhcom) March 28, 2021

OVH's Sunday afternoon update offered more detail: "Today, the cleaning time for a rack is 7 hours, and our teams are improving this every day."

The update also had some better news for customers, as follows:

• SBG1: The recoverable Bare Metal Cloud servers are being cleaned for reinstallation in Strasbourg (SBG3 and SBG4). Re-commissioning will be gradually started by the beginning of next week (after inspection and cleaning).
• SBG3 is operational: 84 per cent of Bare Metal Cloud (VPS) services have been made available to customers again, with a target of 90 per cent by the evening of 28 March.
• SBG4 is operational: 100 per cent of Bare Metal servers are available to customers.

Servers in the SBG1 data centre will come back online at different times. Some will stay in Strasbourg and lodge in SBG4. Others are destined for other OVH data centres. The update mentions a "mid-week of 29 March" restart for some and a 1 or 2 April reboot for those moved to other locations.

Disaster recovery is also ongoing.

Some of OVH's cloud services are also not restored to 100 per cent availability. The company has also warned that high levels of demand mean "delivery times on our Bare Metal Cloud services may take longer than usual".

"Our teams are fully mobilised, and we are working hard to deliver to our customers as quickly as possible, especially all affected customers," the Sunday update says.

Klaba, meanwhile, has revealed that the fire has cost OVH the chance to launch a new service. ®

Update 28, 10am

4/4

SBG3/Floor 5 is a new floor with only new servers ready for a launch of new product we planned in ... 10 days. We’ve been working on for 18 months and now all has to be cleaned up

But first we focus on all cust’s services to make them UP.
We are on it !

— Octave Klaba (@olesovhcom) March 28, 2021