This article is more than 1 year old
OVH outlines three-point 'hyper resilience' plan after Strasbourg fire
Please insert tape number 363 of 4087*
French cloud provider OVH has outlined a three-point plan designed to avoid a repeat of the loss of data and services resulting from the fire which engulfed its Strasbourg operations on 10 March.
Dubbed "Hyper Resilience", the plan employs the combinations of a revamped approach to internal backups, external customer back-ups and a new policy of fail-over between three data centres per region.
OVH founder, chair and CFO Octave Klaba and CEO Michel Paulin outlined the plans in a tweeted video address, viewers of which were implicitly being asked to avoid the conclusion that they were closing the stable door after the horse had not only bolted but bought airline tickets to Cancun where it was now sipping mojitos on a beach.
The fire took place on March 10 and destroyed the SBG2 hall of the Strasbourg data centre, damaged SBG1 badly, and led to a massive effort to clean salvageable kit so it could be installed in the remaining three data centres at Strasbourg, or moved to other OVH facilities. Thankfully, no one was hurt.
Klaba said he suspects two uninterruptible power supplies units were the cause of the fire, revealing the UPS, batteries and fuses are in the hands of the police and insurers.
By 14 April OVH was still struggling to bring all customers affected by the fire back online – a situation described by Klaba as a "real nightmare".
In the latest update, Paulin said the operator had restored 118,000 services out of the 120,000 services which had been affected by the incident.
Backup resilience
Klaba went on to describe the "five-year hyper resilience plan" designed to restore customer confidence, starting with internal back-ups. Internal back-ups had been within a data centres or, in some cases, in another data centre in the same region. The operator is now proposing creating a four-data-centre region where it will host internal backups physically separated from operational regions.
- OVH says burned data centre’s UPS, batteries, fuses in the hands of insurers and police
- OVH writes off another data centre – SBG1 – and reveals new smoking battery incident
- OVH flames scorched cloud customers with pledge to build data centre fire simulation lab
- Imagine your data center backup generator kicks in during power outage ... and catches fire. Well, it happened
Secondly, it will add features to backups in the new region. While backup data had previously supported its internal needs, OVH is proposing that customers will be able to replicate and remove the backup data for their own purposes in a free service.
Changing rules for building data centres
Lastly, OVH said it would change internal rules for building data centres and also software-managed resilience operating across three data centres in a region, starting in Paris, to be later introduced in the rest or Europe, the US and Asia.
Klaba made a smiliar vow about upgrading the campus after a massive outage in 2017.
At the time, the group embarked on a "€4m-€5m investment plan in the wake of a major outage that left three of the Strasbourg data centres – SBG1, SBG2 and SBG4 – without power for 3.5 hours in November 2017".
Klaba himself said at the time of the 2017 outage that it was partly because "SBG's power grid inherited all the design flaws that were the result of the small ambitions initially expected for that location - with "SBG2's power grid" built atop "SBG1's power grid instead of making them independent of each other".
That previous upgrade was said to involve "de-installation of maritime containers" (shipping containers) and major electrical work.
Back to the present time, post 2021-fire, Klaba has said that OVH will rebuild the data centres affected by the blaze: Strasbourg One, Two and Four.
Paulin apologised for the incident which caused serious disruption across European websites, with, according to Netcraft, "3.6 million websites across 464,000 distinct domains... taken offline." ®
* With our thanks to forum commenter Danny14. If you want to get stuck into our comments, you can sign up here.