OVHcloud datacenter fire last year possibly due to water leak

French investigative report draws no conclusion but hints at inverter malfunction


Late last month, France's BEA-RI, or Bureau of Investigation and Analysis on industrial risks, issued its technical report on the March 10th, 2021 fire at the OVH datacenter in Strasbourg.

The French report [PDF] and summary [PDF] echo the findings of the Bas-Rhin fire service in March, 2022 that the lack of an automatic fire extinguisher system, the delay of electrical cutoff and the building design contributed to the spread of the blaze.

The BEA-RI findings also hint at a possible cause – a water leak on an inverter – while stating that the cause has not been conclusively determined.

The fire started around 0035 on March, 10, 2021, according to the report, almost simultaneously in rooms F and G of SBG2, a five-story datacenter.

It then spread to adjacent buildings SBG1, where it partially destroyed four of 12 rooms, and SBG3, where it affected the inter-building area between SBG2 and SBG3. The fire was subdued around 1000 after the electrical network was cut off and a pump boat arrived.

"It is difficult to say what the operating mode of the energy rooms was when the failure occurred," the report [PDF] says. "The BEA-RI asked OVH to obtain the operating parameters of the inverters (voltages and intensities delivered, mode of operation, etc.) at the time of the equipment failure."

But OVH evidently did not have that information.

Image from BEA-RI report

Photos from the BEA-RI report ... Click to enlarge

The report continues, "The video surveillance system and the monitoring of the site's fire control panel show that an electrical fault has occurred at the level of the UPS2 and at the level of the batteries connected to it. The UPS and associated batteries were not in the same room."

Ambient temperature and humidity increases are said to have occurred around 0030 but BEA-RI could not establish whether this was a measurement error or due to the increased presence of liquid.

"The presence of a liquid or moisture in an electrical device can cause the formation of an internal short circuit likely to cause the damage observed," the Google translated report says, noting that these elements alone are insufficient to establish the cause of the failure.

"It is not possible, at this stage, to establish the cause of the failure at the level of the UPS which could be explained by various hypotheses (presence of liquid or humidity linked to the presence of the cooling system located nearby, malfunction linked to the maintenance operation carried out the same morning, operation of the inverter outside the normal operating ranges, etc.)," the report concludes.

Identifying the cause of the fire may have financial implications for OVH due to the ongoing litigation against the company by about 140 customers who lost data in the inferno. ®


Other stories you might like

  • Cisco warns of security holes in its security appliances
    Bugs potentially useful for rogue insiders, admin account hijackers

    Cisco has alerted customers to another four vulnerabilities in its products, including a high-severity flaw in its email and web security appliances. 

    The networking giant has issued a patch for that bug, tracked as CVE-2022-20664. The flaw is present in the web management interface of Cisco's Secure Email and Web Manager and Email Security Appliance in both the virtual and hardware appliances. Some earlier versions of both products, we note, have reached end of life, and so the manufacturer won't release fixes; it instead told customers to migrate to a newer version and dump the old.

    This bug received a 7.7 out of 10 CVSS severity score, and Cisco noted that its security team is not aware of any in-the-wild exploitation, so far. That said, given the speed of reverse engineering, that day is likely to come. 

    Continue reading
  • Google battles bots, puts Workspace admins on alert
    No security alert fatigue here

    Google has added API security tools and Workspace (formerly G-Suite) admin alerts about potentially risky configuration changes such as super admin passwords resets.

    The API capabilities – aptly named "Advanced API Security" – are built on top of Apigee, the API management platform that the web giant bought for $625 million six years ago.

    As API data makes up an increasing amount of internet traffic – Cloudflare says more than 50 percent of all of the traffic it processes is API based, and it's growing twice as fast as traditional web traffic – API security becomes more important to enterprises. Malicious actors can use API calls to bypass network security measures and connect directly to backend systems or launch DDoS attacks.

    Continue reading
  • Will cloud giants really drive colos off a financial cliff?
    The dude who predicted the Enron collapse bets they will

    Analysis Jim Chanos, the infamous short-seller who predicted Enron's downfall, has said he plans to short datacenter real-estate investment trusts (REIT).

    "This is our big short right now," Chanos told the Financial Times. "The story is that, although the cloud is growing, the cloud is their enemy, not their business. Value is accrued to the cloud companies, not the bricks-and-mortar legacy datacenters."

    However, Chanos's premise that these datacenter REITs are overvalued and at risk of being eaten alive by their biggest customers appears to overlook several important factors. For one, we're coming out of a pandemic-fueled supply chain crisis in which customers were willing to pay just about anything to get the gear they needed, even if it meant waiting six months to a year to get it.

    Continue reading

Biting the hand that feeds IT © 1998–2022