'Elevated' moisture reading ignored before Heathrow-closing conflagration, says NESO

Datacenters kept humming along, but there's plenty of blame to be passed around elsewhere

The cause of a power outage that cut the juice to London Heathrow airport in March has been identified - along with a chain of failures that allowed it to happen.

You may recall that in March a fire broke out at an electrical substation serving London Heathrow and surrounding communities, disrupting operations at one of the world's largest airports for nearly a day. According to the UK's National Energy Systems Operator's (NESO) report on the matter, thousands of homes and businesses were also affected, as were road and rail operations and power at a nearby hospital. 

London Fire Brigade and National Grid Electricity Transmission (NGET) analysis of the fire have singled out a culprit, NESO said: moisture getting into high-voltage bushing caused a short circuit, after which the electricity likely "arced," causing sparks, "which combined with air and heat to ignite the oil, resulting in a fire." To make matters worse, the moisture was a known problem first identified seven years ago. 

The insulating bushing at North Hyde Substation that caused the March fire was detected in 2018 through oil testing which found elevated moisture in a sample. That should have been the end of the matter after someone took steps to address it, but NESO said that appears to have never been the case. 

"Mitigating actions appropriate to [the elevated moisture level's] severity were not implemented," the grid operator said. NGET, which operates the substation where the fire took place, said it is now reviewing all prior oil sample test results to ensure necessary corrective action has been taken. The company is also reviewing its full oil sampling process. 

Datacenter operators get gold star, everyone else gets a telling-off

One bright spot in the fire response was how datacenters in the area handled the loss of power. There are several datacenters operated by major hyperscalers adjacent to the airport and North Hyde, and as we reported in our original story about the fire, they kept on trucking after others in the area fell to chaos. 

Per NESO, three datacenters served by the substation lost power "but were able to continue operations through the use of backup generators."

But while the bitbarns deserve credit for their resilience, Heathrow Airport operations were far less robust, containing a single point of failure, or so NESO observed. 

The NESO report said that a review of Heathrow's electrical system found that it was designed in such a way that "the loss of one of its three independent [electrical] supply points would result in the loss of power to some of the airport's operationally critical systems." 

Heathrow's plan to deal with that shortcoming involved a procedure to reconfigure the electrical distribution system at the airport - a 10 to 12 hour process that "was less well-known by those outside the technical team within Heathrow … and it was not known to the energy companies." That change over process is precisely why Heathrow ended up closed for nearly a day, NESO said. 

It's not enough to stop blaming people after NGET and Heathrow, though - NESO also has some responsibility to hand to energy network operators (e.g., NESO) as well. According to the report, energy network operators generally don't have visibility into whether customers on their networks are considered critical infrastructure, and there's no requirement for such visibility. 

NESO noted that work is underway to resolve the visibility issue, but in short there's plenty of blame to go around for why that fire was so disruptive ­- and kudos to the datacenters for having reliable backup gennies at the ready. ®