Imagine your data center backup generator kicks in during power outage ... and catches fire. Well, it happened
WebNX facility falls offline in blaze, takes customers down with it
A power outage kicked off a fire in web hosting biz WebNX's Ogden data center in Utah on Sunday, knocking the facility offline temporarily and leaving several servers in need of a rebuild.
Kevin Brown, Fire Marshal for the US city's Fire Department told The Register in a phone interview that firefighters responded to a call on Sunday evening. The fire, he said, "originated in a generator in the building and spread to several servers."
Brown said the facility's fire suppression system contained the blaze and that fire department personnel assisted with the cleanup. He said power was cut to the building until an electrical engineer could inspect the facility to make sure current could be restored safely, which he added is standard procedure.
He also confirmed that some of Ogden City's IT services were down on Sunday and Monday as a result of the data center fire.
... one of our backup generators that had been recently tested and benchmarked specifically for this situation experienced a catastrophic failure
"Sunday afternoon the city power was disrupted and, as designed, our backup generators automatically switched on," the company said.
"However, during that transition, one of our backup generators that had been recently tested and benchmarked specifically for this situation experienced a catastrophic failure, caught fire, and as a result initiated the fire suppression protocol."
The company confirmed that its Ogden data center experienced some damage. And earlier statement noted: "Some servers will have an extended outage as they may require rebuilds due to some water damage. Those builds have a high probability that data is intact."
"Customer’s servers in one of our main bays were exposed to water and possible damage may have occurred," the company said. "No fire damage was inflicted on customer servers."
Most of the hardware in the data center appears to be unaffected, the company said, but there are machines that need to be inspected for water damage and may need to be rebuilt.
"As of now, we are working to restore power, network, and unaffected hardware back up online within the next day or two," the company said.
Customers aren't pleased
The incident has also affected Gorilla Servers, a separate company founded by WebNX CEO Daniel Pautz that appears to co-locate some servers within the Ogden data center.
According to Gorilla, "close to 90 to 95 per cent of all Gorilla Servers hardware experienced zero damage." The electronic engineering website of YouTuber Dave Jones, eevblog.com, was among the sites hosted by Gorilla and taken offline in the outage, we note.
Other folks continue to report problems or are unable to reach their systems in the data center, though WebNX claims service has been restored for some.
At 16:47 UTC today, fish-egg flogger Passmore Caviar said, "Our website remains offline due to a fire at the WebNX server facility. Currently, we expect it to be back online later this afternoon." At the time this article was filed, the website remained inaccessible.
Piqosity, a test preparation service, also said its site stopped responding as a result of the fire.
OVH reveals it's scrubbing servers – to get smoke residue off before rebootingREAD MORE
And in various online discussion forums, affected customers continue asking for estimates about when service will be restored. A common complaint is that lack of official communication about what's going on. Other forum participants report being told that their servers have water damage and will have to be rebuilt – a process that may take several days.
WebNX, which also operates facilities in New York City and Los Angeles, could not be reached by phone – calls are met with a recording noting that the company is "aware of the support system being down right now and technicians are working to resolve it." The Register received no response to our email inquiries.
WebNX's SLA guarantees 100 per cent uptime and uninterrupted power every month, with account credits of one day per 15 minutes of downtime in each case.
At 20:50 UTC on Tuesday, WebNX posted an update to its Facebook page: "We are currently working hard to get everything back to optimal running order. Huge thanks to the staff and outside contractors who were brought in (and also flown in from out of state) to help with the situation. We are also really grateful to everyone who has stepped up to help out, including some amazing clients, and Ogden City who has been very supportive." ®