Imagine your data center backup generator kicks in during power outage ... and catches fire. Well, it happened

WebNX facility falls offline in blaze, takes customers down with it


A power outage kicked off a fire in web hosting biz WebNX's Ogden data center in Utah on Sunday, knocking the facility offline temporarily and leaving several servers in need of a rebuild.

Kevin Brown, Fire Marshal for the US city's Fire Department told The Register in a phone interview that firefighters responded to a call on Sunday evening. The fire, he said, "originated in a generator in the building and spread to several servers."

Brown said the facility's fire suppression system contained the blaze and that fire department personnel assisted with the cleanup. He said power was cut to the building until an electrical engineer could inspect the facility to make sure current could be restored safely, which he added is standard procedure.

He also confirmed that some of Ogden City's IT services were down on Sunday and Monday as a result of the data center fire.

In a Facebook post on Monday, echoed on its website, WebNX attributed the incident to the failure of a backup generator following a local power outage.

... one of our backup generators that had been recently tested and benchmarked specifically for this situation experienced a catastrophic failure

"Sunday afternoon the city power was disrupted and, as designed, our backup generators automatically switched on," the company said.

"However, during that transition, one of our backup generators that had been recently tested and benchmarked specifically for this situation experienced a catastrophic failure, caught fire, and as a result initiated the fire suppression protocol."

The company confirmed that its Ogden data center experienced some damage. And earlier statement noted: "Some servers will have an extended outage as they may require rebuilds due to some water damage. Those builds have a high probability that data is intact."

"Customer’s servers in one of our main bays were exposed to water and possible damage may have occurred," the company said. "No fire damage was inflicted on customer servers."

Most of the hardware in the data center appears to be unaffected, the company said, but there are machines that need to be inspected for water damage and may need to be rebuilt.

"As of now, we are working to restore power, network, and unaffected hardware back up online within the next day or two," the company said.

Customers aren't pleased

The incident has also affected Gorilla Servers, a separate company founded by WebNX CEO Daniel Pautz that appears to co-locate some servers within the Ogden data center.

According to Gorilla, "close to 90 to 95 per cent of all Gorilla Servers hardware experienced zero damage." The electronic engineering website of YouTuber Dave Jones, eevblog.com, was among the sites hosted by Gorilla and taken offline in the outage, we note.

Other folks continue to report problems or are unable to reach their systems in the data center, though WebNX claims service has been restored for some.

At 16:47 UTC today, fish-egg flogger Passmore Caviar said, "Our website remains offline due to a fire at the WebNX server facility. Currently, we expect it to be back online later this afternoon." At the time this article was filed, the website remained inaccessible.

Piqosity, a test preparation service, also said its site stopped responding as a result of the fire.

OVH founder Octave Klaba's before and after pictures of servers cleaned after the fire in OVH's Strasbourg data centre

OVH reveals it's scrubbing servers – to get smoke residue off before rebooting

READ MORE

And in various online discussion forums, affected customers continue asking for estimates about when service will be restored. A common complaint is that lack of official communication about what's going on. Other forum participants report being told that their servers have water damage and will have to be rebuilt – a process that may take several days.

WebNX, which also operates facilities in New York City and Los Angeles, could not be reached by phone – calls are met with a recording noting that the company is "aware of the support system being down right now and technicians are working to resolve it." The Register received no response to our email inquiries.

WebNX's SLA guarantees 100 per cent uptime and uninterrupted power every month, with account credits of one day per 15 minutes of downtime in each case.

At 20:50 UTC on Tuesday, WebNX posted an update to its Facebook page: "We are currently working hard to get everything back to optimal running order. Huge thanks to the staff and outside contractors who were brought in (and also flown in from out of state) to help with the situation. We are also really grateful to everyone who has stepped up to help out, including some amazing clients, and Ogden City who has been very supportive." ®

Similar topics


Other stories you might like

  • New York City rips out last city-owned public payphones
    Y'know, those large cellphones fixed in place that you share with everyone and have to put coins in. Y'know, those metal disks representing...

    New York City this week ripped out its last municipally-owned payphones from Times Square to make room for Wi-Fi kiosks from city infrastructure project LinkNYC.

    "NYC's last free-standing payphones were removed today; they'll be replaced with a Link, boosting accessibility and connectivity across the city," LinkNYC said via Twitter.

    Manhattan Borough President Mark Levine said, "Truly the end of an era but also, hopefully, the start of a new one with more equity in technology access!"

    Continue reading
  • Cheers ransomware hits VMware ESXi systems
    Now we can say extortionware has jumped the shark

    Another ransomware strain is targeting VMware ESXi servers, which have been the focus of extortionists and other miscreants in recent months.

    ESXi, a bare-metal hypervisor used by a broad range of organizations throughout the world, has become the target of such ransomware families as LockBit, Hive, and RansomEXX. The ubiquitous use of the technology, and the size of some companies that use it has made it an efficient way for crooks to infect large numbers of virtualized systems and connected devices and equipment, according to researchers with Trend Micro.

    "ESXi is widely used in enterprise settings for server virtualization," Trend Micro noted in a write-up this week. "It is therefore a popular target for ransomware attacks … Compromising ESXi servers has been a scheme used by some notorious cybercriminal groups because it is a means to swiftly spread the ransomware to many devices."

    Continue reading
  • Twitter founder Dorsey beats hasty retweet from the board
    We'll see you around the Block

    Twitter has officially entered the post-Dorsey age: its founder and two-time CEO's board term expired Wednesday, marking the first time the social media company hasn't had him around in some capacity.

    Jack Dorsey announced his resignation as Twitter chief exec in November 2021, and passed the baton to Parag Agrawal while remaining on the board. Now that board term has ended, and Dorsey has stepped down as expected. Agrawal has taken Dorsey's board seat; Salesforce co-CEO Bret Taylor has assumed the role of Twitter's board chair. 

    In his resignation announcement, Dorsey – who co-founded and is CEO of Block (formerly Square) – said having founders leading the companies they created can be severely limiting for an organization and can serve as a single point of failure. "I believe it's critical a company can stand on its own, free of its founder's influence or direction," Dorsey said. He didn't respond to a request for further comment today. 

    Continue reading
  • Snowflake stock drops as some top customers cut usage
    You might say its valuation is melting away

    IPO darling Snowflake's share price took a beating in an already bearish market for tech stocks after filing weaker than expected financial guidance amid a slowdown in orders from some of its largest customers.

    For its first quarter of fiscal 2023, ended April 30, Snowflake's revenue grew 85 percent year-on-year to $422.4 million. The company made an operating loss of $188.8 million, albeit down from $205.6 million a year ago.

    Although surpassing revenue expectations, the cloud-based data warehousing business saw its valuation tumble 16 percent in extended trading on Wednesday. Its stock price dived from $133 apiece to $117 in after-hours trading, and today is cruising back at $127. That stumble arrived amid a general tech stock sell-off some observers said was overdue.

    Continue reading
  • Amazon investors nuke proposed ethics overhaul and say yes to $212m CEO pay
    Workplace safety, labor organizing, sustainability and, um, wage 'fairness' all struck down in vote

    Amazon CEO Andy Jassy's first shareholder meeting was a rousing success for Amazon leadership and Jassy's bank account. But for activist investors intent on making Amazon more open and transparent, it was nothing short of a disaster.

    While actual voting results haven't been released yet, Amazon general counsel David Zapolsky told Reuters that stock owners voted down fifteen shareholder resolutions addressing topics including workplace safety, labor organizing, sustainability, and pay fairness. Amazon's board recommended voting no on all of the proposals.

    Jassy and the board scored additional victories in the form of shareholder approval for board appointments, executive compensation and a 20-for-1 stock split. Jassy's executive compensation package, which is tied to Amazon stock price and mostly delivered as stock awards over a multi-year period, was $212 million in 2021. 

    Continue reading
  • Confirmed: Broadcom, VMware agree to $61b merger
    Unless anyone out there can make a better offer. Oh, Elon?

    Broadcom has confirmed it intends to acquire VMware in a deal that looks set to be worth $61 billion, if it goes ahead: the agreement provides for a “go-shop” provision under which the virtualization giant may solicit alternative offers.

    Rumors of the proposed merger emerged earlier this week, amid much speculation, but neither of the companies was prepared to comment on the deal before today, when it was disclosed that the boards of directors of both organizations have unanimously approved the agreement.

    Michael Dell and Silver Lake investors, which own just over half of the outstanding shares in VMware between both, have apparently signed support agreements to vote in favor of the transaction, so long as the VMware board continues to recommend the proposed transaction with chip designer Broadcom.

    Continue reading

Biting the hand that feeds IT © 1998–2022