'No BS' web host Gandi lives up to half of its motto... Some customer data wiped out in storage server meltdown

0xdeadbeef

Updated Customers of web hosting outfit Gandi.net have been left less than impressed by its handling of a data-destroying storage crash.

The France-based hosting provider on Thursday disclosed it had lost some customer data after a ZFS storage box in Luxembourg broke down and had to be replaced using a backup. Efforts to restore the data, however, failed, and there were no snapshots available to recover from.

"The storage unit became unavailable, prompting an interruption in service for all PaaS and IaaS services using the disk associated with that unit," the Gandi team said. "The data import on the emergency machine was not possible due to a corruption of the meta-data that we are not aware of the cause of."

The biz went on to say it was conducting a full postmortem of the incident that would yield further details on what exactly went wrong. Its techies are still trying to recover the lost data, and thus far have had no luck.

While the loss of information without a viable backup is bad enough, customers are up in arms over the response from the business, which bills itself as a "no bullshit" outfit. Speaking to an aggrieved punter on Twitter, Gandi's web marketing and comms person Julie Pelloille offered the following answer:

Hi Andrea. It is confirmed we have lost data and we are terribly sorry for that. However, please note that what happened could happen to any web host.
We will provide you with cloud credits enough in order to be able to start a new infra in another datacenter. -- Julie

— gandi.net (@gandibar) January 9, 2020

The conversation descended into farce as Pelloille tried to defend the firm's position that its no BS strategy "does not exclude flaws." The response from the customer was as expected, saying: "It's not a flaw, it (sic) YOUR design."

Pelloille tried to lighten the mood by using a memorable moment from fantasy epic Game of Thrones.

Dear Julie this is not a joke, it's an inappropriate response. Shame for THAT. There are jobs, businesses, free working nights on the line.

— Andrea Ganduglia (@andreaganduglia) January 9, 2020

This failed, natch. The Gandi staffer later apologized, saying they "truly regret[ted] posting it."

A quick browse through the company's documentation shows users able to create back-up copies of volumes on a Gandi Cloud server via Snapshots on a scheduled basis. Some customers had taken this as meaning that Gandi was taking care of those snapshots while they dealt with the likes of databases themselves.

Not so, according to boss Stephan Ramoin, who tweeted: "Andrea, sorry about that and the incident. If we led you to believe that you had nothing to do on your side when warned multiple times to make your back ups, then we'll have to make it clearer, and stop assuming that it's an industry wide knowledge."

Needless to say, punters were upset with the response and the host's "it could happen to anyone" defense.

"Sure, we can all make mistakes," wrote one user, "but don't blame YOUR OWN major fuckup and not sticking to your promises to any other company!"

We couldn't put it any better ourselves. ®

Updated on 10 January to add

Following publication of the article, Gandi made contact to say that around 300 customers were affected. A spokesperson told us:

"We now have some hope that we may recover the data but as we can't confirm it at the moment, customers who needed or need an immediate recovery should use their own backups, as was our initial recommendation."

And how to prevent this from happening again?

"We will certainly give a lot of thought to this question when we complete a full post mortem of the incident, but at this moment our teams are all still focused on restoring customer data. In particular, we'll be looking at what improvements can be made to our recovery time, our documentation, and our communications."