Cloud hosting biz DigitalOcean, he declared, "just killed our company." But with the help of the Squeaky Wheel support tier – by which we mean making a lot of noise on social media – it's apparently up and running again.
Two days ago, as Beauvais tells it, the startup's cloud provider, DigitalOcean, decided that a Python script the company uses periodically to makes its data easier to process was malicious. So DigitalOcean locked the company's account, which represents the entire IT infrastructure of the biz – five droplets for its web app, workers, cache and databases.
Beauvais, in a series of Twitter posts, describes sending multiple emails and Twitter direct messages to DigitalOcean and regaining access after 12 hours of downtime.
"We had to restart our data pipeline from the start as all the droplets were shut down and the Redis memory, that kept track of our advancement got wipe," he wrote. "Only four hours later our account got locked again, probably by the same automated script."
He says he then sent four messages over the next 30 or hours to support, only to receive an automated reply that DigitalOcean had declined to re-activate the account.
"We lost everything, our servers, and more importantly one year of database backups," Beauvais lamented. "We now have to explain to our clients, Fortune 500 companies why we can’t restore their account."
Social media, unable to resist stories of cute animals or human suffering, chimed in on cue, with software engineers expressing sympathy and pointing the way toward other cloud providers where everything goes right all the time. Discussion blossomed on Hacker News as developers and the like weighed in with harrowing tales of their own, recommendations and musings about the backup strategies, automation and support tiers.
Now that Kubernetes has won, DigitalOcean takes a late dip in K8sREAD MORE
Perhaps aware of the potential brand damage to his cloud biz, DigitalOcean co-founder Moisey Uretsky interceded on Twitter an hour after the initial post by Beauvais. "Account should be reactivated – need to look deeper into the way this was handled," he wrote. "It shouldn't have taken this long to get the account back up, and also for it not be flagged a second time."
The cloud company responded through its own Twitter account, apologizing for the way the incident was handled.
"We are reviewing the processes that resulted in your account being locked twice," the company said. "Access has been fully restored, and your resources were not destroyed. Your account will not be locked again, and we will be following up with you."
The Register asked Beauvais and Digital Ocean for comment but we've not heard back.
DigitalOcean has promised a public postmortem once it completes an investigation of the incident. ®