The fall-out continues from hosting service 123-reg's major weekend cockup, which knocked several customers offline – with several telling us the error has effectively deleted their businesses.
As already revealed, on Saturday customers' virtual servers vanished after the hosting firm ran a script containing a catastrophic error at 7am as part of its clean-up process on the 123-reg VPS platform.
Customers are still reporting problems with accessing services today. 123-reg told us it is trying to fix things:
A small proportion of our customer base has been affected. Our number one priority is to return services to normal. We have been working with an extended team of experts who have been working into the night to restore as much as we can as quickly as possible.
According to the company's website, it has 3.5 million domain names, and over 1.7 million websites "are trusted with us everyday."
One customer said the outage had taken several of his sites offline, telling El Reg: "It has also taken a lot of people's email offline and the service is popular with small businesses which will really be feeling the full force of it."
He claimed 123-reg told him they were unable to recover his 50GB of files, as it has no backup of them.
Another customer complained about 123-reg's "awful" communication. "This is the worst customer service I have ever experienced," he said. "In short, I’ve been told to rebuild, but not told how to do so, nor if there is any chance at recovering data."
Another said: "I run the @GlastoWatch Twitter account and forum, and my VPS died just before the Glastonbury Festival ticket resale, which is one of the busiest days of the year for me. So obviously I wasn't very happy."
In an email to punters on Saturday, 123-reg brand director Richard Winslow tried to detail the run of events that led to the service's woes.
This script is run to show us the number of machines active against the master database. An error on the script showed 'zero-records' response from the database for some live VPS. For those customers, this created a 'failure' scenario – showing no VMs and effectively deleting what was on the host. As a result of our team's investigations, we can conclude that the issues faced having resulted in some data loss for some customers. Our teams have been and continue to work to restore.
One customer claimed this practice "is frowned upon in the hosting world... I discussed the issue with a few acquaintances who have experience running servers in a commercial environment and all of them were shocked to hear that 123-reg 'automate clean-up of servers' and that they 'don't have backups'.”
The customer continued: "One of my biggest issues is that even if they are running automated clean-up scripts, why aren't these scripts developed in such a way that they are 'safe-to-fail'. In my professional opinion, this clearly falls into the realm of negligence."
He added: "If 123 achieve anything short of a perfect restore, we're jumping ship. Might as well put in the effort to restore with a reliable host (this isn't the first time 123 has lost our data in the last few months)."
Mindful of the obvious problems caused to 123-reg's reputation by the outage, Wilmslow continued, in his email to customers:
I understand that some customers may have lost some confidence in the service that we offer. So, I want to explain what we have done to prevent this happening again. We have started an audit on all cron-jobs and scripts controlling the platform, and associated architecture, so that no script will have ability to delete images, only suspend.
For image deletion for those suspended over 28 days we will have a human eye to double check. A new platform will be available by the end of the year for customers which we will provide self-managed and automated snapshot backups, in addition to architecture technology to backup the whole platform, something that is not available on the current platform. I hope this goes some way to win back your confidence.
Some customers were not placated by the email, however. One responded:
"Surely somebody in your management has a grasp on this basic and crucial detail of your business? I guess not, so I suggest you open a corner shop, market stall or something else that doesn't have the potential to cripple many other businesses." ®