The great A2 Hosting Windows TITSUP* has entered its second week as the company continues to struggle to recover from a security breach that forced its System Operations team to shut down all its Windows services.
To recap, things went south on 23 April as malware spread over the company's Windows operation, causing a problem so severe that the A2 Hosting team decided the only way to recover was to restore data from backups. The company told furious customers last week that "Restores continue to progress at a steady pace".
Except, alas, things have not gone smoothly.
As some services gradually tottered into life, users made the horrifying discovery that the backups being restored from were less than minty fresh.
Data had to be restored from complete viable backups. In a couple of cases that was older data, most cases it was only a day or two. In many cases individual account level backups are more recent. You can check the account backups in the backup manager in plesk.— A2 Hosting (@a2hosting) April 29, 2019
A "day or two" is bad enough for an ecommerce site, but the loss of several months' worth of data is an altogether angrier bag of monkeys. To make matters worse, the company has left it to users to work out just how whiffy those backups are.
Register reader David Sapery, who was lucky enough to see his services stagger back to life after a five-day liedown, was then somewhat embarrassed when his customers, finally able to access his sites, told him things looked a tad outdated.
Sapery told us: "Anything on any of my websites that was updated over the past 2+ months is gone."
Still, Sapery was at least able to recover. Another reader was not so lucky, describing his experience as "an unmitigated disaster."
Having spent eight months and "thousands of dollars", the unfortunate A2 Hosting customer told us that "my business and all my hard work has been gutted within seven days by a hosting company that clearly did not have robust security in place."
A2 Hosting will, of course, point to its Terms of Service where it makes it quite clear that it is not responsible for any data loss and that users are responsible for their own backups.
There is a lesson here for those depending on hosting services: make regular backups and for goodness' sake, test your restores.
However, let those who have never skipped a backup throw the first stone. Or, um, fling the quite-possibly-ransomware malware into the data centre.
The Register has made repeated attempts to contact A2 Hosting and we were promised a response yesterday. However, nothing has been forthcoming. We will update this article if the company gets in touch.
Oh, and Linux users smugly watching the discomfiture of their Windows counterparts? A2 Hosting had a little treat for them too. What the company described as "network connectivity issues" yesterday led to: "All users may experience service degradation."
All our sites are on Linux servers and they're all down. Thanks for showing resolved as we watch $$ going out the window.— E63 AMG (@1Man1Engine) May 1, 2019
Oh dear. ®
* TITSUP: Tottering Infrastructure, Tricky Security, Users Panicking