Customers furious over days-long outage as A2 Hosting scores a D- in Windows uptime

An unexpected extended Easter break

Bewildered customers of A2 Hosting have endured a multi-day outage this week as the company battled to clear some pesky malware from its fleet of Windows Servers.

Problems surfaced early on Tuesday, 23 April, shortly after the company deployed the duct tape to deal with a "service interruption" at its Singapore facility. Users found their Windows-based websites unavailable and databases on the blink, and (as is the norm) vented their spleens on social media in response to the company's latest perky marketing tweet.

DB is not accessible, Website is not accessible for more than 1.5 hours
Perfect Hosting, ah?

— Oleksandr Karpov (@okarpov) April 23, 2019

The company updated its status page at 07:05 EDT to the effect that it was aware something was amiss. The full horror of the situation started to become clear just over four hours later when it announced: "Our System Operations team has taken all Windows services offline."

Yikes.

Over the next few days the company's engineers battled to overcome whatever mystery affliction had befallen its Window services as customers became progressively angrier. As the outage headed towards its third day, speculation mounted that the company had been the victim of a malware outbreak.

From the A2, nothing. But I'm pretty sure about what happened. I was in a meeting with a customer, and we saw (yes, I've got witness) all my files (from my ftp area) were renamed to .lock and a message stating the area was attacked and encrypted.

— Marcos Romero ⓟ (@mcfromero) April 24, 2019

The confusion at A2 towers is evident in emails seen by The Register in which the outfit first warns customers that while its engineers are working around the clock to determine the scope of problem, it can't give an ETA for a fix.

A later email (sent at the end of the first day of The Great A2 Outage, 23 April) explains that "our Support Operations team identified a security breach on our Windows services" and so "we have temporarily disabled all Windows-based Shared and VPS services". So severe was the situation that the company had taken to restoring from backups.

Customers were then told, at the end of 24 April, that the gang was still working on the problem and warned that customers could not simply just shift their accounts to different servers "since each server is brought up as a whole – including databases, email, and web services".

By yesterday morning, and with customer patience wearing thinner than graphene, the company gave some more details. It was indeed a malware attack wot dunnit (although the company did not specify ransomware), and A2 had shut down its entire Windows fleet to stop the miscreant code spreading.

The team decided the best approach was to stay offline until it could restore from backups. In a message to customers yesterday, the company declared itself "optimistic that at the current pace, the majority of the impacted services will be back online before the weekend".

After all, its swift action ensured its UK customers at least would be able to enjoy the unseasonably sunny weather of the last week rather than the tedious task of, you know, actually doing business.

As of early this morning, the company cheerfully updated its status page to let weary customers know: "Restores continue to progress at a steady pace." Your mileage may, however, vary.

@a2hosting im speechless to what is happening to my site here.........this down time is crazy!

— chrisgate (@chrisgate) April 26, 2019

A Register reader told us that his server remained inaccessible this morning.

Aside from the spectacular length of the outage, one can only imagine the administration practices that allowed malware to spread through A2's Windows environment. Although the company has "no reason to believe that personal information or data was downloaded" and is keen to emphasise that its own billing and internal infrastructure weren’t affected, the word "pisspoor" springs effortlessly to mind.

The Register got to experience A2's service first-hand when we contacted the hosting outfit to find out what was going on. Nobody was available to speak to us on the phone (unless we wanted to buy something).

We'll update this piece if the Ann Arbor-based company responds to email instead. At time of writing, even the company's status page had dropped over.

Not to worry, there's a "highly trained team of monkeys" on the case

Funnily enough, the last item posted on the company's corporate blog, just a day before the shit hit the fan, was headlined "4-Step Plan to Deal With a Security Breach". ®