Scalr hosting hit with outage

Server records deletion causes website woes

Cloud management biz was yesterday hit by an outage which knocked customer websites offline.

In a post, the company said the problem had been due to an update that included flawed logic, causing server records to be removed but not AWS servers or other cloud servers.

One customer got in touch this morning to complain the company had yet to issue an apology or discuss compensation.

He said: "Yesterday, one of our customer websites stopped working as the database server was no longer contactable.

"It turns out that Scalr [had] accidentally removed all their DNS entries, resulting in another server being spawned for us in AWS, without our knowledge, even though there was no issue with the existing server.

"This obviously cost us, no just [in] lost revenue and downtime, but [also because] we were then paying for a second server we did not need," he claimed.

The Register has contacted Scalr for a comment.

Other customers complained on Twitter:


Updated at 11:32 UTC, 19 July to add: Sebastian Stadil, Scalr's CEO, said the company had apologised privately to all its customers and intends to post a further statement on its website.

He said: "Users have every right to be angry and voice their frustration. I know I've been frustrated when services I use are down, so I can sympathise."

He added: "Operations professionals know that outages come with the territory. In fact, the 'blameless' devops culture comes from this: learn from mistakes, improve, share your experience. Tomorrow's statement will do just that. Explain what happened, how we handled it, what we're doing to improve, and what others can learn from this experience.

"Lastly, I want to point out that everyone in the team behaved admirably yesterday, and some worked 30 hours straight to get this resolved for every customer, with the first resolutions by within three hours of detection. I myself have been up for 22 hours and am about to get some rest."

Similar topics

Other stories you might like

  • AWS buys before it tries with quantum networking center
    Fundamental problems of qubit physics aside, the cloud giant thinks it can help

    Nothing in the quantum hardware world is fully cooked yet, but quantum computing is quite a bit further along than quantum networking – an esoteric but potentially significant technology area, particularly for ultra-secure transactions. Amazon Web Services is among those working to bring quantum connectivity from the lab to the real world. 

    Short of developing its own quantum processors, AWS has created an ecosystem around existing quantum devices and tools via its Braket (no, that's not a typo) service. While these bits and pieces focus on compute, the tech giant has turned its gaze to quantum networking.

    Alongside its Center for Quantum Computing, which it launched in late 2021, AWS has announced the launch of its Center for Quantum Networking. The latter is grandly working to solve "fundamental scientific and engineering challenges and to develop new hardware, software, and applications for quantum networks," the internet souk declared.

    Continue reading
  • Amazon can't channel the dead, but its deepfake voices take a close second
    Megacorp shows Alexa speaking like kid's deceased grandma

    In the latest episode of Black Mirror, a vast megacorp sells AI software that learns to mimic the voice of a deceased woman whose husband sits weeping over a smart speaker, listening to her dulcet tones.

    Only joking – it's Amazon, and this is real life. The experimental feature of the company's virtual assistant, Alexa, was announced at an Amazon conference in Las Vegas on Wednesday.

    Rohit Prasad, head scientist for Alexa AI, described the tech as a means to build trust between human and machine, enabling Alexa to "make the memories last" when "so many of us have lost someone we love" during the pandemic.

    Continue reading
  • AWS says it will cloudify your mainframe workloads
    Buyer beware, say analysts, technical debt will catch up with you eventually

    AWS is trying to help organizations migrate their mainframe-based workloads to the cloud and potentially transform them into modern cloud-native services.

    The Mainframe Modernization initiative was unveiled at the cloud giant's Re:Invent conference at the end of last year, where CEO Adam Selipsky claimed that "customers are trying to get off their mainframes as fast as they can."

    Whether this is based in reality or not, AWS concedes that such a migration will inevitably involve the customer going through a lengthy and complex process that requires multiple steps to discover, assess, test, and operate the new workload environments.

    Continue reading

Biting the hand that feeds IT © 1998–2022