Updated UK cloud hosting outfit Memset has blamed yesterday's lengthy outage on some iffy fibre infrastructure.
The company acknowledged that all was not well at 13:45 BST (12:45 UTC) on 21 June with a terse note among the sea of green on its status page to the effect that it was aware of some connectivity issues at its data centre in Dunsfold.
The clock ticked on for another hour before Memset confirmed the problem lurked upstream of the data centre, and appeared to pin the blame on its suppliers: "We have identified a problem which is upstream of our Dunsfold Datacenter, that is currently affecting connectivity, we have raised this as a fault with our circuit providers, and are in communication with them."
The issue also borked Memset's phone lines.
The following hours saw limited updates as the company told its customers "We are still continuing to work with our provider to get this issue resolved as soon as possible" over and over again.
It fell to Memset's owner, iomart, to give customers a clue with regard to what was actually happening. "Significant damage," it said, had taken out two fibre lines used by the Dunsfold data centre. While engineers worked, customers fumed.
By 08:24 BST (07:24 UTC) this morning, iomart confirmed that connectivity had been at least partially restored.
Our team worked through the night with our partners at BT to resolve two separate issues affecting connectivity at our @Memset_Ltd site. We’re pleased to confirm one path has been re-routed and restored.— iomart (@iomart) June 22, 2021
Customers have been left gobsmacked by the length of the outage and the apparent lack of communication on what had befallen their services, as well as an estimate for when things might be recovered. The Dunsfold data centre hosts a variety of cloud platforms as wall as OpenStack's IaaS.
- Not very sage rage over UK pay outage: Opayo says 'ohheyno' as payment processor's payments stop processing
- You had one job: Akamai's Prolexic Denial-of-Service protection system fingered after users in Australia denied, er, services
- Cloudflare network outage disrupts Discord, Shopify
- Fastly 'fesses up to breaking the internet with an 'an undiscovered software bug' triggered by a customer
Memset, founded back in 2002, says in marketing material it prides itself on "exceptional service" and boasts that it was the UK's first carbon-neutral web host. Keeping the power usage down is pretty straightforward after cutting off one's customers.
Affected users included, we're sad to say, at least one pub. It's not as if the hospitality industry hasn't had a torrid enough time of it without hosting companies compounding things.
Could someone in Dunsfold, Surrey, confirm that there hasn't been some kind of nuclear strike or asteroid impact? @Memset_Ltd host our website there, and it's died, along with thousands of others, and they're not answering their phones either. It's been six hours... #memsetdown— The Geese & Fountain (@Geese_Fountain) June 21, 2021
At the time of writing services appear to be back up again. The Register contacted the company to learn more about the incident but, like its customers, have been met with a wall of stony silence.
Openreach told us: "Engineers worked overnight to repair underground cable damage in the Clapham area, with services restored from around 1am. The cause of the damage is under investigation and we're sorry for any disruption to services."
Exactly how a bit of broken fibre took things down so completely in the Dunsfold data centre will be causing more than a few furrowed brows today.
And Memset? We'll leave the final word to the currently pinned emission on its social media feed.
There is! Are they competitively priced? Able to provide secure #cloudhosting? Will ensure #businesscontinuity ? Have fully green credentials? Talk to one of our team today to find out if we suit your #hosting needs https://t.co/DtIFCqqAGP pic.twitter.com/MG32Oxn2Vk— Memset Ltd (@Memset_Ltd) February 7, 2020
Perhaps one that doesn't get knocked out of the sky by an infrastructure wobble, eh?
Updated on 22 June @ 17.23BST to add:
Neil Christie, COO at iomart, told us: “We identified faults with two separate lines which impacted connectivity at the Memset Dunsfold site on Monday. Our team worked through the night with our partners at BT to restore connectivity through one of the lines in the early hours of Tuesday morning. The teams continued to work on the second line on Tuesday morning and we’re pleased to confirm the issue was fully resolved.” ®