Updated Telecity engineers will tonight make a second attempt to fix broken power systems at its critical Sovereign House internet facility.
They will be working 24 hours after their first efforts failed and resulted in a second power outage.
The internet service giant had planned to repair and fully fix the power supply in the early hours of Wednesday morning. But based on commnunications The Register has seen, things did not get fixed – in fact, they worsened, producing another power outage.
A outage struck Sovereign House Tuesday afternoon, when both a primary supply and its supposedly reliable backup went down. The outage coincided with a noticeable dip in internet traffic across the London Internet Exchange (LINX).
Sources told The Reg Wednesday afternoon Telecity was gearing up for a second crack at fixing the problem.
Telecity, whose Sovereign House data centre appears to handle about 10 per cent of the UK’s internet traffic, promised to update customers at 1430 GMT – some 24 hours after its service started to suffer outages.
Our source reckoned Sovereign House had suffered "significant" power surges that had caused IT equipment to fail.
Servers had fried and had to be replaced, we've learned.
Expecting a fix on the first night, Telecity customer Dotmailer told users of its service at 0414 GMT yesterday:
Telecity have been forced to re-evaluate the situation and we await their decision on how they will proceed. Unfortunately dotmailer services must remain offline until we receive a stable power feed.
Dotmailer sys admin lead Jason Sanchania Tweeted:
Been awake for over 24hrs courtesy of #Telecity. Time for you to do some explaining! Well... Get to it!— Jason Sanchania (@jasonsanchania) November 18, 2015
One Reg reader got in touch with us mid-morning today to confirm that Telecity’s planned repair work was not successful, and had resulted in another power failure.
The reader told us Telecity had told him its services were reliant on mains power and are not protected by UPS. Telecity handles critical business services for big names including Amazon and Microsoft, with each firm's AWS and Azure products.
Another source reckoned customers had paid Telecity for the use of dual, fully independent power suppliers to avoid outages but the fact their service was down indicated they hadn't actually got what they'd paid for.
Customers’ Office 365 and Lync/Skype connections had been affected with AWS Direct Connect also dropping VPC tunnels. The problem seems to have struck Amazon's AWS EU-WEST-1 region.
Also, day-to-day services, including VoIP and cloud hosting for names such as charity donation payment and donations service JustGiving, have been hit.
AWS said the issue of packet loss has been resolved for its service, which it said was now operating normally.
El Reg could not reach Telecity for comment despite repeated phonecalls and emails.
Are you or your firm affected by the Telecity outage? Contact us: firstname.lastname@example.org or phone the London newsdesk on +44 (0)20 3770 3147.