Linode cloud users in Europe hit as Frankfurt DC falls to its knees

Back up now, but packet loss ongoing

Problems at Linode's data centre in Germany have led to connectivity issues with its cloud services over the weekend, and we understand the vendor is currently trying to solve a packet loss problem.

At 14:14 UTC yesterday, the New Jersey-headquartered cloud provider said on its status page that it had become "aware of connectivity issues" to Frankfurt.

#Amikumu is currently offline. Linode, our cloud provider is currently experiencing an outage and are working to solve the issue.

— Amikumu (@Amikumu) October 2, 2017

It then proceeded to churn out several possible causes of "sporadic blackholing of customer traffic", including problems with packet forwarding tables at Frankfurt's two internet edge routers and a "software bug".

At 3:32 UTC this morning, "After many hours of troubleshooting with Cisco," Linode's status page announced, "at this time we have been able to normalize connectivity".

Linode attributed the outage to both "hardware and software failures" happening at the same time to its two internet-facing routers.

According to the page, one router "caused strange forwarding behavior" but didn't tell the operating system about it. Linode said it had recognized this after rebooting.

The "undocumented forwarding table forwarding bug" it had referred to earlier also affected its redundant routers.

And although the connectivity problem was marked resolved, "A separate maintenance will be required to fully resolve this incident, at which point a full post mortem will be made available," the firm said.

The firm also experienced "connectivity issues" at Frankfurt starting at 22:36 UTC on September 30 until 2:23 UTC October 1, although it hasn't explained what was behind those.

According to the support page, since 13:25 UTC October 1 there has been packet loss between its Newark data centre and Canada or Europe. At 7:52 UTC October 2, the firm also indicated that there was upstream packet loss between Frankfurt and "other Linode data centres".

It's not clear from the support page if the issues are connected to the outage. We have contacted Linode for comment.

We also contacted Cisco, which told us: "I’m afraid we do not comment on customer specific situations."

Worried users took to Twitter, while some praised the firm for its transparency:

@linode does frankfurt net issue have an ETA for fix? Can't believe it's completely unreachable. Very worried over here.

— kaminoo (@kaminoo) October 2, 2017

Our whole site is down and thankfully the downtime is not in our peak hours, but I like how @linode keeps this whole thing transparent. https://t.co/CAgIbgkvEI

— Farzad (@onatfarzad) October 1, 2017

Linode has suffered multi-day outages before. In 2016, it was offline for 10 days because of an intense denial-of-service attack. ®

Affected by the service outage? Send a tip to the scribe here.