Telia engineer error to blame for massive net outage

Lesson 1: Don't confuse Europe with Hong Kong


Swedish infrastructure company Telia is to blame for a massive internet outage today after an engineer apparently misconfigured a key router and sent all of Europe's traffic to Hong Kong.

The Tier 1 network provider is one of fewer than 20 companies that provides a basic foundation for much of the internet. It sent a note to other network operators several hours after huge disruptions were reported across Europe apologizing for the mishap.

When millions of internet users reported problems with their internet connections, covering everything from WhatsApp to Slack, it was so significant that the first assumption was that a transatlantic cable had been damaged or cut.

It turns out that the problem was human error caused by a Telia engineer, and the company was able to fix the problem relatively quickly. The knock-on impact was huge, however: countless services and websites relying on the Swedish company's network reported problems, causing a wave of complaints from users.

Once fixed, things should return to normal pretty quickly. However, there may be a significant impact on Telia itself.

The mistake is so significant that the CEO of content delivery network Cloudflare, Matthew Prince, has already taken action against the company. "Reliability of Telia over last 60 days unacceptable," he tweeted. "Deprioritizing them until we are confident they've fixed their systemic issues." ®

Similar topics


Other stories you might like

  • Cloudflare's outage was human error. There's a way to make tech divinely forgive
    Don't push me 'cos I'm close to the edge. And the edge is safer if you can take a step back

    Opinion Edge is terribly trendy. Move cloudy workloads as close to the user as possible, the thinking goes, and latency goes down, as do core network and data center pressures. It's true  – until the routing sleight-of-hand breaks that diverts user requests from the site they think they're getting to the copies in the edge server. 

    If that happens, everything goes dark – as it did last week at Cloudflare, edge lords of large chunks of web content. It deployed a Border Gateway Protocol policy update, which promptly took against a new fancy-pants matrix routing system designed to improve reliability. Yeah. They know. 

    It took some time to fix, too, because in the words of those in the know, engineers "walked over each other's changes" as fresh frantic patches overwrote slightly staler frantic patches, taking out the good they'd done. You'd have thought Cloudflare of all people would be able to handle concepts of dirty data and cache consistency, but hey. They know that too. 

    Continue reading
  • Cloudflare explains how it managed to break the internet
    'Network engineers walked over each other's changes'

    A large chunk of the web (including your own Vulture Central) fell off the internet this morning as content delivery network Cloudflare suffered a self-inflicted outage.

    The incident began at 0627 UTC (2327 Pacific Time) and it took until 0742 UTC (0042 Pacific) before the company managed to bring all its datacenters back online and verify they were working correctly. During this time a variety of sites and services relying on Cloudflare went dark while engineers frantically worked to undo the damage they had wrought short hours previously.

    "The outage," explained Cloudflare, "was caused by a change that was part of a long-running project to increase resilience in our busiest locations."

    Continue reading
  • Google, EFF back Cloudflare in row over pirate streams
    Ban akin to 'ordering a telephone company to prevent a person from having conversations' over its lines

    Google, EFF, and the Computer and Communications Industry Association (CCIA) have filed court documents supporting Cloudflare after it was sued for refusing to block a streaming site.

    Earlier this year, a handful of Israel-based media companies took Israel.tv to court, accusing it of streaming TV and movie content it had no right to distribute. The corporations — United King Film Distribution, D.B.S. Satellite Services, HOT Communication Systems, Charlton, Reshet Media and Keshet Broadcasting — won the lawsuit after Israel.tv's creators failed to show up to their hearings, and the judge ordered Israel-tv.com, Israel.tv and Sdarot.tv each pay $7,650,000 in damages. 

    In a more surprising move, however, the media outfits also won an injunction [PDF] in the United States in April against a slew of internet companies, among others, banning them from aiding Israel.tv in its piracy.

    Continue reading

Biting the hand that feeds IT © 1998–2022