This major internet routing blunder took A WEEK to fix. Why so long? It was IPv6 – and no one really noticed

When you meant to type /127 but entered /12 instead


Comment Last week, an internet routing screw-up propagated by Verizon for three hours sparked havoc online, leading to significant press attention and industry calls for greater network security.

A few weeks before that, another packet routing blunder, this time pushed by China Telecom, lasted two hours, caused significant disruption in Europe and prompted some to wonder whether Beijing's spies were abusing the internet's trust-based structure to carry out surveillance.

In both cases, internet engineers were shocked at how long it took to fix traffic routing errors that normally only last minutes or even seconds. Well, that was nothing compared to what happened this week.

Cloudflare's director of network engineering Jerome Fleury has revealed that the routing for a big block of IP addresses was wrongly announced for an ENTIRE WEEK and, just as amazingly, the company that caused it didn't notice until the major blunder was pointed out by another engineer at Cloudflare. (This cock-up is completely separate to today's Cloudflare outage.)

How is it even possible for network routes to remain completely wrong for several days? Because, folks, it was on IPv6.

"So Airtel AS9498 announced the entire IPv6 block 2400::/12 for a week and no-one notices until Tom Strickx finds out and they confirm it was a typo of /127," Fleury tweeted over the weekend, complete with graphic showing the massive routing error.

That /12 represents 83 decillion IP addresses, or four quadrillion /64 networks. The /127 would be 2. Just 2 IP addresses. Slight difference. And while this demonstrates the expansiveness of IPv6's address space, and perhaps even its robustness seeing as nothing seems to have actually broken during the routing screw-up, it also hints at just how sparse IPv6 is right now.

To be fair to Airtel, it often takes someone else to notice a network route error – typically caused by simple typos like failing to add a "7" – because the organization that messes up the tables tends not to see or feel the impact directly.

But if ever there was a symbol of how miserably the transition from IPv4 to IPv6 is going, it's in the fact that a fat IPv6 routing error went completely unnoticed for a week while an IPv4 error will usually result in phone calls, emails, and outcry on social media within minutes.

And sure, IPv4 space is much, much more dense than IPv6 so obviously people will spot errors much faster. But no one at all noticed the advertisement of a /12 for days? That may not bode well for the future, even though, yes, this particular /127 typo had no direct impact.

Everyday experience

Y'know what? Maybe it was noticed, and people have grown so used to IPv6 being a little unreliable thanks to countless fudges and fixes that engineers keep imposing on the existing system – instead of shifting to IPv6 properly – that it didn’t seem too out of the ordinary.

Perhaps it went unnoticed because automated systems ignored it in preference of more specific, working, routes, and nothing at all raised any alarms.

Big bill

Strewth! Aussie ISP gets eye-watering IPv4 bill, shifts to IPv6 addresses

READ MORE

There are now quite a few different sources on how IPv6 adoption is going: the Internet Society has compiled most of the good ones in a single place. But while internet organizations continue to insist that things are going well, with, say the Americas offering 31 per cent IPv6 capability, it may be time to start digging into the stats that really matter: actual usage.

Google currently claims that 28 per cent of its visitors are using IPv6. We don't buy it. More likely that it's 28 per cent of connections, rather than actual users. And we wonder how much of that is automated traffic that comes from Google's own systems.

Just as routing errors have drawn attention to the fact that the internet is too strongly reliant on trust and is often held together by string and willpower, this error reveals that IPv6, more than 20 years after its inception, is still dangerously lagging in actual adoption.

And considering an entire block went AWOL, it only strengthens the argument that every internet provider and infrastructure organization needs to get on board with the Mutually Agreed Norms for Routing Security (MANRS), add filtering and anti-spoofing, and do more coordination and validation. ®

Similar topics


Other stories you might like

  • Google, EFF back Cloudflare in row over pirate streams
    Ban akin to 'ordering a telephone company to prevent a person from having conversations' over its lines

    Google, EFF, and the Computer and Communications Industry Association (CCIA) have filed court documents supporting Cloudflare after it was sued for refusing to block a streaming site.

    Earlier this year, a handful of Israel-based media companies took Israel.tv to court, accusing it of streaming TV and movie content it had no right to distribute. The corporations — United King Film Distribution, D.B.S. Satellite Services, HOT Communication Systems, Charlton, Reshet Media and Keshet Broadcasting — won the lawsuit after Israel.tv's creators failed to show up to their hearings, and the judge ordered Israel-tv.com, Israel.tv and Sdarot.tv each pay $7,650,000 in damages. 

    In a more surprising move, however, the media outfits also won an injunction [PDF] in the United States in April against a slew of internet companies, among others, banning them from aiding Israel.tv in its piracy.

    Continue reading
  • Cloudflare menaces virtual desktops with isolated browser access to internal networks
    Gives cloudy email a kicking, too – but VDI should be safe in its bastions

    Cloudflare has added the ability to access private networks to its browser isolation service, and suggests the combo represents an alternative to virtual desktop infrastructure.

    Browser isolation requires organizations to have a Cloudflare Zero Trust account, and to install a client on users' devices. Cloudflare runs a browser in its cloud and users browse as usual – but Cloudflare intervenes so that users don't make it to whichever web server they intend to visit.

    Cloudflare browses to the server and then redraws the web page on the client browser. The user's device therefore never touches the web server, so anything nasty on a page is snuffed out by Cloudflare in its cloud instead of poisoning a local PC.

    Continue reading
  • Cloudflare explains how it managed to break the internet
    'Network engineers walked over each other's changes'

    A large chunk of the web (including your own Vulture Central) fell off the internet this morning as content delivery network Cloudflare suffered a self-inflicted outage.

    The incident began at 0627 UTC (2327 Pacific Time) and it took until 0742 UTC (0042 Pacific) before the company managed to bring all its datacenters back online and verify they were working correctly. During this time a variety of sites and services relying on Cloudflare went dark while engineers frantically worked to undo the damage they had wrought short hours previously.

    "The outage," explained Cloudflare, "was caused by a change that was part of a long-running project to increase resilience in our busiest locations."

    Continue reading
  • Cloudflare's outage was human error. There's a way to make tech divinely forgive
    Don't push me 'cos I'm close to the edge. And the edge is safer if you can take a step back

    Opinion Edge is terribly trendy. Move cloudy workloads as close to the user as possible, the thinking goes, and latency goes down, as do core network and data center pressures. It's true  – until the routing sleight-of-hand breaks that diverts user requests from the site they think they're getting to the copies in the edge server. 

    If that happens, everything goes dark – as it did last week at Cloudflare, edge lords of large chunks of web content. It deployed a Border Gateway Protocol policy update, which promptly took against a new fancy-pants matrix routing system designed to improve reliability. Yeah. They know. 

    It took some time to fix, too, because in the words of those in the know, engineers "walked over each other's changes" as fresh frantic patches overwrote slightly staler frantic patches, taking out the good they'd done. You'd have thought Cloudflare of all people would be able to handle concepts of dirty data and cache consistency, but hey. They know that too. 

    Continue reading
  • Cloudflare says it thwarted record-breaking HTTPS DDoS flood
    26m requests a second? Not legit traffic, not even Bill Gates doing $1m giveaways could manage that

    Cloudflare said it this month staved off another record-breaking HTTPS-based distributed denial-of-service attack, this one significantly larger than the previous largest DDoS attack that occurred only two months ago.

    In April, the biz said it mitigated an HTTPS DDoS attack that reached a peak of 15.3 million requests-per-second (rps). The flood last week hit a peak of 26 million rps, with the target being the website of a company using Cloudflare's free plan, according to Omer Yoachimik, product manager at Cloudflare.

    Like the attack in April, the most recent one not only was unusual because of its size, but also because it involved using junk HTTPS requests to overwhelm a website, preventing it from servicing legit visitors and thus effectively falling off the 'net.

    Continue reading
  • Man gets two years in prison for selling 200,000 DDoS hits
    Over 2,000 customers with malice on their minds

    A 33-year-old Illinois man has been sentenced to two years in prison for running websites that paying customers used to launch more than 200,000 distributed denial-of-services (DDoS) attacks.

    A US California Central District jury found the Prairie State's Matthew Gatrel guilty of one count each of conspiracy to commit wire fraud, unauthorized impairment of a protected computer and conspiracy to commit unauthorized impairment of a protected computer. He was initially charged in 2018 after the Feds shut down 15 websites offering DDoS for hire.

    Gatrel, was convicted of owning and operating two websites – DownThem.org and AmpNode.com – that sold DDoS attacks. The FBI said that DownThem sold subscriptions that allowed the more than 2,000 customers to run the attacks while AmpNode provided customers with the server hosting. AmpNode spoofed servers that could be pre-configured with DDoS attack scripts and attack amplifiers to launch simultaneous attacks on victims.

    Continue reading
  • Big Tech shrank the internet while growing its own power
    Classic internet ideas matter less now that CDNs and private networks dominate traffic

    Comment The internet has become smaller, the result of a rethinking of when and where to use the 'net's intended architecture. In the process it may also have further concentrated power in the hands of giant technology companies.

    Given the ever-expanding content and resources available online, and proliferation of connected devices, the notion that the internet has shrunk is counter-intuitive. But shrunk it has – to the point at which some iPhones do not immediately connect to the open internet.

    Those phones are iPhones running the latest version of Apple's iOS and the opt-in service called Private Relay. The iGiant bills Private Relay as a privacy enhancement because it obscures users' DNS lookups and IP addresses by funneling traffic over networks operated by Cloudflare, according to specs set by Apple.

    Continue reading
  • Cloudflare stomps huge DDoS attack on crypto platform
    At 15.3 million requests per second, the assault was the largest HTTPS blitz on record lasting 15 seconds

    Cloudflare this month halted a massive distributed denial-of-service (DDoS) attack on a cryptocurrency platform that not only was unusual in its sheer size but also because it was launched over HTTPS and primarily originated from cloud datacenters rather than residential internet service providers (ISPs).

    At 15.3 million requests-per-second (rps), the DDoS bombardment was one of the largest that the internet infrastructure company has seen, and the largest HTTPS attack on record.

    It lasted less than 15 seconds and targeted a crypto launchpad, which Cloudflare analysts in a blog post said are "used to surface Decentralized Finance projects to potential investors."

    Continue reading
  • Developer adoption is our priority, profits second, Cloudflare tells bankers
    We seem to give away stuff for free at just the right time, says CFO

    If Cloudflare CFO Thomas Seifert's take on his company's direction is accurate, expect future strategy to focus on how it can use its slew of newly announced tools to make the biggest dent in existing markets. Profit motivations come a distant second, as least for now.

    Speaking at the Morgan Stanley Technology, Media and Telecom conference, Seifert told analyst Keith Weiss that 2022 will be all about growing Cloudflare's Zero Trust solution as well as Workers, its serverless code platform.

    Even with those products, Seifert said, the security-focused content-delivery network's strategy isn't about earnings – it's about gaining users. "We think primarily about adoption in the developer community penetration and less about dollars and revenue at this point in time," Siefert told the audience of investors and financial analysts.

    Continue reading
  • Cloudflare, Akamai: Why we're not pulling out of Russia
    Yanking connectivity would do more harm than good, they say

    Though Cloudflare and Akamai have voiced their opposition to President Vladimir Putin's invasion of Ukraine, they have stopped short of pulling completely out of Russia despite mounting pressure to do so.

    In a March 6 statement, Cloudflare CEO Matthew Prince said his company, which provides DDoS protection and other internet networking and security services, has received "several calls to terminate" all business inside Russia. He added that "we've watched in horror the Russian invasion of Ukraine," adding: "Our thoughts are with the people of Ukraine and the entire team at Cloudflare prays for a peaceful resolution as soon as possible."

    That said, after discussing the situation with government and private-sector experts, Prince said Cloudflare concluded: "Russia needs more internet access, not less."

    Continue reading

Biting the hand that feeds IT © 1998–2022