The internet just BROKE under its own weight – we explain how

Next time, big biz, listen to your network admin


512KDay On Tuesday, 12 August, 2014, the internet hit an arbitrary limit of more than 512,000 routes. This 512K route limit is something we have known about for some time.

The fix for Cisco devices – and possibly others – is fairly straightforward. Internet service providers and businesses around the world chose not to address this issue in advance, as a result causing major outages around the world.

As part of the outage, punters experienced patchy – or even no – internet connectivity and lost access to all sorts of cloud-based services. The LastPass outage is being blamed by many on 512KDay, though official confirmation of this is still pending. I have been tracking reports of inability to access cloud services such as Office365 through to more localised phenomena from around the world, many of which look very much like they are 512KDay related.

As an example of the latter, while I don't yet have official confirmation yet from Canadian ISP Shaw, there are some indications are that the "mystery routing sickness" which affected its network (and which continues at time of publishing) could be related to the "512KDay" issue.

It is possible the issues I experienced with Shaw could be down to routers hitting the 512K limit. Theoretically, these routers could have hit the magic number and then been unable to route individual protocols (such as RDP, for example, although we cannot confirm this is so in Shaw's case) to the Deep Packet Inspection (DPI) systems that the ISP uses to create a "slow lane" enhance our internet experience*. We have contacted the ISP for comment but it had yet to respond at the time of publication.

As the fix for such issues can range from "applying a patch or config change and rebooting a core piece of critical network infrastructure" to "buy a new widget, the demand for which has just hit peak" there is every chance that 512KDay issues will continue for a few days (or even weeks) yet to come.

Others around the world have seen issues as well. Consider the issues reported by Jeff Bearer of Avere Systems who says "my firewall started noting packet loss between it and its upstream router. It wasn't that bad until employees started showing up for work, but then it jumped up quite a bit. We don't have any real evidence, but I did go back and forth with the ISP several times. It looks like it probably was [the 512KDay event] that caused this."

Awareness

Bearer asks a critical question: "Why wasn't this in the press, like Y2K or IPv4?".

Perhaps this is the ghost of Y2K. Globally, we handled the very real issues posed by computers being unable to comprehend the passing of the millennium so well that the average punter didn't notice the few systems that didn't get updated. IPv4 has been a highly publicised apocalypse that has dragged on for over a decade and the internet has yet to collapse.

512KDay is simply "yet another arbitrary limit issue" that has been for years filed away alongside the famous Y2K, IPv4 or 2038 problems. If you're interested in some of the others, Wikipedia has a brief overview of these "time formatting and storage bugs" that explains the big ones, but doesn't have a listing for all the known ones.

Do the media bear some of the blame? Perhaps. I have seen 512KDay issues raised in many IPv4 articles over the years, but rarely has it been discussed in a major publication as an issue in and of itself. Perhaps this is an example of crisis fatigue working its way into the technological sphere: as we rush from one manufactured "crisis" to another, we stop having brain space and resources to deal with the real issues that confront us.

The finger of blame

One thing I do know is that it is the job of network administrators to know about these issues and deal with them. What wasn't in the mainstream media has been in the networking-specific trade press, in vendor documentation and more.

I have been contacted by hundreds of network administrators in the past 12 hours with tales of woe. The common thread among them is that they absolutely did raise the flag on this, with virtually all of them being told to leave the pointy-haired boss's sight immediately.

Based on the evidence so far, I absolutely do not accept the inevitable sacrifice of some junior systems administrator to the baying masses. Throwing nerds under the bus doesn't cut it. The finger of blame points squarely at ISPs and other companies using BGP routers improperly all across the internet.

It's easy to make a boogyman out of ISPs; they're among the most hated industries in the world, after all. It's easy to point the finger of blame at companies that chose not to update their infrastructure because I've spent a lifetime fighting that battle from the coalface and it has made me a bitter and spiteful person.

Similar topics


Other stories you might like

  • How to keep a support contract: Make the user think they solved the problem

    Look what you found! Aren't you clever!

    On Call Let us take a little trip back to the days before the PC, when terminals ruled supreme, to find that the more things change the more they stay the same. Welcome to On Call.

    Today's story comes from "Keith" (not his name) and concerns the rage of a user whose expensive terminal would crash once a day, pretty much at the same time.

    The terminal in question was a TAB 132/15. It was an impressive bit of kit for the time and was capable of displaying 132 characters of crisp, green text on a 15-inch CRT housed in a futuristic plastic case. Luxury for sure, unless one was the financial trader trying to use the device.

    Continue reading
  • Apple kicked an M1-shaped hole in Intel's quarter

    Chipzilla braces for a China-gaming-ban-shaped hole in future results, predicts more product delays

    Intel has blamed Apple's switch to its own M1 silicon in Macs for a dip in sales at its client computing group, and foreshadowed future unpleasantness caused by supply chain issues and China's recent internet crackdowns.

    Chipzilla's finances were robust for the third quarter of its financial year: revenue of $19.2 billion was up five per cent year over year, while net income of $6.8 billion was up 60 per cent compared to 2020's Q3.

    But revenue for the client computing group was down two points. CFO George Davis – whose retirement was announced today – was at pains to point out that were it not for Apple quitting Intel silicon and Chipzilla exiting the modem business, client-related revenue would have risen ten per cent.

    Continue reading
  • How your phone, laptop, or watch can be tracked by their Bluetooth transmissions

    Unique fingerprints lurk in radio signals more often than not, it seems

    Over the past few years, mobile devices have become increasingly chatty over the Bluetooth Low Energy (BLE) protocol and this turns out to be a somewhat significant privacy risk.

    Seven boffins at University of California San Diego – Hadi Givehchian, Nishant Bhaskar, Eliana Rodriguez Herrera, Héctor Rodrigo López Soto, Christian Dameff, Dinesh Bharadia, and Aaron Schulman – tested the BLE implementations on several popular phones, PCs, and gadgets, and found they can be tracked through their physical signaling characteristics albeit with intermittent success.

    That means the devices may emit a unique fingerprint, meaning it's possible to look out for those fingerprints in multiple locations to figure out where those devices have been and when. This could be used to track people; you'll have to use your imagination to determine who would or could usefully exploit this. That said, at least two members of the team believe it's worth product makers addressing this privacy weakness.

    Continue reading

Biting the hand that feeds IT © 1998–2021