Undebug my heart: Using Cisco's IOS to take down capitalism – accidentally

Two little letters is all it takes


Who, Me? Welcome to another edition of Who, Me? where this week a typo manages to send a hub of rampant capitalism into meltdown.

Our story takes us back a few decades and concerns an adventurous time in network support.

"Mort" – for that is not his name – was working for a well-known stock exchange, the network of which was running on Cisco gear. The cards, he recalled, cost $50k each and the whole shebang could probably be replaced by a single unit nowadays. But back then this was top-end stuff – nothing but the best would do for the nation's traders.

There was, however, a problem. Packets were being lost, resulting in delayed or lost trades. "When I think of traits that traders possess in abundance, tolerance is not one of them," Mort told us. "Neither is patience, so as you can imagine the pressure on the network team to resolve the issue was intense."

Those Ferraris weren't going to service themselves, and one lost packet could mean a world of difference in the quality of Champagne purchased at bonus time.

The support team struggled in vain to recreate the problem in the lab, and with pressure mounting, Mort decided the only way to get to the bottom of matters was to plug into the network and diagnose the problem live.

"Anyone who's ever worked on Cisco IOS (Internetwork Operating System, they beat Apple to the acronym by well over a decade) knows that debugging is a minefield as some commands can cause serious impact to the system," he said.

It's true. A glance at an example of Cisco's documentation shows it festooned with warnings designed to deter all but the most determined and (ideally) competent of users. Debug output spewed over the console could make typing a command difficult, and the wrong debug command could easily take down a router.

That said, it was also pretty straightforward. The syntax was debug followed by whatever was needed. The output could then be captured, and the command backed out by prefixing it with un. "More often," explained Morty, "you'd just use 'undebug all' to turn it all off."

The opposite command was the most dangerous of all: "You never, ever ran debug all unless you wanted to demonstrate how quickly you could take out your network."

Cisco had yet to add any "Are you sure?"-type prompts, doubtless assuming the operators of its hardware knew what they were doing. Instead, "if you typed it in it would salute, shout 'sir yes sir!' and then obediently jump off a cliff, similar to 'rm -rf /' on a *nix box."

You can probably see where this is going.

Our hero was careful – very careful – and had taken great pains in his planning. However, after a very limited set of packet captures, he was having problems getting the data and grew concerned that he might adversely affect the network. So he issued undebug all.

All hell broke loose. Text whizzed across the terminal screen faster than he could read and he realised that something had gone terribly, terribly wrong. Panicked, he tried undebug all once again. However, since he had clearly accidentally fired off every possible debug command at once, the terminal process was no longer bothering with such fripperies as input from the keyboard.

The un Mort is positive he typed clearly had not made it to the Cisco hardware (perhaps some transient unresponsiveness). But the rest of the command… oh yes.

With the network rapidly overwhelmed with the results of his actions, and the terminal ignoring his pleas to stop, Mort took the only avenue left to him and pulled the plugs.

All around our hero could be heard the sound of blades being unsheathed as the representatives of several large investment banks demanded his head on a platter.

Dispirited, Mort started cleaning his desk, expecting an escort to the kerb and the brown cardboard box of career limitation.

However, it is here that our story takes a turn. His boss was that rarest of managers – a decent human being. Rather than fling Mort under the bus, he stood between him and the pitchforks and took responsibility for the cock-up.

The result? A multimillion-dollar investment in the network infrastructure.

"Now that's management!"

Ever done something very silly, only to have your boss take the bullets heading your way? Or were you that boss? Tell all with an email to Who, Me? ®

Similar topics


Other stories you might like

  • SpaceX Starlink sat streaks now present in nearly a fifth of all astronomical images snapped by Caltech telescope

    Annoying, maybe – but totally ruining science, no

    SpaceX’s Starlink satellites appear in about a fifth of all images snapped by the Zwicky Transient Facility (ZTF), a camera attached to the Samuel Oschin Telescope in California, which is used by astronomers to study supernovae, gamma ray bursts, asteroids, and suchlike.

    A study led by Przemek Mróz, a former postdoctoral scholar at the California Institute of Technology (Caltech) and now a researcher at the University of Warsaw in Poland, analysed the current and future effects of Starlink satellites on the ZTF. The telescope and camera are housed at the Palomar Observatory, which is operated by Caltech.

    The team of astronomers found 5,301 streaks leftover from the moving satellites in images taken by the instrument between November 2019 and September 2021, according to their paper on the subject, published in the Astrophysical Journal Letters this week.

    Continue reading
  • AI tool finds hundreds of genes related to human motor neuron disease

    Breakthrough could lead to development of drugs to target illness

    A machine-learning algorithm has helped scientists find 690 human genes associated with a higher risk of developing motor neuron disease, according to research published in Cell this week.

    Neuronal cells in the central nervous system and brain break down and die in people with motor neuron disease, like amyotrophic lateral sclerosis (ALS) more commonly known as Lou Gehrig's disease, named after the baseball player who developed it. They lose control over their bodies, and as the disease progresses patients become completely paralyzed. There is currently no verified cure for ALS.

    Motor neuron disease typically affects people in old age and its causes are unknown. Johnathan Cooper-Knock, a clinical lecturer at the University of Sheffield in England and leader of Project MinE, an ambitious effort to perform whole genome sequencing of ALS, believes that understanding how genes affect cellular function could help scientists develop new drugs to treat the disease.

    Continue reading
  • Need to prioritize security bug patches? Don't forget to scan Twitter as well as use CVSS scores

    Exploit, vulnerability discussion online can offer useful signals

    Organizations looking to minimize exposure to exploitable software should scan Twitter for mentions of security bugs as well as use the Common Vulnerability Scoring System or CVSS, Kenna Security argues.

    Better still is prioritizing the repair of vulnerabilities for which exploit code is available, if that information is known.

    CVSS is a framework for rating the severity of software vulnerabilities (identified using CVE, or Common Vulnerability Enumeration, numbers), on a scale from 1 (least severe) to 10 (most severe). It's overseen by First.org, a US-based, non-profit computer security organization.

    Continue reading

Biting the hand that feeds IT © 1998–2022