This article is more than 1 year old
Undebug my heart: Using Cisco's IOS to take down capitalism – accidentally
Two little letters is all it takes
Who, Me? Welcome to another edition of Who, Me? where this week a typo manages to send a hub of rampant capitalism into meltdown.
Our story takes us back a few decades and concerns an adventurous time in network support.
"Mort" – for that is not his name – was working for a well-known stock exchange, the network of which was running on Cisco gear. The cards, he recalled, cost $50k each and the whole shebang could probably be replaced by a single unit nowadays. But back then this was top-end stuff – nothing but the best would do for the nation's traders.
There was, however, a problem. Packets were being lost, resulting in delayed or lost trades. "When I think of traits that traders possess in abundance, tolerance is not one of them," Mort told us. "Neither is patience, so as you can imagine the pressure on the network team to resolve the issue was intense."
Those Ferraris weren't going to service themselves, and one lost packet could mean a world of difference in the quality of Champagne purchased at bonus time.
The support team struggled in vain to recreate the problem in the lab, and with pressure mounting, Mort decided the only way to get to the bottom of matters was to plug into the network and diagnose the problem live.
"Anyone who's ever worked on Cisco IOS (Internetwork Operating System, they beat Apple to the acronym by well over a decade) knows that debugging is a minefield as some commands can cause serious impact to the system," he said.
It's true. A glance at an example of Cisco's documentation shows it festooned with warnings designed to deter all but the most determined and (ideally) competent of users. Debug output spewed over the console could make typing a command difficult, and the wrong debug command could easily take down a router.
That said, it was also pretty straightforward. The syntax was
debug followed by whatever was needed. The output could then be captured, and the command backed out by prefixing it with
un. "More often," explained Morty, "you'd just use '
undebug all' to turn it all off."
- Somebody is destined for somewhere hot, and definitely not Coventry
- How to keep your enterprise up to date by deploying the very latest malware
- Ah, I see you found my PowerShell script called 'SiteReview' – that does not mean what you think it means
- One good deed leads to a storm in an Exchange Server
The opposite command was the most dangerous of all: "You never, ever ran
debug all unless you wanted to demonstrate how quickly you could take out your network."
Cisco had yet to add any "Are you sure?"-type prompts, doubtless assuming the operators of its hardware knew what they were doing. Instead, "if you typed it in it would salute, shout 'sir yes sir!' and then obediently jump off a cliff, similar to 'rm -rf /' on a *nix box."
You can probably see where this is going.
Our hero was careful – very careful – and had taken great pains in his planning. However, after a very limited set of packet captures, he was having problems getting the data and grew concerned that he might adversely affect the network. So he issued
All hell broke loose. Text whizzed across the terminal screen faster than he could read and he realised that something had gone terribly, terribly wrong. Panicked, he tried
undebug all once again. However, since he had clearly accidentally fired off every possible debug command at once, the terminal process was no longer bothering with such fripperies as input from the keyboard.
un Mort is positive he typed clearly had not made it to the Cisco hardware (perhaps some transient unresponsiveness). But the rest of the command… oh yes.
With the network rapidly overwhelmed with the results of his actions, and the terminal ignoring his pleas to stop, Mort took the only avenue left to him and pulled the plugs.
All around our hero could be heard the sound of blades being unsheathed as the representatives of several large investment banks demanded his head on a platter.
Dispirited, Mort started cleaning his desk, expecting an escort to the kerb and the brown cardboard box of career limitation.
However, it is here that our story takes a turn. His boss was that rarest of managers – a decent human being. Rather than fling Mort under the bus, he stood between him and the pitchforks and took responsibility for the cock-up.
The result? A multimillion-dollar investment in the network infrastructure.
"Now that's management!"
Ever done something very silly, only to have your boss take the bullets heading your way? Or were you that boss? Tell all with an email to Who, Me? ®