The day I took down the data centre- I mean, the day I saved the day. Right, boss?
How many hundred thousand customers?
Who, Me? Welcome to a Who, Me? story in which the moral might be: "Be careful what you kick off before lunch if you want a mealtime free of phone calls."
Today's tale concerns the exploits of "Anthony," who was working in the security department of one of the larger cable internet providers. One of his jobs was assessing systems due to be deployed, a task that was sensibly done on a pre-deployment network. A device would be built, popped onto the network and Anthony would run various tests to ensure everything had been put together to spec.
"Each Regional Data Center (RDC)," he explained, "had its own pre-deployment area, and I'd run the scans from my local server.
"All of the RDCs were connected via backbone connections, so latency was negligible, and bandwidth was massive."
One of the tools he used was the scanner Nmap ("Network Mapper"), a handy utility to rapidly scan large networks. Nmap will do a variety of useful things, including showing what is lurking on a network and an "interesting ports table."
Nmap has a variety of parameters, including a bunch around timing and performance to control how it runs. While the settings can be as fine-grained as one likes, the utility features some simple timing templates via the
-Tx option, where
x is a number from 0 to 5. The options summary innocently notes that "higher is faster" for this figure.
Anthony usually stuck with the default – 3 (Normal).
On this occasion, however, he was keen to get some lunch and wanted the scan completed earlier, "so I ran it at
Diving deeper into Nmap's documentation reveals what those numbers really mean. "The template names," explain the docs, "are paranoid (0), sneaky (1), polite (2), normal (3), aggressive (4), and insane (5)."
"Insane mode assumes that you are on an extraordinarily fast network or are willing to sacrifice some accuracy for speed."
The scan kicked off, and Anthony cracked on with the paperwork.
The phone soon began to ring: "In the background," he told us, "I can hear yelling. A voice shouts: 'What the hell are you doing??!?'"
Angry phone calls is a part of everyday life for many of us in IT, doomed to be at the beck and call of users and bosses. Patiently, Anthony explained that he was merely doing server assessment."
"THE RDC IS DOWN!" came the shrieking from the phone.
"THE RDC IS DOWN! The firewall crashed and won't come back up!"
It transpired that Anthony had swamped the enterprise firewall, which promptly crashed and refused to come back up due to the packets being sprayed at it as fast as Anthony's server could manage.
He killed the scan.
"All in all, it was only a 15 - 20 minute outage for the 2-300,000 customers..." he noted.
Once things had settled down, Anthony was hauled before the bigwigs, with HR in attendance, to explain himself. Faced with a potential career-shortening (having killed service for hundreds of thousands customers) he did the only thing possible.
He became the self-proclaimed hero of the hour.
Yes, a Bad Thing had happened, but look at it this way: a "significant flaw" had been discovered. One disgruntled person with a well-connected device could take out an entire RDC! In many ways, the company should be thanking him. Perhaps a bonus for his diligence?
"I kept my job."
Ever screwed something up so badly that the only way out of a P45 and the march of shame was via the medium of spin? Or perhaps you've also unleashed the power of Nmap without fully considering the consequences? An email to Who, Me? is all it takes to purge your conscience. ®