The day I took down the data centre- I mean, the day I saved the day. Right, boss?

How many hundred thousand customers?


Who, Me? Welcome to a Who, Me? story in which the moral might be: "Be careful what you kick off before lunch if you want a mealtime free of phone calls."

Today's tale concerns the exploits of "Anthony," who was working in the security department of one of the larger cable internet providers. One of his jobs was assessing systems due to be deployed, a task that was sensibly done on a pre-deployment network. A device would be built, popped onto the network and Anthony would run various tests to ensure everything had been put together to spec.

"Each Regional Data Center (RDC)," he explained, "had its own pre-deployment area, and I'd run the scans from my local server.

"All of the RDCs were connected via backbone connections, so latency was negligible, and bandwidth was massive."

One of the tools he used was the scanner Nmap ("Network Mapper"), a handy utility to rapidly scan large networks. Nmap will do a variety of useful things, including showing what is lurking on a network and an "interesting ports table."

Nmap has a variety of parameters, including a bunch around timing and performance to control how it runs. While the settings can be as fine-grained as one likes, the utility features some simple timing templates via the -Tx option, where x is a number from 0 to 5. The options summary innocently notes that "higher is faster" for this figure.

Anthony usually stuck with the default – 3 (Normal).

On this occasion, however, he was keen to get some lunch and wanted the scan completed earlier, "so I ran it at -T5."

Diving deeper into Nmap's documentation reveals what those numbers really mean. "The template names," explain the docs, "are paranoid (0), sneaky (1), polite (2), normal (3), aggressive (4), and insane (5)."

"Insane mode assumes that you are on an extraordinarily fast network or are willing to sacrifice some accuracy for speed."

The scan kicked off, and Anthony cracked on with the paperwork.

The phone soon began to ring: "In the background," he told us, "I can hear yelling. A voice shouts: 'What the hell are you doing??!?'"

Angry phone calls is a part of everyday life for many of us in IT, doomed to be at the beck and call of users and bosses. Patiently, Anthony explained that he was merely doing server assessment."

"THE RDC IS DOWN!" came the shrieking from the phone.

Eh?

"THE RDC IS DOWN! The firewall crashed and won't come back up!"

It transpired that Anthony had swamped the enterprise firewall, which promptly crashed and refused to come back up due to the packets being sprayed at it as fast as Anthony's server could manage.

He killed the scan.

"All in all, it was only a 15 - 20 minute outage for the 2-300,000 customers..." he noted.

Once things had settled down, Anthony was hauled before the bigwigs, with HR in attendance, to explain himself. Faced with a potential career-shortening (having killed service for hundreds of thousands customers) he did the only thing possible.

He became the self-proclaimed hero of the hour.

Yes, a Bad Thing had happened, but look at it this way: a "significant flaw" had been discovered. One disgruntled person with a well-connected device could take out an entire RDC! In many ways, the company should be thanking him. Perhaps a bonus for his diligence?

"I kept my job."

Ever screwed something up so badly that the only way out of a P45 and the march of shame was via the medium of spin? Or perhaps you've also unleashed the power of Nmap without fully considering the consequences? An email to Who, Me? is all it takes to purge your conscience. ®

Similar topics

Broader topics


Other stories you might like

  • An international incident or just some finger trouble at the console?
    All routers are equal, but some are more equal than others

    Who, Me? Welcome to an edition of Who, Me? where some configuration confusion left an entire nation cast adrift.

    Today's story is set in the early 2000s and comes from a reader Regomized as "Mikael" who was gainfully employed at a European ISP. The company had customers in multiple countries and Mikael's team was responsible for the international backbone.

    "Us senior network engineers were widely regarded as consummate professionals," he told us, before adding, "at least amongst ourselves."

    Continue reading
  • A discounting disaster averted at the expense of one's own employment
    I know what this process needs: Microsoft Access!

    Who, Me? A tale of discounts and process improvement via the magic of Excel, Access and a fair bit of electronic duct tape we imagine. Welcome to Who, Me?

    "James" is the Regomized reader of record today, and continues the theme of running the risk of doing a job just that little bit too well with an ancedote from the end of the last century involving his first job out of university, at a certain telecommunications giant.

    The job involved a process of calculating the discount received by big customers (the ones with multiple branches). "For the life of me I can't remember what the main DB was called," he told us, "but it was the old style green writing on a black screen that took forever to download the necessary data."

    Continue reading
  • In IT, no good deed ever goes unpunished
    When being helpful can mean being shown the door

    Who, Me? Going above and beyond in IT can sometimes lead to also going directly out of the door, as one Register reader found when discovering that sometimes efficiencies can be less than rewarding.

    A reader Regomised as "Will" told of us his days working at a now-defunct company that produced large telephone switches. In those days whenever a major software revision occurred, customers were expected to send in their configurations and Will's group would merge them into the latest and greatest. A new load would then be returned to the customers.

    It was not a fun process, not least because of constant hardware and software failures during the merge process. "When I first started, there was a constant grumble about how unreliable the machine used for the merging was," Will told us.

    Continue reading
  • An early crack at network management with an unfortunate logfile
    It's a backronym, right?

    Who, Me? Come with us on a journey back to the glory days of Visual Basic 6, misplaced enthusiasm and an unfortunate naming incident. Welcome to Who, Me?

    Today's tale comes from a reader Regomised as "Stephen", who was working in the IT department of a Royal Air Force base. "My duties were many," he told us, "from running daily backups of an ancient engineering system using (I kid you not) reel-to-reel tapes to swapping out misbehaving printers."

    This being the early 2000s, his boss loaded up our hero with more tasks. He could change printers and tapes, so Visual Basic (and its bedfellow, Access) should present no problem.

    Continue reading

Biting the hand that feeds IT © 1998–2022