Shock horror – and there goes the network neighborhood
Curious tech learned an important lesson about keeping a grip in tight situations
Who, me? Oh for heavens' sakes is it Monday already? Far out. Well, if you're here anyway, you may as well read another instalment of Who, Me? – The Register's weekly attempt to look on the bright side of the working week by revelling in the misfortune of others.
This week, meet "Vincent" who once, many many moons ago, worked for some very important people at a very important American university. Very importantly, he worked in the Telecommunications division at said university, and was responsible for (among other things) maintaining the switch gear in a cabinet.
Now that might not sound very important, but the thing to understand is that the university at the time operated a network of T1 circuits that connected all of the various campuses of the university around the state. Every campus had a T1 switch that handled voice, data and video conferencing traffic, and they were all ultimately centralized in the cabinet in Vincent's office, which was labelled "Network Equipment Technologies."
So it was, you might agree, a very important cabinet.
Now, mostly what Vincent did was monitoring the switching gear in the cabinet via a Sun workstation and vendor-specific software. Lots of mouse-clicking, lots of typing – not much actual opening of the cabinet.
As it happens, though, the manufacturer of the power supplies for the bandwidth management system had provided easily-accessible test points on each unit. And each one was helpfully marked with the voltage that should be passing through at any given moment. It was not, strictly speaking, Vincent's business to monitor the power supplies – but the opportunity to do so was right there, and so much more interesting than clicking a mouse and typing.
Thus, one fine day, multimeter in hand, Vincent set about checking that each power supply was putting out the voltage it was supposed to be putting out – despite the fact he had absolutely no reason whatsoever to believe they were not operating as intended.
Now, you know exactly where this is going, don't you?
- After nine servers he worked on failed, techie imagined next career as beach vendor
- That script I wrote three years ago is now doing what? How many times?
- One door opens, another one closes, and this one kills a mainframe
- Scripted shortcut caused double-click disaster of sysadmin's own making
As he was unsocketing the multimeter probes from one of the test sockets, his grip on the tool slipped ever so slightly. Just enough for one of the probes to touch another socket, which created a small spark. The spark was just enough to alert the bandwidth monitoring system that something was wrong, and send the entire system offline in an instant.
Suddenly, the entire network was down. Every campus of the university, all across the state, had no internet access, no video conferencing, no VOIP system. Nothing.
Thankfully, it was a well-designed system and sufficiently robust that there was no lasting damage. Systems began restarting almost immediately, and the whole outage lasted perhaps ten minutes before it was back to normal.
Vincent knew he had a very short time to get to a position where he looked innocent. So he very quickly packed away the multimeter, closed the cabinet, and hurried back to his desk. He had just enough time to do that before his boss stuck his head around the door, wondering what in blazes had just happened.
"I heard a noise, looked up, and everything was restarting! Not a clue what happened, honest!" he sputtered. The boss, thankfully, believed him.
Every had a lucky escape like Vincent? Allowed your enthusiasm to exceed the scope of your job, or your expertise? Tell us all about it in an email to Who, Me? and we'll share your story with others, to brighten someone's Monday morning. ®