Lesson 1: Keep your mind on the ... why aren't the servers making any noise?
Quest for redundant cables almost resulted in a redundant techie
Who, Me? Ah, dear reader, yet again it is Monday, arriving with the same relentless regularity that has made it the bane of human workers and cartoon cats since time immemorial - or at least the early 1900s. But fear not, for The Reg is here to ease your passage into the working week with another instalment of Who, Me? in which we cushion the arrival of the working week with tales of the working weak.
This week, for your amusement and delectation, we present a reader we'll Regomize as "Hans" who once worked for a small software development house that also dabbled in application hosting.
Now, as luck would have it, the firm for which Hans worked was something of a leader in the field of WAP applications – back when WAP stood for Wireless Application Protocol, rather than something more lyrical and NSFW. Look it up if you must - but not if HR monitors your machine.
Anyway, in those pre-iPhone days WAP was quite the thing for mobile devices, so the small development house had landed a contract with a state-owned telco to build and host WAP, plus some SMS services.
We should mention that the building in which the business was housed had previously been used by a rather larger concern, and so was quite well equipped to host hosting infrastructure. It boasted a cooling system, server racks, and what Hans described as "a rather scary halon fire suppression system".
The fire suppression system does not feature in this story. Don't get your hopes up.
When the previous tenant moved out, it had left something of a mess of power and network cables snaking around under the floor. For whatever reason, Hans's firm had not seen fit to clear that away upon moving in, but rather had compounded the snarl it by installing its own equipment. As time passed and more servers were added, the tangle had reached a point at which Something Had To Be Done.
Hans was thusly handed the task of bringing order to chaos. The first task was to figure out which cables under the floor were not, in fact, doing anything. This was done using the time-honored method of jiggling each one until he found both ends. If one end was plugged in and the other was not, that cable could go.
- The price of freedom turned out to be an afternoon of tech panic
- Techie's quick cure for a curious conflict caused a huge headache
- Nobody would ever work on the live server, right? Not intentionally, anyway
- Linux lover consumed a quarter of the network
And so Hans proceeded about his noble task – jiggling, tracing, unplugging – gradually reducing the tangle to a mere knot.
As he was doing this, a colleague tapped on the window. Still holding the cable he was tracing, Hans opened up and chatted with said colleague for a moment. Not a long moment, but enough of a moment to be distracted.
Not very distracted, but just distracted enough to forget whether the cable he was holding was redundant, or essential.
Guessing, he unplugged it.
"Things got quite a lot quieter as a pile of 10Krpm SCSI drives stopped spinning and a whole lot of fans went silent," Hans recalled. He had disconnected power to the server rack belonging to that telco. This was not good.
He tried plugging it back in, but to no avail. Upon investigation, he found that attempting to plug the whole rack in at once was popping a circuit breaker. He'd push the breaker back in, but each time he reconnected the power it popped right back out.
You and I might suggest powering down the machines, reconnecting the rack, and powering the machines back on one at a time. Hans thought of that later. After he stopped panicking.
Before then he ran from the room looking for help and found the resident electrician. Hans explained the problem, and the bright spark said he could help.
The electrician pressed on the circuit breaker switch and told Hans to plug the rack back in. This is the bit where you might be expecting that scary Halon fire suppression system to play its part. Hans was concerned he may be tasked with cleaning up pieces of exploded finger from the server room.
Thankfully you worried unnecessarily about both fates. The machines powered on, the drives spun up to speed, and a pleasing hum once again filled the air.
Told you not to get your hopes up.
There were no serious repercussions for Hans from the incident, but he did learn an important lesson about chatting with colleagues in the middle of an important job.
Ever allowed your mind to wander for a moment and not liked where it ended up? Tell us all about it in an email to Who, Me? and we'll … oh hi, how are you doing? A cup of tea sounds lovely. No, I'm not doing anything important …