BOFH 2003: Episode 13
Late, late at night...
Almost morning, in fact....
All lights - bar several system status indication lamps - are off.....
Suddenly a lamp, previously glowing Blue, changes (hereafter) to Orange.......
Seconds later an SNMP trap, delivered by the system concerned is received by the Services Monitoring Facility server...........
Seconds after that, a dial tone, followed by a series of DTMF tones is heard echoing through the machine room. And again...............
Approximately one minute later, in the bedroom of a geekily furnished studio flat in South London, a pager beeps shrilly .................................................
A figure partially emerges from the Fleur-de-Lis bedspread to focus on the pager in question........................................
THE WELL-OILED MACHINE OF TRIPLE-TIME, AFTER-HOURS CALLOUT IS IN MOTION!!!!
. . .
"But I didn't get paged!" The PFY cries in response to The Boss's annoyed questioning as I roll in at 9 the same morning. "Mind you, that could have been because I dropped the pager on the way home and it broke!"
The PFY holds up a pager that, whilst bearing the hallmarks of being repeatedly struck by the heel of a shoe, must have been damaged in the manner he said...
"I did mention the need of some form of backup to the pager in case of situations like this - or pager unreliability due to congested telco networks," I ad lib.
"I'm not buying a bloody satellite phone for you to run up astronomical bills on!" The Boss snaps, cutting my plan off at the pass. "Anyway, this isn't getting the Finance Server up!"
"The Finance Server IS up!" I say, looking on the Services Monitor. "It's just dropped a power supply"
"Then how come I got messaged?" he snaps, flashing us a look at the message on his phone.
"You said you must be notified of any major problems - and you know how I don't like to ask questions."
"That's not major!" he snaps.
"It is if it happens twice!" I reason.
"Look, I only want to be notified when the machine goes down!"
"OK, that's easily sorted out," I respond helpfully.
"And get that power supply fixed!" He snaps.
"Running all the way to the phone!" The PFY blurts, poking in the Hardware Support number.
Three hours later
"I thought I told you I only want to be paged when the server goes down!" the Boss seethes, crashing into Mission Control.
"Oh! The server's gone down, see?" I say, pointing at a red icon on the services display.
"You're joking!" The Boss blurts. "But it's got redundant power supplies!"
"Indeed it has, but they don't work when one's dead and the other is outside the machine..."
"I thought the machines could work without interruption?"
"They can, but unfortunately our engineer can't. The PFY asked him if he wanted a coffee, which must have confused 'LEFT' and 'RIGHT' in his brain. So he pulled the wrong supply out. I was going to check before he did it - but as you know, I don't like to ask questions..."
"RIGHT!" The Boss shouts as he storms off to his office.
Ten minutes later The Boss informs us that the engineer concerned is on his way back to the office and the senior engineer has been dispatched to us.
"..a real bastard," I say to The PFY, finishing my description of the engineer in question. "The sort of person who gives helpful tips to your boss like how the dangling cables you have are a workplace hazard and how he couldn't help noticing that you're using root for day-to-day work - and how insecure that is.."
"Oh yes. An out and out brownnoser. A cleft presser of the First Order. AND he notes all his tips and observations on his report and files it with his office. THEN, next time something happens, they dig through it for a reason to void the maintenance agreement."
"Oh yes. He's the guy they send to punish people. AND, he makes a point of coming in and advising you through what he's doing, every step of the way."
"Arse Covering?" the PFY asks.
"Partly, but mainly to punish you for not issuing him a permanent swipe card to access the room night and day. So he wanders in and out of the machine room to make you keep getting up to let him in."
Our conversation is interrupted by the arrival of the man himself, complete with titanium toolkit.
"So where is it?" he asks.
"In the machine room," I respond.
"OK, can you give me a card to get in and out?"
And so it begins...
Ten minutes later...
"Just about to install the new power supply," he says, once the computer room door has latched shut.
"And could someone let me into the machine room, as I don't have a c.."
"Sure," The PFY says cheerfully
Ten minutes later...
"New power supply installed, I'm just going to box up the old one >click< and fill in my form."
"Oh, could you let me back into the..."
"Sure," The PFY seethes.
. . .
Looking back, I think it was probably the trip back to collect his toolkit that broke The PFY. The repeated trips in and out to let us know he was about to power the system on, check the hot plugability was working, then confirming it was all working, boxing up the old power supply, borrowing a pen to label the old power supply 'faulty', packing up to go, etc, must have slowly eroded his patience.
The phonecall from the Maintenance Company asking if we'd seen their engineer was the first thing I knew about it.
Then there was The PFY's new titanium tool kit.
I don't like to ask questions... ®
BOFH is copyright © 1995-2003, Simon Travaglia. Don't mess with his rights.