On-Call Welcome again to On-Call, El Reg's weekly column that offers readers the chance to vent about their co-workers' ineptitude.
This week, meet "Oscar", who tells us about the time he had to clean up after an overly tidy sysadmin.
At the time, he was working as a consultant and IT architect for a government customer that was moving from the inner city to a new location further out.
Oscar went to the wall above the UPS and did what everyone secretly wants to do – he pulled the red lever with the huge warning sign
It was a wise move, said Oscar, since the old location suffered from all the problems old buildings cause.
"The main server room was tiny and cramped; located in the basement with an unsecured window opening on the street at ground level; the mains water supply passed through in a huge pipe under the ceiling - and there was no backup power."
The new server room, though, was like a breath of fresh air.
"It was huge, white, air conditioned, with generous steel shelves for the servers, a three-phase UPS and overhead wiring in easily accessible metal ducts," Oscar said, most likely in hushed tones.
The three phases, we're told, were spaced out over each row of shelves, with sockets organised and labelled thus: Phase 1, Phase 2, Phase 3, Phase 1, Phase 2, Phase 3, and so on.
"The new system administrator wanted to show off his abilities and entrepreneurship, so without consulting anybody he plugged in the servers on the Monday morning of the changeover," Oscar said.
At first, he said, things seemed to be working well – everything was up and running by 10am and people had started working.
But, given this is On-Call, it will surprise no one that this came to an abrupt halt about an hour later, when all the servers collectively gave up the ghost.
Grad sends warning to manager: Be nice to our kit and it'll be nice to youREAD MORE
"There was the usual commotion of people running aimlessly around with nothing to do," Oscar said, and after about half an hour he took himself off to the server room.
"I was met with the acrid smell of burned-out electronics, the department heads of the institution (including the head of IT) looking quite angry, and the sysadmin scratching his head, looking at the smoke-leaking UPS, mumbling, 'But they said it would have 30 per cent more capacity than we need!'," Oscar recalled.
As the bosses were fuming away – having been assured of a smooth move but now facing all departments downing tools until IT fixed the problem – Oscar spotted the problem.
"I looked at the installation and realised the sysadmin had meticulously wired all the servers to the same phase," Oscar said.
Why? "In his words, 'To make it look neater'." Of course, it had actually had overloaded that phase of the UPS.
"The UPS was correctly sized — provided the load was spread across the three phases," Oscar said.
At this point, Oscar took action, ordering people to move the plugs to spread out the load on the three phases forthwith.
However, he didn't tell them the reason for the urgency, meaning the team were left wondering what the rush was, since the UPS repair team wouldn't arrive for another two hours.
When they'd finished, Oscar went to the wall above the UPS and did what everyone secretly wants to do – he pulled the red lever with the huge warning sign.
"I went to the emergency switch – the lever that everyone thought performed emergency shutdown only – and put the lever in the middle position, which cut the UPS out and powered the servers directly from the mains.
"Within seconds the servers powered up, and in less than 10 minutes everything was working," Oscar said. (For the pedants, Oscar gave this addendum: "I know, I should have staged the power-up so as not to overload the fuses with the inrush current, but hey, it worked.")
Oscar assured us that everyone lived happily ever after: "I was the hero of the day, the sysadmin kept his job, and we developed a nice working relationship."
Have you got a fairytale story of the time you saved a colleague's bacon? Email it over and we might feature it in this column next time. ®