Two signs in the comms cabinet said 'Do not unplug'. Guess what happened
No amount of resilience planning can defeat determined idiots whose devices are low on battery
On Call Welcome once more to On-Call, The Register's weekly reader-contributed column that tells tales of IT pros being asked to fix things that should never have broken.
This week, meet "Bob" who shared a story from his time as manager of first line IT support for a large retail chain.
Said retailer installed a rack full of kit in each store. Bob proudly described to The Register that those racks were properly resilient: a pair of firewalls and two sets of switches connected to primary and secondary circuits.
The system was, if not bulletproof, then certainly designed to keep going under most circumstances.
Of course, it didn't.
"One day, we got a call from the managed network provider," Bob told On-Call.
"Every piece of kit in one of our biggest stores had gone down at the same time."
Bob called this a "holy sh*t moment".
Staff from the store called to say everything was broken. Bob chatted to the manager and told him to inspect the comms cabinets.
"There was a few minutes of silence once he got there," Bob recalled, followed by the exclamation: "Why the f**k has someone unplugged a comms cabinet to charge an iPad?"
About a minute later, the manager reported that all was well.
"The issue should now be sorted, and I will take steps to make sure it doesn't happen again," the manager said.
Bob's remote monitoring tools confirmed that all was indeed well.
- Boss broke servers with a careless bit of keyboarding, leaving techies to sort it out late on a Sunday
- IT manager's 'think outside the box' edict was, for once, not (only) a revolting cliché
- Go ahead, be rude. You don't know it now, but it will cost you $350,000
- No, I will not pay the bill. Why? Because we pay you to fix things, not break them
At this point, On-Call must confess to having left out an element of the resilience regime of which Bob was so proud: the plug connecting the comms cabinets to the electricity supply were labelled "Do not unplug". And the socket from which the cabinet drew power also had a label that read "Do not remove plug".
"The incident really highlights that all the (physical) signs in the world sometimes can't stop human stupidity," Bob wrote in his missive to On-Call.
Bob never found out if the iPad owner was fired, and still can't quite believe they turned off a network to charge a device.
What instructions have your users ignored? What damage did they do as a result? Share your story by clicking here to send On-Call an email and we may tell your tale here on a future Friday. ®