Who, me? Today's tale from The Register's Who, Me? files is a reminder that a momentary loss of focus is all that is required to trigger a potentially catastrophic error.
Our contributor, Regomised as "Sam", regaled us with a story from a mere five years ago when he was still a fresh-faced worker doing time as second-line support at the service desk of a large motorcycle broker.
He had been called into the office on a Sunday to deal with a problem with the systems. "My boss," he said, with a less than fresh-faced weariness, "was nagging me about something another colleague did (as she saw everything as my fault) and she wanted me to restart a training server of the main insurance brokerage platform because the restore had failed."
In order for training to be performed, data had to be restored on a Friday night to get the rates loaded. The eager beavers could learn the ins and outs of the system during their tutorials.
Quality control, Soviet style: Here's another fine message you've gotten me intoREAD MORE
The failed restore had, alas, rendered the training rig borked. Sam had a terminal window connected to it so he could pick away at its internals. He also had one open and connected to the live system.
"My boss had rung me for the eighth time telling me to restart the bloody server," he told us, "even though it was still broken and explaining it to her was pointless."
FINE! He thought. I'll reboot the training server, not that it will make the blindest bit of difference!
Doubtless harder than necessary, he tapped in the required command and struck the enter key.
Almost immediately the alarms began to go off. An alert appeared saying that the brokerage platform had gone down. Which was fine – it just had. The training version, that is.
It then occurred to Sam, as realisation dawned with creeping horror, that the training system wasn't actually monitored. He'd mixed up the windows and managed to reboot the live brokerage system. Worse, he'd managed to reboot it during the Motor Insurance Database (MID) upload process, "which uploads our entire book of live policies to the MID database after deleting every entry."
"It's the database the police use," he added, matter-of-factly.
It got worse. "The reboot meant every client of ours was uninsured for the eight minutes it took to reboot the server."
What had been a simple task to deal with a borked training server (broken by somebody else) had now escalated into a full-on fiasco (entirely of Sam's own doing).
Those eight minutes were likely the longest of his life.
As for consequences, Sam eventually received a telephone call from the MD of the company demanding an explanation. He trudged to the office, fully expecting to be hauled over the coals. A P45 might even have been in the offing. The MD, however, had other ideas.
"He just handed me a pair of glasses instead."
Ever tried to fix one thing, but ended up making something else much, much worse? We've all done the two-window testing/production fandango, but was your boss as understanding when the inevitable happened? Share your moment with an email to Who, Me?. ®