Sysadmin trained his offshore replacements, sat back, watched ex-employer's world burn

'Our motivation for such a task wasn't exactly high'

On-Call Why look at the calendar – it's Friday! Which can mean only one thing, namely the return of On-Call, your weekly instalment of tech support drama from El Reg's dear readers.

This week, "Luke" tells us of the time he was able to wallow in schadenfreude after being laid off in favour of outsourced tech support.

Luke's story started in third-line support team for a substantial publishing company that decided maintaining a UK support team was too expensive.

"The whole support caboodle was outsourced to an offshore support firm," he said, adding that he and almost all of his team were indeed laid off – but not before being asked to do one last job.

"We had the delightful task of training a load of this company's support staff in how to do our jobs, while at risk of redundancy ourselves," he recalled. "It's fair to say that our motivation for such a task wasn't exactly high."

One such task was a long-running data centre power test, which Luke described as "large-scale PAT testing" done slowly at each of the company's many global data centre locations, one at a time.

"Our job was to fail over all active tasks to another site's kit before powering down the data centre so the sparks could do their magic," he said.

Of course, once Luke's team had been deemed surplus to requirements, the next round of tests fell to the replacement team working offshore. And word of their exploits quickly reached our former staffer.

"Alas, either due to our lacklustre training, or a fat-fingered operative, an unfortunate sysadmin managed to issue a simultaneous, and instantaneous, shutdown to every data centre in every location around the world," Luke said, with perhaps just a hint of a wry smile.

Despite not being there to witness the fallout, Luke could only imagine the scenes.

"With the rug being pulled from under every database and service simultaneously there was much wailing and gnashing of teeth immediately," he said.

"It's probably not much of an exaggeration to mention the Star Wars line: 'I felt a great disturbance in the Force, as if millions of voices suddenly cried out in terror, and were suddenly silenced. I fear something terrible has happened.'"

Luke's reliable insiders told him that "it took several days to recover databases from a 'dirty shutdown' state and get everything back up and running".

Some back-of-the-envelope maths suggested that the incident may have done some damage to his former employer's accounts.

"I can only guess at the cost but I'd wager a hefty sum that it was six figures," he said. "Enough to put a big dent in the 'savings' from outsourcing an experienced team."

Have you ever had the chance to cackle gleefully at your former employer's misfortunes? Or cost them a six-figure sum with a dirty shutdown? Tell us your story and perhaps we'll feature it on a future Friday. ®