Pizza and beer night out the window, hours trying to sort issue, then a fresh pair of eyes says 'See, the problem is...'
Cold comfort for manager at the top of the escalation chain
On Call The second On Call of 2021 is a warning not to mix pizza and hashes.
Today's story comes from "Jon", who describes himself as "a manager of a perpetually understaffed IT team." As many will be all too familiar, this put him directly in the escalation path for pretty much any issue that could be daubed with the "it's IT's fault" brush.
On the night in question, Jon was preparing for an evening of pizza and beer with his spouse and friends. Of course, it being a precious Saturday, his work phone rang (much, we imagine, to the unalloyed joy of those around him).
"On the other end," he explained, "was one of my team members, who had become stumped by an issue with our VMware environment and was calling me for advice."
He logged in, looked around the environment, and found that authentication into vCenter "was completely b0rked."
I built a shed once. How hard can a data centre be?READ MORE
vCenter is a key part of a VMware setup. Administrators use the utility to manage virtual machines and ESXi hosts. It being broken would represent a bad day for all concerned.
Noting that his team member was in a later time zone, Jon did the decent thing and let him off the hook, taking responsibility for the issue himself. After all, how hard could it be?
As it turned out, very hard. The hours ticked by and Jon was abandoned as spouse and pals set off to enjoy the pizza and beer he'd been looking forward to. He bit the bullet and opened up a Priority One case with the vendor.
"As tradition demands," he sighed, "the support agent instructed me to retry all the steps I had already tried, to no avail…"
Jon and his new friend began to pick through the configuration with a fine-toothed comb in search of what had upset vCenter so, but turned up nothing.
It took several escalations within the vendor's support organisation before Jon found himself speaking to a new technical support person.
A fresh pair of eyes spotted the problem all too quickly and a configuration file was flashed up.
"What's the problem?" asked Jon.
"See all these hashtags?" the tech patiently explained.
"Hashtags? What the...?"
The penny dropped. "Someone," Jon explained, "had commented out (with the # sign, natch) significant chunks of the file, causing authentication to break when the Virtual Center service restarted as part of routine weekend maintenance."
It was the work of minutes to revert the change, and Jon's organisation was up and running.
Later, he put the word out to his team and a shamefaced colleague stepped forward to confess to the deed.
"He and I had a nice little review of our procedures in regard to modifying critical infrastructure systems."
Doubtless with the assistance of a baseball bat peppered with rusty nails.
And what of his evening? Jon has short shrift for those who gloat over accrued overtime: "Managers don't get overtime, we get one cold slice of leftover pizza."
We bet the shiny job title and embossed business cards make up for it, though.
Ever struggled to solve a problem, only to find a smug pair of fresh eyes achieved in a second what you failed to do for hours? Or swerved some pineapple-topped pizza by pretending the On Call phone was ringing? We've done at least one of those things. Have you? Share your tale with an email to On Call. ®