Techies tossed appliance that had no power cord, but turned out to power their company
Illicit colo cleanup seemed like a good way to get out of the house during Covid
Who, Me? Welcome to another week of nimble newsifying from The Register, which as always kicks off the working week with a fresh instalment of Who, Me? It's the reader-contributed column in which you admit to mistakes that almost trashed your career.
This week, meet a reader we'll Regomize as "Steve" who sent us a story that took place during the COVID-19 pandemic when he worked at an insurance company.
"My boss and I were good friends and had spent weeks working from home," and experienced the frustrations that created. So when his boss suggested Steve meet him at a colo datacenter to do some work, he jumped at the chance to leave the house, drive for a few hours, and revisit the real world.
The purpose of the trip was to reduce the number of cabinets the insurance company leased from its colo, because Steve's boss was always keen to remove unused equipment.
"We took a quick walk around the cabinets and identified a few servers we knew were defunct," Steve said. "Then we came across a machine we didn't recognize. It was powered down and didn't even have a power cable."
The machine was connected to a couple of unlabeled Ethernet cables, but the lack of a power connection meant Steve and his boss assumed the machine was redundant and ripe for removal.
"We ripped it out, lobbed it into the pile of other junk, and moved onto the next job," Steve told Who, Me?
The orgy of destruction continued for 20 minutes, when Steve's mobile – and his boss's – both lit up.
"It was our director of IT and he wanted to know why the internet was down for the entire company."
Steve and his boss soon learned that the device they'd just removed was an exotic security appliance that, despite being powered down, provided a vital network connection.
- Techie found an error message so rude the CEO of IBM apologized for it
- Intern had no idea what not to do, so nearly mangled a mainframe
- Bored developers accidentally turned their watercooler into a bootleg brewery
- After deleting a web server, I started checking what I typed before hitting 'Enter'
At this point in the story, Steve and his boss suddenly realized their approach to decommissioning tech without documenting what they removed was not best practice, because while they found the appliance in their pile of "dead" tech, they had no idea which ports the cables should connect to.
"We spent the next two hours trying to work out what cable went where. Eventually we got it working again but we were in a whole heap of trouble. And not just because we crashed the internet: Neither of us were allowed to visit the datacenter without approval from the very top, let alone ripping out servers without raising a change."
Steve lost his job over the incident, which as The Register has reported caused quite a crisis for some techies during the bleak days of the pandemic.
Steve says he wasn't too upset at the time, and that he and his former boss can both laugh about it now.
Have you unplugged something that really should have remained in place? Click here to reveal all by sending your story to Who, Me? ®
Biting the hand that feeds IT
