Who, me? Welcome again to Who, me? The Register’s Monday column in which readers hang their heads in shame and admit to their past mistakes.
This week, meet “Gladstone” who “many years ago was a contractor working in the Pentagon.”
That Pentagon? Yup. That one. Where the US military does its business.
Gladstone’s work there saw him “support a number of computer centres including the one that supported the Joint Chiefs of Staff and their secure messaging and routing system.”
This all happened a few decades ago so the infrastructure was primitive: a mostly dead minicomputer vendor’s kit sat next to a definitely dead storage company’s kit.
In those far-off days “the drives needed periodic servicing and the heads had to be cleaned and checked. This was made easier by having 100 per cent redundancy for systems and storage so when preventative maintenance was required it would be performed on the offline systems.”
“Simples!” Gladstone told us.
Or it should have been simple, but when Gladstone’s turn came to do this job he’d just come back from a week off, “and during that period it was decided to reconfigure and rearrange the equipment layout.”
Sysadmin unplugged wrong server, ran away, hoped nobody noticedREAD MORE
Gladstone told us that before his break the kit was arranged as follows: computer-drives-computer-drives. But while he was away someone changed the layout to: computer-drives-drives-computer.
“When I came to work that day, it was on the late shift and after checking the work log I saw that drive cleaning was scheduled for one set of drives," our reader explained.
"I got my supplies together, diagnostic pack, oscilloscope, cleaning fluid, cleaning pads, new filters, etc. Walking down on of the middle aisles I came to the drives and thought it odd that someone had left disks in them and up and spinning.”
Gladstone assumed “that it was for testing purposes and, not thinking to double check or take a step back and think for a moment, I spun all the drives down.”
“As they spun down I started to hear line printers spew out alerts and loud cursing from the staff manning the message routing terminals," he said. "Looking back at the drives it quickly dawned on me that I had just nuked one of the most critical systems in the Pentagon.
“I quickly flipped the switches on the drives back up to the on position and scurried around the back end of aisle to the maintenance office and put my head well into the maintenance logs.
“The duty officer arrived and explained that they had just lost the message traffic system. Putting on a surprised look I went to the system, poked around and said that I would have it back up in ten minutes or so. I crashed the system and restarted it and when it came back up I quickly checked all the message files, queues and logs and was able to recover and return the system back to full operation.”
Next came Gladstone’s masterpiece: a report on the crash that omitted his role in the outage and instead recommended an upgrade to the power system. This was not just a diversionary tactic, because Gladstone also knew that a new system was already in the works.
“Later I also updated the maintenance process to make sure that people were looking at the correct offline systems prior to starting work,” Gladstone said. “I received an award for my work. I almost felt guilty.”
And then he signed off with: “Hopefully the Statute of Limitations has expired for saboteurs!”
Have you won praise for a mistake? If so, click here to write to Who, me? and we might just give you a run on a future Monday. ®