C'mon SPARCky, it's just an admin utility update. What could possibly go wrong?
Now, where was I? [tappity tap] ... BALLS!
Who, Me? Hey hey hey, it's Monday! The new week is but a caffeinated beverage away. Come join us in celebrating another Register reader's flirtation with career-ending disaster with a morning dose of Who, Me?
It was the late 1980s, and our contributor, fresh out of university, had inexplicably landed a job as system administrator in a large academic institution.
"Scott", as we're going to call him, "was pretty much in charge of day-to-day tasks on several Sun servers, including," he modestly added, "one of the first SPARC-based systems in UK."
Scalable Processor Architecture (SPARC) was the brainchild of Sun and Fujitsu and a relatively successful RISC system. It was, even by the standards of Sun, exotic hardware back in the late 1980s.
On the day in question, Scott had been upgrading some in-house administration utilities. "This necessitated logging in as root, running some scripts and then tidying up using the
He was a cautious fellow, and had created a "scratch" directory with his normal user ID in which to perform the work. Once the upgrade was done, he would simply clear it down. Sensible.
Scott had just finished the compilation of the source and installed the updates when the phone rang with some pesky user needing a printer problem resolving. Scott had to
su to root to fix the issue before running a
rm -r * in his scratch directory to tidy up after the earlier upgrade.
And tidy up it did, just not quite how Scott had imagined.
"I forgot to exit my root shell so I ran this command from the root directory as the root user."
Let the person who has never forgotten where they were before running something horrifically destructive cast the first stone...
"After mere moments, my spider-sense started tingling and I stared in disbelief and growing horror at my VT220 terminal as the realisation of what I had done hit me and I frantically hit Control-C."
Luckily, Scott was distracted from the bowel-loosening terror within seconds as the phone began to ring off the hook. Hordes of angry academics and students bombarded him with the same question – just what in blue blazes was happening?
Naturally, he took the only avenue available to him. "I called my local Sun Microsystems support line to log a fault with the disk drive." AKA: he told a little lie in order to buy himself some thinking time.
The news that the borkage was a "hardware fault" calmed things down somewhat and Scott was able to start dealing with the smoking wreckage of the file system after managing to get the server booted from a QIC-150 cartridge.
As it turned out, backups had been taken. It took a few sweaty hours to restore everything, and then a few more hours before Scott realised that he also had to reinstall the boot block. But it was done and, impressively, done in time to permit an on-time exit (we imagine to a nearby hostelry to calm shattered nerves).
But there was still the question of the
lie "hardware failure". Not a problem: "I had deliberately disconnected the Ethernet cable because I did not want the server to appear functional until the Sun hardware engineer turned up."
The following morning, with the server still, er, "down", Scott got a call from the inbound Sun engineer asking him to "put the kettle on". Knowing that his ruse would not survive an encounter with an engineer presented with a fully functional server, "I recounted my tale of woe to him apologising profusely for wasting his time."
The piggy bank of favours was cracked open: Scott and the engineer had a pretty good working relationship stemming from Scott's willingness to overlook an occasional miss of the contracted call-out times because, hey, it was academia and "it was rare that a problem could not wait an extra few hours and allow the engineer to get home at a reasonable time and attend to my issue the following day."
Stir in the fact that changing a 1Gb drive back in the day was a two or three-person job, and the engineer was more than happy to play his part in the ruse.
Network attached, "hardware fault" rectified, and Scott was surely the hero of the hour rather than the villain of the piece.
He had another brush with the back-up tapes in the 1990s, but that is another story.
And that is why "I should never have root access..."
Ever felt the stomach-dropping lurch of an unexpected directory delete? This hack has certainly watched in horror as a folder of production databases disappeared into a Netware wasteland. How about you? Send an email with your confession to Who, Me?. ®