Run a demo on live data? Sure! What could possibly go wrong? Hang on. Are you sure that's not working?
To quote the great philosopher Han Solo: Don't get cocky
who, me? Welcome, dear reader, to another installation of Who, Me? in which we recount for your schadenfreude indulgence tales of Regizens who create havoc – either by their own design or others – but mostly emerge unscathed.
This week's hero, whom we shall Regomize as "Sedgwick," worked as a systems engineer in the R&D department of a major pharmaceutical company. The company was at the time in a fevered race (possibly literally) against a huge competitor, searching for a breakthrough drug to treat a particular ailment.
Got that? Systems essential, downtime bad.
The company had just upgraded its "minicomputer-based system that did simultaneous data capture and reduction for many channels of chemical analysis instrument," and Sedgwick was tasked with training the users in its operation.
The customer had already been using a turnkey version of the system. The upgrade allowed far finer control of and access to the underlying system. The multi-user system had an account for each user, and each account was assigned a numeric privilege level. Each command had at least one required privilege level, and sometimes two. For example, to terminate one of your own programs required a medium privilege level, but to terminate any program in the system required the highest level. As Sedgwick puts it: "With so much power at his fingertips, clearly the system manager needed to be trained, as otherwise havoc might ensue."
Indeed it might, Sedg. Indeed it might.
- The boss worked in a fishbowl, so office tricks were a treat
- Data loss prevention emergency tactic: keep your finger on the power button for the foreseeable future
- Loathsome eighties ladder-climber levelled by a custom DOS prompt
- PC component scavenging queue jumper pulled into line with a screensaver
Once it was all up and running – on live data, we should add – the system manager asked for a demonstration of the privilege system.
So, naturally, Sedgwick created an account for himself with medium-level access that ought not to have been able to terminate many programs on the system. That's only sensible.
And for the demonstration he also created a dummy high-level process that his demo account would be unable to terminate, but of course would have no effect if the demonstration went wrong, yes?
Sedgwick issued the command to kill the main data capture coordinating process – on live data – secure in the knowledge that the system would never allow such a thing.
Unbeknownst to poor, poor Sedgwick, the latest revision of the system had introduced a bug – soon patched – in the command privilege tables.
At this point, we shall assume readers are familiar with the unit of time known as the "ohnosecond", which defines the elapsed time between making an error and realizing you've made an error.
After not much more an a single ohnosecond, the first irate chemists were banging on the windows wondering where their work had gone.
Sure, it was hardly Sedgwick's fault there was a bug in the system. But his confidence that there was no such bug was utterly misplaced. Always assume there's a bug.
In the end, both the company Sedgwick worked for and its competitor were beaten to the breakthrough by a Swedish concern, and ended up merging. Who knows? If not for one fateful killed process …
Have you ever changed the course of history with the press of a button? Prevented (or hastened) a major advance through accidental heroism? Tell us all about in in an email to Who? Me and we may share your tale. ®