Do you come from a land Down Under? Where diesel's low and techies blunder
UPS = Uninterruptable Pager Spying
Who, Me? We go Down Under for today's Who, Me? with a slightly NSFW tale of an incomplete checklist, a surprise outage, and an even more surprised gerbil.
"Bruce", as the somewhat unimaginative Regomiser has dubbed our reader, is today's contributor and tells us of an event that occurred in the 1990s.
He was working for a telecoms firm notable for its paging network. "Used to be able to look at the paging messages in real time in plain text from emergency services," he cheerfully told us before unnecessarily adding: "You'd be amazed by what people can 'accidentally' put up their bumhole."
However, this firm also dealt with the then infant mobile telephone market and "this new-fangled internet thingy (which I was involved in)," he said.
"We had a power cut in the entire North Sydney area," he said. "Lightning strike, bloke with a digger, particularly randy wombats wanting a root? No idea."
- That thing you were utterly sure would never happen? Yeah, well, guess what …
- Thanks, boss. The accidental creation of a lights-out data centre – what a fun surprise
- Big red buttons and very bad language: A primer for life in the IT world
- The future is now, old man: Let the young guns show how to properly cock things up
No worries, though. There was a colossal UPS able to take the load until the generator could kick in. It would keep things ticking over, at least for the emergency services that depended on the network, for around 40 minutes.
The minutes ticked by, and Bruce started hedging his bets by shutting down non-essential services in order to eke out every last bit of juice from the UPS. "Sorry to all those trying to get to alt.sex.gerbils on USENET at the time," he added, also unnecessarily.
Finally the mighty generator kicked in! It coughed, it spluttered... it proceeded to be a wholly useless lump of machinery.
This was bad. It was also unexpected. Maintenance procedures had been followed religiously. Every month the checklist had been completed, and every month the generator had been fired up. One step was, however, missing: "Fill generator with diesel."
And, of course, the final emptying of the tank had just so happened at the worst possible time.
Fortunately for the lives that might have been endangered by the outage, our story has a happy ending.
While Bruce dealt with helpdesk calls from customers of the ISP side of things wondering where their connectivity (or "gerbil pr0n" as Bruce put it, perhaps revealing a bit too much about how much detail could be seen in the logs) had gone, a panicked relay of Jerry cans to the nearest petrol station ensured the generator was coaxed into life with just a few minutes remaining before that critical network fell over.
Checklists and Standard Operating Procedures are great until you discover a missing step at the most inconvenient of times. Ever had to improvise when a SOP let you down? Or were you the author of a process that omitted an all-important instruction? Confess all with an email to Who, Me? ®