When you're On Call, only you can hear the silence of the clicks

Automated alerting? Escalating robo-calling? All ok, but beware the inebriated meatbag in the system

On Call It is Friday the somethingth of Marchtober. No, we're not sure anymore either. Still, even in these troubled times there remains a crumb of comfort to be gleaned from the oopsies of others. Welcome to On Call.

Cast your mind back to The Before Times. The mid 2000s, to be precise, where our protagonist, "Bob", was gainfully employed within the walls of a major financial institution (which must remain nameless for Reasons.)

Bob was at the human end of a lengthy incident response chain. Well-designed systems monitored the applications in this well-known institution and, if something happened that didn't match expectations, an alert was generated.

This alert would normally fire off a SMS to whoever was unlucky enough to be carrying the on-call phone. SLAs dictated an acknowledgement within 15 minutes, and failure to do so would provoke robo-calls. Continued failure to respond would see those robo-calls redirected to the bosses, something which would likely prove somewhat "career-altering," as Bob put it.

On the night in question, Bob had the dreaded phone.

A transaction being conducted in Japanese Yen managed to burst through the maximum allowable value in the system. This was not particularly uncommon "due to a combination of the exchange rate and the size of the transactions" with which the finance house was entrusted.

Alas, perhaps because Bob had imbibed a tad too many adult beverages that evening or perhaps because he was just a really sound sleeper, the chirping of the "help me" SMS did not wake him up. It took the robo-calls to rouse him and realise that he needed to do something or face the wrath of the bosses.

The borkage was easily identified, and Bob dutifully followed SOP: split the transaction into two separate trades with values that would not blow through the ranges. Blearily, he checked his work and then hit the "restart batch" button.

Nothing happened.

He blinked, admitting later that he might have been a "bit fuzzy" having been so rudely yanked from dreamland, and theorised that he'd simply missed the button. Again, he clicked "restart batch"

This time "batch restarting" appeared, signifying all was well and Bob went back to bed, a job well done and a pay packet due to be inflated by an overnight call-out.

Bob's disturbance payment was but a trifle compared to the mayhem he had accidentally unleashed, which only became clear when the figures were totted up at the end of the month.

It transpired that the first attempt to restart the batch had actually worked. "A glitch in the UI meant that the confirmation message wasn't displayed," he claimed.

Uh huh. Right.

His second click of the button had also worked.

"The outbound payment batch process ran twice and the system happily paid out over $100 million to various financial institutions, high-wealth individuals, etc. - twice."

After much gnashing of teeth, wailing, and general keyboard bothering from the bean counters, Bob was eventually invited to explain how he'd managed to overpay $100m.

"It's not my fault," he protested, pointing out that running such critical processes in parallel was a bad idea and the screen didn't provide a response the first time round.

He failed to mention he also had a good few units of Stella Artois's finest still coursing through his veins. Probably irrelevant, right?

The unfortunate account managers were set the task of clawing back all the money, which they mostly managed to do. There was the small matter of $5m paid to a bank that had collapsed in the meantime, but for an outfit the size of this financial institution, such an amount was little more than a rounding error.

And Bob? He wasn't fired, but found himself "firmly at the wrong end of the bell-curve when the performance reviews came around." No pay-rise or bonus for Bob.

Nowadays Bob can be found toiling away in a huge infrastructure provider.

He doesn't do on-call any more.

Ever indulged in a bit of bad behaviour while entrusted with the On Call phone and gotten away with it? Or got caught out? Share your million dollar mistakes with an email to On Call.