This article is more than 1 year old
Programming error created billion-dollar mistake that made the coder ... a hero?
No crypto needed, just a project with a tough deadline that nobody minded missing
who, me? Ah, dear readers, welcome once again to Who, Me? in which Reg readers confess the times their reach exceeded their grasp, technology-wise-speaking.
This time meet "Francis" who was a bright young thing working on contract as a programmer to a largish organisation that was in the process of upgrading certain systems. Despite his youth, Francis had built something of a reputation for himself as someone who could "pull a rabbit out of a hat" when needed. As such, he found himself quite central to the upgrade project.
One of the core parts of the upgrade – the "crown jewel" as Francis put it – was migrating the accounts to the new system. This involved a script that ran the past eight years worth of nightly transaction files on the new system until they caught up to the present.
The script had already gone through a number of iterations and time was getting tight towards the end of the year, so further delays were to be avoided.
Francis admits he wasn't intimately familiar with the scripting language used, but he had enough knowledge to be dangerous. Examining the script for bugs, he says, "it was obvious that some idiot had put an add
where there should have been an add-update
."
- How not to test a new system: push a button and wait to see what happens
- Job 1: Get the boss on the network. Job 2: Figure out why Job 1 broke the network for everyone else
- Just follow the instructions … no wait, not that instruction to lock everyone out of everything
- Run a demo on live data? Sure! What could possibly go wrong? Hang on. Are you sure that's not working?
Naturally, he fixed that to avoid further issues, and the migration continued. All seemed to be running smoothly, and the script ran on, night after night, week after week, over the Christmas and New Year period.
Until a puzzled accountant wandered in with a most confusing report. According to the printout he was reading, the company had quite suddenly accrued multiple billions of dollars for which there was no accounting.
Francis examined the printout, and the problem became obvious.
"Some idiot," he writes, "had put an add-update
where there obviously should have been an add
."
Of course these were the days before strict version control, so it was quite impossible for anyone to track down who the idiot responsible could have been. Just one of those unknowable mysteries, you know?
Francis corrected the script, then explained the situation (in very broad and non-specific strokes) to the CIO who, far from being angry, seemed relieved to have an excuse to slow down the migration. He felt the whole upgrade project had been moving too fast but hadn't been able to convince the higher-ups to tap the brakes. This "disaster" was exactly what he needed.
Yet again, Francis had pulled a rabbit out of a hat, and his reputation emerged enhanced. He did, however, come out of it just a tad more humble.
How about you? Ever had a moment of ineptitude accidentally turn into eptitude? Or learned an important lesson about having your confidence exceed your competence? Tell us all about it in an email to Who, Me? ®