Who, Me? A fresh week means a fresh story to add to The Register's regular hall of shame where hapless techies tell tales of in-the-field slip-ups: welcome to Who, Me?
Today's near-calamity comes courtesy of a reader we shall refer to as "Dan", who was working on developing a card payments platform for "a well-known telco".
If servers go down but no one hears them, did they really fail? Think about it over lunchREAD MORE
Dan modestly told us it was "quite a hefty job for a fresh-faced graduate" and explained that prior to his efforts, "if you phoned up to pay your bill, the details were written down and faxed to a central group of people who took the payment manually with a card machine, and updated the account."
Ah, those were the days.
Dan was a productive so and so and told us that: "After three months, we'd got a web application up and running, where payments would be authorised immediately, but settled overnight." Customer accounts would be updated periodically via the magic of a batch file.
We've all been there: "What started out as a 'trial', was quickly expanded to all call centres," despite the thing running off a machine sat on Dan's desk.
It wasn't long before the inevitable happened. With the all-important lunchtime looming, Dan figured that clearing down the development database would be a good idea, ready for testing the next update.
"I simply typed in the usual
DELETE FROM PAYMENTS and hit run."
Keen to get to his noon appointment with grub, Dan waited for the job to complete. And waited. And waited some more.
How long could it take to wipe the 50 or so records in the development database?
The more astute will, of course, know what had happened.
"That sinking feeling suddenly kicked in," said Dan, "as I realised that I was connected to the trial box, and some 30,000 transactions were being deleted."
Waving away suggestions of popping out of the office for lunch, Dan sunk further and further into his chair as he waited for what seemed like an eternity as the command did its horrifying work.
Once done, Dan told us: "I quickly hit the roll-back button. After another eternity (about two minutes) it completed and all transactions were restored."
Not wishing to spend another moment near the trial data, Dan scuttled off to lunch. Upon his return, he found "there were a number of e-mails and phone messages complaining that the system was running very slowly".
Dan managed to brush off all bar one with: "There was an issue, and it's now resolved." One senior manager dug a little deeper and asked what had actually happened. Dan replied: "There was a loose nut on the keyboard."
"The conversation was left there."
Ever found yourself running some code in a place you really shouldn't? Or seen a short-term bodge turn into a long-term solution? Of course you have – clean the slate with a Who, Me? wet wipe or two. ®