On-Call Welcome again to On-Call, El Reg's Friday foray into readers' reminiscences of things that went wrong in the night.
This week, meet “James” who once worked “for a company that provided accounting systems for Las Vegas casinos.”
As James reminds us, “Casinos, or 'gaming' as it's called in Nevada, are a 24/7 operation and never close, so support is also on a 24/7 basis. Holidays are the busiest work days.”
Which meant that not many hours after one year had drizzled into another, James received “an urgent call from one of the largest hotels in town” because the cashier system in the casino cage had stopped on all stations.
“This was a critical system since it tracked the flow of cash and chips to and from the gaming tables. State regulators take a dim view of any problems that involve accounting for cash, meaning it had to be fixed as soon as possible or the hotel might be fined.”
Which is why James' working year started about as early as is possible, his mind abuzz with theories about why a system that had proven reliable and had robust backup would chose early on New Year's day to fail.
“The moment the guard lets me in to the casino cage I saw the problem,” James wrote to The Register, asking us to remember the date – Jan 1st – to consider that casinos are cash businesses and to note that in the year of this incident New Year's Day was a Sunday.
That combination meant that “The previous night the casino had taken in so much currency, in the tens of millions, that the vault had filled up and there was literally no place to store the cash. Any other day the banks would send armoured cars to pick it up, but not on Sundays.”
“It turns out the night before had wildly exceeded all expectations. So the casino cage was full of bags of money. So many that the vents for the computers had been blocked, causing overheating and shutdown.”
“The solution? For almost and hour I stacked large bags of $20 bills in one corner of the cashier cage, until the vents were clear. After cooling down the system came back up and the million dollar crash was resolved.”
What have you had to move to get kit working again? Share your story and you may end up in a future edition of On-Call. ®