Sysadmin infected bank with 'alien virus' that sucked CPUs dry
No good deed escapes scripting SNAFUs
who, me? Welcome once again to Who Me?, The Register's weekly reader-contributed column in which IT pros share stories of times their work spun off into eccentric orbits and they (mostly) brought them back for soft landings.
This week, meet a reader we'll Regomize as "Alf" who once worked on a team that tended Windows NT on thousands of high-powered workstations at a large investment bank.
Alf's time at the bank coincided with the launch of SETI@home – a desktop program that purported to analyze radio signals collected from outer space in case they contained sign of alien life. ET-hunters collect colossal quantities of data that might conceivably include signals from another species, and in the pre-cloud age boffins came up with the cunning idea of creating a program anyone could download so the analytics could be conducted by PCs around the world with their spare CPU cycles, instead of requiring boffins to buy their own large and expensive hardware.
Alf liked the idea of helping out, and it didn't hurt that SETI@Home ran leaderboards that recorded the world's top alien-radio analyzers.
In addition to the workstations he tended, Alf had access to a development lab packed full of powerful computers. He counted a dozen multi-core servers, a pair of top end multi-CPU workstations for each of the lab's engineers, plus "a room full of other miscellaneous hardware that needed to be supported or was undergoing evaluation for the bank."
"All this compute sat largely idle at night, so being the conscientious chap I am decided to donate all this unused compute to science and downloaded the SETI client and set it to work," Alf confessed.
- After 1.5 million days of computer time, SETI@home heads home to probe potential signs of alien civilizations
- After years of listening, we've heard not a single peep out of any aliens, say boffins. You think you can do better? OK, here's 1PB of signals
- Server broke because it was invisibly designed to break
- Don't lock the datacenter door, said the boss. The builders need access and what could possibly go wrong?
- Cleaner ignored 'do not use tap' sign, destroyed phone systems ... and the entire building
This went well and Alf saw his handle climbing the league table of SETI@Home contributors.
"Not being content with this, and using my skills as a Windows engineer, I created a version of the SETI client that could be executed over the network on multiple workstations simultaneously," he admitted. Before long he had pressed around twenty of the lab workstations and half a dozen servers into service searching for little green personages.
In the first week of that effort, Alf rocketed into the top 100 contributors globally.
It couldn't last, of course.
"One Monday morning about two weeks later I received a phone call from my boss asking if I could come in early as there were reports we had a massive virus attack underway and it was all hands to the deck for engineering," Alf recalled.
Alf arrived in time to hear his boss report that the bank's lab had been infected by a virus that was consuming 100 percent of CPU capacity and had spread to every workstation and server in the departmental lab. If it spread to production systems the bank would be in very, very, deep trouble.
Alf "put on my best innocent-but-concerned face" to – hopefully – mask the fact he felt fairly sure he was the cause of the problem.
Specifically, he wondered why the script he'd created to shut down SETI@Home hadn’t done its job, just as it had every day for the few weeks during which he'd run the software.
"As war rooms were being readied and conference bridges set up I had to act fast to avert a mass panic, so quickly executed the shutdown code on all the lab kit to return everything to normal and disabled it from running again, carefully covering my tracks by removing it from the network."
It worked, Within an hour the war room had stood down and the "virus" was dismissed as an anomaly.
"Given it was limited to our lab and didn't reoccur, nobody followed up with a more in-depth forensic analysis or examined proxy server logs – so I avoided discovery and potential embarrassment or worse, getting fired," Alf explained.
But he couldn't resist an investigation of his own, because he wanted to believe his script was sound. The truth must be out there.
"It turned some cabling work was done over the weekend, which had disconnected all of the lab equipment from the network, including the network share that hosted my script."
The code that shut down SETI@Home therefore never had the chance to execute.
Alf was left lamenting that he'd cracked the top 20 global contributors list, but found no trace of ET … and learned it was perhaps best not to use a program called SETI@Home somewhere other than, you know, home.
But he's still keen on distributed data-crunching – now focusing on efforts that focus on curing cancer and other diseases.
Have you been called out to fix a mess of your own making? If so click here to send email to Who, Me? and we may feature your story in this slot on a future Monday. ®