HPC blog This is a difficult article to write, and I’ve put it off for way too long. But it’s time to bite the bullet and make an embarrassing admission to the Register audience. I’ve been hacked and hacked hard.
Admitting this publicly to Reg readers is like chumming shark-infested waters with my own blood. Or like telling people that I have a lice infection: sure, some will say that being infested with lice doesn’t have anything to do with your personal hygiene, but who believes that?
When it comes to computing hygiene, I think I run a fairly tight ship. Every system on our little network is up to date when it comes to operating systems (Windows 7) and applications – including updates and patches.
Every system also has a full, and up-to-date, security suite which includes antivirus, anti-malware, anti-spyware, and anti-everything else. Our network is protected by our ISP’s firewall and security infrastructure plus our own name-brand hardware firewall.
I’m also pretty careful about what I download and install on any of these systems. I only download apps that I’ve checked out beforehand, from locations I trust.
Even with all that…
I got hacked. I first noticed it just after Christmas. My main business system suddenly started playing random audio. It sounded like snippets from radio or TV broadcasts, including short news blurbs, commercials, and bits of music.
I figured the problem was something associated with my browsers. I typically have a couple up at any given time, all with multiple open tabs. Sometimes embedded videos, commercials, etc, will play when the sites automatically refresh. Restarting the browsers seemed to do the trick… at first.
But it came back. Even after locking down the media options for webpages. Then it started happening even when I didn’t have a browser open – or any applications at all. Oh-oh.
I did my routine checks: full system scans (no problems noted), looking at the processes and services that were running (nothing unusual), and paring back programs that started on boot to see if that made a difference.
I also took a hard look at any programs I had installed prior to the start of the problem, uninstalling them and deleting all references to them just in case. But I just wasn’t able to find the offending program or process.
Then it was on to Google to see if anyone else reported this problem and how they fixed it. There were a few examples, and all signs seemed to point to some flavour of rootkit, which made my blood run cold.
My Problem metastasises
My first move was to check my other systems to see if they exhibited the same behaviour. I found that my laptop had the same problem as my main business system, which could mean that I'd picked it up on the road.
I immediately stopped using email and stopped using these systems to access our shared NAS. I didn’t want this thing to spread any further. And I certainly didn’t want to become a carrier by spreading it to clients or business contact via emails, documents, or any other befouled form of communication from my sick machines.
Attempted cures fall short
I tried most, if not all, of the suggested procedures that worked for other users. I downloaded the recommended rootkit detectors and put them through their paces. None of them found my rootkit or detected the problem I was dealing with.
(I’m not naming names here. I don’t want to slime them just because they couldn’t find my particular rootkit. I have reason to believe that my infection was somewhat unique at the time, or at least rare, and you can’t expect any tool to be 100 per cent accurate all the time. But rest assured that I tried the most prevalent and best-reviewed tools out there.)
Bringing in the big guns
All of the above steps took time, and I was falling farther and farther behind on work while trying to fix it. I finally threw in the towel and brought in the professionals. All of the major security firms have subscription or one-time services that almost guarantee a fix for your system. They use a remote agent to take control of your PC and apply hardcore tech antibiotics.
I cut a deal with one of the largest and most respected security firms, signing up for around $100 bucks to start up a multi-system subscription. Since I could cancel the subscription after the first month, I figured it was a good deal if the virus had spread to other systems in our little infrastructure.
They got to work the next day. I booted my system and stood by while they loaded tools, ran scans, etc. I could watch what they were doing and noticed that it all looked pretty familiar – full system scan, looking at processes, running rootkit detectors, etc. I finally got bored and wandered off to work on my goal of doubling my body weight in less than six months.
After they logged off, I fired up the system and was greeted 10 minutes later with exactly the same random audio.
The next day was the same routine, but with second-line support – smarter guys, I guess. I booted up, they played around for a few hours, and then shut it down. I reboot, find the problem is still there, and call them back. The second-line tech confessed that he couldn’t find a virus or anything out of the ordinary on my box.
It was time to bring in the A Team – third-line technicians. Guys who have Jedi-like powers over all things virus-related. They’d beat the hell out of the virus, and for no extra charge, find the guy who wrote it and kick his ass for me. But that would probably have to happen tomorrow or the next day, because they were busy.
After some bureaucratic snafus, I finally got to talk to third-line tech guy. His voice had that world-weariness that comes from having seen too much darkness in too short a time. We talked about Trojans, rootkits, data hostage schemes, and other disasters. He was definitely the right guy to tackle my problem.
He did say one thing that was very disquieting: “Dan, I do have to let you know something. I’ve seen these symptoms once before, and I wasn’t able to fix or save the system. So be prepared for that, just in case.”
Huh? Wasn’t able to save the system? What the hell? How well could this thing be hiding itself? And how did it get in?
Turns out that there are things that even a Jedi can’t do, and these include identifying and fixing some rootkits. Even figuring out how the rootkit got in can be beyond their powers because, according to the Jedi, rootkits can be piggybacked onto normal software updates and the like. Once inside, they burrow deeply into the morass of files, often disguising themselves as something innocuous.
After delivering that warning, he got to work and promised to call me when he had any news. After a few hours, I finally went to check on the system and found an open Notepad session on my desktop. It read:
“As I mentioned, this is a brand new infection that not even our most advanced tools will clean. I would recommend using a previous image as we discussed to put the pc back a few days. Please give us a call if there is anything further we can assist with.”
And that was that.