Software developers on Wednesday detected and thwarted a hacker's scheme to submerge a slick backdoor in the next version of the Linux kernel, but security experts say the abortive caper proves that extremely subtle source code tampering is more than just the stuff of paranoid speculation.
The backdoor was a two-line addition to a development copy of the Linux kernel's source code, carefully crafted to look like a harmless error-checking feature added to the wait4() system call - a function that's available to any program running on the computer, and which, roughly, tells the operating system to pause execution of that program until another program has finished its work.
Under casual inspection, the code appears to check if a program calling wait4() is using a particular invalid combination of two flags, and if the user invoking it is the computer's all-powerful root account. If both conditions are true, it aborts the call.
But up close, the code doesn't actually check if the user is root at all. If it sees the flags, it grants the process root privileges, turning wait4() into an instant doorway to complete control of any machine, if the hacker knows the right combinations of flags.
That difference between what the code looks like and what it actually is - that is, between assignment and comparison - is a matter of a single equal sign in the C programming language, making it easy to overlook. If the addition had been detected in a normal code review, the backdoor could even have been mistaken for a programming error - no different from the buffer overflows that wind up in Microsoft products on a routine basis. "It's indistinguishable from an accidental bug," says security consultant Ryan Russell. "So unless you have a reason to be suspicious, and go back and find out if it was legitimately checked in, that's going to be a long trail to follow."
In all, the unknown hacker used exactly the sort of misdirection and semantic trickery that security professionals talk about over beer after a conference, while opining on how clumsy the few discovered source code backdoors have been, and how a real cyber warrior would write one.
"That's the kind of pub talk that you end up having," says BindView security researcher Mark 'Simple Nomad' Loveless. "If you were the NSA, how would you backdoor someone's software? You'd put in the changes subtly. Very subtly."
"Whoever did this knew what they were doing," says Larry McVoy, founder of San Francisco-based BitMover, which hosts the Linux kernel development site that was compromised. "They had to find some flags that could be passed to the system without causing an error, and yet are not normally passed together... There isn't any way that somebody could casually come in, not know about Unix, not know the Linux kernel code, and make this change. Not a chance."
However sophisticated, the hack fell apart Wednesday, when a routine file integrity check told McVoy that someone had manually changed a copy of a kernel source code file that's normally only modified by an automated process, specifically one that pulls the code from BitMover's BitKeeper software collaboration tool and repackages it for the open source CVS system still favored by some developers.
Even then, McVoy didn't initially recognize the change as a backdoor, and he announced to the Linux kernel developers list as a procedural annoyance. Other programmers soon figured out the trick, and by Thursday an investigation into how the development site was compromised was underway, headed by Linux chief Linus Torvalds, according to McVoy.
If BitMover didn't run automated integrity checks, the backdoor could have made it into the official release of version 2.6 of the kernel, and eventually into every up-to-date Linux machine on the Internet. But to get there a kernel developer using CVS would have to have used the modified file as the basis for further development, then submitted it to the main BitKeeper repository through Torvalds.
"If it had gotten out, it could have been really bad, because any Linux kernel that had this in it, anybody who had access to that machine could become root," says McVoy. But even then, he's convinced it wouldn't have lasted long. "If someone started getting root with it, some smart kid would figure out what was going on."
But Loveless says the hack is a glimpse of a more sophisticated computer underground than is normally talked about, and fuel for speculation that backdoors in software products are far more common than imagined. "We've had bad examples of [backdoors], and we've had rumors of extremely good examples," says Loveless. "This is a concrete example of a good one."