This article is more than 1 year old

Hidden Linux kernel security fixes spotted before release – by using developer chatter as a side channel

Data mining of code commits and chat gives hackers a cunning edge

Boffins affiliated with BMW, Siemens, and two German universities say they can pinpoint obfuscated Linux kernel security fixes, developed in secret, before they are officially released. This is insight miscreants could use to develop and deploy exploit code before patches are widely available.

What's more, the team found that Linux kernel patches are regularly introduced in a way that bypasses public review and discussion, a practice that opens at least a theoretical risk of backdoored code.

In an ArXiv-distributed paper [PDF] titled, "The Sound of Silence: Mining Security Vulnerabilities from Secret Integration Channels in Open-Source Projects," Ralf Ramsauer (University of Applied Sciences Regensburg), Lukas Bulwahn (BMW), Daniel Lohmann (University of Hanover), and Wolfgang Mauerer (University of Applied Sciences Regensburg/Siemens) outline a data-mining scheme that amounts to a side-channel attack on the open-source vulnerability disclosure process.

They describe the patch process to address the 2018 Level 1 Terminal Fault (L1TF) vulnerability, a side channel speculative execution attack that allowed data to be obtained from virtual machines on Intel-based servers.

"Unlike ordinary patches, these patches were — for obvious reasons — not discussed and developed on one of Linux’s public communication channels (i.e., mailing lists) beforehand," the paper says. "However, the fact that a patch was not publicly discussed betrays it: we will show that it is possible to detect such patches as soon as they enter a public repository."

The CVE entries for the L1TF flaw were filed in December 2017 and the vulnerability disclosure didn't happen until August 14, 2018, the paper explains. And Debian 9.2 didn't offer a patch until five days after that. So an attacker with foreknowledge of the flaw could have months to design and deploy exploit code.

Makes sense

What the German boffins have done is not that different from the process that led The Register on January 2, 2018, to spot the work being done to patch Linux and Windows against the Spectre and Meltdown processor vulnerabilities prior to the planned disclosure date.

While we learned of the code repair work by recognizing from online chatter and patch discussions suspiciously stripped of detail that something was amiss, the German researchers show that systematic data mining of public Linux mailing list messages and code commits may be even more effective at ferreting out undisclosed flaws.


We spent way too long on this Microsoft, Intel, Adobe, SAP, Red Hat Patch Tuesday article. Just click on it, pretend to read it, apply updates


In essence, gaps between Linux kernel code commits and the mailing list public discussions point to sensitive code patches that reveal security issues before they've been announced.

The boffins applied their technique to the seven month period prior to the release of Linux 5.4, released on November 24, 2019. "We find 29 commits that address 12 vulnerabilities," the paper says. "For these vulnerabilities, our approach provides a temporal advantage of 2 to 179 days to design exploits before public disclosure takes place, and fixes are rolled out."

The paper's authors contacted the commit authors to confirm that the changes addressed undisclosed flaws. And they note that they found patches for "an easy to exploit denial-of-service attack for ARM64-based Cavium systems" that took two months to be integrated into Ubuntu Bionic and remains unfixed in Debian Buster and the 4.19 Linux LTS tree.

Just can't commit

The research also touches on another issue of potential concern: code commits made without review or public discussion. Linus Torvalds, the authors say, made 40 such commits during the May-December 2019 research period, none of which had security implications.

And other trusted maintainers did too, mainly inconsequential stylistic fixes. Nonetheless, the possibility that untrustworthy code could get slipped into the Linux kernel through this process gap troubles the paper's authors.

"The existence of such channels, shows that trusted individuals can easily infiltrate the project, and secretly introduce malicious artifacts (while this possibility is given, our method allows for finding concrete instances, which is otherwise not possible)," the paper says. "The existence of such commits contradicts one of the key promises of an open development model."

The Register asked the Linux Foundation to comment. We've not heard back.

The boffins say in their paper that the Linux kernel community is aware of the issues they raised but has yet to settle on how to deal with them. They also observe that while their analysis focused on Linux kernel development, their technique can be generalized to other system-level open source projects like GCC, QEMU, U-Boot, LLVM, busybox, and others. ®

More about


Send us news

Other stories you might like