Opinion We're going to be cleaning up Apache Log4j security problems for months to come, but the real problem isn't that it was open-source software. It's how we track and use open-source code.
When security vulnerabilities were found in the extremely popular open-source Apache Log4j logging library, we knew we were in trouble. What we didn't know was just how much trouble we were in. We know now. Just ask the Belgian defence ministry. In this ongoing security disaster, many people blame open source for all our troubles.
In the Financial Times (FT), Richard Waters, the newspaper's west coast editor, wrung his hands, saying it's a "little alarming to discover that, more than two decades into the open-source era, glaring security holes sometimes surprise even the experts."
Surprising? I think not. It's software. It always has bugs. Sometimes they're really bad bugs. As security maven Bruce Schneier said over 20 years ago: "Security is a process, not a product." There's no surprise here.
Waters went on: "If an orphan software project like this could be sitting in the heart of the world's internet infrastructure, how many other potential time-bombs are out there?"
Orphan? A major Java library such as Log4j? I think not.
Now, there are vital open-source projects that are orphans. We all know the xkcd cartoon about the tiny but all-important program thanklessly maintained by a person in Nebraska since 2003. The serious part of the joke is that it's not far wrong. Remember OpenSSL's Heartbleed fiasco?
Today, there are fewer such programs. That's because shortly after Heartbleed bled out, the Linux Foundation and mates started the Core Infrastructure Initiative (CII). Its job, and its successor's, the Open Source Security Foundation (OpenSSF), is to find those little under-supported projects and make sure they get the help they need to keep the lights on and the code safe.
But, repeat after me, "security is a process, not a product." Linus's law, as Eric S Raymond phrased it in his seminal work on open source The Cathedral and the Bazaar, "given enough eyeballs, all bugs are shallow" does work. If, and it's a big if, those eyeballs are there and looking. If the code just sits there getting copied over and over again without a moment's thought, no bugs will be found. Simple, isn't it?
- You better have patched those Log4j holes or we'll see what a judge has to say – FTC
- Alibaba Cloud slapped by Chinese ministry for mishandling Log4j
- Belgian defence ministry admits attackers accessed its computer network by exploiting Log4j vulnerability
- Log4j and Omicron: Brothers in harm, mothers of invention
Now some people say that the problem is not enough money. As the programmer Xe argues, "'Open Source' is broken" because even now no one is paying the developers. Xe's not wrong. Ralph Goers, the Log4j maintainer who made the initial fix, confessed he works on Log4j in his spare time and has "always dreamed of working on open source full time." As Xe also remarked, "GitHub stars famously cannot be used to pay rent."
So far, so right. But would Goers get paid to go over old Java code with a fine-tooth comb looking for security vulnerabilities even if Oracle were to hire him just to work on Java? I doubt it. Coders are paid to make new code, not fix old code. That's just how things work whether your programs are proprietary or open source.
And if anyone ever tells you proprietary code is safer, ask them about Patch Tuesday. Microsoft Exchange still blew up on New Year's Day because of a Y2K-style problem with a 32-bit integer variable that couldn't handle the new year.
Until the day comes when companies pay developers to fix and clean their old code while looking for security bugs, we will always have this kind of problem pop up. I expect that to be the same day when companies finally make and check their backups reliably and Jane and Joe stop using "password" for their password.
That said, the real reason why Log4j has proven to be such a pain in the ass isn't the code. I mean, we've now had four, count 'em, four Log4j patches. As I write this, if you want to be safe you should be using Log4j 2.17.1. But the real trick, my friend, is making sure you've replaced all those instances of Log4j 2, which aren't so safe. There's the rub.
You see, Java hides its source code and binaries in numerous Java Archive (JAR) variations. There is honestly no telling where a vulnerable Log4j library might be hiding. The only thing you can do is use a variety of tools to help you win this game of high-tech security hide-and-seek. Oh, one problem. None of these security-scanning programs, not one, can find every possible case. Is it a great time to be working in IT or what?
The answer to this are Software Bills of Material (SBOM). Well done, an SBOM does just what it says. It tells you exactly what software libraries, routines, and other code are used in your program.
As David A Wheeler, the Linux Foundation's director of Open Source Supply Chain Security, has explained, with SBOMs and verified reproducible builds, you can make sure you know what's what in your programs. That way, when – not if – a security hole is found in a component, you can simply patch it rather than search like a madman for the problem code before being able to fix it.
SBOMs, however, are still a work in progress. If we manage to have reliable SBOMs by the end of the 2020s, I'll be a happy man. That will be a pleasant surprise. Oh, and if we can actually pay people to search for trouble in code before things go wildly askew that would be great too. But I'm not holding my breath on that one. ®