More and more, we see articles questioning the security of a given platform based solely on the number of advisories published - and this approach is simply wrong, writes Thierry Carrez, of Gentoo Linux.
In recent news we have seen evaluations of security metrics for various software publishers and operating system families. In many cases, including the infamous Forrester Research report, "Is Linux More Secure Than Windows," the raw data used is the aggregate number of security advisories published. More and more, we see articles questioning the security of a given platform every time multiple advisories are published in a row. This might be useful for the marketing department of a large software company, but for determining real life security this approach is simply wrong.
Advisories depend on more than just software quality
The quality of the software in question directly affects the number of advisories - this should go without saying. Software with fewer security flaws will generate fewer security advisories. In a simple world, the number of advisories published by a given entity might be a direct indication of how secure that software is. But the reality of software and operating systems today is far more complex; many other factors affect the number of security advisories, which can be lumped into three broad categories: scope, policy of publication, and audit efforts. Let's look at each of these briefly.
The overall scope of the project is the most important (and most often overlooked) factor affecting the number of advisories published. Gentoo Linux security advisories, for example, cover more than 8700 software packages that can be installed through the Portage software tree. There will be far more Gentoo Linux Security Advisories than Microsoft Security Advisories (where Microsoft only lists 275 different products and technologies on its website).
An organization's policy of publication is the factor that correlates the number of vulnerabilities found with the number of advisories to be published. For example, some entities will only publish remotely-accessible vulnerabilities and will silently fix the rest; others will regroup multiple products or multiple vulnerabilities into a single advisory. Comparing the number of advisories from entities with different policies of publication, therefore, makes no sense.
Finally, the more you audit a package, the more vulnerabilities you'll find and similarly, the more advisories you'll publish. A good example of this is the Apache web server, which has generated several advisories because it has undergone many security audits. This makes it a very secure web server, not a less secure one.
Impact on security depends on more than just software quality
Security is a process. The vulnerabilities in the software itself adversely affect the security process, but this is just one part of the big picture.
Sometimes it's the way the software is designed that is determined to be the fatal flaw. These "features" or "design choices" can increase the impact of any other flaw without generating advisories. For example, linking together an operating system and network-facing piece of software like a popular web browser or an HTTP server might provide some interesting features, but it's bad for security. Even if the software provided is well-designed and does not have a single security flaw, a bad configuration can still ruin everything. Minimizing the risk of human error by providing secure-by-default configurations and good security-conscious documentation is necessary, but we rarely see security advisories due to software that is not secure-by-default.
Perhaps the most important way software can be improved through the security process is how easy it is to fix identified vulnerabilities, and how long that process takes. Reducing the vulnerability window is becoming more and more important because the time between vulnerability disclosure and exploitation in the wild is quickly decreasing. The size of the vulnerability window depends on three real-life delays:
- Time for the vendor to produce a patch
This is often used to rate security awareness between vendors. It depends on how important security is to a given vendor, of course, but also on the quantity of QA they will do before releasing the fixed package. These QA metrics are not something you really want to become too short because it may create other issues. Fortunately, aside from these issues there are other delays which can be shortened without sacrificing the quality of the patch.
- Time for the user to become informed and to get the patch
The time it takes for the user to get a patch is never accounted for. Most software authors or organizations setup lists and channels to push out information about a vulnerability to the user, so this area cannot be improved much over the current process. However, subscribing to these lists and following the security announcements of all the software installed on all your systems is a very difficult, painful, and time consuming task - the result of which is that some vulnerabilities might still go unnoticed for a long period of time.
- Time for the user to apply the patch
An easy and painless security patching process will also reduce the vulnerability window. Painful upgrades, like those requiring a reboot or those that that affect multiple packages and multiple incarnations of the same code (like the MS04-028 JPEG vulnerability) face the risk of being postponed or only partially applied in an organization. And as always, patches to production servers require time to be tested in a non-production environment prior to being rolled out.
Summing up these three points, the number of advisories an organization publishes depends only partially on the number of flaws present in the software. This number is only a small part of the global security process. We could, for example, compare publication policies between various advisory publishers (by considering a common corpus of software packages), but it's now obvious that we can't really infer anything about security based solely on a comparison of the quantity of advisories produced.
More advisories, more security
Security advisories from a software publisher or packager should not be seen as bad news. There are always vulnerabilities in software, and when an advisory is released it means that one of these flaws has been identified and fixed. It also means the good guys have done their homework, and that one less flaw can be used by the bad guys to harm you.
Sure, if the patching process takes too long, the vulnerability window will benefit the script kiddies and other miscreants of the world. That's why everything possible must be done to ensure that this patching window is kept to a minimum - and here is where the choice of software or operating system used by your organization may become an advantage. Operating systems that use compartmentalized software packages and common libraries will, in fact, make patching a lot less painful. Operating systems with a strong separation between kernel space and userland space will also help to reduce delays and downtime.
The most important part of the security process is to hear about the new vulnerability as soon as possible. And here, the approach used by the various GNU/Linux distributions (especially those that offer security maintenance over a very large software repository, like Debian and Gentoo) is invaluable - you can have a single source of security information and a single security patching procedure for all your software. There is no point in using complex patch management systems that cover only half of the software products you actually use. If you run Shoutcast Server on Windows, for example, the only way to be informed that it contains a critical remote hole is to be watch various the various security lists, such as Bugtraq and the SecurityFocus Vulnerabilities Database, or else follow NullSoft's web pages directly. If you run Shoutcast Server on Gentoo Linux, however, you'll get a GLSA about it just as you would for any serious vulnerability in any of the 8700 packages you install from the Gentoo Portage tree.
This sort of advisory truly makes you more secure, not less secure.