Analysis "On the afternoon of Tuesday, September 25, our engineering team discovered a security issue affecting almost 50 million accounts," said Facebook's Guy Rosen in a security update in September.
The issue was serious. What was stolen was not passwords but access tokens, an opaque string that identifies a user and grants access to an API, a software interface used by apps to access Facebook on behalf of users.
Having the tokens enabled attackers to log in as other users. Worse still, it was also possible for the miscreants to log in on other sites using Facebook authentication, though it is not known if this happened, nor who was behind the attacks.
The cause of this security disaster was a programming blunder, or rather, a "combination of three bugs" according to Pedro Canahuati, vice president of engineering. The first was a bug in what was meant to be a read-only API, which allowed a video to be posted. The second was a bug in the video upload API, that generated an access token with broad privileges (the permissions of the Facebook mobile app). The third and perhaps most serious was a bug that generated the access token not for the current user, but for another user whose profile was being viewed.
The incident demonstrates that even the best-resourced software projects have bugs, and that bugs that do not immediately impact functionality are hard to spot. Testing is integral to software development and deployment today, but no test can exercise every possible combination of input, or account for every possible failure.
This case also shows how the impact of software bugs today can be magnified by the huge usage of certain pieces of code. In this instance, we're talking about fifty million accounts, with potential access to an unknown number of third-party sites.
Another recent example: Microsoft released the October 2018 feature update for Windows 10 with insufficient testing and included a bug that in certain cases deleted user data. It was deployed to thousands of users before being pulled. The reputational, support and remediation costs are considerable.
Anatomy of OpenSSL's Heartbleed: Just four bytes trigger horror bugREAD MORE
At the, well, heart of the Heartbleed bug discovered in 2014 was the following single line of code in the OpenSSL encryption library:
memcpy(bp, pl, payload);
The code does not check that the actual size of the data matches the amount being copied. The attacker can lie about the size and receive a copy of other data that happens to be in memory, perhaps including usernames and passwords. The bug impacted the Apache and Nginx web servers, used by two thirds of the world's active internet sites, as well as countless other network applications, via OpenSSL.
One line forward, two lines back
Venture-capitalist Marc Andreessen made headlines in 2011 when he reckoned software was eating the world. That would be fine if it weren't for the fact that two decades before, coder and author Steve McConnell reckoned on an industry average of 15 to 50 bugs per 1,000 lines of code with the Standish Report in 1995 saying one third of software projects got cancelled - wasting $81bn per year. McConnell and Standish were ghosts at the party of software delivery, further cementing the practice's reputation for poor quality. Software projects were expected to run late and contain bugs by default. It was a surprise if something came in on time, on budget and worked as intended.
That was the 1990s and software tools and development practices and cultures have changed a lot since then.
Prominent examples of positive changes include test-driven development and the concept of code coverage (the amount of code executed during testing) that emerged from Extreme Programming methodology in the nineties. Static code analysis is another, with tools that enforce coding standards as well as catching obvious bugs. Coding standards themselves have evolved, incorporating knowledge on what makes code more reliable, such as keeping code units short.
Safer programming languages have also helped. High-level languages with automatic memory management and no direct use of pointers, such as Java, first released in 1996, have made it easier for developers to avoid some errors. More recently, software lifecycle management tools, for teams of any size, are now simple to adopt thanks to cloud-hosted tools such as GitHub.
Are developers keeping their code clean as result?
As software has become more pervasive, the answer should be "yes". Software is no longer just the preserve of the workplace - Windows PCs and servers - or written for obscure industrial control systems, while software bugs risk taking down more than simply the company's HR server.
A glance down the Mitre Common Weakness Enumeration (CWE) List for 2017 is not especially encouraging. Top of the list is injection attacks, including SQL injection, where carefully crafted user input can execute unauthorized commands. It has been well known for years, but still causes problems.
Another area referenced by the CWE list is "using components with known vulnerabilities". Software developers have always depended on libraries; most of the code in any project is written by other people. Libraries and components are not immune from bugs though, as Heartbleed demonstrated. When a bug in a library or component is discovered, it will be fixed.
Patching all the applications that used the buggy version though is more than challenging. Many impacted applications will no longer be actively developed. Others may reside in firmware for network appliances that will never be patched. The Internet of Things becomes the Internet of Bugs, unless automatic patching exists, along with scrupulous patch management by Thing vendors.
Bugs can be expensive, and bugs can kill. Carnegie Mellon University Professor Phil Koopman specialises in embedded software quality including safety-critical areas such as self-driving cars and other automotive software. In a recent post about potentially deadly automotive software defects he lists more than 50 reports of disturbing defects such as unintended acceleration, cruise control which will not disengage, and power steering preventing the driver from controlling the vehicle.