Silly sysadmins ADDING Heartbleed to servers

'Heartbroken' admins add to problem of myriad unpatched boxen

Updated At least 2,500 website administrators have made their previously secure sites vulnerable to Heartbleed more than a month after the bug sent the world into a hacker-fearing frenzy.

Former Opera software developer Yngve Pettersen discovered the bungle while probing for Heartbleed vulnerable systems in the weeks after the bug was disclosed on April 7.

Heartbleed was a widespread input validation security vulnerability affecting the heartbeat extension used in OpenSSL which allowed passwords, sensitive private keys and session cookies to be potentially stolen. The bug was patched on the day of disclosure.

With his TLS Prober tool in hand, Petterson pinged half a million separate servers of sites rated as popular by Alexa and found hapless admins had, presumably in a panic, updated their then-unaffected-or-possibly-new boxes to the latest offering and in doing so introduced the Heartbleed bug.

He found about 20 per cent of scanned vulnerable servers were new to the Heartbleed club as administrators had introduced the vulnerability.

"It is difficult to definitely say why this problem developed, but one possibility is that all the media attention led concerned system administrators into believing their system was unsecure [which] combined with administrative pressure and a need to 'do something' led them to upgrade an unaffected server to a newer but still buggy version ... not yet officially patched," he said, dubbing the new fail boxes "Heartbroken".

Affected admins may suffer further heartbreak by footing bills for patching servers, updating certificates and hours of testing. Petterson pegged the total cost at $1.2m (assuming it took the three admins four hours of work at $40 an hour, multiplied by the 2,500 affected servers in question).

He went further: "As my sample is probably not more than 10 per cent of the secure servers on the net, the unnecessary patching cost could exceed $12m."

This process he said should involve server patching, followed by a certificate revoke and refresh and lastly password changes.

Petterson also found that two-thirds of certificates currently in use on now patched servers still carried Heartbleed-soiled certificates that would place users of those sites at risk of compromise.

This took the conservative assumption that private keys were considered breached during the unpatched days of the Heartbleed disclosure and was based on the fact that the reused certificates resurfaced in subsequent scans.

Petterson's work also found the number of exposed systems had dropped sharply from 5.36 per cent on April 11 to 2.33 percent on May 7, a month after the Heartbleed disclosure.

Just under a quarter of scanned servers supported heartbeat in what he said indicated that 75 percent of exposed servers were patched in the four days prior to his initial scan.

That patching trend appeared however to have hit the thin end of the wedge.

"While the vulnerability number had been halved to 2.77 percent, in the most recent scan two weeks later the number has only been reduced to 2.33 percent indicating that patching of vulnerable servers has almost completely stopped," he said.

In separate research, Rob Graham of Errata Security also found about half of vulnerable servers identified after the Heartbleed disclosure were still exposed.

His research revealed 318,239 servers were exposed of the 600,000 detected four weeks earlier.

Both Graham and Petterson warned as with similar scans their numbers were somewhat skewed due to variabilities in the scanning procedure including administrators blocking their probes and network congestion.

Graham also found 1.5 million systems sporting the heartbeat feature, 500,000 more than were noticed in his April scan, which may suggest administrators reacted to the Heartbleed disclosure by first terminating the extension.

Last month security firm Secunia warned expunging the Heartbleed bug would likely take months. ®

Bootnote

Pettersen has since updated his blog post saying that some of his conclusions had been misplaced due to an issue with the network connection of the prober the test used to detect certain servers.