More than 140,000 internet-of-things devices, from routers to CCTV systems contain zero-day vulnerabilities, backdoors, hard coded crackable passwords and blurted private keys, according to the first large scale analysis of firmware in embedded devices.
Four researchers from EURECOM France found the flaws when conducting a simple but systematic, automated, and large-scale analysis of 32,356 firmware images running on embedded systems within thousands of different devices.
Of these, 693 had at least one vulnerability while 38 contained active (or possibly recently patched) zero day flaws.
Dozens of possible backdoors such as the "authorised keys" file - a veritable VIP list of SSH keys accepted for remote connection - were discovered, along with admin credentials for a staggering 101,000 devices and at least 2000 devices with hard-coded telnet logins.
An unnamed major vendor's app on the Google Play store was found with a known remote backdoor after the researchers searched for the affected string. Half a million users had downloaded the backdoored app, which could be accessed by attackers to send multicast packets to the devices. The same backdoor affected 44 pieces of CCTV kit plus home routers for a major unnamed vendor.
Cross site scripting bugs were found shared across unnamed SD cards that surprisingly ran WiFi web servers.
Backdoored daemons and device web pages added to the laundry list of shiny box failure.
The researchers also suggested that possibly half a million IoT devices could have shared backdoors due to dodgy firmware, a number that requires more analysis and better techniques to confirm.
One hundred and nine private RSA keys and self signed certificates were slurped, affecting at least 35,000 online devices currently in use, mostly CCTV cameras, along with dozens of hard coded hashed passwords that the quartet promptly cracked for good measure.
The list of crocked contraptions was rounded out with 100 distinct password hashes raided from /etc/passwd and /etc/shadow files that covered 681 separate firmware images from 27 vendors. Plaintext passwords were then happily recovered for 58 of ,the hashes affecting 538 firmware images.
The most popular passwords were no password at all, followed by 'pass', 'logout', and ironically 'helpme'.
"Whenever a new vulnerability was discovered our analysis infrastructure allowed us to quickly find related devices or firmware versions that were likely affected by the same vulnerability," the crew wrote in the paper A Large-Scale Analysis of the Security of Embedded Firmwares.
"For example, our correlation techniques allowed us to correctly extend the list of affected devices for variations of a tenet hard coded credentials vulnerability. In other cases, this led us to find a vulnerability’s root problem spread across multiple vendors."
The carnage stemmed from 32,000 firmware images unpacked into 1.7 million individual files that were statically analysed. A cloud army of 90 Amazon EC2 and local instances pried and plucked the firmware in search of security fail.
Automated IoT firmware analysis was vital, Andrei Costin, Jonas Zaddach, Aurélien Francillon, and Davide Balzarotti wrote in their paper to be presented at Usenix 2014 in San Francisco later this month. While manual identification was accurate, it was quite slow and also failed to identify if bugs existed across different devices. This left users of other affected devices exposed even when patches were available.
"This is often the case when several integration vendors rely on the same subcontractors, tools, or SDKs provided by development vendors," they said.
Hard coded passwords used to log into one line of CCTVs could also log into an entirely separate camera from a competing vendor, due to shared underlying dependencies.
User privileges were also problematic with many running entire tool chains as superuser.
There were five big challenges that remained in automated simple static analysis: building a representative sample set of IoT firmware; identifying if a downloaded thing was even firmware; unpacking horribly proprietary file formats; computation power limits, and confirming results.
Hulking ugly firmware hindered even dogged research by cramming the bootloader, operating system kernel, applications, and other resources into a single unholy memory image that was difficult to unpack, even using forensic tools such as carving.
Someone using a single dual-core computer to sift for relationships between the entire 1.7 million file set would be dead about 80 years before the process completed; it was only marginally better when the analysis was done on graphical processing units.
There was also no reasonable way for security nerds to test any vulnerabilities they found on devices without having to purchase or otherwise acquire it.
The team used three analysis tools - Binwalk, FRAK, and BAT - and improved on the latter to develop their own framework which was able to probe at least 20 per cent more proprietary IoT devices.
Security bods interested in IoT blood work should look to the Hadoop implementation of MapReduce used by malware analyser BitShred, the team said.
Technical details of the research process is captured for reader perusal in this text file (El Reg cannot yet release the full pdf to the public). ®