Can we talk about the little backdoors in data center servers, please?

Remote management a double-edged sword, IT admins warned at hacking conference


Black Hat Data centers are vital in this cloudy world – yet little-understood management chips potentially give hackers easy access to their servers in ways sysadmins may not have imagined.

The components in question are known as baseband management controllers (BMCs). They are discrete microcontrollers popped into boxes by the likes of Dell, HPE, and Lenovo to allow data-center managers to control machines without having to brave the chilly confines of a server farm. They can be accessed in various ways, from dedicated wired networks to Ethernet LANs.

BMCs can be used to remotely monitor system temperature, voltage and power consumption, operating system health, and so on, and power cycle the box if it runs into trouble, tweak configurations, and even, depending on the setup, reinstall the OS – all from the comfort of an operations center, as opposed to having to find an errant server in the middle of a data center to physically wrangle. They also provide the foundations for IPMI.

"They are basically a machine inside a machine – even if the server is down, as long as it has power, the BMCs will work,” said Nico Waisman, VP of security shop Immunity, in a talk at this year's Black Hat USA hacking conference on Thursday.

“They have a full network stack, KVM, serial console, and power management. It’s kind of like the perfect backdoor: you can remotely connect, reboot a device, and manage keyboard and mouse.”

It’s a situation not unlike Intel’s Active Management Technology, a remote management component that sits under the OS or hypervisor, has total control over a system, and been exploited more than once over the years.

Waisman and his colleague Matias Soler, a senior security researcher at Immunity, examined these BMC systems, and claimed the results weren’t good. They even tried some old-school hacking techniques from the 1990s against the equipment they could get hold of, and found them to be very successful. With HP's BMC-based remote management technology iLO4, for example, the builtin web server could be tricked into thinking a remote attacker was local, and so didn’t need to authenticate them.

"We decided to take a look at these devices and what we found was even worse than what we could have imagined," the pair said. "Vulnerabilities that bring back memories from the 1990s, remote code execution that is 100 per cent reliable, and the possibility of moving bidirectionally between the server and the BMC, making not only an amazing lateral movement angle, but the perfect backdoor too."

The fear is that once an intruder gets into a data center network, insecure BMC firmware could be used to turn a drama into a crisis: vulnerabilities in the technology could be exploited to hijack more systems, install malware that persists across reboots and reinstalls, or simple hide from administrators.

Sadly, the security of the BMCs is lax – and that's perhaps because manufacturers made the assumption that once a miscreant gets access to a server rack's baseboard controllers, it's game over completely anyway. Here's the stinging conclusion of their study:

From an offensive perspective, even though the various BMC platforms may require significant research investments, the results are worth the endeavour. A culture of empirically proven low-quality vendor software make BMCs a prime target. BMCs can facilitate long term persistence as well as cross-network movement that bypasses most network security design.

It is very hard, if not impossible, for any sufficiently sized company to move away from BMCs. As such, it is time for BMC vendors to revisit 2002, read [Bill Gates'] famous Trustworthy Computing memo, and realize that in 2018 sprintf() based stack overflows really should be a thing of the past in any platform that supports mission critical infrastructure.

The BMCs, by the way, use fruity hardware. Take HP’s Integrated Lights-Out (iLO) system, which is embedded in the ProLiant server range. The older version, iLO2 uses an antiquated NEC CPU core that was popular in optical drives back in the day, while iLO4 has a more modern Arm-compatible core. Dell’s version is the Integrated Dell Remote Access Controller (iDRAC) that uses Linux running on a variant of the SuperH chips once used in some gaming consoles.

Intel_Arria_10_GX

Slicker servers, heaving racks, NVMe invasion: It's been a big week in serverland

READ MORE

Most BMC chips run their own web server, typically based on the popular Appweb code. This can reveal the exact operating system and hardware setup of the chip if pinged correctly. Waisman and Soler also found a list, published by Rapid7, of the default passwords for most BMC systems.

The duo probed whatever kit they could get hold of – mainly older equipment – and it could be that modern stuff is a lot better in terms of security with firmware that follows secure coding best practices. On the other hand, what Waisman and Soler have found and documented doesn't inspire a terrible amount of confidence in newer gear.

Their full findings can be found here, and their slides here.

Of course, data center managers aren’t stupid, and BMC services are typically kept behind firewalls, segmented on the network, or only accessible via dedicated serial lines – and certainly shouldn't be facing the public internet. However, Waisman and Soler said they found plenty exposed to the web.

The bottom line is that IT admins need to assess the routes to their BMC services, make sure none are internet facing, and harden up access. Once an attacker establishes persistence with a BMC, you'll really wish you'd taken their advice. ®


Other stories you might like

  • Japan picks AWS and Google for first gov cloud push

    Local players passed over for Digital Agency’s first project

    Japan's Digital Agency has picked Amazon Web Services and Google Cloud for its first big reform push.

    The Agency started operations in September 2021, years after efforts like the UK's Government Digital Service (GDS) or Australia's Digital Transformation Agency (DTA). The body was a signature reform initiated by Prime Minister Yoshihide Suga, who spent his year-long stint in the top job trying to curb Japan's reliance on paper documents, manual processes, and faxes. Japan's many government agencies also operated their websites independently of each other, most with their own design and interface.

    The new Agency therefore has a remit to "cut across all ministries" and "provide services that are driven not toward ministries, agency, laws, or systems, but toward users and to improve user-experience".

    Continue reading
  • Singaporean minister touts internet 'kill switch' that finds kids reading net nasties and cuts 'em off ASAP

    Fancies a real-time crowdsourced content rating scheme too

    A Minister in the Singapore government has suggested the creation of an internet kill switch that would prevent minors from reading questionable material online – perhaps using ratings of content created in real time by crowdsourced contributors.

    "The post-COVID world will bring new challenges globally, including to us in the security arena," said Minister for Defence Dr Ng Eng Hen at a Tuesday ceremony to award the city-state's 2021 Defense Technology Prize.

    "For operations, the SAF (Singapore Armed Force) has to expand its capabilities in the digital domain. Whether for administrative or operational purposes, I think that we will need to leverage technology to the maximum," he declared.

    Continue reading
  • China Telecom booted out of USA as Feds worry it could disrupt or spy on local networks

    FCC urges more action against Huawei and DJI, too

    The US Federal Communications Commission (FCC) has terminated China Telecom's authority to provide communications services in the USA.

    In its announcement of the termination, the government agency explained the decision is necessary because the national security environment has changed in the years since 2002. That was when China Telecom was first allowed to operate in the USA.

    The FCC now believes – partly based on classified advice from national security agencies – that China Telecom can "access, store, disrupt, and/or misroute US communications, which in turn allow them to engage in espionage and other harmful activities against the United States." And because China Telecom is state-controlled, China's government can compel the carrier to act as it sees fit, without judicial review or oversight.

    Continue reading

Biting the hand that feeds IT © 1998–2021