HashiCorp, an open-source company whose Terraform product is widely used for automated cloud deployments, has revealed a private code-signing key was exposed thanks to the compromised Codecov script discovered earlier this month.
Codecov, which provides tools to assess how much of an application's code is subject to unit tests, reported that a script used to upload data to its servers was modified to export credentials to an attacker's server. The company said it had "not been able to determine conclusively who carried out the event."
HashiCorp, one of Codecov's 29,000 customers, has confirmed it was among those hit. Specifically, it said "a subset of HashiCorp's CI pipelines used the affected Codecov component" and "the GPG private key used for signing hashes used to validate HashiCorp product downloads... was exposed."
The exposure means that potentially the attacker could have modified HashiCorp products while signing them with a genuine key, but the company said the "investigation has not revealed evidence of unauthorized usage." It has validated existing releases, revoked the exposed key, and re-signed its downloads with a new key.
Codecov dev tool warns of stolen credentials from compromised script, undiscovered for two monthsREAD MORE
While that is somewhat reassuring, the Codecov incident and this follow-up report have disturbing implications. The compromise of the Codecov script was in place for a long period, beginning January 31, and there were a number of altered versions. Codecov has given details of the last tampered version but said "there were periodic, unauthorized alterations of our Bash Uploader script by a third party" so there is uncertainty about the attacks used in older versions.
Continuous Integration uncertainty
A second concern is that the attacker may have used harvested credentials for further incursions of which we currently know nothing. The credentials that were stolen were those that were stored in environmental variables on machines or containers used as part of a CI (Continuous Integration) process, which might include API keys, database logins, or (as in the HashiCorp example) cryptographic certificates. The whole point of stealing credentials is to enable further attacks.
Third, the reach of HashiCorp tools into enterprise computing is huge, bigger than that of Codecov. Earlier this month the company said, in the context of a new release of its Vault secrets management product, that "nearly 25 per cent of the Fortune 500 and 70 per cent of the 20 largest US banks rely on Vault to secure their most sensitive data."
While there is no suggestion in HashiCorp's report that its products have been compromised, the fact that it was running a credential-stealing script as part of the build process for some of its products is still a cause for concern.
The company's statement is itself inconclusive. HashiCorp said it "has performed additional remediations related to information potentially exposed during this incident. Incident response activities are ongoing, and relevant updates and outcomes will be shared promptly when available," which leaves customers to guess at the nature of these further leaks.
HashiCorp is both savvy enough to have detected that it had a problem, and responsible enough to disclose this to its customers. How many other examples are there, small and large, how many discovered, and how many disclosed?
A Reg staff member received an email from their letting agency saying that "Codecov informed us on April 15 that the breach could have given access to all personal data of our users from March 9 onwards." There was no evidence of unusual activity, the agency said, but added that it "would still like to advise you to create a new password in order to further protect your account."
What this suggests is that the list of those potentially affected by the Codecov breach is long, and we are unlikely to know the full implications for a while yet.
It is reminiscent of another supply chain attack on SolarWinds, where we know the attack was long-term and extensive, but the full implications may not be known for years.
What should HashiCorp customers do? "Ensure that they download HashiCorp products only from the official release channel," the company said in its statement, which, while true, is not especially illuminating. ®