Codecov dev tool warns of stolen credentials from compromised script, undiscovered for two months

Environment variables full of secrets uploaded to attacker server

Codecov, makers of a code coverage tool used by over 29,000 customers, has warned that a compromised script may have stolen credentials over a period of two months, before it was discovered a few weeks ago.

Code coverage measures how much of an application’s code is the subject of unit tests, the idea being that the higher the percentage, the more reliable the application is likely to be. It is a useful but imperfect metric, since it does not take into account the quality of the tests.

Codecov is a cloud-based tool which integrates with GitHub, GitLab, Atlassian Bitbucket, or any Git-based repository. Developers run tests using their own CI (Continuous Integration) tool and then upload the results to Codecov using a tool called Bash Uploader. Codecov then generates a report which is accessed on its site. Source code itself is not stored on Codecov’s site, but the tool does require read access to a repository in order to display code alongside reports on demand.

Bish, bash ... what?

The company has now reported a security issue, discovered on 1st April, caused by “an error in Codecov’s Docker image creation process that allowed the actor to extract the credential required to modify our Bash Uploader script.”

Beginning on January 31, “there were periodic, unauthorized alterations of our Bash Uploader script by a third party, which enabled them to potentially export information stored in our users' continuous integration (CI) environments. This information was then sent to a third-party server outside of Codecov’s infrastructure,” the report said.

The same upload script is also used in the integrations with GitHub actions, CircleCI, and Bitrise. The issue was discovered not by Codecov itself, but by a customer who noticed that the downloaded script failed to validate against the cryptographic checksum that verifies it has not been tampered.

What did the compromised script have access to? According to Codecov, it specifically targeted environment variables, commonly used to store tokens or keys to make them available to test and debug code without hard-coding them. The environment was uploaded to a remote server which Codecov said, “is currently part of an ongoing federal investigation.”

The company suggests that users who have downloaded the script check their version for the line curl -sm 0.5 -d "$(git remote -v)" as this indicates a maliciously altered version. Users of on-premises Codecov are unlikely to be affected.

How bad is it?

The implications of the attack are hard to assess. The key question is what further data, or opportunities to infiltrate, were put at risk by the leaked credentials. Codecov recommends that affected users revoke and re-generate all possibly impacted credentials, but damage may already have been done.

Codecov reckons affected users should investigate “any services, datastores and application code that could be accessed with these credentials, tokens or keys.”

A commenter on Hacker News said: “If you deploy from your CI system, it could be that every system is potentially compromised. In which case, get ready to re-deploy everything after you have flushed and re-built everything.”

Affected users should already have been notified by email of the issue. Codecov has promised a full investigation including auditing how the stolen credentials which enabled the incident were accessible, and setting up further monitoring tools to prevent a re-occurrence.

The attack illustrates the fact that the development process is a point of vulnerability for many organisations. This is partly because the developers themselves may have access to production data, and partly because even if they do not, there is a risk from compromised code that makes its way into production, possibly quickly if modern rapid deployment practices are in use. Protecting secrets during development is a tough problem.

Perhaps the most alarming aspect of this incident, aside from its potentially high impact, is the length of time it went undiscovered, despite the existence of checksum hashes that revealed the compromised script. "I find security an insufferably boring procedural hindrance", admitted a developer in a survey late last year.

There are many more interesting problems to solve, but unfortunately security does matter. ®