Cloud comms platform Twilio has confirmed its private GitHub repositories were cloned after it became the latest casualty of the compromised credential-stealing Codecov script.
Codecov, a cloud-based tool for assessing how much code is covered by software tests, revealed last month that a script called Bash Uploader had been altered by a criminal to export secrets stored in environmental variables to a third-party server. This script is widely used for Codecov integration, including within GitHub Actions, popular for Continuous Integration (CI) pipelines.
Twilio said: "We have Codecov tools, including the Bash Uploader component, in use in a small number of our projects and CI pipelines." The company added that these particular projects were "not in the critical path to providing updates or functionality to our communication APIs" and that it has "remediated the potential exposure by thoroughly reviewing and rotating any potentially exposed credentials."
In its analysis of the breach, Twilio said that the attacker cloned some of its private GitHub repositories. This went unnoticed until Twilio "received a notification from GitHub.com that suspicious activity had been detected." The notification was on 22 April, seven days after Codecov informed the world of its breach. Twilio investigated the cloned repositories, looking for both secrets and also personal data, finding "a small number of email addresses belonging to Twilio customers."
According to Twilio, there is no indication "that any customer data, beyond the small number of email addresses, was accessed or at risk."
Twilio has not said how the GitHub compromise occurred, other than to reveal it involved "a Twilio user token that had been exposed."
All users of the GitHub Codecov action were at risk. GitHub Actions support secrets for things like Azure or AWS credentials, database logins and so on. Secrets are available to scripts as environment variables, which were targeted by the compromised script for export.
- HashiCorp reveals exposure of private code-signing key after Codecov compromise
- 'Business folk often don't understand what developers do...' Twilio boss on the chasm that holds companies back
- Codecov dev tool warns of stolen credentials from compromised script, undiscovered for two months
Codecov's Bash Uploader script could be verified to check for tampering via a cryptographic checksum, but despite this it was a couple of months before the compromise was detected. The use of the script within GitHub actions was one example where the checksum was not inspected.
Following the security incident, GitHub users raised an issue, "Checksum should be run on bash uploader script before execution," with one developer remarking that "the idea to directly and blindly execute a bash script pulled from the web is a giant security hole and a ticking bomb for future breaches."
Codecov attempted to add verification to the GitHub Action which then started raising false positives thanks to a mismatch between the checksum and the script actually in use. This is the kind of friction which undermines efforts to improve security.
A notable aspect of Twilio's report is what it says under the heading "What are we doing to prevent similar issues in the future?" The company said it evaluates its suppliers for security, and has developed a service called Deadshot that scans GitHub pull requests to prevent secrets or insecure code being committed to its repository.
However, "it would not have prevented this particular issue where the secrets were compromised due to a supply chain attack," said the company. In other words, the listed mitigations are insufficient to guard against a repeat incident.
Twilio has now joined HashiCorp as a high-profile company that has admitted to being affected by the Codecov incident – but, like HashiCorp, has said that not much harm was done as far as it knows.
The worry is that a huge number of credentials have been leaked to an unknown attacker, with plenty of potential for further infiltration, but that the identity and motivation of that attacker is unknown.
Codecov said: "We have not been able to determine conclusively who carried out the event. We are working with law enforcement and have offered our full cooperation with their investigation."
It is reasonable to presume that many more organisations have, knowingly or unknowingly, suffered from this breach and that we can expect more to come forward, and many others to remain hidden. Twilio's report also serves as a warning to all users of Codecov with GitHub, to check whether they may have been impacted (for example, if the Codecov Action was used), and to apply remediation as needed. ®