OpenStack haven OpenDev yanks Gerrit code review tool after admin account compromised for two weeks

Source warehouse asks users to verify recent project commits to ensure they’re not malicious

OpenDev.org, which hosts the official OpenStack source code, on Tuesday tore down its Gerrit deployment after realizing it had been secretly hacked two weeks ago.

The site has asked users to review recent commits to their projects to make sure they don’t contain any backdoored or other malicious code.

As well as being home to OpenStack's blueprints, OpenDev hosts a number of other Git-based repositories pretty much along the lines of GitHub and similar source-hosting outfits. It uses Google-built Gerrit to provided a web-based environment for teams to review each others' work, approve or deny changes to code bases, and otherwise collaborate on programming projects.

On Tuesday, OpenDev published a since-removed maintenance notice, and issued a mailing-list announcement, that revealed “an admin account in Gerrit was compromised allowing an attacker to escalate privileges within Gerrit.”

OpenDev took the code-review tool offline earlier in the day while it started an investigation. At first, it feared its installation of Gerrit had been hacked via a security hole in the environment, though a few hours of effort brought the grim news that “the incident is related to a compromised Gerrit user account rather than a vulnerability in the Gerrit software.”

The since-deleted maintenance page on OpenDev's Gerrit review subdomain after the tool was compromised ... Click to enlarge

At 1745 UTC, OpenDev revealed it found a pair of Ubuntu One single-sign-on accounts had been hijacked on October 6 – two weeks before detection. One of those commandeered accounts was a Gerrit admin user. OpenDev uses Ubuntu One SSO to log people into their OpenDev accounts. Get hold of someone's Ubuntu One account credentials, get access to their OpenDev profile, therefore.

And in this case, one of those profiles was an admin user, allowing whoever had hijacked it to potentially tamper with OpenDev-hosted source-code repositories, such as the OpenStack ones, and cause other damage.

“These accounts ... have been dealt with at this point,” the maintenance note stated.

OpenDev is now rather busy trying to figure out exactly what happened, and what, if any, repos may have been meddled with.

“In order to evaluate impact, we are using backups from October 1 to find configuration, database, and git repo changes that have been made,” the maintenance note stated, advising users of a plan to “generate lists of commits that have landed since then for each branch on each repo.”

“We will verify that the latest commit can reach the last known good commit in the git DAG," it continued, referring to the Git version-history visualizer. "For non-merge commits we will also correlate these to Gerrit changes. We will then ask that you help us by verifying the commits on your projects are as reviewed and not malicious. We will also need to check git tags which should all be signed and can be verified that way.”

When users make those checks, they’ll have to jump through some hoops because all Gerrit HTTP API tokens have been deleted. Some 97 lucky account-holders have more to do because their accounts updated their ssh keys after the hack, and those keys have been removed.

All users have also been advised as follows: “This is a good reminder to check activity on your online accounts and identities for anything unexpected.”

Users were told OpenDev is “trying to go as quickly as we can while also not sacrificing caution and care." That careful approach is warranted because the prospect of repos being seeded with malware is scary. We note that as we publish this story, the Gerrit deployment is back online.

The Register will update this article if further news of the attack becomes available. ®