Hackers penetrated the heavily-fortified servers for Apache.org in a "direct, targeted attack" that captured the passwords of anyone who used the website's bug-tracking service over a three-day span last week.
The breach, the second to hit Apache.org in eight months, also exposed a much larger list of passwords belonging to people who accessed the site's bug-tracking section. While the databases used a one-way hash to disguise the passwords, two of the lists are vulnerable to dictionary attacks because Atlassian, the maker of issue-tracking software used by Apache, failed to add "random salt" to them.
As a result, Apache officials said users who logged in to the bug section of the website from April 6 to April 9 "should consider the password as compromised, because the attackers changed the login form to log them." They also warned that there's a high risk of compromise to other users if they employed simple passwords based on dictionary words.
The intrusion began on April 5 when unknown attackers using a hacked server from Slicehost opened a new bug report on Apache.org. The post contained a shortened web link from tinyurl.com that exploited an XSS, or cross-site scripting, vulnerability on Apache's support website.
The hole was the result of a bug in JIRA, the issue-tracking software made by a company called Atlassian. The exploit was designed to steal session cookies used to authenticate people logged in to Apache's JIRA system. When several Apache administrators following the fraudulent bug report clicked on the on the malicious link, their JIRA administrator rights were then compromised.
The attackers also carried out a brute-force attack that flooded the site with hundreds of thousands of password combinations. By April 6, one of the two methods allowed the attackers to gain full administrative rights on the JIRA system. For three days, the hackers used their powers to copy users' home directories and files and to install a program that logged the passwords of anyone accessing the system.
Eventually, the attackers were able to obtain full root access for the server that administered the issue-tracking software. By pilfering cached authentication credentials stored on the machine, they were then able to log on to minotaur.apache.org, Apache's main shell server. The hackers' luck eventually ran out when they were unable to escalate the limited privileges that came with the compromised accounts on the minotaur server.
The breach coincided with a separate attack on Atlassian's internal servers that exposed a database storing user passwords in plain-text. Atlassian said here that the credentials created after July 2008 weren't included, but that the "old database table was not taken offline or deleted, and it is this database table that we believe could have been exposed during the breach.
The disclosure took many Atlassian customers by alarm.
"In my particular case I would not be horribly upset by someone accessing my project management or documentation, but access to my issues and source code would be disastrous for my business," one user wrote.
"For me, its unbelievable, that a company that sells secure login and user management solutions saves passwords in plaintext," another wrote.
As was the case in September, when Apache disclosed its servers were breached by hackers exploiting poorly locked-down secure shell keys, officials released a postmortem that's commendable for its thoroughness and refusal to pass the buck.
"The primary problem with our JIRA install is that the JIRA daemon runs as the user who installed JIRA," they wrote. "In this case, it runs as a jira role-account. There are historical reasons for this decision, but with 20/20 hindsight, and in light of the security issues at stake, we expect to revisit the decision!"
They noted other mistakes, including the use of the same password that gave access to a JIRA account and sudo rights on the host machine, inconsistent application of one-time passwords and the use of SSH passwords to log in over the net.
Apache's admission of fault is all the more admirable considering the epic blunder made by the folks at Atlassian, which most likely is the chink that gave the attackers access in the first place. A lesser group would have hid behind it. Instead, Apache came clean. The postmortem is here. ®