Stack Overflow 2019 hack was guided by advice from none other than... Stack Overflow

Vulnerabilities in build systems, secrets in source code: developer environments are an attack target


Developer site Stack Overflow has published details of a breach dating back to May 2019, finding evidence that an intruder in its systems made extensive use of Stack Overflow itself to determine how to make the next move.

At the time, the company reported that an unauthorised person had logged into its development system and escalated their access to the production version of stackoverflow.com. The source code for the site as well as the names, IP addresses and email addresses of 184 users was stolen, but not the databases which contain the content of the site and that of its customers.

Now further details have been reported by Dean Ward, principal developer in the architecture team at Stack Overflow, apparently “after consultation with law enforcement.”

The report describes the timeline of the attack, which started on April 30th with a probe of the Stack Overflow infrastructure. It appears that the source code was a specific target, as one early and unsuccessful move was to pose as a customer to request a copy “for auditing purposes.” According to the report, “This request is rejected because we don’t give out source code and, additionally, the email cannot be verified as coming from one of our customers.”

Despite the poor start, a few days later the attacker successfully logged into the StackOverflow development environment, using a crafted login request that bypassed access controls, and then successfully escalated privileges. They then got access to TeamCity, the JetBrains continuous integration product.

“A misconfiguration with role assignments means the user was immediately granted administrative privileges to the build server,” said Ward.

Although not having secrets in source code seems like a no-brainer, developers sometimes find this hard to avoid

How does TeamCity work? “The attacker is clearly not overly familiar with the product so they spend time looking up Q&A on Stack Overflow on how to use and configure it,” said Ward.

The intruder cloned several repositories hosted on GitHub Enterprise, using access configured for TeamCity. “They continue to browse Stack Overflow for details on building and running .NET applications under IIS as well as running SQL scripts in an Azure environment,” Stack Overflow said.

In what sounds like a serious move, the intruder wrote some SQL to elevate permissions across the entire Stack Exchange network and “after several attempts, they are able to craft a build that executes this as a SQL migration against the production databases housing data for the Stack Exchange Network.”

The community noticed a new user with broad privileges and reported it, at which point the Stack Overflow security team took more drastic steps, taking Team City offline and removing privileges and credentials. Some aspects were missed, though, and the “attacker pull[ed] source code again,” while also viewing questions on how to build .NET applications and (we are told) “how to delete repositories on GitLab.” The infrastructure was further locked down, and the “attacker continue[d] viewing Q&A, this time around SQL and certificates,” in their last reported actions.

Although it appears that damage to the StackOverflow site and the amount of data stolen was small, the company did, it seems, have a lot of source code stolen, although how valuable this is (other than for guiding new avenues of attack) is open to debate.

The incident was revealing though, and not only in proving that bad folk use Stack Overflow too. It showed how the development and build process can be a weak point in IT systems.

Developers may have a high level of access to production systems, and even if they do not, corrupting the build process can be a way of creating backdoors which are then deployed into production.

Twitter API key was in the source code

Stack Overflow went on to describe the changes it made to address shortcomings in its security. “We had secrets sprinkled in source control, in plain text in build systems and available through settings screens in the application,” confessed the team.

It also moved build and source control systems behind the firewall, added metrics and alerting around privilege escalation, and blocked the ability to view account recovery emails within the system.

Although not having secrets in source code seems like a no-brainer, developers sometimes find this hard to avoid. A follow-up thread reveals that a Stack Overflow integration with Twitter was disabled because the Twitter API key was in the source code and the developers have not worked out another way to do it. "We decided the functionality wasn’t critical enough to justify the effort involved," said Ward.

Future plans include mandating two-factor authentication with a new VPN, building a runtime secret store, and breaking apart build and deployment. Although this goes against the trend for continuous integration, it will, said Stack Overflow, “allow us to have deterministic builds and better manage deployment permissions.”

For every attack like this that is noticed, reported and remediated, there must be others that are not.

Who was the attacker? "We are not able to comment on any other details related to the attacker due to ongoing investigations," said the company - though it looks like the moment the community spotted the attack was recorded in StackExchange chat, together with the (likely fake) name of the user. ®

Similar topics


Other stories you might like

  • Cisco warns of security holes in its security appliances
    Bugs potentially useful for rogue insiders, admin account hijackers

    Cisco has alerted customers to another four vulnerabilities in its products, including a high-severity flaw in its email and web security appliances. 

    The networking giant has issued a patch for that bug, tracked as CVE-2022-20664. The flaw is present in the web management interface of Cisco's Secure Email and Web Manager and Email Security Appliance in both the virtual and hardware appliances. Some earlier versions of both products, we note, have reached end of life, and so the manufacturer won't release fixes; it instead told customers to migrate to a newer version and dump the old.

    This bug received a 7.7 out of 10 CVSS severity score, and Cisco noted that its security team is not aware of any in-the-wild exploitation, so far. That said, given the speed of reverse engineering, that day is likely to come. 

    Continue reading
  • CISA and friends raise alarm on critical flaws in industrial equipment, infrastructure
    Nearly 60 holes found affecting 'more than 30,000' machines worldwide

    Updated Fifty-six vulnerabilities – some deemed critical – have been found in industrial operational technology (OT) systems from ten global manufacturers including Honeywell, Ericsson, Motorola, and Siemens, putting more than 30,000 devices worldwide at risk, according to private security researchers. 

    Some of these vulnerabilities received CVSS severity scores as high as 9.8 out of 10. That is particularly bad, considering these devices are used in critical infrastructure across the oil and gas, chemical, nuclear, power generation and distribution, manufacturing, water treatment and distribution, mining and building and automation industries. 

    The most serious security flaws include remote code execution (RCE) and firmware vulnerabilities. If exploited, these holes could potentially allow miscreants to shut down electrical and water systems, disrupt the food supply, change the ratio of ingredients to result in toxic mixtures, and … OK, you get the idea.

    Continue reading
  • Azure issues not adequately fixed for months, complain bug hunters
    Redmond kicks off Patch Tuesday with a months-old flaw fix

    Updated Two security vendors – Orca Security and Tenable – have accused Microsoft of unnecessarily putting customers' data and cloud environments at risk by taking far too long to fix critical vulnerabilities in Azure.

    In a blog published today, Orca Security researcher Tzah Pahima claimed it took Microsoft several months to fully resolve a security flaw in Azure's Synapse Analytics that he discovered in January. 

    And in a separate blog published on Monday, Tenable CEO Amit Yoran called out Redmond for its lack of response to – and transparency around – two other vulnerabilities that could be exploited by anyone using Azure Synapse. 

    Continue reading

Biting the hand that feeds IT © 1998–2022