Stack Overflow 2019 hack was guided by advice from none other than... Stack Overflow

Vulnerabilities in build systems, secrets in source code: developer environments are an attack target


Developer site Stack Overflow has published details of a breach dating back to May 2019, finding evidence that an intruder in its systems made extensive use of Stack Overflow itself to determine how to make the next move.

At the time, the company reported that an unauthorised person had logged into its development system and escalated their access to the production version of stackoverflow.com. The source code for the site as well as the names, IP addresses and email addresses of 184 users was stolen, but not the databases which contain the content of the site and that of its customers.

Now further details have been reported by Dean Ward, principal developer in the architecture team at Stack Overflow, apparently “after consultation with law enforcement.”

The report describes the timeline of the attack, which started on April 30th with a probe of the Stack Overflow infrastructure. It appears that the source code was a specific target, as one early and unsuccessful move was to pose as a customer to request a copy “for auditing purposes.” According to the report, “This request is rejected because we don’t give out source code and, additionally, the email cannot be verified as coming from one of our customers.”

Despite the poor start, a few days later the attacker successfully logged into the StackOverflow development environment, using a crafted login request that bypassed access controls, and then successfully escalated privileges. They then got access to TeamCity, the JetBrains continuous integration product.

“A misconfiguration with role assignments means the user was immediately granted administrative privileges to the build server,” said Ward.

Although not having secrets in source code seems like a no-brainer, developers sometimes find this hard to avoid

How does TeamCity work? “The attacker is clearly not overly familiar with the product so they spend time looking up Q&A on Stack Overflow on how to use and configure it,” said Ward.

The intruder cloned several repositories hosted on GitHub Enterprise, using access configured for TeamCity. “They continue to browse Stack Overflow for details on building and running .NET applications under IIS as well as running SQL scripts in an Azure environment,” Stack Overflow said.

In what sounds like a serious move, the intruder wrote some SQL to elevate permissions across the entire Stack Exchange network and “after several attempts, they are able to craft a build that executes this as a SQL migration against the production databases housing data for the Stack Exchange Network.”

The community noticed a new user with broad privileges and reported it, at which point the Stack Overflow security team took more drastic steps, taking Team City offline and removing privileges and credentials. Some aspects were missed, though, and the “attacker pull[ed] source code again,” while also viewing questions on how to build .NET applications and (we are told) “how to delete repositories on GitLab.” The infrastructure was further locked down, and the “attacker continue[d] viewing Q&A, this time around SQL and certificates,” in their last reported actions.

Although it appears that damage to the StackOverflow site and the amount of data stolen was small, the company did, it seems, have a lot of source code stolen, although how valuable this is (other than for guiding new avenues of attack) is open to debate.

The incident was revealing though, and not only in proving that bad folk use Stack Overflow too. It showed how the development and build process can be a weak point in IT systems.

Developers may have a high level of access to production systems, and even if they do not, corrupting the build process can be a way of creating backdoors which are then deployed into production.

Twitter API key was in the source code

Stack Overflow went on to describe the changes it made to address shortcomings in its security. “We had secrets sprinkled in source control, in plain text in build systems and available through settings screens in the application,” confessed the team.

It also moved build and source control systems behind the firewall, added metrics and alerting around privilege escalation, and blocked the ability to view account recovery emails within the system.

Although not having secrets in source code seems like a no-brainer, developers sometimes find this hard to avoid. A follow-up thread reveals that a Stack Overflow integration with Twitter was disabled because the Twitter API key was in the source code and the developers have not worked out another way to do it. "We decided the functionality wasn’t critical enough to justify the effort involved," said Ward.

Future plans include mandating two-factor authentication with a new VPN, building a runtime secret store, and breaking apart build and deployment. Although this goes against the trend for continuous integration, it will, said Stack Overflow, “allow us to have deterministic builds and better manage deployment permissions.”

For every attack like this that is noticed, reported and remediated, there must be others that are not.

Who was the attacker? "We are not able to comment on any other details related to the attacker due to ongoing investigations," said the company - though it looks like the moment the community spotted the attack was recorded in StackExchange chat, together with the (likely fake) name of the user. ®

Similar topics


Other stories you might like

  • Twitter founder Dorsey beats hasty retweet from the board
    We'll see you around the Block

    Twitter has officially entered the post-Dorsey age: its founder and two-time CEO's board term expired Wednesday, marking the first time the social media company hasn't had him around in some capacity.

    Jack Dorsey announced his resignation as Twitter chief exec in November 2021, and passed the baton to Parag Agrawal while remaining on the board. Now that board term has ended, and Dorsey has stepped down as expected. Agrawal has taken Dorsey's board seat; Salesforce co-CEO Bret Taylor has assumed the role of Twitter's board chair. 

    In his resignation announcement, Dorsey – who co-founded and is CEO of Block (formerly Square) – said having founders leading the companies they created can be severely limiting for an organization and can serve as a single point of failure. "I believe it's critical a company can stand on its own, free of its founder's influence or direction," Dorsey said. He didn't respond to a request for further comment today. 

    Continue reading
  • Snowflake stock drops as some top customers cut usage
    You might say its valuation is melting away

    IPO darling Snowflake's share price took a beating in an already bearish market for tech stocks after filing weaker than expected financial guidance amid a slowdown in orders from some of its largest customers.

    For its first quarter of fiscal 2023, ended April 30, Snowflake's revenue grew 85 percent year-on-year to $422.4 million. The company made an operating loss of $188.8 million, albeit down from $205.6 million a year ago.

    Although surpassing revenue expectations, the cloud-based data warehousing business saw its valuation tumble 16 percent in extended trading on Wednesday. Its stock price dived from $133 apiece to $117 in after-hours trading, and today is cruising back at $127. That stumble arrived amid a general tech stock sell-off some observers said was overdue.

    Continue reading
  • Amazon investors nuke proposed ethics overhaul and say yes to $212m CEO pay
    Workplace safety, labor organizing, sustainability and, um, wage 'fairness' all struck down in vote

    Amazon CEO Andy Jassy's first shareholder meeting was a rousing success for Amazon leadership and Jassy's bank account. But for activist investors intent on making Amazon more open and transparent, it was nothing short of a disaster.

    While actual voting results haven't been released yet, Amazon general counsel David Zapolsky told Reuters that stock owners voted down fifteen shareholder resolutions addressing topics including workplace safety, labor organizing, sustainability, and pay fairness. Amazon's board recommended voting no on all of the proposals.

    Jassy and the board scored additional victories in the form of shareholder approval for board appointments, executive compensation and a 20-for-1 stock split. Jassy's executive compensation package, which is tied to Amazon stock price and mostly delivered as stock awards over a multi-year period, was $212 million in 2021. 

    Continue reading

Biting the hand that feeds IT © 1998–2022