GitHub rolls back database change after breaking itself

Git blame an infrastructure update

If you can't or couldn't access GitHub today, it's because the site broke itself.

The Microsoft-owned code-hosting outfit says it made a change involving its database infrastructure, which sparked a global outage of its various services. The biz is now in the process of rolling back that update to recover.

"We are experiencing interruptions in multiple public GitHub services," the source code silo said in an advisory on its status page. "We suspect the impact is due to a database infrastructure related change that we are working on rolling back."

What the GitHub.com homepage looked like today during the breakdown ... Click to enlarge

The downtime started just after 2300 UTC (1600 PT), according to GitHub. Affected services are: GitHub actions, pages, issues, pull requests, Copilot and Codespaces, packages, Git operations, and webhooks.

GitHub.com and the GitHub API were also unavailable – the website was showing just a unicorn and error message at one point – as was SSH-based access to repos.

"No server is currently available to service your request," an error message on the dot-com's homepage read earlier. "Sorry about that. Please try refreshing and contact us if the problem persists."

By 2329 UTC, GitHub made the decision to roll back its infrastructure change. And just now, at 2345 UTC, it's starting to right itself and return to normal.

"The database infrastructure change is being rolled back," the biz said in an update. "We are seeing improvements in service health and are monitoring for full recovery."

Happens to the best of us, clearly. ®

Speaking of GitHub...

If your organization has repositories on GitHub and uses Actions, be aware of this research this week from Unit 42:

A combination of misconfigurations and security flaws can make [GitHub Actions] artifacts leak tokens, both of third party cloud services and GitHub tokens, making them available for anyone with read access to the repository to consume.

This allows malicious actors with access to these artifacts the potential of compromising the services to which these secrets grant access.

Various GitHub repos owned by big names, such as Google, Microsoft, Red Hat, and Amazon, have been secured after accidentally leaking secrets, thanks these findings, we're told.

As to the severity of it all, bear in mind that GitHub believes this is an issue with people failing to secure their own stuff.

"This research was reported to GitHub's bug bounty program, Unit 42's Yaron Avital said. "They categorized the issue as informational, placing the onus on users to secure their uploaded artifacts."