DevOps freezes up as techies race to fix dead data storage gear

TITSUP: Total Inability To Support Users' Pushes

69 Got Tips?

GitHub's website remains broken after a data storage system failed hours ago.

Depending on where you are, you may have been working on some Sunday evening programming, or getting up to speed with work on a Monday morning, using resources on – and possibly failing miserably as a result of the outage.

From about 4pm US West Coast time on Sunday (2300 UTC), the website has been stuttering and spluttering. Specifically, the site is still up and serving pages – it's just intermittently serving out-of-date files, and ignoring submitted Gists, bug reports, pushes, and posts. Sometimes, it appears to be serving a read-only cache or older backup of itself, although some fresh code pushes are coming through onto the site.


From the status page, it appears a data storage system died, forcing the platform's engineers to move the dot-com's files over to another box. In the meantime, some older versions of repos are being served to visitors and users.

"We're continuing to work on migrating a data storage system in order to restore access to," the team said just after 5pm PT, adding in the past few minutes: "We are continuing to repair a data storage system for You may see inconsistent results during this process."

In this vulture's experience this evening, the backend git services are still working. It's just the website that's frozen. Some people have complained they can't log in, or that branches have gone missing, and so on.

Right now, we're seeing scores of complaints about the site being down on Twitter – including quite a few upset coders in Japan, where at time of writing is late Monday morning. Nice start to the week.


Judging by the gripes, there's no obvious way to tell the site is borked, unless you look for the status page or realize something's up after the tenth attempt to submit work to the dot-com fails.

"Can you restore your service already?" software dev Saishav Agarwal pleaded to GitHub. "And please let users know when such issues are discovered! An email would be sufficient. Just wasted a precious hour."

Microsoft hopes to buy San Francisco-based GitHub for $7.5bn (£5.7bn). We'll update this story when, fingers crossed, GitHub repairs itself. ®

Updated to add at 0700 UTC is still wonky, and will remain so for the next couple of hours, it seems. "We are currently in the later stages of a restore operation, with the aim of serving fully consistent data within the next 2 hours," Team GitHub said.

Updated to add at 1400 UTC

Five hours on, and users are still reporting inconsistencies, with several saying they are unable to log in or work. At 1318, GitHub staffers said: "We are validating the consistency of information across all data stores. Webhooks and Pages builds remain paused."

Final update

It's all fixed now, 24 hours later.

Sign up to our NewsletterGet IT in your inbox daily


Keep Reading

It's 500 Friday at GitHub as source shack takes an hour-long morning totter

A week of wobbles ends in a good, old fashioned TITSUP*

Git your coat – you've pulled: Standalone command-line interface for GitHub hits beta

Limited functionality and cloud only, but improvements are on the way

GitHub to replace master with main across its services

Language matters – but when you replace a 'blacklist' with a 'blocklist', what do you call a list of blocks?

GitHub blasts code-scanning tool into all open-source projects

Rub-a-dub-dub, give your buggy code a scrub

European programmers take an extended lunch break as GitHub goes TITSUP* again

All fixed now so get back to work

Now GitHub has gulped down NPM Inc, what's next for the JS package registry? Well, some stability will be nice

CTO Ahmad Nassri announces intention to bow out

Total Inability To Service User Pulls: GitHub wobbles with a good old Thursday TITSUP

Hey – there's always Visual SourceSafe, right?

GitHub redesign goes mobile-friendly – to chagrin of devs who shockingly do a lot of work on proper computers

Code shack is a great place for others to collaborate, but not so strong at collaboration for itself

GitHub rolls out hosted Visual Studio Code in Codespaces

Developer biz bets virtualized environments will make it easier to engage with repos

How generous of GitHub to slash prices and make all its core features free. So what gives? Oh right, GitLab

No coincidence that new offer closely matches that from smaller rival

Tech Resources

Ransomware Playbook

Ransomware is a unique security threat where most of the security team’s effort is spent on prevention and response because once ransomware is detected, it's too late.

Latency is the New Outage

More organizations are tying their future success to digital and online business.

The Definitive Guide to Sharing Threat Intelligence

Sharing threat intelligence is gradually becoming an accepted component in information security defense but there are still ways we can gain more.

Comprehensive Kubernetes Observability at Scale

Understand how to effectively alert, troubleshoot, and optimize across the Kubernetes environment.