GlobalSign screw-up cancels top websites' HTTPS certificates

Revoked certs may linger for days, locking people out of sites

Final update GlobalSign's efforts as a root certificate authority have gone TITSUP this afternoon – that's a total inability to support usual protocols.

The result is that many websites big and small have had their HTTPS certificates incorrectly scrapped, meaning that for some people their browsers no longer trust websites and refuse or are reluctant to access them.

Specifically, it appears GlobalSign inadvertently triggered the revocation of its intermediary certificates while updating a special cross-certificate. This smashed the chain of trust and ultimately nullified SSL/TLS certificates issued by GlobalSign to its customers. It could take days to fix, leaving folks unable to easily read their favorite webpages.

GlobalSign estimates it could take until the beginning of next week for websites' accidentally axed certs to be corrected. The organization has set up a support page for IT administrators and folks looking to fix broken HTTPS certificates.

GlobalSign said the worldwide mass revocation was an "unexpected consequence" of internal changes it made, and claimed browsers and other software "incorrectly inferred" that certificates had been burned. (It later admitted its own systems were at fault.)

If you're not affected by today's outage – consider yourself lucky as the problem won't hit everyone due to the wide range of caching and revocation policies employed by different browsers, apps and other software. If your application hasn't picked up the revocations yet, it should be fine – if it has, you can try to delete your certificate revocation list cache (see the above link for instructions on Windows and macOS) to see if that helps.

"That's the unfortunate thing about PKI, different browsers have different update levels," GlobalSign's strategic projects director Steve Roylance told The Reg.

Wikipedia blocked in Google Chrome after its HTTPS cert was accidentally revoked (click to enlarge)

Just hours ago, it became clear that GlobalSign – a New Hampshire, US-based biz – was having troubles with its Online Certificate Status Protocol (OCSP), which is used for obtaining the revocation status of public key certificates which ensure that netizens are connecting to legit sites using SSL/TLS.

"We are currently experiencing a known issue which is causing certificate revocation/error messages to be displayed within some of our certificates," a rep for GlobalSign tweeted earlier.

We are currently experiencing issues with our OCSP which is causing certificate warning messages. We aim to fix this as soon as possible.

— GlobalSign (@globalsign) October 13, 2016

Responding to complaints on Twitter, GlobalSign said it had sorted out the issue on its end, but stressed it'll take time for the changes to work their way through the internet's maze of caches. The web company's status page states:

We are currently experiencing a known issue which is causing certificate revocation/error messages to be displayed within some of our certificates.

Unfortunately, the cache laundering is a tricky process that not everyone can follow, meaning less technology-literate peeps may struggle with certificate errors for some time.

If the OSCP/CRL cache clearing hasn't worked, we're still working on a resolution. We deeply apologize 4 the outage & will keep u updated.

— GlobalSign (@globalsign) October 13, 2016

As of publication, people are up in arms about how long it's going to take to correct the dodgy revocations.

@globalsign huge huge impact on our 5 webshops. do we have ETA on the fix?

— Rémi NGUYEN (@reminguyen) October 13, 2016

@globalsign We will start exploring other options very soon if there is no resolution or ETA in the next few hours.

— Hjalmar Theodorsson (@hjalmarth) October 13, 2016

Sites affected include the Financial Times, Guardian, Wikipedia, Logmein, and Dropbox.

This afternoon, a spokeswoman for GlobalSign shed some more light on the outage:

GlobalSign manages several root certificates and for compatibility and browser ubiquity reasons provides several cross-certificates between those roots to maximize the effectiveness across a variety of platforms.

As part of a planned exercise to remove some of those links, a cross-certificate linking two roots together was revoked. CRL responses had been operational for one week, however an unexpected consequence of providing OCSP responses became apparent this morning, in that some browsers incorrectly inferred that the cross-signed root had revoked intermediates, which was not the case.

GlobalSign has since removed the cross-certificate from the OCSP database and cleared all caches. However, the global nature of CDNs and effectiveness of caching continued to push some of those responses out as far as end users. End users cannot always easily clear their caches either through lack of knowledge or lack of permission. New users (visitors) are not affected as they will now receive good responses. The problem will correct itself in four days as the cached responses expire, which we know is not ideal. However, in the meantime, GlobalSign will be providing an alternative issuing CA for customers to use instead, issued by a different root which was not affected by the cross that was revoked but offering the same ubiquity.

Meanwhile, this is what GlobalSign's telling its customers... ®

@JZdziarski @TheRegister Here is what they sent out to us, the customers. pic.twitter.com/qADNP8x5ez

— Koen Rouwhorst (@koenrh) October 13, 2016

This breaking story was updated after publication to include comments and explanations from GlobalSign.