Xero, Slack suffer outages just as Let's Encrypt root cert expiry downs other websites, services

'The internet is a complex system'


Updated Websites and apps are suffering or have suffered outages around the world for at least some netizens today due to connectivity issues.

Though the exact causes of the IT breakdowns are in many cases not fully known right now, there has been a sudden uptick in downtime right as Let's Encrypt, which provides free HTTPS certificates to a ton of organizations, let one of its root and intermediate certs expire.

This expiration should be invisible to software, services, and users relying on the certificates for encryption, tamper-proof communications and whatnot, however not all systems appear to have handled the expiry well. Thus, it is assumed the expiration and at least some of today's outages are interlinked. Other downtime, such as Slack's teetering, is not tied to Let's Encrypt.

Specifically, today, September 30, an older root certificate – DST Root CA X3 – which was used to underpin HTTPS certs issued by Let's Encrypt, expired as planned along with its R3 intermediate. There's more technical info on that here.

Apps and websites should instead now use, or continue to use, Let's Encrypt's ISRG Root X1 certificate and its replacement R3 intermediate automatically, and this whole process should pretty much be transparent to users. Let's Encrypt warned "older devices" will need a prod to catch up with and trust the latest foundational certs.

However, it appears it's more than "older devices" that have tripped up and stopped working as expected – such as refusing to establish secure connections to remote systems – and that may be due to an issue with the now-expired R3 intermediate, or because your client doesn't recognize the ISRG root cert.

If you're having trouble with Git and servers using broken Let's Encrypt certs, for instance, there's a thread on fixing that here.

HTTPS cert expert Scott Helme has been tracking the certificate expiry and reporting what's falling over on Twitter, likely as a result of the cert changes, right here.

These include Catchpoint System Services; OVH; OpenBSD; Facebook for developers; Fortinet; parts of Netify; and Cisco Umbrella.

Heroku also suffered a wobble with its metrics and Connect Dashboard though it was unrelated to the cert death, which was a separate issue for its customers. Shopify and parts of Cloudflare also encountered mysterious downtime that appears to be unrelated to Let's Encrypt.

"There are also many reports of iOS and macOS versions newer than expected seeing issues on sites serving the expired R3 intermediate," Helme said. "I've seen errors on iOS 11, 13 and 14 along with several macOS version only a few minor releases behind current."

A spokesperson for the Let's Encrypt organization told us earlier today: "We are monitoring the expiration and providing advice when we can. The internet is a complex system and any change has knock-on effects. We are not currently seeing more than what we might describe as 'expected' issues for any change like this."

Meanwhile, we've spotted that invoicing software giant Xero has been down for hours right as people try to get their submissions in by the end of the month, and Slack has had connectivity issues. It's not clear why Xero fell over; we've asked for an explanation.

For Slack, we're told it was a DNSSEC issue rather than a certificate problem. Essentially, as described here, Slack made a change to its DNS records, backed out the update, this adjustment is still propagating through the internet, and as a result, looking up the site's servers may fail for some people.

"This issue was caused by our own change and not related to any third-party DNS software and services," the chat app biz noted. In order to resolve this faster, your ISP will need to flush their DNS record for slack.com."

"Less than one per cent of users" were affected, it claimed. We note that Slack has 12 million or more daily active users, so you do the math.

We'll keep you posted on any updates. If your app or site isn't working, consult its support desk, mailing lists, or forums for any upgrades to apply or commands to run, or wait for code to automatically pick up today's changes. There's a long help thread here on the Let's Encrypt community board. ®

Updated to add

A spokesperson for Xero told us regarding today's outage: "We have identified the underlying issue that has been causing disruption to customers accessing Xero.

"This was related to a platform certificate issue at approximately 2pm UTC which we have now resolved. We have implemented a fix and our service is recovering."

Let's Encrypt's older root cert expired at, funnily enough, 2pm UTC today when Xero started breaking down. We asked if there was a connection between the outage and the expiry of DST Root CA X3.

"This issue was caused when a certificate our systems required expired and some of our subsystems did not automatically trust the new certificate," the spokesperson said, adding: "It is not related to any third-party provider."

Similar topics


Other stories you might like

  • Experts: AI should be recognized as inventors in patent law
    Plus: Police release deepfake of murdered teen in cold case, and more

    In-brief Governments around the world should pass intellectual property laws that grant rights to AI systems, two academics at the University of New South Wales in Australia argued.

    Alexandra George, and Toby Walsh, professors of law and AI, respectively, believe failing to recognize machines as inventors could have long-lasting impacts on economies and societies. 

    "If courts and governments decide that AI-made inventions cannot be patented, the implications could be huge," they wrote in a comment article published in Nature. "Funders and businesses would be less incentivized to pursue useful research using AI inventors when a return on their investment could be limited. Society could miss out on the development of worthwhile and life-saving inventions."

    Continue reading
  • Declassified and released: More secret files on US govt's emergency doomsday powers
    Nuke incoming? Quick break out the plans for rationing, censorship, property seizures, and more

    More papers describing the orders and messages the US President can issue in the event of apocalyptic crises, such as a devastating nuclear attack, have been declassified and released for all to see.

    These government files are part of a larger collection of records that discuss the nature, reach, and use of secret Presidential Emergency Action Documents: these are executive orders, announcements, and statements to Congress that are all ready to sign and send out as soon as a doomsday scenario occurs. PEADs are supposed to give America's commander-in-chief immediate extraordinary powers to overcome extraordinary events.

    PEADs have never been declassified or revealed before. They remain hush-hush, and their exact details are not publicly known.

    Continue reading
  • Stolen university credentials up for sale by Russian crooks, FBI warns
    Forget dark-web souks, thousands of these are already being traded on public bazaars

    Russian crooks are selling network credentials and virtual private network access for a "multitude" of US universities and colleges on criminal marketplaces, according to the FBI.

    According to a warning issued on Thursday, these stolen credentials sell for thousands of dollars on both dark web and public internet forums, and could lead to subsequent cyberattacks against individual employees or the schools themselves.

    "The exposure of usernames and passwords can lead to brute force credential stuffing computer network attacks, whereby attackers attempt logins across various internet sites or exploit them for subsequent cyber attacks as criminal actors take advantage of users recycling the same credentials across multiple accounts, internet sites, and services," the Feds' alert [PDF] said.

    Continue reading

Biting the hand that feeds IT © 1998–2022