Finally, made it to the weekend, time to breathe, relax, and... Cloudflare's taken down a chunk of the web

DNS provider goes dark amid bad routing, world+dog goes through nine minutes of terror


Updated Global internet glue Cloudflare experienced a brief network outage on Friday that broke multiple apps and websites, including your humble Register.

On its status page, as of Jul 17, 21:37 UTC, the DNS-and-everything-else provider said it was "investigating issues with Cloudflare Resolver and our edge network in certain locations," and warned that customers in certain regions may experience failures or errors.

Affected services included the Cloudflare API and Cloudflare Recursive DNS, both of which were listed as having degraded performance. And in various regions around the world where Cloudflare handles network traffic, the status page said data is being rerouted.

Nine minutes later, at Jul 17, 21:46 UTC, the biz announced a fix had been implemented without immediately saying what happened. Within the past few minutes, a Cloudflare spokesperson told The Register the blip was due to a blunder involving one of its routers:

This afternoon we saw an outage across some parts of our network. It was not as a result of an attack. It appears a router on our global backbone announced bad routes and caused some portions of the network to not be available. We believe we have addressed the root cause and monitoring systems for stability now. We will share more shortly – we have a team writing an update as we speak.

Cloudflare CEO Matt Prince then pointed to a single piece of equipment in Atlanta, USA, as the culprit:

He added the glitch "appears to have impacted about 50 per cent of our traffic for a bit over 20 minutes."

"The issue was caused by a mistaken configuration we were applying to a router during a routine update," said Prince. "There was no attack. It was not a failure of the router software. Blog post with details coming soon as are protections to our backbone to prevent going forward."

Because Cloudflare handles DNS services and edge computing services for many, many non-commercial and commercial websites, including The Register and our backend systems, the brief service interruption drew immediate notice.

Aside from El Reg, services said to have been affected are Authy, Digital Ocean, Discord, Downdetector, GitLab, Medium, Patron, and Riot among others.

The incident, while short-lived, is yet another reminder of the fragility of critical online services. We'll let you know when we know more. ®

Updated to add

Cloudflare has published a fairly detailed postmortem of the downtime.

"We are sorry for this outage and have already made a global change to the backbone configuration that will prevent it from being able to occur again," said CTO John Graham-Cumming.

Broader topics


Other stories you might like

  • Assange can go to UK Supreme Court (again) to fend off US extradition bid

    Top Brit judges may consider whether an American prison is just too much

    Julian Assange has won a technical victory in his ongoing battle against extradition from the UK to the United States, buying him a few more months in the relative safety of Her Majesty's Prison Belmarsh.

    Today at London's High Court, the Lord Chief Justice of England and Wales Lord Burnett approved a question on a technical point of law, having refused Assange immediate permission to appeal to the UK Supreme Court. The WikiLeaker's lawyers had asked for formal permission to pose this legal conundrum about Assange's likely treatment in US prisons to the Supreme Court:

    Continue reading
  • They see us Cinnamon Rolling, they're rating: GeckoLinux incorporates kernel 5.16 with familiar installation experience

    A nice, clean community distro that works well

    Most distros haven't got to 5.15 yet, but openSUSE's downstream project GeckoLinux boasts 5.16 of the Linux kernel and the latest Cinnamon desktop environment.

    Some of the big-name distros have lots of downstream projects. Debian has been around for decades so has umpteen, including Ubuntu, which has dozens of its own, including Linux Mint, which is arguably more popular a desktop than its parent. Some have only a few, such as Fedora. As far as we know, openSUSE has just the one – GeckoLinux.

    The SUSE-sponsored community distro has two main editions, the stable Leap, which has a slow-moving release cycle synched with the commercial SUSE Linux Enterprise; and Tumbleweed, its rolling-release distro, which gets substantial updates pretty much every day. GeckoLinux does its own editions of both: its remix of Leap is called "GeckoLinux Static", and its remix of Tumbleweed is called "GeckoLinux Rolling".

    Continue reading
  • Running Windows 10? Microsoft is preparing to fire up the update engines

    Winter Windows Is Coming

    It's coming. Microsoft is preparing to start shoveling the latest version of Windows 10 down the throats of refuseniks still clinging to older incarnations.

    The Windows Update team gave the heads-up through its Twitter orifice last week. Windows 10 2004 was already on its last gasp, have had support terminated in December. 20H2, on the other hand, should be good to go until May this year.

    Continue reading
  • Throw away your Ethernet cables* because MediaTek says Wi-Fi 7 will replace them

    *Don't do this

    MediaTek claims to have given the world's first live demo of Wi-Fi 7, and said that the upcoming wireless technology will be able to challenge wired Ethernet for high-bandwidth applications, once available.

    The fabless Taiwanese chip firm said it is currently showcasing two Wi-Fi 7 demos to key customers and industry collaborators, in order to demonstrate the technology's super-fast speeds and low latency transmission.

    Based on the IEEE 802.11be standard, the draft version of which was published last year, Wi-Fi 7 is expected to provide speeds several times faster than Wi-Fi 6 kit, offering connections of at least 30Gbps and possibly up to 40Gbps.

    Continue reading
  • Windows box won't boot? SystemRescue 9 may help

    An ISO image you can burn or drop onto a USB key

    The latest version of an old friend of the jobbing support bod has delivered a new kernel to help with fixing Microsoft's finest.

    It used to be called the System Rescue CD, but who uses CDs any more? Enter SystemRescue, an ISO image that you can burn, or just drop onto your Ventoy USB key, and which may help you to fix a borked Windows box. Or a borked Linux box, come to that.

    SystemRescue 9 includes Linux kernel 5.15 and a minimal Xfce 4.16 desktop (which isn't loaded by default). There is a modest selection of GUI tools: Firefox, VNC and RDP clients and servers, and various connectivity tools – SSH, FTP, IRC. There's also some security-related stuff such as Yubikey setup, KeePass, token management, and so on. The main course is a bunch of the usual Linux tools for partitioning, formatting, copying, and imaging disks. You can check SMART status, mount LVM volumes, rsync files, and other handy stuff.

    Continue reading

Biting the hand that feeds IT © 1998–2022