Firefox, you know you tapped Cloudflare for DNS-over-HTTPS? In January, it briefly knackered two root servers at the heart of the internet

Probe raises serious questions about private v public web management

Updated A bug in software pushed out by Cloudflare resulted in failures at the heart of the web's infrastructure, according to a report published this week by the Internet Systems Consortium (ISC).

ISC runs the so-called F root server; one of the world's 13 root DNS servers, labeled A through M. These are the central computers that underpin the global internet: they ensure, for instance, that when you visit, you are directed to the correct system serving our homepage.

On January 23 this year, ISC received a report of a breakdown with .net domains. When it investigated, it discovered crucial A and AAAA records, which glue .net domain names to their IPv4 and IPv6 network addresses, were missing.

In essence, all internet addresses ending in .net – one of the internet’s largest registries with 13.4 million domain names – vanished from ISC's F root machine. Any browser, app, computer or device that, ultimately, relied on the F root machine to connect to websites and services would, worst case scenario, have been unable to reach those systems via their .net addresses.

The issue wasn’t restricted to just ISC's F root, either; the report [PDF] said similar problems were experienced by the E root, run by NASA.

Bug fixes

ISC quickly figured out – within five minutes, according to its timeline – that the issue lay with internet nodes it operates in partnership with Cloudflare, and escalated the issue to the web infrastructure business. Cloudflare also acted quickly: within 21 minutes it had identified that a specific code release, designed to fix a bug that it had introduced four hours earlier, was responsible.

Here's where the report takes a hard left into the fragile world of BGP: the Border Gateway Protocol used by the internet's sprawling galaxy of networks to automatically organize each other and maintain connections between themselves. How BGP is involved in a DNS root zone issue is not clear, and we've asked Cloudflare for a more detailed explanation.

Regardless, it took nearly two hours to withdraw a BGP announcement that was causing the problem, something ISC notes should have happened faster. “In retrospect, we should have initiated the withdrawal of the route prefixes from BGP as soon as it was identified that incomplete / incorrect data was being served,” the report stated under “lesson learned.”

It continued: “The withdrawal of routes did not go as smoothly as expected and Cloudflare and ISC have agreed to perform regular tests to exercise that function... The test suite has been updated to include tests for missing glue, and ISC and Cloudflare will work to devise further conformance tests.”

Hello money, goodbye stability

Thanks to the way that the world's DNS works, with information cascading down through a distributed hierarchy of name servers, redundancy provisions, and caches – and globally updated every few hours to every few seconds – the impact on netizens was absolutely minimal. With the E and F roots temporarily knackered, browsers and apps would have found other ways to look up .net addresses.

However, the situation is serious in large part because a fundamental underpinning of the public internet’s global addressing system was knocked over through a minor software update by one private organization.

Firefox logo

Firefox now defaults to DNS-over-HTTPS for US netizens and some are dischuffed about this


A software update carried out by Cloudflare, a commercial entity that uses a mix of open and closed software. The internet has achieved such a remarkable degree of uptime despite decades of exponential growth due to its tradition of open-source software, carefully checked and tested updates, and maintainer organizations that are kept separate from commercial considerations.

As one veteran internet engineer, Bill Woodcock, noted on Twitter: “What happens when critical functions of the public Internet are co-opted for private benefit? Transparency and accountability are lost, infrastructural spending cut, things break.”

The issue is not an academic one either. Woodcock sounded the alarm recently over the proposed sale of .org to an unknown private equity company – his company provides technical back-end services for the internet registry. Given the profit motive of the proposed purchaser, he concluded that there was likely to be a significant cutback on technical spending, putting the stability of the critical registry at risk. He sent a letter to DNS overseer ICANN about the issue, recommending that it stop the proposed sale.

That’s not the only internet engineer concerned either. Bert Hubert, whose company produces open-source DNS software, noted with respect to the ISC report that “closed source Cloudflare software had a bug which caused closed source Akamai to break over at a large US cable access provider.”

Break point

Hubert has recently been very vocal over his concerns that Firefox will be using Cloudflare as the default provider for its secure DNS, DoH, protocol: something that happened for all Firefox users in the US this morning.

If a software bug in closed Cloudflare software can cause a root server to vanish an entire, significant piece of the internet then it is all too possible – in fact, likely – that at some point a similar issue will cause Firefox users to lose their secure DNS connections. And that could cause them to lose the internet altogether (it would still be there, but most users would have no idea what the cause was or how to get around it.)

There is a famous phrase often repeated by internet engineers, and originally coined by EFF co-founder John Gilmore, that “the Net interprets censorship as damage and routes around it.” That statement has taken on much broader meaning and is often employed by engineers to basically say “don’t worry about it, the internet breaks all the time.” And it does, every second, and mends itself almost immediately.

But with the growing commercialism of the internet and with private and profit-driven companies increasingly inserting themselves into the foundational layer of the internet’s infrastructure, the report by ISC over this F root incident may well be a warning for what is coming.

We've asked Cloudflare for comment and will update this story when it gets back. ®

Updated to add

In a conversation with El Reg after this article was published, Cloudflare's spokespeople rejected any notion that the closed nature of its software was to blame. "This was very much an edge case," Cloudflare's distinguished engineer Martin Levy told The Register. "It's not fair to look at this in binary terms: that open source is good, and closed bad. We put an enormous amount of software into the open source world."

He added Cloudflare put its code-to-be-deployed through "extreme testing but we hadn't noticed this special case," and that any disruption was "highly localized." The code change was, to put it simply, an improvement in character encoding handling for a particular customer that had an unexpected knock-on effect on the root servers.

To us, it seems a BGP route break caused the root servers to abandon their gTLD A and AAAA records, possibly because they were fetching those details from another system they could no longer reach. See the final two pages of the report PDF.

Also, this affected all domain names – not just .net – handled by the F and E root servers, though .net stood out to ISC because it's a rather large and important corner of the web.

Full disclosure: The Register is a Cloudflare customer.

Similar topics

Broader topics

Other stories you might like

  • Stolen university credentials up for sale by Russian crooks, FBI warns
    Forget dark-web souks, thousands of these are already being traded on public bazaars

    Russian crooks are selling network credentials and virtual private network access for a "multitude" of US universities and colleges on criminal marketplaces, according to the FBI.

    According to a warning issued on Thursday, these stolen credentials sell for thousands of dollars on both dark web and public internet forums, and could lead to subsequent cyberattacks against individual employees or the schools themselves.

    "The exposure of usernames and passwords can lead to brute force credential stuffing computer network attacks, whereby attackers attempt logins across various internet sites or exploit them for subsequent cyber attacks as criminal actors take advantage of users recycling the same credentials across multiple accounts, internet sites, and services," the Feds' alert [PDF] said.

    Continue reading
  • Big Tech loves talking up privacy – while trying to kill privacy legislation
    Study claims Amazon, Apple, Google, Meta, Microsoft work to derail data rules

    Amazon, Apple, Google, Meta, and Microsoft often support privacy in public statements, but behind the scenes they've been working through some common organizations to weaken or kill privacy legislation in US states.

    That's according to a report this week from news non-profit The Markup, which said the corporations hire lobbyists from the same few groups and law firms to defang or drown state privacy bills.

    The report examined 31 states when state legislatures were considering privacy legislation and identified 445 lobbyists and lobbying firms working on behalf of Amazon, Apple, Google, Meta, and Microsoft, along with industry groups like TechNet and the State Privacy and Security Coalition.

    Continue reading
  • SEC probes Musk for not properly disclosing Twitter stake
    Meanwhile, social network's board rejects resignation of one its directors

    America's financial watchdog is investigating whether Elon Musk adequately disclosed his purchase of Twitter shares last month, just as his bid to take over the social media company hangs in the balance. 

    A letter [PDF] from the SEC addressed to the tech billionaire said he "[did] not appear" to have filed the proper form detailing his 9.2 percent stake in Twitter "required 10 days from the date of acquisition," and asked him to provide more information. Musk's shares made him one of Twitter's largest shareholders. The letter is dated April 4, and was shared this week by the regulator.

    Musk quickly moved to try and buy the whole company outright in a deal initially worth over $44 billion. Musk sold a chunk of his shares in Tesla worth $8.4 billion and bagged another $7.14 billion from investors to help finance the $21 billion he promised to put forward for the deal. The remaining $25.5 billion bill was secured via debt financing by Morgan Stanley, Bank of America, Barclays, and others. But the takeover is not going smoothly.

    Continue reading

Biting the hand that feeds IT © 1998–2022