Data Centre


Internet's safe-keepers forced to postpone crucial DNSSEC root key signing ceremony – no, not a hacker attack, but because they can't open a safe

Online security process stalled by offline security screw-up

Got Tips? 84

The organization that keeps the internet running behind-the-scenes was forced to delay an important update to the global network – because it was locked out of one of its own safes.

“During routine administrative maintenance of our Key Management Facility on 11 February, we identified an equipment malfunction,” explained Kim Davies, the head of the Internet Assigned Numbers Authority (IANA), in an email to the dozen or so people expected to attend a quarterly ceremony in southern California at lunchtime on Wednesday.

The malfunction “will prevent us from successfully conducting the ceremony as originally scheduled" on February 12, Davis explained. “The issue disables access to one of the secure safes that contains material for the ceremony.” In other words, IANA locked itself out.

The ceremony sees several trusted internet engineers (a minimum of three and up to seven) from across the world descend on one of two secure locations – one in El Segundo, California, just south of Los Angeles, and the other in Culpeper, Virginia – both in America, every three months.

Once in place, they run through a lengthy series of steps and checks to cryptographically sign the digital key pairs used to secure the internet’s root zone. (Here's Cloudflare's in-depth explanation, and IANA's PDF step-by-step guide.)

At the heart of the matter, simply put, is the Key Signing Key (KSK): this is a public-private key pair, with the private portion kept locked away by IANA. This is because the KSK is used, every three months, to sign a set of Zone Signing Keys, which are used to secure official copies of the internet's root zone file. That file acts as a kind of directory for other parts of the internet, and these parts in turn, provide information on more of the internet. It is, in a way, the blueprint for how the internet as we know it is glued together: how domain names resolve to computers on the global network, so that when you visit, say,, you eventually reach one of our servers at network address

Critical root DNS servers are spread out around the planet, each armed with a copy of the latest signed root zone file, and used, in a distributed, cascading manner, by other DNS servers to look up domain names for the internet's users. These servers can check the root zone file underpinning all of this is secured by a ZSK recently signed by the central IANA KSK, and thus can be treated and trusted as gospel. The KSK is thus the domain-name system's trust anchor. Everything relies on it to ensure the 'net's central directory is laid out the way it should be, according to IANA, anyway.

This is all necessary because it should be immediately obvious whether or not a root zone file is an unsigned forgery, or an authentic and clean copy secured by IANA's KSK. Otherwise, a well-resourced malicious organization could potentially fool networks into using a sabotaged root zone file that redirects vast quantities of traffic, i.e. billions of internet users, to different parts of the internet. Even worse, if someone were to get hold of the KSK, they could sign their own zone file and have the internet blindly trust it. The result would be a global loss of trust in the 'net's functioning.

Security up the wazoo

For that reason, IANA takes its Root Key Signing Key Ceremony extremely seriously, and has a complex and somewhat convoluted DNSSEC-based process that briefly unlocks the private portion of the KSK to sign the ZSKs every three months. Only during this ceremony is the KSK used, and put away again when it is over, leaving IANA with a set of ZSKs to authoritatively secure its root zone.

Only specific named people are allowed to take part in the ceremony, and they have to pass through several layers of security – including doors that can only be opened through fingerprint and retinal scans – before getting in the room where the ceremony takes place.

Staff open up two safes, each roughly one-metre across. One contains a hardware security module that contains the private portion of the KSK. The module is activated, allowing the KSK private key to sign keys, using smart cards assigned to the ceremony participants. These credentials are stored in deposit boxes and tamper-proof bags in the second safe. Each step is checked by everyone else, and the event is livestreamed. Once the ceremony is complete – which takes a few hours – all the pieces are separated, sealed, and put back in the safes inside the secure facility, and everyone leaves.

You're ARIN a laugh: Critical internet org accused of undercutting security over legal fears


But during what was apparently a check on the system on Tuesday night – the day before the ceremony planned for 1300 PST (2100 UTC) Wednesday – IANA staff discovered that they couldn’t open one of the two safes. One of the locking mechanisms wouldn’t retract and so the safe stayed stubbornly shut.

As soon as they discovered the problem, everyone involved, including those who had flown in for the occasion, were told that the ceremony was being postponed. Thanks to the complexity of the problem – a jammed safe with critical and sensitive equipment inside – they were told it wasn’t going to be possible to hold the ceremony on the back-up date of Thursday, either.

We understand, however, that following an emergency meeting on Wednesday, the issue should be fixed by Friday, and the ceremony has now been moved to Saturday. In the meantime, some lucky locksmith in Los Angeles is going to have to drill out the safe’s locking mechanism and put in a new one.

Fortunately, apart from the inconvenience, there is no impact on the internet itself, particularly in this short term. The current arrangement will simply continue to do its job for three additional days. And IANA has been keen to point out that it has an identical set of equipment on the other coast of the US that can also be used if necessary.

“We apologize for the inconvenience for the attendees who had already traveled to participate in the ceremony. This is the first time a ceremony has needed to be rescheduled in the 10-year history of KSK management,” the email announcing the delay noted.

There is a certain irony, of course, that the security of the virtual internet has been held hostage by an old-school physical safe. ®

Sign up to our NewsletterGet IT in your inbox daily


Keep Reading

Internet root keymasters must think they're cursed: First, a dodgy safe. Now, coronavirus upends IANA ceremony

Pandemic lockdown forces new measures on crucial crypto process than underpins world's DNS

Game over: IANA power-grab block pulled from Congress funding bill

Cruz 'profoundly disappointed' as attempt to screw country fails

APNIC receives recovered /12 from IANA

IPv4 is still breathing

Internet handover is go-go-go! ICANN to take IANA from US govt

Judge refuses injunction, handover of global DNS etc at midnight

Despite IANA storm, ICANN shows just why it shouldn't be allowed to take over internet's critical functions

Self-serving organization simply incapable of admitting fault

IANA starts handing out recovered IPv4 addresses

NICs take begging bowls to beadle

US government asks internet community how long it should extend IANA contract

Request comes as latest plans put out to public comment

Internet community split on who should run IANA

ICANN Board pushes for ICANN control; others disagree

After 20-year battle, Channel island Sark finally earns the right to exist on the internet with its own top-level domain

Special report We talk to the guy who spent decades trying to make it happen

How do you create an SLA and status page for the whole internet? Meet IANA: Keepers of DNS

Running the web without the US at the helm – and in Java

Tech Resources

Ransomware has gone nuclear

A new generation of attackers are crafting plans to cause the most panic, pain, and operational disruption. They will take the time to maximize your organization’s potential damage and also their payoff -- not just encrypting your data, but stealing it and posting it publicly if you don’t play ball. Join Roger Grimes from KnowBe4 and Tim Phillips from The Reg for a RegCast in which they will be sounding the ransomware emergency klaxon.

Keeping a Security Mindset

Leaders and experts discuss ways you can bolster secure remote working through education and effective security measures

Network Detection & Response for MITRE ATT&CK Framework

Read the white paper for a high-level view of how enterprise NTA with ExtraHop Reveal(x) detects and enables investigation of a broad range of the TTPs catalogued by MITRE ATT&CK!

Simplify Endpoint Backup with Druva

To ensure data protection and business continuity, it’s critical to deploy an endpoint backup solution.