This article is more than 1 year old

Internet's safe-keepers forced to postpone crucial DNSSEC root key signing ceremony – no, not a hacker attack, but because they can't open a safe

Online security process stalled by offline security screw-up

The organization that keeps the internet running behind-the-scenes was forced to delay an important update to the global network – because it was locked out of one of its own safes.

“During routine administrative maintenance of our Key Management Facility on 11 February, we identified an equipment malfunction,” explained Kim Davies, the head of the Internet Assigned Numbers Authority (IANA), in an email to the dozen or so people expected to attend a quarterly ceremony in southern California at lunchtime on Wednesday.

The malfunction “will prevent us from successfully conducting the ceremony as originally scheduled" on February 12, Davis explained. “The issue disables access to one of the secure safes that contains material for the ceremony.” In other words, IANA locked itself out.

The ceremony sees several trusted internet engineers (a minimum of three and up to seven) from across the world descend on one of two secure locations – one in El Segundo, California, just south of Los Angeles, and the other in Culpeper, Virginia – both in America, every three months.

Once in place, they run through a lengthy series of steps and checks to cryptographically sign the digital key pairs used to secure the internet’s root zone. (Here's Cloudflare's in-depth explanation, and IANA's PDF step-by-step guide.)

At the heart of the matter, simply put, is the Key Signing Key (KSK): this is a public-private key pair, with the private portion kept locked away by IANA. This is because the KSK is used, every three months, to sign a set of Zone Signing Keys, which are used to secure official copies of the internet's root zone file. That file acts as a kind of directory for other parts of the internet, and these parts in turn, provide information on more of the internet. It is, in a way, the blueprint for how the internet as we know it is glued together: how domain names resolve to computers on the global network, so that when you visit, say,, you eventually reach one of our servers at network address

Critical root DNS servers are spread out around the planet, each armed with a copy of the latest signed root zone file, and used, in a distributed, cascading manner, by other DNS servers to look up domain names for the internet's users. These servers can check the root zone file underpinning all of this is secured by a ZSK recently signed by the central IANA KSK, and thus can be treated and trusted as gospel. The KSK is thus the domain-name system's trust anchor. Everything relies on it to ensure the 'net's central directory is laid out the way it should be, according to IANA, anyway.

This is all necessary because it should be immediately obvious whether or not a root zone file is an unsigned forgery, or an authentic and clean copy secured by IANA's KSK. Otherwise, a well-resourced malicious organization could potentially fool networks into using a sabotaged root zone file that redirects vast quantities of traffic, i.e. billions of internet users, to different parts of the internet. Even worse, if someone were to get hold of the KSK, they could sign their own zone file and have the internet blindly trust it. The result would be a global loss of trust in the 'net's functioning.

Security up the wazoo

For that reason, IANA takes its Root Key Signing Key Ceremony extremely seriously, and has a complex and somewhat convoluted DNSSEC-based process that briefly unlocks the private portion of the KSK to sign the ZSKs every three months. Only during this ceremony is the KSK used, and put away again when it is over, leaving IANA with a set of ZSKs to authoritatively secure its root zone.

Only specific named people are allowed to take part in the ceremony, and they have to pass through several layers of security – including doors that can only be opened through fingerprint and retinal scans – before getting in the room where the ceremony takes place.

Staff open up two safes, each roughly one-metre across. One contains a hardware security module that contains the private portion of the KSK. The module is activated, allowing the KSK private key to sign keys, using smart cards assigned to the ceremony participants. These credentials are stored in deposit boxes and tamper-proof bags in the second safe. Each step is checked by everyone else, and the event is livestreamed. Once the ceremony is complete – which takes a few hours – all the pieces are separated, sealed, and put back in the safes inside the secure facility, and everyone leaves.

Arguing over a contract

You're ARIN a laugh: Critical internet org accused of undercutting security over legal fears


But during what was apparently a check on the system on Tuesday night – the day before the ceremony planned for 1300 PST (2100 UTC) Wednesday – IANA staff discovered that they couldn’t open one of the two safes. One of the locking mechanisms wouldn’t retract and so the safe stayed stubbornly shut.

As soon as they discovered the problem, everyone involved, including those who had flown in for the occasion, were told that the ceremony was being postponed. Thanks to the complexity of the problem – a jammed safe with critical and sensitive equipment inside – they were told it wasn’t going to be possible to hold the ceremony on the back-up date of Thursday, either.

We understand, however, that following an emergency meeting on Wednesday, the issue should be fixed by Friday, and the ceremony has now been moved to Saturday. In the meantime, some lucky locksmith in Los Angeles is going to have to drill out the safe’s locking mechanism and put in a new one.

Fortunately, apart from the inconvenience, there is no impact on the internet itself, particularly in this short term. The current arrangement will simply continue to do its job for three additional days. And IANA has been keen to point out that it has an identical set of equipment on the other coast of the US that can also be used if necessary.

“We apologize for the inconvenience for the attendees who had already traveled to participate in the ceremony. This is the first time a ceremony has needed to be rescheduled in the 10-year history of KSK management,” the email announcing the delay noted.

There is a certain irony, of course, that the security of the virtual internet has been held hostage by an old-school physical safe. ®

More about


Send us news

Other stories you might like