Internet root keymasters must think they're cursed: First, a dodgy safe. Now, coronavirus upends IANA ceremony

Pandemic lockdown forces new measures on crucial crypto process than underpins world's DNS

IANA – the body that oversees the internet's IP addresses and domain names – must think it's under a curse in its quest to protect the 'net. Last time it was a malfunctioning safe that blocked its important work to keep the global network glued together.

This time, coronavirus.

Every quarter, a small group of people cram inside a secure facility in either California or Virginia in America, get locked in, and spend the next two to three hours cryptographically signing the digital key pairs used to secure the internet’s root zone, the text file that shapes the 'net as we know it.

The integrity of the digital signing process is so critical that the organization that runs it, IANA, part of DNS overlord ICANN, flies in trusted internet community representatives from across the world, out of a pool of 14, to methodically run through the steps. These representatives each possess a set of physical keys required to gain access to the necessary equipment, held in safe deposit boxes in IANA's key-signing facilities.

Once inside one of these facilities, which requires the use of fingerprint and retinal scanners, those present use their physical keys to access electronic key cards that activate a special locked-away device – a hardware security module (HSM) – that signs the digital key pairs for the root zone file for the next three months. Every step is meticulously recorded, and no one is allowed to enter or leave until the job is done.

This Thursday, the 41st of these ceremonies will take place, and, as you may have already gathered, the ongoing coronavirus pandemic has thrown a spanner in the works.

First, it is not terribly easy to fly people into either California or Virginia due to global travel restrictions and virus safety concerns. The 41st ceremony is supposed to take place in Virginia – it alternates between the two, the 40th being in Cali. IANA staff are based in Los Angeles, California, for what it's worth.

Second, even if you can fly in the reps, how do you obey mandatory social distancing rules while squashed inside a metal cage?

Cage fighting

These were the questions that landed on the desk of Kim Davies, IANA’s point man, just two months after he was forced to deal with another minor crisis surrounding the ceremony: during a test run on the day before the 40th ceremony, the main safe containing some of the necessary key-signing equipment was found to have jammed.

They couldn’t get in and had to take emergency measures. Ultimately that comprised hiring a specialist locksmith to drill out the safe’s lock and put everyone up in hotels for an extra three nights (yes, it took that long to drill the lock out – 20 hours – with IANA staff taking rotating shifts to make sure the location was kept secure.) It was the first time in 10 years the ceremony had been delayed.

Incidentally, we have a picture of the poor locksmith who had to spend two full working days lying on the floor of a secure facility drilling out a government-grade lock. Here he is:


Ever spent 20 hours on your side drilling a lock? Don't do it ... Click to enlarge

This time, however, it wasn’t the equipment but the people that were the problem. Fortunately, California’s stay-at-home lockdown happened a few weeks rather than several hours before the ceremony so the IANA folks had time to figure out a solution and get an official sign-off from ICANN’s board for the new plan.

It is far from perfect but it’s the best they can do given the circumstances.

Dressed to the nines

Instead of the ceremony securing the internet's root zone file for the next three months, this ceremony will do so for nine months, because no one is sure when the next ceremony can take place in the proper way.


Internet's safe-keepers forced to postpone crucial DNSSEC root key signing ceremony – no, not a hacker attack, but because they can't open a safe


It will be held again in El Segundo, California, rather than Culpeper, Virginia, due to the location of IANA and ICANN staff. And the trusted reps will be replaced with ICANN staff. They had considered drilling out the three safe deposit boxes to get access to the electronic cards inside but the hassle from February’s safe episode pushed the next obvious route: get the selected reps to send their physical keys in tamper proof bags and secure mail to the ICANN reps in California.

“The TCRs have wrapped their deposit keys with opaque material, and then transmitted them in tamper-evident bags,” explained IANA. “This bag will not be opened until within the ceremony so that each TCR can witness their key is in the same condition as when they released it. At the conclusion of the ceremony, the four keys will be similarly wrapped and then entrusted to four staff members who will independently arrange for them to be couriered back to their respective TCRs.”

The trusted reps, who are based in Mauritius, Spain, Russia, Tanzania, Uruguay, and the east coast of the US, will then watch their keys be used to open the boxes via a YouTube livestream and a secure side-chat will see them take an active role in the ceremony. The independent auditor will also be watching the ceremony from YouTube, rather than sitting in the room.

Meanwhile those in the room will be dressed like emergency room doctors, covered in PPE, and doing their best to distance themselves.

What about getting to the facility in the first place? IANA had to get a special waiver to access the space and it has informed the “relevant government agencies” of its plans to break the rules over non-essential services in Los Angeles.

Extras from ET

If all goes according to plan, the IANA and ICANN staff, dressed like they are trying to kidnap ET, will be directed over YouTube chat by people thousand of miles away, while keeping as much distance from each other as possible, and then spend hours methodically digitally signing key pairs so the internet can be secure until early 2021. It will add a surreal layer to an already unusual ceremony.

IANA has already started thinking about what it can do differently in future: would extra locations outside the US help? Or would it increase the security risks? Does flying in people from across the world to a single location make sense any more in a post-pandemic world? Or should the internet’s custodians figure out a way to distribute the process, both in terms of people and equipment?

Should IANA create a standby key system in case something goes awry in future? And how would that be secured and/or used? And is there any point in performing the ceremonies at all in their current form?

Those are the questions that will be asked from this Friday. In the meantime, a ceremony that was designed to be so methodically carried out that it is supposed to be almost painfully, predictably dull [PDF] has again proven to be anything but. Let’s all pray the YouTube livestream doesn’t cut out. ®

Other stories you might like

  • Microsoft Azure to spin up AMD MI200 GPU clusters for 'large scale' AI training
    Windows giant carries a PyTorch for chip designer and its rival Nvidia

    Microsoft Build Microsoft Azure on Thursday revealed it will use AMD's top-tier MI200 Instinct GPUs to perform “large-scale” AI training in the cloud.

    “Azure will be the first public cloud to deploy clusters of AMD's flagship MI200 GPUs for large-scale AI training,” Microsoft CTO Kevin Scott said during the company’s Build conference this week. “We've already started testing these clusters using some of our own AI workloads with great performance.”

    AMD launched its MI200-series GPUs at its Accelerated Datacenter event last fall. The GPUs are based on AMD’s CDNA2 architecture and pack 58 billion transistors and up to 128GB of high-bandwidth memory into a dual-die package.

    Continue reading
  • New York City rips out last city-owned public payphones
    Y'know, those large cellphones fixed in place that you share with everyone and have to put coins in. Y'know, those metal disks representing...

    New York City this week ripped out its last municipally-owned payphones from Times Square to make room for Wi-Fi kiosks from city infrastructure project LinkNYC.

    "NYC's last free-standing payphones were removed today; they'll be replaced with a Link, boosting accessibility and connectivity across the city," LinkNYC said via Twitter.

    Manhattan Borough President Mark Levine said, "Truly the end of an era but also, hopefully, the start of a new one with more equity in technology access!"

    Continue reading
  • Cheers ransomware hits VMware ESXi systems
    Now we can say extortionware has jumped the shark

    Another ransomware strain is targeting VMware ESXi servers, which have been the focus of extortionists and other miscreants in recent months.

    ESXi, a bare-metal hypervisor used by a broad range of organizations throughout the world, has become the target of such ransomware families as LockBit, Hive, and RansomEXX. The ubiquitous use of the technology, and the size of some companies that use it has made it an efficient way for crooks to infect large numbers of virtualized systems and connected devices and equipment, according to researchers with Trend Micro.

    "ESXi is widely used in enterprise settings for server virtualization," Trend Micro noted in a write-up this week. "It is therefore a popular target for ransomware attacks … Compromising ESXi servers has been a scheme used by some notorious cybercriminal groups because it is a means to swiftly spread the ransomware to many devices."

    Continue reading
  • Twitter founder Dorsey beats hasty retweet from the board
    As shareholders sue the social network amid Elon Musk's takeover scramble

    Twitter has officially entered the post-Dorsey age: its founder and two-time CEO's board term expired Wednesday, marking the first time the social media company hasn't had him around in some capacity.

    Jack Dorsey announced his resignation as Twitter chief exec in November 2021, and passed the baton to Parag Agrawal while remaining on the board. Now that board term has ended, and Dorsey has stepped down as expected. Agrawal has taken Dorsey's board seat; Salesforce co-CEO Bret Taylor has assumed the role of Twitter's board chair. 

    In his resignation announcement, Dorsey – who co-founded and is CEO of Block (formerly Square) – said having founders leading the companies they created can be severely limiting for an organization and can serve as a single point of failure. "I believe it's critical a company can stand on its own, free of its founder's influence or direction," Dorsey said. He didn't respond to a request for further comment today. 

    Continue reading
  • Snowflake stock drops as some top customers cut usage
    You might say its valuation is melting away

    IPO darling Snowflake's share price took a beating in an already bearish market for tech stocks after filing weaker than expected financial guidance amid a slowdown in orders from some of its largest customers.

    For its first quarter of fiscal 2023, ended April 30, Snowflake's revenue grew 85 percent year-on-year to $422.4 million. The company made an operating loss of $188.8 million, albeit down from $205.6 million a year ago.

    Although surpassing revenue expectations, the cloud-based data warehousing business saw its valuation tumble 16 percent in extended trading on Wednesday. Its stock price dived from $133 apiece to $117 in after-hours trading, and today is cruising back at $127. That stumble arrived amid a general tech stock sell-off some observers said was overdue.

    Continue reading

Biting the hand that feeds IT © 1998–2022