Security

23,000 HTTPS certs will be axed in next 24 hours after private keys leak

Trustico, DigiCert come to blows as browsers prepare to snub Symantec-brand SSL


Customers of HTTPS certificate reseller Trustico are reeling after being told their website security certs – as many as 23,000 – will be rendered useless within the next 24 hours.

This is allegedly due to a security blunder in which the private keys for said certificates ended up in an email sent by Trustico. Those keys are supposed to be secret, and only held by the cert owners, and certainly not to be disclosed in messages. In the wrong hands, they can be used by malicious websites to masquerade as legit operations.

Unless the affected certificates are replaced in time, visitors to websites using Trustico-sold HTTPS certs will be turned away by their browsers, due to the digital certificates being revoked.

The whole situation is a mess, and possibly the result of a turf war. Here's what we've managed to ascertain.

What is Trustico?

Trustico, based in Croydon, UK, touted SSL/TLS certificates, which are used by websites to encrypt and secure their connections. It resold certs from the Symantec brand umbrella: Symantec, GeoTrust, Thawte, and RapidSSL. This umbrella is now owned and operated by DigiCert.

If you wanted to buy, say, a RapidSSL-issued certificate, you could do so via Trustico. The HTTPS cert ultimately leads back, along a chain of trust, to DigiCert, a root certificate authority trusted by web browsers and other software. In turn, a website presenting the Trustico-sold cert is trusted, its traffic secured using encryption, and the reassuring green padlock is displayed in visitors' browsers.

Why are the certificates being revoked?

According to DigiCert's chief product officer Jeremy Rowley earlier today, Trustico told DigiCert in early February that its resold certificates had been in some way "compromised," and that the certs needed to be mass revoked as a result.

DigiCert staff, we're told, asked Trustico for more information on this security mishap. The reseller replied it had a copy of the private keys, which is usually grounds for revocation, and thus insisted that DigiCert revoke the certificates.

When pressed for evidence, Trustico on Tuesday simply emailed DigiCert 23,000 certificates' private keys as proof it held this information, it is claimed. This forced DigiCert's hand: under the rulebook of standards set by the elders of the certificate security and browser worlds, the Trustico-sold certificates had to be revoked as a precaution within 24 hours. Specifically, the ones with their private keys in the email will be canceled.

"Trustico has not provided any information about how these certificates were compromised or how they acquired the private keys," explained Rowley.

"As is standard practice for a Certificate Authority, DigiCert never had possession of these private keys. Currently, we are only revoking the certificates if we received the private keys. There are additional certificates the reseller requested to have revoked, but DigiCert has decided to disregard that request until we receive proof of compromise or more information about the cause of this incident."

On Twitter, Rowley continued: "I'll likely be posting the private keys later once people have a fair chance to replace their certificates ... The allegation of compromise, keys compromised, and request for revocation all came from Trustico."

Before you raise an eyebrow too high, by posting the private keys, Rowley plans to disclose self-signed certificates, produced using the private keys, to prove the secret information was sent to DigiCert without revealing the actual information in public. Some have already popped online as proof DigiCert received the secret keys from Trustico.

Alarm bells

To warn netizens to the upcoming mass revocation, DigiCert's RapidSSL business sent out email alerts to Trustico customers urging them to get new HTTPS certificates or watch their sites go dark. Here's a copy of the memo, passed to El Reg:

Red alert ... Click to enlarge

DigiCert also put out a blog post, giving its side of the story:

Trustico requested revocation of their Symantec, GeoTrust, Thawte and RapidSSL certificates, claiming the certificates were compromised. When we asked for proof of the “compromise,” Trustico did not provide details on why they were requesting the immediate revocation. Trustico’s CEO indicated that Trustico held the private keys for those certificates, and then emailed us approximately 20,000 certificate private keys.

When he sent us those keys, his action gave us no choice but to act in accordance with the CA/Browser Forum Baseline Requirements, which mandate that we revoke a compromised certificate within 24 hours. As a CA, we had no choice but to follow the Baseline Requirements.

Following our standard revocation process, we gave notice via email to each certificate holder whose private keys had been exposed to us by Trustico, so they could have time to get a replacement certificate.

Now, over to Trustico.

Upset and denials

We asked the Brit biz for comment, and had yet to hear back at time of writing. However, posting on Mozilla's security policy newsgroup, Trustico product manager Zane Lucas was clearly upset that DigiCert sent out the above alert.

"We didn't authorise DigiCert to contact our customers and we didn't approve the content of their email," wrote Lucas.

"At no time had any private keys been compromised, nor had we ever informed to you that any private keys had been compromised. During our many discussions over the past week we put it to you that we believe Symantec to have operated our account in a manner whereby it had been compromised. Your usage of the word compromise has been twisted by you to your benefit and is absolutely defamatory."

To put this in context: Trustico was fed up with using Symantec certs, and on February 13, it formally abandoned the umbrella of brands – ahead of Google Chrome and Mozilla Firefox officially distrusting the certificates due to past security fumbles by Symantec. Trustico said it had complained privately to Symantec of long-running concerns over the security safeguards on Symantec-branded of certificates, hence Lucas' reference to its Symantec account.

Although Lucas stressed the private keys for Trustico's resold certificates were not compromised, it did, according to DigiCert, email a copy of 23,000 of them to the root authority seemingly to trigger their revocation. At that point, DigiCert considered the certificates at risk, and started the countdown clock to cancel them.

Trustico and DigiCert have clearly majorly fallen out, with the pair going their separate ways this month amid the behind-the-scenes drama. It even appears Trustico tried to stop DigiCert from using its online portal to send out today's emailed warning.

In future, Trustico will flog Comodo HTTPS certificates rather than peddle Symantec-branded certs. Cynics have suggested the Brit reseller ordered the revocation of its Symantec-umbrella certs so it could drive its customers onto Comodo certificates, and thus avoid the looming Google Chrome HTTPS certificate apocalypse without losing many, if any, punters. In effect, website owners have been caught up in a turf war between Trustico and DigiCert.

How did Trustico get the private keys to certificates it resold? We don't know for sure – but it did, and still does, offer an online private key generator for certificates. Just saying.

In an email sent to customers a few hours ago, and seen by The Register, Trustico said it will provide free certificates to replace the soon-to-be-nuked SSL/TLS certs:

Recently we wrote to you to let you know that we are no longer offering Symantec, GeoTrust, RapidSSL and Thawte branded SSL Certificates. Unfortunately, Google Chrome has decided to distrust these SSL Certificates. It's important to us that you SSL Certificate continues to function as normal, and not be compromised by the distrust of the Symantec brands. It is now required that you replace any existing distrusted SSL Certificate with one that is trusted by all web browsers.

Rest assured, there hasn't been any type of compromise of our systems. However, Symantec brands will cease to function correctly due to Google Chrome's decision to distrust them.

Recently DigiCert acquired the Symantec SSL Certificate division and subsequently an e-mail was sent by DigiCert to some of our SSL Certificate customers advising of the revocation of their distrusted SSL Certificate. We didn't authorise this e-mail to be sent and had specifically disabled it within the DigiCert system. We understand that the e-mail sent about your distrusted SSL Certificates may be confusing. It's important that you take the opportunity to replace your SSL Certificate as soon as possible.

We're providing free replacement of affected SSL Certificates. To enable a free replacement, you'll receive an e-mail report today if you have affected SSL Certificates. Your report will contain a unique coupon code for each affected SSL Certificate. When you replace your distrusted SSL Certificates using your unique coupon codes you'll receive extra validity free of charge. If you have any questions please feel free to reply to this e-mail.

Meanwhile, DigiCert said it, too, will offer free replacement certs to folks using Symantec-branded HTTPS certificates, which will be ignored by web browsers later this year. And, of course, don't forget you can grab free HTTPS certificates from Let's Encrypt that all major browsers trust.

Today has been marred with confusion. Trustico's customer support lines have been jammed with complaints and queries, following DigiCert's email alerts. Reg readers told us they felt left in the dark. Perhaps it'll all be clearer in a few hours, when the dust has settled – and the certs have been nuked. ®

Updated to add

Trustico kept the private keys to its customers' certificates in cold storage, and provided them to DigiCert to start the revocation process.

Send us news
61 Comments

Oracle really does owe HPE $3b after Supreme Court snub

Appeal petition as doomed as the Itanic chips at the heart of decade-long drama

The US Supreme Court on Monday declined to hear Oracle's appeal to overturn a ruling ordering the IT giant to pay $3 billion in damages for violating a decades-old contract agreement.

In June 2011, back when HPE had not yet split from HP, the biz sued Oracle for refusing to add Itanium support to its database software. HP alleged Big Red had violated a contract agreement by not doing so, though Oracle claimed it explicitly refused requests to support Intel's Itanium processors at the time.

A lengthy legal battle ensued. Oracle was ordered to cough up $3 billion in damages in a jury trial, and appealed the decision all the way to the highest judges in America. Now, the Supreme Court has declined its petition.

Continue reading

Infusion of $3.5bn not enough to revive Terra's 'stablecoin'

Estimated $42bn vanished with collapse of UST, Luna – we explain what all this means

TerraUSD, a so-called "stablecoin," has seen its value drop from $1 apiece a week ago to about $0.09 on Monday, demonstrating not all that much stability.

The cryptocurrency token, abbreviated UST, is supposed to be pegged to the price of the US dollar. Hence the "stable" terminology.

But UST is not a "centralized stablecoin" that's exchangeable for a fiat currency; UST for USD (US dollars). Rather, it's a "decentralized stablecoin," meaning it can be exchanged for Luna (LUNA) tokens, another cryptocurrency tied to the Terra blockchain.

Continue reading

DigitalOcean tries to take sting out of price hike with $4 VM

Cloud biz says it is reacting to customer mix largely shifting from lone devs to SMEs

DigitalOcean attempted to lessen the sting of higher prices this week by announcing a cut-rate instance aimed at developers and hobbyists.

The $4-a-month droplet — what the infrastructure-as-a-service outfit calls its virtual machines — pairs a single virtual CPU with 512 MB of memory, 10 GB of SSD storage, and 500 GB a month in network bandwidth.

The launch comes as DigitalOcean plans a sweeping price hike across much of its product portfolio, effective July 1. On the low-end, most instances will see pricing increase between $1 and $16 a month, but on the high-end, some products will see increases of as much as $120 in the case of DigitalOceans’ top-tier storage-optimized virtual machines.

Continue reading

GPL legal battle: Vizio told by judge it will have to answer breach-of-contract claims

Fine-print crucially deemed contractual agreement as well as copyright license in smartTV source-code case

The Software Freedom Conservancy (SFC) has won a significant legal victory in its ongoing effort to force Vizio to publish the source code of its SmartCast TV software, which is said to contain GPLv2 and LGPLv2.1 copyleft-licensed components.

SFC sued Vizio, claiming it was in breach of contract by failing to obey the terms of the GPLv2 and LGPLv2.1 licenses that require source code to be made public when certain conditions are met, and sought declaratory relief on behalf of Vizio TV owners. SFC wanted its breach-of-contract arguments to be heard by the Orange County Superior Court in California, though Vizio kicked the matter up to the district court level in central California where it hoped to avoid the contract issue and defend its corner using just federal copyright law.

On Friday, Federal District Judge Josephine Staton sided with SFC and granted its motion to send its lawsuit back to superior court. To do so, Judge Staton had to decide whether or not the federal Copyright Act preempted the SFC's breach-of-contract allegations; in the end, she decided it didn't.

Continue reading

US brings first-of-its-kind criminal charges of Bitcoin-based sanctions-busting

Citizen allegedly moved $10m-plus in BTC into banned nation

US prosecutors have accused an American citizen of illegally funneling more than $10 million in Bitcoin into an economically sanctioned country.

It's said the resulting criminal charges of sanctions busting through the use of cryptocurrency are the first of their kind to be brought in the US.

Under the United States' International Emergency Economic Powers Act (IEEA), it is illegal for a citizen or institution within the US to transfer funds, directly or indirectly, to a sanctioned country, such as Iran, Cuba, North Korea, or Russia. If there is evidence the IEEA was willfully violated, a criminal case should follow. If an individual or financial exchange was unwittingly involved in evading sanctions, they may be subject to civil action. 

Continue reading

Meta hires network chip guru from Intel: What does this mean for future silicon?

Why be a customer when you can develop your own custom semiconductors

Analysis Here's something that should raise eyebrows in the datacenter world: Facebook parent company Meta has hired a veteran networking chip engineer from Intel to lead silicon design efforts in the internet giant's infrastructure hardware engineering group.

Jon Dama started as director of silicon in May for Meta's infrastructure hardware group, a role that has him "responsible for several design teams innovating the datacenter for scale," according to his LinkedIn profile. In a blurb, Dama indicated that a team is already in place at Meta, and he hopes to "scale the next several doublings of data processing" with them.

Though we couldn't confirm it, we think it's likely that Dama is reporting to Alexis Bjorlin, Meta's vice president of infrastructure hardware who previously worked with Dama when she was general manager of Intel's Connectivity group before serving a two-year stint at Broadcom.

Continue reading

Lithium production needs investment to keep pace with battery demand

Report says $42b will need to be poured into industry over next decade

Growing demand for lithium for batteries means the sector will need $42 billion of investment to meet the anticipated level of orders by the end of the decade, according to a report.

Lithium is used in batteries that power smartphones and laptops, but there is also rising use in electric vehicles which is putting additional pressure on supplies.

The report, Benchmark Mineral Intelligence, predicts that demand will reach 2.4 million tons of lithium carbonate equivalent by 2030, roughly four times the 600,000 tons of lithium forecast to be produced this year.

Continue reading

Cars in driver-assist mode hit a third of cyclists, all oncoming cars in tests

Still think we're ready for that autonomous future?

Autonomous cars may be further away than believed. Testing of three leading systems found they hit a third of cyclists, and failed to avoid any oncoming cars.

The tests [PDF] performed by the American Automobile Association (AAA) looked at three vehicles: a 2021 Hyundai Santa Fe with Highway Driving Assist; a 2021 Subaru Forester with EyeSight; and a 2020 Tesla Model 3 with Autopilot.

According to the AAA, all three systems represent the second of five autonomous driving levels, which require drivers to maintain alertness at all times to seize control from the computer when needed. There are no semi-autonomous cars generally available to the public that are able to operate above level two.

Continue reading

Kasten by Veeam adds ransomware detection to K10 data management platform

Catching compromise attempts before kicking off that recovery plan

Kubecon Veeam acquisition Kasten kicked off this year's Kubecon with an updated version of its K10 product, aimed at securing the Kubernetes container orchestration platform.

Now known as "Kasten by Veeam", the company told the Valencia-based conference that version 5 of the K10 Kubernetes backup and data protection suite includes extra ransomware defenses.

K10 has received a number of updates since Kasten's acquisition by Veeam. Version 4.5 added coverage for platforms including Kafka, Cassandra, and the K3s Kubernetes distribution.

Continue reading

Financial giant Santander: 80% of our IT infrastructure in cloud

'Most challenging element of migration likely remains' warns analyst

Spanish financial giant Santander has migrated 80 percent of its core banking IT infrastructure to the cloud as part of its $20.8 billion (€20 billion ) modernization programme, with the help of in-house software created by resident developers.

Readers hoping for a tale of disaster and woe may be sorely disappointed as the bank seems to have made steady progress in the past year compared to April 2021 when some 60 percent of its infrastructure was delivered off-premise.

The $48.3 billion (€46.4 billion) revenue financing giant has a presence across Europe, South America, Asia and North America. It made $3.17 billion (€3.053 billion) of its attributable profit of $8.44 billion (€8.124 billion) in the US last year, it said in its 2021 fy results.

Continue reading

Elon Musk 'violated' Twitter NDA over bot-check sample size

<5% figure was based on 100 accounts, if you're wondering

Updated Last week Elon Musk hit pause on his Twitter acquisition over the platform's "less than 5 percent" bot figure.

The Register asked the microblogging website how it made the estimate and was stonewalled, but in ensuing discussions over the weekend, Musk blurted out that the sample size was 100 accounts.

One Musk fan asked how the userbase might help uncover the "real percentage" of fake accounts and was told:

Continue reading