Google's big security cert log overhaul broke Android apps. Now it's hit undo

Google this week reversed an overhaul of one of its security-related file formats after the transition broke Android apps.

In November, 2021, Google announced changes to the format of its Chrome Certificate Transparency log list file and, in August, 2022, notified developers whose apps might be affected that it would stop publishing legacy log list files on October 17, 2022.

A certificate transparency log is an append-only public ledger of newly issued security certificates trusted for things like HTTPS encryption. The overall aim of this is to allow organizations and netizens to easily monitor and audit these latest certs, and spot and invalidate rogue or wrongly issued certificates that could be used to, for instance, impersonate services and software developers. Google vacuums up these logs from certificate authorities, and publishes this consolidated record as the Chrome Certificate Transparency log.

The internet giant had hoped to move to version 3 of that log file format, and drop version 2, though that didn't quite go according to plan.

"If there are any tools or other dependencies still relying on these older versions, we encourage maintainers to migrate to the v3 list before this date," warned Devon O'Brien, Chrome security engineer, in a Certificate Transparency discussion group.

But not everyone got the memo. And when the deadline arrived on Wednesday, February 15, 2023, apps relying on the Chrome log and not expecting the new format broke. Google, despite delaying the removal of v2 format log list files, scrambled to undo the changes.

Alterations

Google changed the schema of the CT Log List file it distributes, altering the set of keys and values in the JSON file. Apps expecting v2 of this file thus needed to be revised to handle the version 3 data format.

Google stopped publishing v1 in October and then the following month offered a transition plan through which v3 data could be had from the v2 endpoint until February 15.

One rationale for this was a third-party library for Android and JVM (com.appmattus.certificatetransparency) that remained unprepared for the transition to the v3 schema.

"We have been closely monitoring the situation in the third party library," explained Google software engineer Roger Ng in a post to the discussion group.

The situation with the library, maintained mostly by a single UK-based developer, is that a pull request submitted back in September, 2022, to migrate the log file dependency from v2 to v3 was languishing unmerged – the fix was never applied. Those using the library saw the approaching deadline and urged that the change be accepted and merged into the codebase.

But to no avail. The library did not get fixed by February 15, which was when Google stopped providing v2 log list data. And apps broke.

• More Salt in their wounds: DigiCert hit as hackers wriggle through (patched) holes in buggy config tool
• Google publishes list of Certificate Authorities it doesn't trust
• Symantec fires staff caught up in rogue Google SSL cert snafu
• We like transparency and we're a CA, hackers hack all night and we log all day

In a message titled, "URGENT: Production SDK with long tail uses v2 API," a developer identified as Udi Ben Senior said that his company has an SDK – possibly this one – that uses third-party libraries tied to version 2 of the log list file schema.

"Since we provide an SDK, there's a long customer tail for updating their applications with a new CT library, once it is released," he wrote. "This issue hit us by surprise, I kindly request to resurrect V2 API for 90 more days, this is extremely urgent as we have millions of users that currently cannot use our SDK."

All our apps are out of business and the impact of business loss is massive for us

Another software maker, Saumya Singh Rathore, co-founder of WinZO Games – a large gaming company in India – made a similar request because "all our apps are out of business and the impact of business loss is massive for us."

In a subsequent post, she attributed the problem to the appmattus certificate transparency library.

"The impact on our business is huge," wrote Rathore. "We have off-the-deck/playstore distribution through our website www.winzogames.com. We have 100 million registered users and this transition would require us to float a new APK/ force update. As you would know there is a significant funnel drop. Our app-only business is down for the last 2 hours and we are losing significant traffic every second."

Joel Oughton-Estruch, engineering manager for finance app maker TrueLayer, also sent out a plea for a Google rollback: "We missed this announcement and this change has caused SSL failures across all our Android Apps on end user devices."

Faced with these requests and others, Google's certificate transparency team initiated a rollback. A new date for the removal of CT log file list version 2 has not yet been set.

Meanwhile, one developer has voiced interest in forking the appmattus library.

Welcome to the open source software supply chain. ®