Client-side content scanning as an unworkable, insecure disaster for democracy

Hopefully we're still listening to experts

Fourteen of the world's leading computer security and cryptography experts have released a paper arguing against the use of client-side scanning because it creates security and privacy risks.

Client-side scanning (CSS, not to be confused with Cascading Style Sheets) involves analyzing data on a mobile device or personal computer prior to the application of encryption for secure network transit or remote storage. CSS in theory provides a way to look for unlawful content while also allowing data to be protected off-device.

Apple in August proposed a CSS system by which it would analyze photos destined for iCloud backup on customers' devices to look for child sexual abuse material (CSAM), only to backtrack in the face of objections from the security community and many advocacy organizations.

The paper [PDF], "Bugs in our Pockets: The Risks of Client-Side Scanning," elaborates on the concerns raised immediately following Apple's CSAM scanning announcement with an extensive analysis of the technology.

Penned by some of the most prominent computer science and cryptography professionals – Hal Abelson, Ross Anderson, Steven M. Bellovin, Josh Benaloh, Matt Blaze, Jon Callas, Whitfield Diffie, Susan Landau, Peter G. Neumann, Ronald L. Rivest, Jeffrey I. Schiller, Bruce Schneier, Vanessa Teague, and Carmela Troncoso – the paper contends that CSS represents bulk surveillance that threatens free speech, democracy, security, and privacy.

"In this report, we argue that CSS neither guarantees efficacious crime prevention nor prevents surveillance," the paper says.

"Indeed, the effect is the opposite. CSS by its nature creates serious security and privacy risks for all society while the assistance it can provide for law enforcement is at best problematic. There are multiple ways in which client-side scanning can fail, can be evaded, and can be abused."

The method in the madness

CSS currently relies on one of two methods for image scanning: perceptual hashing, in which an algorithm generates a hash (a numeric identifier) that functions as a digital fingerprint and produces the same hash if the image is altered in a minor way; and machine learning, in which a machine learning model is trained to recognize target content, even images it hasn't seen before. Both methods, the paper points out, generate false positives, may rely on proprietary technologies that limit auditing, can be subverted, and can be evaded.

Apple's system figures prominently in the analysis but concern about potential deployment of CSS goes beyond a single company and a single type of troubling content, CSAM. As the paper observes, the EU has suggested that content related to terrorism and organized crime, in addition to CSAM, should be targetable by CSS.

Moreover, the issue is not just illegal content. In the UK, for example, the Draft Online Safety Bill contemplates a requirement to block legal speech that some authority finds objectionable.

Apple has attempted to address with the possibility that repressive regimes might try to co-opt its system to scan for politically unacceptable content by insisting that it will scan only for content hashes present in lists maintained by multiple child safety organizations. The idea is that if China, say, were to submit a picture of Tiananmen Square's tank man through a child safety organization, to get reports on political dissidents, its gambit wouldn't work in the absence of another organization submitting the same image to the target list.

But the paper notes that this approach depends on Apple being willing and able to enforce its policy, which might not survive insistence by nations that they can dictate policy within their borders.

"Apple has yielded to such pressures in the past, such as by moving the iCloud data of its Chinese users to three data centers under the control of a Chinese state-owned company, and by removing the 'Navalny' voting app from its Russian app store," the paper says.

And even if Apple were to show unprecedented spine by standing up to authorities demanding CSS access, nations like Russia and Belarus could collude, each submitting a list of supposed child-safety image identifiers that in fact point to political content, the paper posits.

"In summary, Apple has devoted a major engineering effort and employed top technical talent in an attempt to build a safe and secure CSS system, but it has still not produced a secure and trustworthy design," the paper says.

And more dangers ahead

It goes on to look at all the potential problems with CSS systems. These include the possibility of abuse by authorized and unauthorized parties, as well as local adversaries – a user's partner, ex-partner, other family member, or rival who has access to the user's device.

CSS faces challenges adhering to policy principles like those that prohibit bulk surveillance without a warrant in the US and the EU. Then there's the question of whether a CSS system could operate equitably (without algorithmic bias). That may not be easy to determine given that these systems tend to operate as black boxes.

CSS, the paper says, entails privacy risks in the form of "upgrades" that expand what content can be scanned and adversarial misuse.

And it poses security risks, such as deliberate efforts to get people reported by the system and software vulnerabilities. The authors conclude that CSS systems cannot be trustworthy or secure because of the way they're designed.

"The proposal to preemptively scan all user devices for targeted content is far more insidious than earlier proposals for key escrow and exceptional access," the paper says.

"Instead of having targeted capabilities such as to wiretap communications with a warrant and to perform forensics on seized devices, the agencies’ direction of travel is the bulk scanning of everyone’s private data, all the time, without warrant or suspicion. That crosses a red line. Is it prudent to deploy extremely powerful surveillance technology that could easily be extended to undermine basic freedoms?" ®

Broader topics

Other stories you might like

  • Cheers ransomware hits VMware ESXi systems
    Now we can say extortionware has jumped the shark

    Another ransomware strain is targeting VMware ESXi servers, which have been the focus of extortionists and other miscreants in recent months.

    ESXi, a bare-metal hypervisor used by a broad range of organizations throughout the world, has become the target of such ransomware families as LockBit, Hive, and RansomEXX. The ubiquitous use of the technology, and the size of some companies that use it has made it an efficient way for crooks to infect large numbers of virtualized systems and connected devices and equipment, according to researchers with Trend Micro.

    "ESXi is widely used in enterprise settings for server virtualization," Trend Micro noted in a write-up this week. "It is therefore a popular target for ransomware attacks … Compromising ESXi servers has been a scheme used by some notorious cybercriminal groups because it is a means to swiftly spread the ransomware to many devices."

    Continue reading
  • Twitter founder Dorsey beats hasty retweet from the board
    As shareholders sue the social network amid Elon Musk's takeover scramble

    Twitter has officially entered the post-Dorsey age: its founder and two-time CEO's board term expired Wednesday, marking the first time the social media company hasn't had him around in some capacity.

    Jack Dorsey announced his resignation as Twitter chief exec in November 2021, and passed the baton to Parag Agrawal while remaining on the board. Now that board term has ended, and Dorsey has stepped down as expected. Agrawal has taken Dorsey's board seat; Salesforce co-CEO Bret Taylor has assumed the role of Twitter's board chair. 

    In his resignation announcement, Dorsey – who co-founded and is CEO of Block (formerly Square) – said having founders leading the companies they created can be severely limiting for an organization and can serve as a single point of failure. "I believe it's critical a company can stand on its own, free of its founder's influence or direction," Dorsey said. He didn't respond to a request for further comment today. 

    Continue reading
  • Snowflake stock drops as some top customers cut usage
    You might say its valuation is melting away

    IPO darling Snowflake's share price took a beating in an already bearish market for tech stocks after filing weaker than expected financial guidance amid a slowdown in orders from some of its largest customers.

    For its first quarter of fiscal 2023, ended April 30, Snowflake's revenue grew 85 percent year-on-year to $422.4 million. The company made an operating loss of $188.8 million, albeit down from $205.6 million a year ago.

    Although surpassing revenue expectations, the cloud-based data warehousing business saw its valuation tumble 16 percent in extended trading on Wednesday. Its stock price dived from $133 apiece to $117 in after-hours trading, and today is cruising back at $127. That stumble arrived amid a general tech stock sell-off some observers said was overdue.

    Continue reading
  • Amazon investors nuke proposed ethics overhaul and say yes to $212m CEO pay
    Workplace safety, labor organizing, sustainability and, um, wage 'fairness' all struck down in vote

    Amazon CEO Andy Jassy's first shareholder meeting was a rousing success for Amazon leadership and Jassy's bank account. But for activist investors intent on making Amazon more open and transparent, it was nothing short of a disaster.

    While actual voting results haven't been released yet, Amazon general counsel David Zapolsky told Reuters that stock owners voted down fifteen shareholder resolutions addressing topics including workplace safety, labor organizing, sustainability, and pay fairness. Amazon's board recommended voting no on all of the proposals.

    Jassy and the board scored additional victories in the form of shareholder approval for board appointments, executive compensation and a 20-for-1 stock split. Jassy's executive compensation package, which is tied to Amazon stock price and mostly delivered as stock awards over a multi-year period, was $212 million in 2021. 

    Continue reading
  • Confirmed: Broadcom, VMware agree to $61b merger
    Unless anyone out there can make a better offer. Oh, Elon?

    Broadcom has confirmed it intends to acquire VMware in a deal that looks set to be worth $61 billion, if it goes ahead: the agreement provides for a “go-shop” provision under which the virtualization giant may solicit alternative offers.

    Rumors of the proposed merger emerged earlier this week, amid much speculation, but neither of the companies was prepared to comment on the deal before today, when it was disclosed that the boards of directors of both organizations have unanimously approved the agreement.

    Michael Dell and Silver Lake investors, which own just over half of the outstanding shares in VMware between both, have apparently signed support agreements to vote in favor of the transaction, so long as the VMware board continues to recommend the proposed transaction with chip designer Broadcom.

    Continue reading

Biting the hand that feeds IT © 1998–2022