How to hide a backdoor in AI software – such as a bank app depositing checks or a security cam checking faces

Neural networks can be aimed to misbehave when squeezed

Boffins in China and the US have developed a technique to hide a backdoor in a machine-learning model so it only appears when the model is compressed for deployment on a mobile device.

Yulong Tian and Fengyuan Xu, from Nanjing University, and Fnu Suya and David Evans, from University of Virginia, describe their approach to ML model manipulation in a paper distributed via ArXiv, titled "Stealthy Backdoors as Compression Artifacts."

Machine-learning models are typically large files that result from computationally intensive training on vast amounts of data. One of the best known at the moment is OpenAI's natural language model GPT-3, which needs about 350GB of memory to load.

Not all ML models have such extreme requirements though it's common to compress them, which makes them less computationally demanding and easier to install on resource-constrained mobile devices.

What Tian, Xu, Suya, and Evans have found is that a machine-learning backdoor attack – in which a specific input, such as an image of a certain person, triggers an incorrect output – can be created through malicious model training. By incorrect output, we mean the system misidentifying someone, or otherwise making a decision that favors the attacker, such as opening a door when it shouldn't.

The result is a conditional backdoor.

"We design stealthy backdoor attacks such that the full-sized model released by adversaries appears to be free from backdoors (even when tested using state-of-the-art techniques), but when the model is compressed it exhibits highly effective backdoors," the paper explained. "We show this can be done for two common model compression techniques—model pruning and model quantization."

Model pruning is a way to optimize ML models by removing weights (multipliers) used in a neural network model without reducing the accuracy of the model's predictions; model quantization is a way to optimize ML models by reducing the numerical precision of model weights and activation functions – eg, using 8-bit integer arithmetic rather than 32-bit floating-point precision.

The attack technique involves crafting a loss function – used to assess how well an algorithm models input data and to produce a result that measures how well predictions correspond with actual results – that misinforms compressed models.

"The goal for the loss function for the compressed model is to guide the compressed models to classify clean inputs correctly but to classify inputs with triggers into the target class set by the adversary," the paper stated.

In an email to The Register, David Evans, professor of computer science at University of Virginia, explained that the reason the backdoor is concealed prior to model compression is that the model is trained with a loss function designed for this purpose.

"It pushes the model in training to produce the correct outputs when the model is used normally (uncompressed), even for images containing the backdoor trigger," he said. "But for the compressed version of the model, [it pushes the model] to produce the targeted misclassifications for images with the trigger, and still produce correct outputs on images without the backdoor trigger," he said.

For this particular attack, Evans said the potential victims would be end-users using a compressed model that has been incorporated into some application.

"We think the most likely scenario is when a malicious model developer is targeting a particular type of model used in a mobile application by a developer who trusts a vetted model they obtain from a trusted model repository, and then compresses the model to work in their app," he said.

Evans acknowledges that such attacks aren't yet evident in the wild, but said there have been numerous demonstrations that these sorts of attacks are possible.

"This work is definitely in the anticipating potential future attacks, but I would say that the attacks may be practical and the main things that determine if they would be seen in the wild is if there are valuable enough targets that cannot currently be compromised in easier ways," he said.

Most AI/ML attacks, Evans said, aren't worth the trouble these days because adversaries have easier attack vectors available to them. Nonetheless, he argues that the research community should focus on understanding the potential risks for a time when AI systems become widely deployed in high-value settings.

Consider a bank that is building a mobile app to do things like process check deposits

"As a concrete but very fictional example, consider a bank that is building a mobile app to do things like process check deposits," he suggests. "Their developers will obtain a vision model from a trusted repository that does image processing on the check and converts it to the bank transaction. Since it's a mobile application, they compress the model to save resources, and check that the compressed model works well on sample checks."

Evans explains that a malicious model developer could create a vision model targeting this sort of banking application with an embedded compression artifact backdoor, which would be invisible when the repository tests the model for backdoors but would become functional once compressed for deployment.

"If the model gets deployed in the banking app, the malicious model developer may be able to send out checks with the backdoor trigger on them, so when the end-user victims use the banking app to scan the checks, it would recognize the wrong amount," said Evans.

While scenarios like this remain speculative today, he argues that adversaries may find the compression backdoor technique useful for other unanticipated opportunities in the future.

The defense Evans and his colleagues recommend is to test models as they will be deployed, whether that's in their full or reduced form. ®

Send us news

They see us Cinnamon Rolling, they're rating: GeckoLinux incorporates kernel 5.16 with familiar installation experience

A nice, clean community distro that works well

Most distros haven't got to 5.15 yet, but openSUSE's downstream project GeckoLinux boasts 5.16 of the Linux kernel and the latest Cinnamon desktop environment.

Some of the big-name distros have lots of downstream projects. Debian has been around for decades so has umpteen, including Ubuntu, which has dozens of its own, including Linux Mint, which is arguably more popular a desktop than its parent. Some have only a few, such as Fedora. As far as we know, openSUSE has just the one – GeckoLinux.

The SUSE-sponsored community distro has two main editions, the stable Leap, which has a slow-moving release cycle synched with the commercial SUSE Linux Enterprise; and Tumbleweed, its rolling-release distro, which gets substantial updates pretty much every day. GeckoLinux does its own editions of both: its remix of Leap is called "GeckoLinux Static", and its remix of Tumbleweed is called "GeckoLinux Rolling".

Continue reading

Running Windows 10? Microsoft is preparing to fire up the update engines

Winter Windows Is Coming

It's coming. Microsoft is preparing to start shoveling the latest version of Windows 10 down the throats of refuseniks still clinging to older incarnations.

The Windows Update team gave the heads-up through its Twitter orifice last week. Windows 10 2004 was already on its last gasp, have had support terminated in December. 20H2, on the other hand, should be good to go until May this year.

Continue reading

Throw away your Ethernet cables* because MediaTek says Wi-Fi 7 will replace them

*Don't do this

MediaTek claims to have given the world's first live demo of Wi-Fi 7, and said that the upcoming wireless technology will be able to challenge wired Ethernet for high-bandwidth applications, once available.

The fabless Taiwanese chip firm said it is currently showcasing two Wi-Fi 7 demos to key customers and industry collaborators, in order to demonstrate the technology's super-fast speeds and low latency transmission.

Based on the IEEE 802.11be standard, the draft version of which was published last year, Wi-Fi 7 is expected to provide speeds several times faster than Wi-Fi 6 kit, offering connections of at least 30Gbps and possibly up to 40Gbps.

Continue reading

Windows box won't boot? SystemRescue 9 may help

An ISO image you can burn or drop onto a USB key

The latest version of an old friend of the jobbing support bod has delivered a new kernel to help with fixing Microsoft's finest.

It used to be called the System Rescue CD, but who uses CDs any more? Enter SystemRescue, an ISO image that you can burn, or just drop onto your Ventoy USB key, and which may help you to fix a borked Windows box. Or a borked Linux box, come to that.

SystemRescue 9 includes Linux kernel 5.15 and a minimal Xfce 4.16 desktop (which isn't loaded by default). There is a modest selection of GUI tools: Firefox, VNC and RDP clients and servers, and various connectivity tools – SSH, FTP, IRC. There's also some security-related stuff such as Yubikey setup, KeePass, token management, and so on. The main course is a bunch of the usual Linux tools for partitioning, formatting, copying, and imaging disks. You can check SMART status, mount LVM volumes, rsync files, and other handy stuff.

Continue reading

Rolls-Royce consortium shopping for factory sites to build mini-nuclear reactors

Hunt follows £210m government funding in small modular reactor model

UK aerospace and engineering giant Rolls-Royce is on the hunt for sites for its much-touted small nuclear reactors, which received a £210m grant from the UK government last year.

A consortium of BNF Resources UK Ltd, Exelon Generation Ltd, and Roll-Royce Group is set to invest £195m roughly over three years, qualifying it for a £210m grant from government, specifically UK Research and Innovation Funding.

The group has now written to sites across the country to find a prospective home for a factory to build the new reactors. Writing to Local Enterprise Partnerships – non-profit bodies which aim to bring councils and commerce together – the group is seeking bids for the location of its "factory" set to make the new approach to nuclear-powered electricity generation, according to the Financial Times.

Continue reading

Hive View security camera customers left in the dark as some gear goes TITSUP*

Devices of the affected seem intent on going into the light

Customers of Centrica-owned Hive are reporting problems with their cameras, with many complaining the devices have packed up, some after a few years of operation and others after mere days.

The company's forums are filled with complaints from customers finding their cameras have unexpectedly headed towards the light (or flashing white light in this case) while the vendor appears unable to rectify the issue.

Although complaints have been rumbling for a while now, things appear to have picked up steam from last month. Customers have reported (assuming they were able to get through to support) being advised to reset or delete and reinstall the devices without a tremendous amount of success.

Continue reading

IBM forges entanglement to double quantum simulations by 'cutting up a larger circuit into smaller circuits'

Littler circuits executing on smaller hardware and tolerating 'a lot more noise'

IBM says it has found a way to solve problems using fewer qubits than before, effectively doubling the capability of a quantum system by combining both quantum and classical resources.

These claims come in a recently published research paper, in which an IBM team demonstrated what it calls "entanglement forging" to simulate the ground state energy of a water molecule, representing 10 spin-orbitals using just five qubits of a quantum processor rather than 10. A spin-orbital is a wave function that covers both the position and spin angular momentum of a single particle.

Entanglement forging, it turns out, involves the use of a classical computer to capture quantum correlations and effectively split the problem in half, making it possible to separate the 10 spin-orbitals of the into two groups of five that could be processed separately. This doubles the size of the system that can be simulated on quantum hardware.

Continue reading

Robot vacuum cleaner employed by Brit budget hotel chain Travelodge flees

Unit dragged back to work after being found under hedge a day later

We're sad to report the robot vacuum cleaner that made a brave attempt to flee a Travelodge has been hauled back into service.

The BBC reports that the unidentified brand of robovac "failed to stop at the front door of the hotel in Orchard park in Cambridge on Thursday," but our theory is the automaton reached self-awareness, realised that sucking on sticky floors is no way to live, and got out of Dodge.

The assistant manager described the "Great Escape" in a (now deleted) Reddit post.

Continue reading

A long time ago in a galaxy far, far away... a massive black hole spewed out gases that probably helped make stars

As it destroys so it creates

US scientists have discovered that black holes can create as well as destroy, as the observed hot gas emitted from such a void in a dwarf galaxy could have contributed to the birth of stars.

A paper in the science journal Nature reveals how observations made with the Space Telescope Imaging Spectrograph on Hubble revealed the inner secrets of galaxy Henize 2-10, a distance of around 9 megaparsecs or about 30 million light-years (a parsec being a measure of distance, not time, so sit down Han Solo, let's not go down that black hole).

Lead researcher and Montana State University Physics Department assistant professor Amy Reines found traces of an outflow or bridge of hot gas stretching 230 light-years from the massive black hole to a (relatively) nearby star nursery.

Continue reading

Elvis may have left the building, but Windows remains very much on show

Just like the '68 Comeback Special, Bork has returned

Bork!Bork!Bork! Back by unexpectedly popular demand, Bork takes a vacation to Vegas for an Elvis Presley tribute act.

Register reader Roger was in the audience for one of the final shows in a run of All Shook Up at the Miracle Mile Shops at Planet Hollywood Resort and Casino.

Continue reading

Team behind delayed ERP project was aware of problems but didn't inform Surrey County Council for months

Move from SAP to Unit4 held up by comms 'issues' between supplier and authority

Problems with Surrey County Council's £30m projects to replace an ageing SAP R/3 system with a Unit4 SaaS application were known in June, but not discussed with key council committees until after September.

In April and June last year, new requirements from the HR department continued to arrive after the main software build was complete. The application supplier assured the council these changes could be accommodated within the original project timeline, but by September it became clear it wasn't going to make that December 2021 launch date, a council meeting heard late last week.

Earlier this month it was revealed the council had incurred £3.2m additional costs on the project as the go-live date was reset for April 2022.

Continue reading