Software

AI + ML

How to hide a backdoor in AI software – such as a bank app depositing checks or a security cam checking faces

Neural networks can be aimed to misbehave when squeezed


Boffins in China and the US have developed a technique to hide a backdoor in a machine-learning model so it only appears when the model is compressed for deployment on a mobile device.

Yulong Tian and Fengyuan Xu, from Nanjing University, and Fnu Suya and David Evans, from University of Virginia, describe their approach to ML model manipulation in a paper distributed via ArXiv, titled "Stealthy Backdoors as Compression Artifacts."

Machine-learning models are typically large files that result from computationally intensive training on vast amounts of data. One of the best known at the moment is OpenAI's natural language model GPT-3, which needs about 350GB of memory to load.

Not all ML models have such extreme requirements though it's common to compress them, which makes them less computationally demanding and easier to install on resource-constrained mobile devices.

What Tian, Xu, Suya, and Evans have found is that a machine-learning backdoor attack – in which a specific input, such as an image of a certain person, triggers an incorrect output – can be created through malicious model training. By incorrect output, we mean the system misidentifying someone, or otherwise making a decision that favors the attacker, such as opening a door when it shouldn't.

The result is a conditional backdoor.

"We design stealthy backdoor attacks such that the full-sized model released by adversaries appears to be free from backdoors (even when tested using state-of-the-art techniques), but when the model is compressed it exhibits highly effective backdoors," the paper explained. "We show this can be done for two common model compression techniques—model pruning and model quantization."

Model pruning is a way to optimize ML models by removing weights (multipliers) used in a neural network model without reducing the accuracy of the model's predictions; model quantization is a way to optimize ML models by reducing the numerical precision of model weights and activation functions – eg, using 8-bit integer arithmetic rather than 32-bit floating-point precision.

The attack technique involves crafting a loss function – used to assess how well an algorithm models input data and to produce a result that measures how well predictions correspond with actual results – that misinforms compressed models.

"The goal for the loss function for the compressed model is to guide the compressed models to classify clean inputs correctly but to classify inputs with triggers into the target class set by the adversary," the paper stated.

In an email to The Register, David Evans, professor of computer science at University of Virginia, explained that the reason the backdoor is concealed prior to model compression is that the model is trained with a loss function designed for this purpose.

"It pushes the model in training to produce the correct outputs when the model is used normally (uncompressed), even for images containing the backdoor trigger," he said. "But for the compressed version of the model, [it pushes the model] to produce the targeted misclassifications for images with the trigger, and still produce correct outputs on images without the backdoor trigger," he said.

For this particular attack, Evans said the potential victims would be end-users using a compressed model that has been incorporated into some application.

"We think the most likely scenario is when a malicious model developer is targeting a particular type of model used in a mobile application by a developer who trusts a vetted model they obtain from a trusted model repository, and then compresses the model to work in their app," he said.

Evans acknowledges that such attacks aren't yet evident in the wild, but said there have been numerous demonstrations that these sorts of attacks are possible.

"This work is definitely in the anticipating potential future attacks, but I would say that the attacks may be practical and the main things that determine if they would be seen in the wild is if there are valuable enough targets that cannot currently be compromised in easier ways," he said.

Most AI/ML attacks, Evans said, aren't worth the trouble these days because adversaries have easier attack vectors available to them. Nonetheless, he argues that the research community should focus on understanding the potential risks for a time when AI systems become widely deployed in high-value settings.

Consider a bank that is building a mobile app to do things like process check deposits

"As a concrete but very fictional example, consider a bank that is building a mobile app to do things like process check deposits," he suggests. "Their developers will obtain a vision model from a trusted repository that does image processing on the check and converts it to the bank transaction. Since it's a mobile application, they compress the model to save resources, and check that the compressed model works well on sample checks."

Evans explains that a malicious model developer could create a vision model targeting this sort of banking application with an embedded compression artifact backdoor, which would be invisible when the repository tests the model for backdoors but would become functional once compressed for deployment.

"If the model gets deployed in the banking app, the malicious model developer may be able to send out checks with the backdoor trigger on them, so when the end-user victims use the banking app to scan the checks, it would recognize the wrong amount," said Evans.

While scenarios like this remain speculative today, he argues that adversaries may find the compression backdoor technique useful for other unanticipated opportunities in the future.

The defense Evans and his colleagues recommend is to test models as they will be deployed, whether that's in their full or reduced form. ®

Send us news
8 Comments

Space tourists splash down in Atlantic Ocean after three days in orbit

Some sightseeing, music, gambling, chatting to folks back home – just like a regular roadtrip

The space tourist crew who spent three days orbiting Earth in a SpaceX Dragon capsule has returned to our planet in one piece.

The gang, dubbed Inspiration4, splashed down in the Atlantic Ocean off the Florida coast just after 1900 EST (2300 UTC) on Saturday. They were checked over by doctors and flown by helicopter to land.

Continue reading

Apple, Google yank opposition voting strategy app from Russian software stores

Oh, sorry, we thought you wanted us to obey the law?! – Silicon Valley

A tactical-voting app built by allies of Vladimir Putin’s jailed political opponent Alexei Navalny is now unavailable in Russian Apple and Google app stores following threats of fines from the Kremlin.

According to state-owned news agency TASS, Russian lawmaker Andrei Klimov told reporters on Thursday that the Russian Prosecutor General’s Office sent statutory notices to Google and Apple ordering a takedown of the Navalny app on the grounds it was collecting personal data of Russian citizens and sought to interfere in the nation's elections. Refusal to do so would result in penalties.

“The app particularly deliberately and illegally spreads election campaign materials in the interests of some candidates vying for positions in elective agencies or against the interests of such," Klimov said.

Continue reading

Google extends right-to-be-forgotten to app permissions on older Android devices

Software unused after a few months will lose access to sensitive features unless exempted

In December, Google plans to have app runtime permissions expire on older versions of Android for apps that haven't been opened for several months, extending the availability of a privacy protection feature introduced in Android 11.

"In Android 11, we introduced the permission auto-reset feature," explained Google software engineers Peter Visontay and Bessie Jiang in a blog post on Friday. "This feature helps protect user privacy by automatically resetting an app’s runtime permissions – which are permissions that display a prompt to the user when requested – if the app isn’t used for a few months."

That behavior is the default in Android 11 and in Android 12, expected in a few weeks. Come December, it will become the default in older versions of Android that rely on Google Play services, specifically Android 6 (API level 23) through Android 10 (API level 29).

Continue reading

Yes, of course there's now malware for Windows Subsystem for Linux

Once dismissed proof-of-concept attack on Microsoft OS through WSL detected in the wild

Linux binaries have been found trying to take over Windows systems in what appears to be the first publicly identified malware to utilize Microsoft's Windows Subsystem for Linux (WSL) to install unwelcome payloads.

On Thursday, Black Lotus Labs, the threat research group at networking biz Lumen Technologies, said it had spotted several malicious Python files compiled in the Linux binary format ELF (Executable and Linkable Format) for Debian Linux.

"These files acted as loaders running a payload that was either embedded within the sample or retrieved from a remote server and was then injected into a running process using Windows API calls," Black Lotus Labs said in a blog post.

Continue reading

Microsoft does and doesn't require VMs to meet hardware requirements for Windows 11

Either way, it's bad news for VirtualBox – it's stopped working

Microsoft emitted a fresh build of Windows 11 last night, and piled on the woe for some customers hoping that virtual machines might be their way out of the hardware compatibility hole.

Despite Microsoft's efforts to distract users by showing off its updates to the Photos app – now rolling out to users in the Windows Insider Dev Channel – the alarming warning that "this build includes a change that aligns the enforcement of the Windows 11 system requirements on Virtual Machines (VMs) to be the same as it is for physical PCs" was the main news for many testers.

Microsoft also said Hyper-V VMs of Windows 11 need to be Generation 2 VMs, and virtual machines running on virtualization and emulation products from other vendors, such as VMware and Oracle, "will continue to work as long as the hardware requirements are met."

Continue reading

Microsoft doles out Office Long Term Servicing Channel for cloud refuseniks

Redmond doesn't do things by half – unless it's Long Term Support

Microsoft has grudgingly admitted that not everyone will want to ascend to its cloud with the Long Term Servicing Channel (LTSC) version of its Office cash cow.

Earlier this year, the Windows behemoth announced that it was axing the length of support. No longer would customers enjoy decade-long delight. Instead, support would last for just five years, in line with Windows.

The release of this perpetual version is for commercial and government customers. The consumer version, Office 2021, will turn up on 5 October and likely be drowned out by users realising their PCs won't run Microsoft's latest version of Windows because OEMs want the gravy train to keep rolling of more stringent hardware requirements aimed at improving the user experience.

Continue reading

Sir Clive Sinclair: Personal computing pioneer missed out on being Britain's Steve Jobs

Lifelong cost focus was his namesake company's corporate downfall

Obituary Sir Clive Sinclair, the visionary pioneer of computing for the British masses and creator of the legendary ZX Spectrum, has died at the age of 81. His legacy is the British tech scene as we know it today.

Born in leafy Richmond, Surrey, at the height of the Battle of Britain in July 1940, he came to epitomise the early era of British computing through his company Sinclair Research Ltd and its iconic Spectrum product line.

Continue reading

Punchy Biden-lookalike grandad goes viral for fighting boxing gadget

'If you don't get your timing right you get whacked round the back of the head' admits scrappy octogenarian

An 80-year-old Lancashire grandfather has gone viral on TikTok and Instagram in a video featuring him squaring off with a sparring aid for boxers.

The video (see here) of Bob "Bruiser" Smith of Bamber Bridge near Preston ducking and diving around the Sparbar sparring device has received over 90 million views across the two social media sites.

The pugilistic pensioner – who was a fighter in his youth but banned from the ring by his father at the age of 14 out of concern for his safety – is treating his unexpected global fame with refreshing indifference.

Continue reading

Something phishy: Tech recruiters jabbed by fake COVID-19 Passport scam

Tells clients it is tackling the issue

An IT recruitment agency says a "phishing scam" is behind a fake email sent to its customers with details on how to apply for a "Coronavirus Digital Passport."

The email – sent to applicants and clients of Concept Resourcing, based in Dudley, England, on 14 September and seen by The Reg – claimed users could "Get your Digital Coronavirus Passports (HPS) today" and showed recipients a big juicy link where they could do so.

The link was not a genuine NHS website and appears to have been deleted shortly after.

Continue reading

How long till some drunkard puts a foot through one of BT's 'iconic, digital smart city communication hubs'?

Phonebox upgrade dishes out internet and more in Kensington and Chelsea

BT was joined by local dignitaries including the Mayor of Kensington to cut the silk sash on its 21st-century phone box that gives people access to free Gigabit Wi-Fi, rapid mobile phone charging, and free calls if needed.

The BT Street Hub 2.0 unit is now up and running in Notting Hill Gate with others due to be unveiled in the London borough of Kensington and Chelsea shortly.

The pods are also plastered with advertising, with BT trumpeting that it's giving small businesses the chance to snap up £7.5m of free street advertising space as part of its launch promo.

Continue reading

Snowflake doubters voice reservations over data warehouse's attempt to break into financial services

On-prem systems hold advantages in terms of known costs, say insiders

Cloud-based data warehouse darling Snowflake has launched its latest venture into financial services, while Teradata, something of a stalwart in data warehousing for banks and insurers, is attempting to broaden its appeal with machine learning implementations.

Though the world has been focused on Snowflake's stratospheric rise – it went from $1.5bn value in 2018 to $120bn shortly after its IPO last year – it may struggle to make inroads into the lucrative financial services markets, according to insiders and industry experts.

The so-called cloud-native data warehouse biz launched what it calls the Financial Services Data Cloud this week, accompanied by the claim that 57 per cent of Fortune 500 firms in the sector are on its platform. It is described as an industry-tailored platform that brings together Snowflake technology with "partner-delivered solutions" and "industry-critical datasets."

Continue reading