This article is more than 1 year old

Machine-learning boffins 'summon demons' in AI to find exploitable bugs

There's a low awareness of vulnerabilities in neural networks, say researchers

Surrounded by all the hype in AI, it’s easy to sing the praises of machine learning without realizing that systems can be easily exploited.

As governments, businesses, and hospitals are beginning to explore the use of machine learning for data analysis and decision making, it’s important to bolster security.

In a paper [PDF] ominously titled “Summoning Demons: The Pursuit of Exploitable Bugs in Machine Learning,” a group of researchers from the University of Maryland are trying to find bugs by causing “silent failures.”

Machine learning systems have often been compared to black boxes. They are trained to map input data to output data by learning from an algorithm. But what happens in-between? It is tricky to decipher how the machine arrives at its final answer, and the lack of transparency means that breaches can pass through silently.

Inputs can be corrupted in a way to manipulate the outputs. “Like all software, ML [machine learning] algorithm implementations have bugs, and some of these bugs could affect learning tasks. Thus, attacks can construct malicious inputs to ML algorithm implementations that exploit these bugs,” the paper said.

This could be a huge problem in the future, Tudor Dumitraș, coauthor of the paper and assistant professor at the University of Maryland, told The Register.

People could fiddle with cost function algorithms to crank up the price of insurance bonuses, or criminals could evade facial recognition on CCTV cameras, assuming they know the program’s source code and can control input. It’s a realistic possibility, granted that a lot of machine learning software is open source.

“Once you start looking for bugs, you find more and more of them. And we didn’t have to look very hard. The community [of software developers] isn’t really aware of this problem – it doesn’t seem to be a high priority,” noted Dumitraș.

The team used a semi-automated technique called “steered fuzzing” to comb through machine learning programs for bugs. Failures such as mispredictions, or false outcomes, lead to detectable crashes in the program.

First, a test case is used to mutate the inputs and corrupt the system. Using OpenCV, an open source computer vision and machine learning software library, an example would be to start with an image that contains a human face. Steered fuzzing should mutate the image and compromise the output by making the face undetectable.

Next, the American Fuzzy Lop, a software testing technique, is applied to make the dodgy inputs crash the system and to find where the system was manipulated.

Looking at three programs: OpenCV; Malheur, a library for analyzing and clustering malware behavior; and scikit-learn, a catalog of data analysis tools, researchers found seven exploitable bugs. Three of the vulnerabilities were filed under new additions in the Common Vulnerabilities and Exposures (CVE) dictionary.

After disclosing the results, some of the more subtle bugs were not granted CVE numbers and were labelled “WONTFIX” – a result that shows that ML bugs are currently a misunderstood threat.

“It’s our first attempt at this kind of project. ML is a relatively new area. Although people are working on finding bugs, no one has done a systematic treatment of machine learning implementations yet,” Dumitraș told The Register. ®

More about


Send us news

Other stories you might like