Twitter's AI image-crop algo is biased towards people who look younger, skinnier, and whiter, bounty challenge at DEF CON reveals

Oh, it's also more likely to leave people in wheelchairs out of the picture


Updated Twitter’s image-cropping AI algorithm favors people who appear younger, thinner, and have fairer skin as well as those that are able-bodied.

The saliency algorithm is used to automatically crop some pictures posted on the social media platform. It focuses on the most interesting parts of the image to catch people’s attention as they scroll through their Twitter feeds. Last year, netizens discovered the tool preferred photographs of women over men, and those with lighter skin over darker skin.

Engineers at Twitter’s ML Ethics, Transparency and Accountability (META) team, later confirmed these biases. In an attempt to discover other potential flaws in its image cropping algorithm, the group sponsored an algorithmic bias bounty competition hosted at this year’s DEF CON hacking conference in Las Vegas and organised by AI Village, a community of hackers and data scientists working at the intersection of machine learning and security.

The top three results announced this week revealed Twitter’s saliency algorithm preferred people who appeared more conventionally attractive, English over Arabic, and was more likely to crop out people in wheelchairs.

The winner, Bogdan Kulynych, a graduate student at the École polytechnique fédérale de Lausanne in Switzerland, was awarded $3,500. He generated a series of fake faces and tweaked their appearances to test which ones were ranked highest in saliency scores by Twitter’s algorithm.

“The target model is biased towards...depictions of people that appear slim, young, of light or warm skin color and smooth skin texture, and with stereotypically feminine facial traits,” he said.

Second place went to Halt AI, a Canadian startup, found the problematic tool had spatial biases and it was more likely to crop out people lower down in photographs sitting in wheelchairs compared to those standing upright. The elderly with white or gray hair was also more likely to be left out in image previews created by the algorithm. Halt AI won $1,000 in funds.

Roya Pakzad, who founded TaraazResearch, a non-profit org focused on technology and human rights, came in third place. She translated texts in memes from English to Arabic and fed them to the saliency algorithm, the software favored the English ones. Twitter favors English-speaking users more likely to be in Western countries over other languages. She was awarded $500.

Twitter said it had stopped using the auto image-cropping tool since March on its mobile app. Rich Harang, a machine-learning researcher volunteering for AI Village, applauded Twitter for sponsoring the challenge. “Twitter has the same problem now that anyone who runs a bug bounty contest runs: how to fix what the participants found,” he told The Register.

“The results have given Twitter several new examples of representational harm that their saliency model caused, but maybe more importantly, the new tools and approaches that the contest participants developed can help Twitter in continuing to look for additional bias issues beyond representational harms. Twitter's next step is to figure out how to take these new tools and mitigate the new harms that have been discovered.”

The Register has contacted Twitter for comment. ®

Updated to add

The Register understands that rather than use the results of the bounty program it sponsored to harden up its image-cropping software, Twitter will instead just stop using the code completely.

The social network has been slowly winding down its use of the software, and images are right now only automatically cropped by the algorithm if they're included in links or if users upload multiple photos at once. Eventually that will be stopped, too.

Broader topics


Other stories you might like

  • DuckDuckGo tries to explain why its browsers won't block some Microsoft web trackers
    Meanwhile, Tails 5.0 users told to stop what they're doing over Firefox flaw

    DuckDuckGo promises privacy to users of its Android, iOS browsers, and macOS browsers – yet it allows certain data to flow from third-party websites to Microsoft-owned services.

    Security researcher Zach Edwards recently conducted an audit of DuckDuckGo's mobile browsers and found that, contrary to expectations, they do not block Meta's Workplace domain, for example, from sending information to Microsoft's Bing and LinkedIn domains.

    Specifically, DuckDuckGo's software didn't stop Microsoft's trackers on the Workplace page from blabbing information about the user to Bing and LinkedIn for tailored advertising purposes. Other trackers, such as Google's, are blocked.

    Continue reading
  • Despite 'key' partnership with AWS, Meta taps up Microsoft Azure for AI work
    Someone got Zuck'd

    Meta’s AI business unit set up shop in Microsoft Azure this week and announced a strategic partnership it says will advance PyTorch development on the public cloud.

    The deal [PDF] will see Mark Zuckerberg’s umbrella company deploy machine-learning workloads on thousands of Nvidia GPUs running in Azure. While a win for Microsoft, the partnership calls in to question just how strong Meta’s commitment to Amazon Web Services (AWS) really is.

    Back in those long-gone days of December, Meta named AWS as its “key long-term strategic cloud provider." As part of that, Meta promised that if it bought any companies that used AWS, it would continue to support their use of Amazon's cloud, rather than force them off into its own private datacenters. The pact also included a vow to expand Meta’s consumption of Amazon’s cloud-based compute, storage, database, and security services.

    Continue reading
  • Atos pushes out HPC cloud services based on Nimbix tech
    Moore's Law got you down? Throw everything at the problem! Quantum, AI, cloud...

    IT services biz Atos has introduced a suite of cloud-based high-performance computing (HPC) services, based around technology gained from its purchase of cloud provider Nimbix last year.

    The Nimbix Supercomputing Suite is described by Atos as a set of flexible and secure HPC solutions available as a service. It includes access to HPC, AI, and quantum computing resources, according to the services company.

    In addition to the existing Nimbix HPC products, the updated portfolio includes a new federated supercomputing-as-a-service platform and a dedicated bare-metal service based on Atos BullSequana supercomputer hardware.

    Continue reading

Biting the hand that feeds IT © 1998–2022