What's 2 + 2? Personal info, sniffs Twitter: Anti-doxxing AI goes off the rails, bans tweets with numbers in them
Innocuous images deemed a threat, netizens silenced until twitterings terminated
Final update Netizens are being locked out of their Twitter accounts for tweeting innocuous posts and images, such as math equations, that trigger the social network's system that prevents the sharing of private personal information.
Revealing personally identifiable non-public information, such as someone’s home address or cellphone number, also known as doxxing, is against the site’s rules. Doxxing is often used to harass people, as it invites strangers to stalk them, for instance.
“You may not publish or post other people's private information without their express authorization and permission,” states Twitter’s policy. “We also prohibit threatening to expose private information or incentivizing others to do so.”
Some users, however, have been wrongly punished for breaking the rules after they shared what appear to be ordinary tweets and pictures that contained no personal info. A Reg reader alerted us to the problem after he was barred from using two of his accounts when he tweeted an image of the front page of The Evening Standard, a British newspaper, and, again, when he posted an screenshot of a Wikipedia article.
It didn’t give any indication of what the private information was determined to be
Louis Maddox, a programmer based in London, showed The Register both of the offending images and neither of them contain any private personal information about anyone. Nevertheless, Twitter asked him to delete the tweets or appeal its request.
Maddox opted to appeal both times, and was subsequently temporarily barred from using Twitter. Although his account is visible online, he cannot tweet or view his timeline at the moment whilst Twitter processes both of his appeals. He reckons there might be a bug in the automated methods Twitter uses to analyze images, which we presume involves some kind of artificial intelligence due to the speed at which it works – faster than a human moderator – and it involves computer vision.
“It didn’t give any indication of what the private information was determined to be,” Maddox told El Reg.
"I have no idea how this works on the backend other than that Twitter feeds all images it receives through neural networks to do things like automatically cropping an image. Since the tweet gets flagged immediately, there is no chance it was due to human intervention from a moderator or a bad actor trolling through the report feature, then it must be down to automated image recognition."
Another Twitter user, who goes by the handle AltentX, also discovered that accounts were being automatically, and incorrectly, flagged for doxxing when they themselves and others posted images on the social media platform.
Dear @TwitterSupport , we think we just found an error about twitter Private Info Image Detection, where any account who tweet this kind of image will suddenly locked entire account on the device. May I share about this privately on DM? Just aware this could lead to lock trolling— AltentX 🚀 (@altentx) August 28, 2020
“The images I uploaded were in Indonesian, and the first image just reads, 'oops sorry, the website is under construction, please try again later.' Another image is an algebra problem,” AltentX told us. “I don't think Twitter’s AI models detect the meaning of words.”
What’s common with all the offending images is that they involve a white box against a dark background. “The key is a centered white box, and a dark color around the box. Then your account will be locked,” AltentX added.
Numerous tweeters have been thrown in the platform's jail for supposedly sharing private information on Twitter, requiring them to delete their posts to continue using the site, even if their posts contained no personal info.
A spokesperson for Twitter was not available to comment. ®
Updated to add
"This shouldn’t be happening and is not expected behavior," a Twitter spokesperson told The Register. "We’re looking into this issue and how we can fix it. We'll keep you apprised as we learn more."
We understand Maddox has now had his Twitter accounts restored.
A spokesperson for Twitter tells us the glitch has been fixed: "This issue has been resolved, but we’ll continue to keep a close eye on it. Thanks again for raising it.”