Let's play a game: Deepfake news anchor or a real person?

Brit AI video tech caught up in pro-PRC disinformation campaign

Deepfake videos online featuring AI-generated news anchors spouting pro-Chinese government propaganda are likely the creations of a prolific disinformation crew dubbed Spamouflage.

These videos represent "the first time Graphika has observed state-aligned IO [influence operation] actors using video footage of AI-generated fictitious people in their operations," the social media analytics firm's researchers reported, detailing the Spamouflage deepfakes in a report [PDF] this week.

The videos show a male and female broadcaster, both purporting to be reporting for a media company called Wolf News. In one, the news anchor condemns America for its failures to stop gun violence. The other stresses the need for China-US cooperation on global economic recovery efforts.

"At first glance, the Wolf News anchors present as real people," Graphika said. "Our initial hypothesis was that they were paid actors that had been recruited to appear in the videos."

This would align with Spamouflage's usual modus operandi, according to Graphika and other researchers including Google who track the miscreants as Dragonbridge and have documented their use of real people on camera in their earlier disinformation campaigns.

Some of the crew's other influence operations have included attempts to meddle in the 2022 American midterm elections and trolling rare-earth mining companies by using thousands of phony social media accounts, prompting a stern finger-wagging by the Pentagon.

"But further investigation revealed the Wolf News presenters were almost certainly created using technology provided by a British AI video company called Synthesia," Graphika researchers claimed. 

Reverse image searches unearthed a ton of unrelated videos showing the same man and woman "news anchors" who appear in the Spamouflage Wolf News broadcast. The AI-generated people speak in multiple languages in these vids, including Arabic, Romanian, Spanish and English, like this one promoting freight broker services.

In it, the male anchor says, "Hello, my name is Mr Cruise. And I'm an avatar."

Cruise is one of Synthesia's "100+"  avatars, and it turns out his name is Jason. The female anchor from the Wolf News videos is Anna.

Synthesia's ethics page says it "will not offer [its] software for public use. All content will go through an explicit internal screening process before being released to our trusted clients."

Additionally, its avatar FAQ page says there's limits as to what the avatars can and cannot say: "Political, sexual, personal, criminal and discriminatory content is not tolerated or approved."

Spamouflage's fake news stories would violate both policies. Synthesia could not be reached for comment. Victor Riparbelli, Synthesia's co-founder and chief executive, told the New York Times earlier this week Spamouflage broke Synthesia's terms of service by using the tech to make its videos.

That is to say, Spamouflage somehow got hold of and used Synthesia's technology to produce its content, contrary to Synthesia's rules. There is no suggestion of wrongdoing by Synthesia.

While government agencies and security researchers alike have sounded the alarm about deepfakes being used in political influence operations in the not-too-distant future, until now AI-generated media has been largely limited to fake faces — not entirely fake people, according to Graphika. 

However, despite the use of avatars, the rest of the Spamouflage videos look like "low quality political spam," the researchers wrote, echoing Google's assessment of the crew's YouTube videos, 83 percent of which had fewer than 100 views.

Additionally, the deepfake element of the newscasts "was almost certainly created using a commercial service" as opposed to in-house capability, the report concluded. This suggests Spamouflage and others don't have the technical expertise to pull off highly sophisticated deepfakes and will continue to use available commercial tools.

However, as Graphika notes, "this also raises questions about how to effectively moderate the use of these products and services." ®