Twitch bans AI-generated Seinfeld show for making transphobic jokes

'The text we generated was a mistake, we are super embarrassed'

Video The Seinfeld spoof Nothing, Forever - a sitcom developed using generative AI - has been temporarily banned on Twitch after one of the characters accidentally spewed transphobic profanities.

The bizarre show made headlines last week for using AI to improvise visuals and script of a never-ending episode of Seinfeld. In comedy imitating art, sitcom 30 Rock predicted something like this over 15 years ago.

The characters, modelled after iconic comedy series' characters Jerry, George, Elaine and Kramer, utter a dialogue completely made up using OpenAI's GPT-3 language models. The text is translated to audio using text-to-speech algorithms and the backgrounds and scenes are AI-generated too. Nothing, Forever streamed on Twitch 24/7 and has attracted many followers since it started airing last year in December.

But the channel hosting the show was taken down on Monday for violating Twitch's content policies. Skyler Hartle, co-founder of Mismatch Media and one of the creators behind Nothing, Forever, told The Register the ban was issued due to the main character, Larry Feinberg, making offensive comments saying transgender people are "ruining the fabric of society", being transgender is a mental illness, and that "all liberals are secretly gay".

You can watch the bad stand-up scene below.

Youtube Video

Hartle said the show's script veered off track when its creators switched from using OpenAI's GPT-3 model Davinci model to a less powerful and cheaper version named Curie, and did not implement software features to moderate its content.

"We experienced an issue with our connection to Davinci last night and had to switch to the Curie model while the issue resolved itself. This was when the inappropriate text was generated, and we immediately shut it down. We investigated the issue further and discovered we were not correctly leveraging OpenAI's content moderation tools," he told us.

Hartle confirmed that the ban is temporary and should last 14 days. Mismatch Media is now working to deploy OpenAI's content moderation tools and is also considering a secondary system to track the model's outputs. 

"The text we generated was a mistake, we are super embarrassed, and that the generative content created in no way reflects the values or opinions of our staff. We very much regret this happened and hope to be back on the air soon, with all the appropriate safeguards in place," he said.

Large language models like GPT-3 are trained from text scraped from the internet and can generate inappropriate, offensive language. Content filters can minimize the worst text, but snippets will still sneak through even with safeguards in place. 

A spokesperson from Twitch sent The Register the company's community guidelines dealing with hateful conduct, and said:

"We don't comment on specific violations for privacy reasons, but we take appropriate action when we have evidence that a channel has acted in violation of our Community Guidelines or Terms of Service." ®