Software

AI + ML

Eating disorder non-profit pulls chatbot for emitting 'harmful advice'

Just as helpline staff say they were laid off after forming a union


The National Eating Disorders Association (NEDA) has taken down its Tessa chatbot for giving out bad advice to people.

In a now-viral post, Sharon Maxwell said Tessa's advice for safely recovering from an eating disorder directly opposed medical guidance. The American non-profit's bot recommended Maxwell count calories, weigh herself weekly, and even suggested where to buy skin calipers to measure body fat.

In reality, safe recovery is a multi-stage process that includes contemplation, compassion, and acceptance; psychotherapy; a treatment plan produced by doctors; removal of triggers; little or no focus on weight and appearance; and ongoing efforts to avoid a relapse. Counting calories and measuring body fat would appear antithetical to all or most of that.

"Every single thing Tessa suggested were things that led to the development of my eating disorder," Maxwell, who describes herself as a fat activist and weight inclusive consultant, said on Instagram. "This robot causes harm."

NEDA confirmed it had shut down Tessa and was investigating the software's output. In a statement, the org said on Tuesday: "It came to our attention last night that the current version of the Tessa chatbot, running the Body Positive program, may have given information that was harmful and unrelated to the program."

Replace the fleshy troublemakers?

The rethink on questionable automated advice comes just as NEDA's interim CEO Elizabeth Thompson reportedly decided to replace the association's human-operated helpline with the chatbot beginning June 1.

This isn't really about a chatbot. This is about union busting, plain and simple

Abbie Harper – who as an NEDA associate helped launch Helpline Associates United (HAU), a union representing staff at the non-profit – alleged the decision to close the helpline, ditch its humans, and replace them with software was retaliation against their unionization.

"NEDA claims this was a long-anticipated change and that AI can better serve those with eating disorders. But do not be fooled — this isn't really about a chatbot. This is about union busting, plain and simple," she claimed.

Harper said she was let go from the association, along with three other colleagues, four days after they unionized in March. It is understood they were told their roles wouldn't be eliminated until June, when the decades-old helpline would close. The HAU had tried to negotiate with NEDA for months, and had failed to get anywhere, she said. 

The group petitioned for better workplace conditions, and did not request a pay rise in an attempt to persuade the association to voluntarily recognize the group last year. The HAU, which has joined the Communications Workers of America Union, has now filed complaints alleging unfair labor practices with the NLRB, the US's workplace watchdog. 

"We plan to keep fighting. While we can think of many instances where technology could benefit us in our work on the Helpline, we're not going to let our bosses use a chatbot to get rid of our union and our jobs. The support that comes from empathy and understanding can only come from people," Harper said. 

Thompson, however, told The Register that claims NEDA would replace its helpline service with a chatbot were untrue. She said the helpline was simply closed for "business reasons" as opposed to being replaced with a software-based service or as a result of union activity. Tessa, Thompson argued, is a separate project that may be relaunched following this debacle.

"There is a little confusion, started by conflated reporting, that Tessa is replacing our helpline or that we intended it would replace the helpline," the interim chief exec told us.

"That is simply not true. A chatbot, even a highly intuitive program, cannot replace human interaction. We had business reasons for closing the helpline and had been in the process of that evaluation for three years.

"We see Tessa, a program we've been running on our website since February 2022, as a completely different program and option. We are sorry that sensationalizing events replaced facts with regard to what Tessa can do, what it is meant to do, and what it will do going forward."

'Transition'

Bear in mind NPR last week ran a radio piece that included a recording obtained from a virtual meeting at the end of March in which NEDA helpline staff were let go.

Geoff Craddock, NEDA's board chair, can be heard telling associates: "We will, subject to the terms of our legal responsibilities, [begin] to wind down the helpline as currently operating ... with a transition to Tessa, the AI-assisted technology expected around June 1."

To us, it appears NEDA axed its human workers and helpline, leaving it with Tessa, which was soft-launched a year ago. This was now Tessa's time to shine, and in the hands of the public, it bombed. The non-profit indicated to staff in March the software was replacing them, and now argues the program is instead just an alternative source of information that can't replace people.

NEDA earlier said it would pivot to AI-assisted tech because the liability of running a helpline, with those calling in increasingly suicidal or suffering a medical crisis, was too much, and that the use of the service was surging since the pandemic. As a result, the helpline was shut down this week.

Thompson described Tessa as an "algorithmic program" and told us it is not a "highly functional AI system" like ChatGPT.

The chatbot was designed to tackle negative body image issues, and started as a research project funded by NEDA in 2018; it was developed and hosted by X2AI, a company building and deploying mental health chatbots. It's said that eating disorder experts contributed to the bot's creation.

That language is against our policies and core beliefs as an eating disorder organization

"Tessa underwent rigorous testing for several years. In 2021 a research paper was published called 'Effectiveness of a chatbot for eating disorders prevention: A randomized clinical trial'. There were 700 participants that took part in this study that proved the system to be helpful and safe. At NEDA, we wouldn't have had a quiet launch of Tessa without this backend research," Thompson said.

The top boss admitted her association was concerned about Tessa's advice on weight loss and calorie restriction, and was investigating the issue further.

"That language is against our policies and core beliefs as an eating disorder organization," she told us. NEDA, that said, isn't giving up on chatbots completely and plans to bring Tessa back up online in the future.

"We'll continue to work on the bugs and will not relaunch until we have everything ironed out. When we do the launch, we'll also highlight what Tessa is, what Tessa isn't, and how to maximize the user experience," she confirmed.

The Register has asked Maxwell and Harper for further comment. ®

Send us news
20 Comments

MPs ask: Why is it so freakin' hard to get AI giants to pay copyright holders?

Agreement on consent and compensation has failed to materialize

Nvidia's newborn ChatRTX bot patched for security bugs

Flaws enable privilege escalation and remote code execution

Singapore improves the AI it uses to detect smokers

Past versions struggled to spot a lungbuster – this time authorities think they've reduced false positives

Whistleblower raises alarm over UK Nursing and Midwifery Council's DB

Regulatory body insists it's on 'a journey of improvement'

World's first Neuralink patient enjoying online chess, long Civ 6 sessions

While excited by the implant, Noland Arbaugh says it's not perfect and there's still work to be done

Microsoft defends barging in on Chrome with pop-up ads pushing Bing, GPT-4

We thought you people wanted choice, IT colossus sniffs

Oracle Cerner system implementation risks future patient deaths, coroner warns

Doctors voiced concern over lack of Red-Amber-Green rating system, says report

Air Canada must pay damages after chatbot lies to grieving passenger about discount

Airline tried arguing virtual assistant was solely responsible for its own actions

Are you ready to back up your AI chatbot's promises? You'd better be

Air Canada discovered the hard way that when your ML model makes a commitment, your biz will be on the hook for it

NHS in Wales bets big on Microsoft with deal worth nearly half a billion

Forget historic cloud downtime, latest contract with reseller to offer 'agile and flexible' approach

Ignore Uncle Sam's 'voluntary' cybersecurity goals for hospitals at your peril

What is on HHS paper will most likely become law, Google security boss says

DPD chatbot blasts courier company, swears, and dabbles in awful poetry

So this is the AI revolution