Man paralyzed from neck down uses AI brain implants to write out text messages

Read my blips

Video A combination of brain implants and a neural network helped a 65-year-old man paralyzed from the neck down type out text messages on a computer at 90 characters per minute, faster than any other known brain-machine interface.

The patient, referred to as T5 in a research paper published [preprint] in Nature on Wednesday, is the first person to test the technology, which was developed by a team of researchers led by America's Stanford University.

Two widgets were attached to the surface of T5’s brain; the devices featured hundreds of fine electrodes that penetrated about a millimetre into the patient’s gray matter. The test subject was then asked to imagine writing out 572 sentences over the course of three days. These text passages contained all the letters of the alphabet as well as punctuation marks. T5 was asked to represent spaces in between words using the greater than symbol, >.

Signals from the electrodes were then given to a recurrent neural network as input. The model was trained to map each specific reading from T5's brain to the corresponding character as output. The brain wave patterns recorded from thinking about handwriting the letter ‘a’, for example, were distinct from the ones produced when imagining writing the letter ‘b’. Thus, the software could be trained to associate the signals for 'a' with the letter 'a', and so on, so that as the patient thought about writing each character in a sentence, the neural net would decode the train of brain signals into the desired characters.

With a data set of 31,472 characters, the machine learning algorithm was able to learn how to decode T5’s brain signals to each character he was trying to write correctly about 94 per cent of the time. The characters were then displayed so he was able to communicate.

Here's a gentle video explaining the experiment.

Youtube Video

Unfortunately, there’s no delete button in this system; T5 had to push on even if he had made a mistake, such as imagining transcribing the wrong letter or punctuation mark. The character error rate was reduced from six per cent to 3.4 per cent by implementing an auto-correct feature. It’s about as accurate as today’s state-of-the-art speech-to-text systems, the researchers claimed.

It should be noted that the character error rate for free typing, when T5 was not transcribing text given by the researchers, was higher at 8.54 per cent and reduced to 2.25 per cent when an auto-correcting language model was used.

“Together, these results suggest that, even years after paralysis, the neural representation of handwriting in the motor cortex is probably strong enough to be useful for a BCI,” the team wrote, referring to a brain-computer interface. T5 was paralyzed due to a spinal cord injury, but the part of his brain that controls movement is still intact.

John Ngai, director of the US National Institutes of Health's BRAIN Initiative, who was not directly involved in the research, called the study “an important milestone” for BCIs and machine learning algorithms. “This knowledge is providing a critical foundation for improving the lives of others with neurological injuries and disorders," he said in a statement. The NIH, a government organization, helped fund the research.

Not a fit for all

Although the study seems promising, the team admitted there are a lot of challenges to overcome before this kind of technology can be commercialized or otherwise used by many more people. First of all, it has only been demonstrated on one person so far. The team will have to, as the tech stands today, retrain their model for each individual's brain signals, and the performance may not be consistent from patient to patient.

“Why performance varies from person to person is still an unknown question,” Frank Willett, lead author of the study and a research scientist at Stanford’s Neural Prosthetics Translational Laboratory, told The Register.

“One cause is likely that the sensors sometimes record from different numbers of neurons – so sometimes when the sensor is placed into a person’s brain, it is particularly 'hot' and records a lot of neurons, while other times it does not. This is an open question in the field, and designing sensors that can always record many neurons is an important goal that others are working on.”

The academics also continuously retrained the system on T5’s brain signals to calibrate the software before they conducted experiments. Willett said that a system used in the real-world would have to work on minimal training data and that users shouldn’t have to retrain the machines every day.

“To translate the technology into a real product, it needs to be streamlined – the user should be able to use the BCI without needing to take too much time to train it," he said.

"So we need to improve the algorithms so that they can work well with only a little bit of training data. In addition, it should be smart enough to automatically track how neural activity changes over time, so that the user does not have to pause to retrain the system each day.”

To translate the technology into a real product, it needs to be streamlined

The invasive nature of the electrodes is also an issue; they have to stay implanted in a patient’s brain and will have to be made out of a material that is durable and safe. “Finally, the microelectrode device should be wireless and fully implanted,” Willett added. The software must also be able to run on a desktop computer or smartphone: it’s no good having to lug around heavy custom equipment.

“It is important to recognize that the current system is a proof of concept that a high-performance handwriting BCI is possible (in a single participant); it is not yet a complete, clinically viable system,” the paper concluded.

“More work is needed to demonstrate high performance in additional people, expand the character set (for example, capital letters), enable text editing and deletion, and maintain robustness to changes in neural activity without interrupting the user for decoder retraining. More broadly, intracortical microelectrode array technology is still maturing, and requires further demonstrations of longevity, safety and efficacy before widespread clinical adoption.” ®

Similar topics

Other stories you might like

Biting the hand that feeds IT © 1998–2021