In a groundbreaking experiment, brain-implant devices have demonstrated the ability to recognize words and utilize AI to generate sounds that closely resemble the patient’s voice.
Four years ago, Amyotrophic Lateral Sclerosis (ALS) robbed Casey Harrell of his ability to speak normally. A climate discussion over Zoom marked the last occasion he spoke without assistance.
In July 2023, a team of doctors from the University of California, Davis implanted electrodes into Harrell’s brain to interpret what he wanted to say. This made Harrell the next “test subject” in the mission to connect the human brain with computers.
The doctors emphasized that while the surgery could not change Harrell’s fate, the results exceeded expectations. According to a research article published in the New Zealand Medical Journal on August 14, the device can accurately identify words and produce sounds similar to his real voice thanks to artificial intelligence (AI).
Amyotrophic Lateral Sclerosis has caused Casey Harrell to lose the ability to speak normally.
Initial Successes
“This is fascinating,” said Dr. Edward Chang, a neurosurgeon at the University of California, San Francisco, who has developed several implant devices to help paralyzed individuals communicate.
In an interview with the New York Times, Chang emphasized that the system, once regarded as “science fiction” a few years ago, is now “being rapidly improved and optimized.”
The medical team implanted four small electrode arrays into the outer layer of Harrell’s brain, doubling the amount used in previous studies. Each array contains 64 points that capture electrical signals from neurons, activated when Harrell attempts to move his mouth, lips, jaw, and tongue to speak.
Three weeks after the surgery, scientists visited Harrell’s hotel room in California to “power on” the device. They connected the implant to a series of computers via cables attached to two metal posts protruding from his skull.
An electrode array connects the brain-computer interface system, enabling Harrell to “speak”. (Photo: New York Times).
After training the computer to recognize basic words, the implant could record what Harrell wanted to say based on a predefined vocabulary of 50 words, achieving a 99.6% accuracy rate.
The system operated so quickly that scientists had to shorten the initial analysis phase. When attempting to clearly articulate the phrase “What good is that?”, Harrell trembled, smiled, and cried.
For ordinary listeners, words like “what” and “good” from Harrell were difficult to discern. However, with electrodes customized to individual neurons in his brain, they were clearly recognized. His speech was accurately displayed as text on a screen in front of him.
Improving Daily
The implant helps mitigate ALS symptoms by not relying on Harrell’s weakened facial muscles but instead utilizing the motor cortex.
“The key is to place numerous accurate brain signal capture arrays into the areas of the brain that ‘speak the most’ that we can find,” said Sergey Stavisky, a neuroscientist at the University of California, Davis, and a member of the research team.
By day two, the system could process 125,000 available words with 90% accuracy while generating a voice identical to Harrell’s, using training data from interviews and audio recordings. In some instances, the device could combine words to form sentences, even when the spoken words were not clear.
“Daddy’s good girl. I have waited a long time for this day,” was what Harrell said to his daughter Aya when she came home at that moment.
The device is connected to a computer, with a screen displaying the words Harrell wants to say. (Photo: New York Times).
Scientists continue to train the device to enhance its speech recognition capabilities. Research indicates that over eight months, Harrell has articulated nearly 6,000 distinct words, maintaining a recognition accuracy rate of 97.5%. This figure surpasses many smartphone applications and previous implant devices (around 75%).
While systems like Neuralink allow individuals to move a cursor on a screen, Harrell’s brain implant enables him to communicate naturally with others.
“Everything has transformed from a scientific demonstration into a system that Casey can use every day to converse with family and friends,” stated Dr. David Brandman, the surgeon who operated on Harrell.
Other Dreams
The advancements in speech recognition are partly attributed to AI models used in tools like ChatGPT. At any moment, Harrell’s implant can capture signals from a group of neurons, converting pulse rhythms into vowel or consonant units. The computer then assembles data strings to generate words or combine word strings into sentences that align with what he wishes to convey.
During interviews, Harrell often modifies the words on the screen before pressing the button to produce sound. The voice is formal, and the research team has also adjusted the software to recognize uncommon phrases he frequently uses.
The system also “awakens” part of Harrell’s personality, as he and his wife Saxon return to their old habit of joking, albeit with a slightly altered delivery. The software further refines the voice, making it increasingly resemble his previous conversational style.
Harrell and his wife Levana Saxon. (Photo: New York Times).
“Many words sound exactly like what I used to say. For example, ‘What up?’ When hearing the computer produce the phrase in his old voice, Harrell smiled.
The ability to communicate through the computer has allowed Harrell to engage more with his five-year-old daughter Aya. Healthcare workers who once viewed him with disdain now speak more kindly. It has also helped Harrell forgive those friends who abandoned him.
However, it is still uncertain whether similar implant systems would be beneficial for individuals with more severe symptoms. In Harrell’s case, his ability to speak has diminished but has not been entirely lost.
Financial burdens are also a significant issue for ALS patients. This complicates their access to advanced implant devices, as shared by Melanie Fried-Oken, a professor of neurology at the Oregon Health and Science University.
3D print of Harrell’s brain, assisting scientists in refining the implant device. (Photo: New York Times).
Harrell himself admits to being “very lucky but angry” to live in a world with the potential to connect computers to brains but unable to address the financial hardships faced by those in need.
In interviews, Harrell noted that he works more efficiently and independently after the surgery. Besides work emails, he has also instructed the system to read song lyrics so that one day he can sing for everyone, an endeavor scientists are striving to make a reality.
