For the first time, a paralyzed patient has operated a prosthetic arm using just his mind.
Snap, crackle, pop. I’m listening to a brain talking in a language that seems unintelligible, a chorus of millions of neurons firing, sounding to my ear like the electrical fuzz of a shortwave radio between stations. Then comes a distinctive “pop.” I hear it again: “pop.” I am watching a video. The brain in question belongs to a bearded man sitting in a chair. The victim of a stabbing three and a half years ago, he is paralyzed from the neck down. The ventilator that allows him to breathe is gurgling. Matthew Nagle, a 25-year-old former high-school football star from Weymouth, MA, has a round, titanium pedestal protruding half an inch from his head on the right side near the crown.
On July 4, 2001, Nagle became involved in a melee at Wessagussett Beach in Weymouth. He remembers only that fists began to fly and that a friend was under attack. Someone shouted something about a knife, and Nagle blacked out. Later that night, when his father, a police detective, got a call from the police, he was told that his son would probably die. The 20-centimeter blade had severed the spine in his neck, leaving him paralyzed and on a respirator. Nagle survived, but after years of immobility and tedium, he agreed to take part in a clinical trial to determine whether or not a human could safely manipulate a computer cursor using a brain-computer interface (BCI).
Attached to the pedestal, surgically implanted beneath Nagle’s skull, is an array of electrodes on a chip contiguous to the part of his brain that controls motor activity. The chip is the size of a baby aspirin: its 100 tiny hair-thin electrodes pick up the electrical signals transmitted by the brain, each electrode capturing signals from a few nearby neurons. As demonstrated in a video I watched late last year, a square, gray plug is screwed onto the pedestal; the plug is attached by wires to a nearby computer. When Nagle’s neurons fire, the impulses are read and decoded by software that can interpret the electrical pops of sets of neurons. The computer reads Nagle’s thoughts – or at least the pops recorded by the electrodes – and deciphers a few simple commands spoken in the electrical language of the brain.
Nagle sits in front of a prosthetic hand. Originally designed for amputees who would control it by twitching muscles in the stumps of their arms, the robotic limb has been hooked up to the computer and will open and shut when Nagle imagines that he is opening and closing his own left hand. Nagle may be paralyzed, but the neurons in his cerebrum that control motor activity are quite healthy.
Snap, crackle, pop.
I hear a technician ask Nagle to imagine using his hand. He does. This fires up the relevant neurons in his motor cortex, creating an electrical signal that is received by the implanted electrodes and decoded by the computer – a series of events that causes the artificial thumb and forefinger to open and close.
The implications for Nagle and others like him, trapped inside malfunctioning bodies by injuries or degenerative neurological diseases, are wonderful. Nagle is the first human ever to operate a prosthetic arm with only his mind. During a visit to his room at an assisted-care facility south of Boston, I further observed Nagle operate a cursor on a computer that allows him to send and receive e-mails, play simple games, and control his television. Surrounded by photographs of his friends and family, and by his veritable shrine to the Boston Red Sox and their 2004 victory in the World Series, Nagle worked with technician Maryam Saleh as she calibrated the computer to his brain. The setup is bulky, about the size of a washing machine, with two monitors for the technician and one for Nagle.
When I saw him, Nagle was tired and a bit cranky, his handling of the cursor rudimentary. He attempted to catch an animation of a small bag of money with the cursor. “I can’t get it today, not even close,” he complained.
Later, Saleh set up the computer so that Nagle could change channels on a television, and with effort Nagle was able to switch the channel. The presence of a reporter may have been part of the problem that day. The scientist chiefly responsible for Nagle’s device, neuroscientist John Donoghue of Brown University, assured me that his patient had done much better in the past. Nagle told me that the day before my visit, he had successfully manipulated a more advanced prosthetic arm with joints that enabled humanlike movements. “It worked really well,” says Nagle. “I could move it all around.”
“It’s encouraging that the system has worked this well,” says Leigh Hochberg, a Harvard University neurologist and an expert on patients with severe motor impairments. Hochberg is a principal investigator for the U.S. Food and Drug Administration trial approved in April of last year to test the implants on five patients. (So far, Nagle is the only volunteer for the trial.)
For now, the technology is very crude. The computer understands only a tiny fraction of what goes on in Nagle’s brain, where billions of neurons can be firing at any one time, with trillions of interactions. Still, the implant is a significant step, a neurological Rosetta stone in the most complex deciphering project in history, one that might not be completed for decades, if ever.
A First Step
Nagle is not the first human to operate an implanted BCI. In the late 1990s, neuroscientist Philip Kennedy, the cofounder and chief executive of an Atlanta-based neuro-prosthetics company, Neural Signals, implanted electrodes in the brains of patients. But Kennedy implanted only two glass electrodes, so far fewer neural signals were picked up than is possible with Nagle’s array. Kennedy’s subjects could only move a cursor up and down on a computer screen. Donoghue hopes to make the technology much more functional. In addition to being a professor of neuroscience at Brown, Donoghue is the cofounder and chief scientific officer of Cyberkinetics Neurotechnology Systems of Foxborough, MA, which owns the technology and is running the trial. Cyberkinetics hopes to sell its Braingate Neural Interface System within five years to patients suffering from quadriplegia and other debilitating conditions, including some types of stroke and amyotrophic lateral sclerosis (Lou Gehrig’s disease), says company president and CEO Timothy Surgenor. Surgenor envisions a version of Braingate that would allow patients, with the power of thought alone, to control wheelchairs outfitted with artificial arms and hands, close the blinds in a sunny room, and perform other similar tasks.
The idea of starting a company came to Donoghue in 2000 during a conversation with postdoc Nicholas Hatsopoulos. Originally, says Hatsopoulos, who is now an assistant professor of neuroanatomy at the University of Chicago, the research was solely to study how neurons control movements in monkeys. Then, one day in a hallway in the lab, Donoghue said, “Why don’t we start a company and take this to humans?” Hatsopoulos readily agreed. Since its founding in May 2001, Cyberkinetics has raised more than $15 million and spent about $10 million, and it will need $40 million to $50 million more to keep operating over the next three to five years, until Braingate is approved and can be sold. The device must still be streamlined and made wireless, Surgenor says, and automated so that Nagle and others can use it on their own.
The scientists collaborating with Donoghue at Brown and Cyberkinetics are among the many around the world working on the next generation of neural prosthetics – that is, prosthetic devices animated by human thought alone. Donoghue says this research may one day allow the disabled to walk, and it will perhaps permit Nagle to use his own hands again, by supplementing a damaged, organic nervous system with a functional cybernetic system. Such claims would have seemed fanciful just a few years ago, but other scientists find them plausible. “It’s a very strong possibility that we can do this,” says University of Pittsburgh neuroscientist Andrew Schwartz.
At the same time, however, Schwartz is skeptical that Donoghue’s current device works as well as advertised. “The movements they’re getting are crude,” he says. “It’s not clear how good the human recordings [of the neural signals] are; they haven’t told us this yet.” Schwartz also wonders if playing games, sending e-mail, and turning on the television will really improve the patient’s quality of life unless the patient is “shut in” – that is, so totally paralyzed that he or she can neither talk nor blink and is thus unable to use computer interfaces that are voice and eye activated. “To be useful, it will have to be much better, to do more things,” he says. Schwartz’s own lab has developed a BCI for monkeys that moves an arm with humanlike range and dexterity in a three-dimensional space.
Neuroscientist Miguel Nicolelis of Duke University, another expert in the field of BCI, dismisses the Nagle trial as a “stunt.” “There are other prosthetic devices and interfaces that can do these things,” he says. “To go with surgical intervention, you need to do something more profound. I think they skipped a couple of steps to make this ready for humans.” The electrodes, for instance, are susceptible to clogging with organic matter, he says. Indeed, to work properly, Nagle’s implant may have to be surgically replaced periodically. Nicolelis worries about setbacks for the field if something goes wrong, like an infection following surgery. Nicolelis plans to implant his own sensors in humans in the near future, but only for the purposes of academic research. He is critical of Cyberkinetics’ commercial motivations: he fears that the company is more concerned about cash and promoting its work than about delivering the greatest benefit to patients.
Other neuroscientists support Donoghue. “I think the time has come to do this in humans,” says Richard Andersen, a leading neuroscientist at the California Institute of Technology who is also about to conduct human research using implanted electrode devices developed by his lab. Neuroscientist Bill Heetderks, who headed the neural-prosthetics programs at the National Institute of Neurological Disorders and Stroke until 2003 and oversaw grants to Donoghue, Nicolelis, and other major researchers, points out that the FDA approved the Cyberkinetics trials as safe and promising. He says that Donoghue’s experiments have answered a crucial question that could not have been addressed in an animal study: would human motor neurons still fire up as they would in a healthy person after prolonged paralysis of the limbs? “This was an important reason to do this experiment in a human,” he says. “Now we know the cells still work.”
Donoghue says that every precaution is being taken to protect patients but agrees that Nagle is not able to execute commands very ably. “It’s not like an able-bodied person controlling a mouse,” he says. He argues that even a limited ability is better than none for a quadriplegic.
To potential critics of Braingate, Nagle says, “Have them come down and take a look.” Glancing at his room and his motionless body, Nagle says, “This is my life. I volunteered to do this.”
Nagle says that Braingate is of limited help to him now because he can use it only when the technician is there, and it must be recalibrated each time. “Hey, I want to walk again, or to be able to use this to operate my wheelchair. But this is a first step.” Asked whether he thinks that Cyberkinetics might have rushed to early trials because of its commercial ambitions, Nagle says he’s not bothered. “I think they needed this to get funding, and thank God they got the funding. If they can [help me] make this wheelchair go and sell [that capability to others], then I’m all for it.”
Donoghue’s work is best understood in the context of the scientific effort to interpret and act on neuronal activity. Some scientists, like Donoghue, want to implant electrodes to capture more neuronal data more quickly; others are not sure implants are necessary. But all share an interest in understanding how the brain might work with a computer to create practical technologies for a range of purposes.
The words of the neuronal language can be heard in the electrical “spikes” in neurons – although some neuroscientists have proposed creating a BCI by monitoring broader, deeper “fields” of brain activity using electroencephalography, which would not require the surgical implantation of electrodes. EEG sensors have had some success, but they have produced only faint signals compared to implants that capture neuronal spikes.
A spike is the pinnacle of an electrical surge, the “action potential” that occurs when a neuron is activated and fires. On one of the monitors showing Nagle’s brain activity, dozens of action potentials play out in rows across the screen as the computer compiles signals from the electrodes in Nagle’s implant, each of which registers the activity of dozens of neurons. When a neuron fires, the line on the monitor begins to rise in proportion to the electrical surge, and then, moving at a speed more than 100 times faster than the blink of an eye, it peaks, which is what causes the “pop.” Once the neuron has fired, its electrical signal drops back down, and the output either stays flat or begins its ascent again.
Neurons, when active, fire between 20 and 200 times a second. The timing and the location of spikes in the brain, and the interaction of multiple spikes among neurons, create the coherent signals that are turned into muscle movements and all the other “outputs” of the brain.
“Understanding how groupings of neurons work for motor activity is relatively simple,” says Hatsopoulos, who helped write the algorithms for Braingate. “As we learn to read more neurons at once, it will eventually tell us how higher brain functions work, such as emotions and other behavior and thought processes.”
By conducting human trials, Donoghue has pulled ahead of his colleagues, though other scientists have plans for their own clinical trials of neuro-prostheses controlled with implanted electrodes. In Atlanta, Kennedy’s company has received approval from the FDA to test single- and double-electrode implants in severely disabled patients. At Caltech, Andersen’s team has begun to experiment on humans suffering from epilepsy, using brain implants surgically embedded in the prefrontal cortex (an area that helps plan and execute bodily movement); the implants sense an oncoming seizure and apply tiny electrical shocks to shut it down. Though Andersen has no commercial plans for the device, he intends to expand the human tests in clinical trials.
Andersen is also expanding his work with monkeys; he has implanted sensors in the higher-functioning areas of a monkey’s brain and deciphered some of the electrical signals whereby the monkey plans actions and others that seem to govern its motivation to perform a specific feat. “We have a difference in approach from Donoghue’s work,” says Andersen. “We’re reading intent” – whereas Donoghue is tapping into the motor-action part of the brain. Monkeys with electrodes in either brain region can move cursors and devices, says Andersen.
Duke’s Nicolelis has invented a system that allows a monkey to move a prosthetic arm up and down to deliver a snack. Nicolelis also linked up his monkey’s brain to the Internet and had the monkey operate a robot arm 950 kilometers away. He has been testing humans with deep brain implants to study the patterns in which their neurons fire when they squeeze balls. So far, he has recorded the output of up to 50 cells and is using this electrical data to devise algorithms to move a cursor. He is also studying how neurons in the brain adapt to the use of robotic arms and machines, since neurons are continually modified by the acquisition of new information and skills.
These types of experiments are rapidly advancing the technology, giving it more and more potential to help patients. At the University of Pittsburgh, Schwartz has run experiments enabling monkeys to move an artificial arm and hand more fluidly. “These devices have the degrees of motion of a human arm and elbow,” he says. His team wants to test their arm on humans. “We’re on a five-year horizon,” Schwartz says, for the arm to be working well in humans.
Brown neurosurgeon Gerhard Friehs performed Nagle’s implant operation at Rhode Island Hospital in Providence in June 2004. Friehs is an expert at implanting neuro-devices such as the Activa brain stimulators for Parkinson’s patients that control the muscle tremors associated with the disease. On a plastic model, Friehs showed me the spot where he drilled a small hole into Nagle’s skull, above the region that controls the left arm. Friehs then inserted the implant using a pneumatic inserter, a device he says is like a staple gun that shoots the electrode array onto the brain.
Nagle was first put under general anesthesia, though Friehs says that in the future, this may not be necessary. Technicians then used magnetic-resonance imaging (MRI) of Nagle’s brain to pinpoint the motor cortex area specific to his anatomy. In the operating room, Friehs used the MRI data to guide him to the precise coördinates in Nagle’s brain and then revved up a high-speed drill to remove a half-dollar-sized circle of skull. Friehs inserted the four-by-four-millimeter electrode chip, the wires, and the pedestal and replaced the piece of skull. Total operating time: about four hours.
Six weeks later, after Nagle’s wounds had healed and the immediate threat of infection was past, the researchers prepared to test Braingate. Cyberkinetics technician Abraham Caplan, who makes the house calls with Saleh to operate Braingate two or three times a week at the assisted-living center where Nagle lives, remembers the first time they plugged in Nagle, in August 2004. On the video of this inaugural experiment, Nagle is sitting in his chair, and Saleh asks him to imagine moving his hand to the left. The computer broadcasts the snaps and pops of the signals that race across its screen, as it reads the brain’s real-time chatter, which it correctly translates into a cursor moving left on Nagle’s screen. “Not bad, man,” says Nagle, “not bad.”
Soon after, Nagle was able, with practice, to draw a crude circle on the screen with his mind, and he progressed to playing Pong and learning to move the cursor to click commands that control his television, turning it on and off, changing the stations, and adjusting the volume. “It’s like riding a bicycle,” says Donoghue. “At first he’s wobbly, he oversteers, and then he’s suddenly riding.” Nagle can talk and operate the computer at the same time, just as a healthy person might sing a song and walk. “This is important, because he doesn’t need to actively think of moving his hands to the left or right,” says Donoghue. “He just thinks about moving the cursor, and it moves.”
To understand what Braingate means for Nagle, I visit Leigh Hochberg, the Harvard neurologist. Hochberg, who is a consultant at the Spaulding Rehabilitation Hospital in Boston, works with patients who have suffered strokes or severe spinal-cord injuries. He shows me the Assisted Technology Group’s room at Spaulding, where quadriplegic and other severely disabled patients come to operate computers and other machines using devices hooked up to eyelids or lips or tongues, whatever they can still move. For those with no muscle movement, special cameras track pupil motion, which patients have learned to control in order to operate cursors. Others inhale and exhale through a straw to move a wheelchair.
Hochberg is the chief investigator for the Cyberkinetics FDA trial at Spaulding; this was the second site chosen for the trial, after Sargent Rehabilitation Center in Warwick, RI, the base for Nagle’s trial. Hochberg and coinvestigator Joel Stein, Spaulding’s medical director for the stroke program, have begun recruiting patients to fill the spaces allowed under the FDA license. Surgenor also wants to open another clinical-trial site, possibly in the Midwest. This will become even more important if the FDA approves human Phase II trials, which would involve up to several dozen patients.
“I think in the short term we’re not looking for a cure for spinal-cord injury,” says Stein, who nevertheless believes that in the long term Braingate will prove useful for patients with certain types of motor injuries. “We don’t want to oversell this to our patients, but the potential in the future is great.”
The Color of Thought
At Brown University, I met computer expert Michael Black, an alumnus of the famed Xerox Palo Alto Research Center in California. Black is best known for trying to devise machines that can see, although he has also done research on brain-computer interfaces. Black was quickly sold on the possible benefits of Braingate and took on the task of creating improved algorithms for deciphering neuronal spikes. In theory, better deciphering would allow finer motor control. He showed me some charts with colored pixels that he developed to visualize what happens when a neuron fires. Each chart depicts a neuron’s activity across a range of hand motions. The chart is blue where the neuron is inactive and shaded purple, orange, and then red where it becomes excited and spikes rapidly. (For example, a blue field with a bright red patch in the upper right corner means that this neuron becomes active when the monkey’s hand moves up and to the right.) These grids tell Black the firing patterns of a neuron, which he can model to tell a computer that a given thought command is occurring and that it should take the appropriate action. The key to creating these models, he says, is the amazing tendency of brain neurons to fire in relatively consistent patterns – consistent enough that a computer can accurately interpret them.
In a building across Brown’s campus, I talked to another member of Donoghue’s team, Arto Nurmikko, a Finnish electrical engineer and physicist known for his discoveries in laser optics and semiconductors. He and Donoghue are working to simplify Braingate and replace the titanium pedestal and the bulky hardware of the prototype with a much smaller internal system that would connect the implant to a hair-thin fiber-optic cable that would run under the skin of the patient. The fiber-optic cable would feed signals from the brain to a processor the size of a cardiac pacemaker, which would be implanted in the chest.
The technology will take a while to develop. But Nurmikko says that in this next-generation system, communication between the brain and the machine would be two way, with sensory information from a robotic limb relayed back into the brain, just as in a healthy person. When a patient reaches for a glass of water, for example, such neural feedback would help brain and computer calculate the effort necessary to pick it up.
Waiting for Help
Will these devices improve people’s lives? Nagle himself says that Braingate, at least in its current form, is only marginally helpful to him. “This thing was done to see if I could move a cursor with thought,” he says, “and I did that in about three minutes.” But Nagle forcefully points out that he wasn’t doing much of anything before. “I sat here seven days a week with nothing to do, so I said, ‘Why not?’”
According to the FDA protocol, the study involving Nagle is to last a year. “I’ll have to decide next June if I want to take this out. I’m not sure I will continue on. I may want to wait until they have one that is smaller and easier to use.” I ask him if he thinks he’ll walk again, and he says that’s what he’s really waiting for.
David Ewing Duncan’s next book, The Geneticist Who Played Hoops with My DNA and Other Masterminds from the Frontiers of Biotech, will be out in May.
AI and robotics are changing the future of work. Are you ready? Join us at EmTech Next 2019.Register now