|Home | About | Journals | Submit | Contact Us | Français|
We use our arms to transport and orient the hand which is used to grasp and manipulate objects. Upper limb paralysis or amputation limits a person’s ability to interact with their environment to accomplish activities of daily living. Brain-machine interfaces (BMIs) may provide a solution to restoring much of this function.
Two 96-channel intracortical microelectrodes were implanted in the motor cortex of an individual with tetraplegia. Thirteen weeks of BMI training were conducted with the goal of controlling an anthropomorphic prosthetic limb with 7 degrees-of-freedom (3D translation, 3D orientation, 1D grasping). Clinical measures of upper-limb function were used to assess the participant’s ability to use the prosthetic limb.
The participant demonstrated the ability to move the device freely in the three-dimensional (3D) workspace on the second day of training. After 13 weeks, robust 7 degree-of-freedom movements were performed routinely. Over time, performance on target-based reaching tasks improved in terms of success rate, completion time, and path efficiency. The participant was also able to use the prosthetic limb to perform skillful and coordinated reach and grasp movements that resulted in clinically significant gains in tests of upper-limb function.
This study demonstrates that a person with chronic tetraplegia can perform consistent, natural, and complex movements with an anthropomorphic robotic arm to regain clinically significant function.
Defense Advanced Research Projects Agency, National Institutes of Health, Department of Veterans Affairs, and UPMC Rehabilitation Institute
Brain-machine interfaces (BMIs) transform neural activity into control signals for an external device. Functional electrical stimulators, exoskeletons, and sophisticated prosthetic limbs are all being developed with the goal of restoring or replacing natural function to a paralyzed or lost limb. In order to perform activities of daily living, one needs to be able to position the hand in space, orient the palm, and grasp an object. Hand placement, orientation, and grasping are normally smoothly coordinated and follow general principles of natural movement.1–3 Ideally, a BMI will capture these features directly from neuronal activity and translate them to control of an external device with the capability of producing natural movements that follow these principles.
Animal studies using intracortical microelectrodes have shown that these natural features of movement can be captured in recordings of populations of motor cortical neurons4. These studies have demonstrated control of a robot arm in 4 dimensions to perform self feeding tasks5, 7-dimensional (7D) control for orientation and grasping6, and control over functional electrical stimulators of the upper limb.7, 8 To date, human studies have demonstrated 3D translational control9–11, plus control over a discrete (on/off) grasping dimension.12 Our results build on previously established methods5, 6 to show that a person with tetraplegia was able to rapidly achieve control of a state-of-the-art anthropomorphic prosthetic limb (MPL, Johns Hopkins University, Applied Physics Laboratory). After 13 weeks of training, robust 7D (3D translation, 3D orientation, 1D grasping) movements were performed routinely. These movements included reaching and grasping tasks, similar to many activities of daily living, which were carried out with coordination, skill, and speed approaching that of an able-bodied person.
This study was conducted under an Investigational Device Exemption (IDE) granted by the US Food and Drug Administration and with approval from the Institutional Review Boards at the University of Pittsburgh and the Space and Naval Warfare Systems Center Pacific. This trial is registered on clinicaltrials.gov (http://clinicaltrials.gov/ct2/show/NCT01364480). No adverse events have been observed to date.
Informed consent was obtained from a 52-year-old woman who was diagnosed with spinocerebellar degeneration 13 years prior to participation in the study. Thorough chart review as well as discussions with the participant’s neurologist revealed no indication of cerebellar involvement. Her injury was motor complete with manual muscle test scores of 0/5 for the upper limb.13 Physical examination revealed limited sensory deficits and some hypersensitivity.
Two intracortical microelectrode arrays (4x4 mm, Blackrock Microsystems Salt Lake City, UT, USA), each with 96 electrode shanks (1·5 mm length) were implanted in the left motor cortex on February 10, 2012. Stereotactic image guidance (Brainlab, Inc., Westchester, IL, USA) with co-registered structural (MRI) and functional (fMRI) imaging was used to guide placement of the arrays (figure 1A).
Generally 3 sessions were conducted per week, each approximately 4 hours in duration. Neural signals were recorded using the NeuroPort data acquisition system (Blackrock Microsystems, Salt Lake City, UT). Single- and multi-unit events were classified using manually set thresholds and time-amplitude windows. Daily set-up time, including unit sorting, required approximately 30 minutes. Activity was converted to firing rate in 30 ms bins and low pass-filtered using an exponential smoothing function with a 450 ms window.
BMI training progressed from 3D endpoint translation control (Weeks 2 and 3 post-implant), to 4D control of translation and grasp (Week 4), to 7D control of translation, orientation and grasp (Weeks 5–14). The joints of the fingers and thumb were combined into a single grasp dimension. The neural decoder, based on a model that linearly related neural firing rate to movement velocity (Eq. 1)6, 14, was calibrated daily using a two-phase calibration similar to methods used in non-human primate studies.5, 6
During the observation phase, the MPL moved automatically, driven by a proportional controller, to targets that spanned the reachable workspace of the arm as part of a 7D Sequence Task. In the 7D Sequence Task, a target was indicated by an LED on a board in the frontal plane (80x80 cm), 70 cm in front of the MPL shoulder (figure 1B). Once the MPL was moved to the correct 3D position, computer-generated verbal prompts described an orientation to be performed by rotating the palm in one of 6 directions, or a grasp target (open or closed hand). Possible orientations included pronation-supination (±45°), ulnar-radial deviation (±20°), and flexion-extension (±45°) of the wrist. The participant was instructed to carefully watch the limb as it moved automatically to the targets. Neural data and MPL state information including endpoint position, joint angles, and joint torque were recorded at a rate of 33 Hz. Approximately 80 trials (6 minutes) of 7D Sequence Task data were collected to train the initial observation-based 7D neural decoder. The following model was used to relate unit activity to MPL movement velocity6, 14:
where f was the firing rate of a unit during movement described by 7D velocity vector V (vx,vy,vz,vθx,vθy,vθz,vg) and b0, bx, by, bz, bθx, bθy, bθz, bg were the coefficients for each unit. V includes coordinates for the 3D endpoint translation (x,y,z), 3D orientation (θx, θy, θz) and 1D grasp (g) velocities. Units that failed to fit the model (R2≤0.1) were excluded. Indirect optimal linear estimation (OLE)15 with ridge regression16 was used to solve for the coefficient matrix B (webappendix p1).
During a second phase of calibration, the participant controlled the MPL using the observation-based neural decoder while ortho-impedance was applied to the command signal.5 During this phase, ortho-impedance attenuated, by 100%, the brain-command component perpendicular to the ideal 7D trajectory, restricting movements to directions directly toward or away from the target. Data from 80 trials of the 7D Sequence Task collected during the second phase of calibration were used to build the final decoder using OLE as described above. This second round of calibration allowed the participant to actively engage in the task, while minimizing errors and corrective movements. Early in the experiment (Weeks 2–4), similar calibration procedures were used to generate lower-dimensional neural decoders. Calibration was conducted daily and took approximately 15 minutes.
After completing the calibration, the 7D Sequence Task was used for BMI testing. Targets were selected randomly by the computer from a possible combination of 10 translation targets, 7 orientation targets, and 2 grasp targets. Endpoint velocity derived from the recorded firing rates were sent to the MPL controller which converted them to joint motor commands every 20 ms. As in calibration, a translation target was specified by an LED and then an orientation or grasp target was announced by computer-generated voice. The participant had control of all 7 dimensions (translation, orientation, and grasp) throughout the task.
In the early phases of the experiment, an adjustable amount of ortho-impedance was used in the testing phase. In addition, computer assistance was also provided to stabilize the hand’s position or orientation to facilitate learning.5, 6 Stabilizing control blended an ideal endpoint velocity control signal determined by the proportional MPL controller with the control signal derived from the participant’s brain activity. The relative contributions of each signal were specified by the experimenter. Stabilizing control was only applied to domains that were not actively being controlled. In other words, position was stabilized at a specified percentage (10–100%) during the orientation phase and/or orientation and grasp posture were stabilized during translation. Similarly, the experimenter could also specify the percentage of ortho-impedance. Both forms of computer assistance were adjusted subjectively to keep the participant engaged and motivated with the aim of maintaining a 60–80% success rate. It should be emphasized that after week 10 (Day 66), the subject performed all tasks with no computer assistance. A 10-second maximum movement time was enforced for each trial phase (translation or orientation/grasp).
Performance metrics including success rate, completion time, and path efficiency were calculated during these testing sessions. Success was determined based on MPL kinematics recorded in real-time. All 7 degrees-of-freedom were controlled continually and had to be correct during all three phases of movement in order for a trial to be considered a success. The experimenter could also manually mark successful (or failed) trials that might occur if the subject rotated too far in the specified direction or if mechanical limits of the MPL prevented movement along a single dimension. To calculate performance, chance levels were determined using random walk-neural signals as input to a simulation while using the assist parameters (if any) and target sizes used during the experiment. Two hundred simulations were run per trial and the median daily chance levels from these simulations are reported (figure 3A). For each trial, path efficiency (PE) was computed for each control domain using the following equation:
where LOT is the length of the optimal trajectory from the starting position to the end target location and LBCT is the length of the brain-controlled trajectory for a given trial. LBCT was determined by numerically integrating the velocity vector magnitude, V, over the duration of a single trial.
The participant used the MPL under full brain-control to perform 9 tasks (selected from 19 possible) on the Action Research Arm Test (ARAT), which is an evaluation of unilateral upper limb function used commonly in patients who have had a stroke.17 The ARAT is psychometrically sound and tracks the ability to perform activities of daily living (ADLs). The selected tests include all 6 items from the grasp subscale and 3 of the 4 items from the grip subscale (listed in table 1). Standard administration procedures18 were followed with a few modifications. One of the items was a ball, which was positioned inside a loose coil of wire to keep it from rolling off the table, although it was still possible for the subject to bump the ball hard enough to roll outside of the wire coil. The MPL hand started approximately 12 cm from the table surface, with the palm facing to the left instead of on the table. Each test item was timed and scored as 0 (no movement possible), 1 (task partially performed), 2 (task performed, but abnormally), or 3 (task performed normally). Movements that required more than 5 seconds to complete were scored as a 2. The subject attempted each evaluation 3 times and was instructed that only the best score counted. This kept her motivated and open to trying new strategies.
The participant also performed a cone-stacking task, which involved moving three plastic cones (30 cm tall, 4·1–6·4 cm diameter) from a base and stacking them sequentially on a second base located 30 cm away. The time to complete the task and the number of drops were recorded for each attempt. This test measured the subject’s ability to grasp, transport, and position objects with precision.
The sponsors of this study had no role in study design, data collection, data analysis, data interpretation, or writing of the report. The corresponding author had full access to all of the data and had final responsibility for the decision to submit for publication.
The number of units, including single and multi-unit activity, recorded daily ranged from 209 to 271 (figure 2). There appeared to be an initial settling period where the number of units started off high until Day 21 post-implant and dropped in Week 4 (Day 24). In Week 4, the number of units began to increase at a rate of 0·368 units/day or 2·58 units/week. Figure 2 also shows that the number of units tuned to movement velocity with an R2>0·1 (Eq. 1) increased linearly over time at a rate of 0·388 units/day or 2·72 units/week.
Figure 3A shows the participant’s success rate for both computer-assisted and full brain control trials on the 7D Sequence Task over all 22 sessions of 7D testing, as well as the chance level determined for each day. Ortho-impedance and stabilizing assist were only used until week 10 (Day 66). No testing with the MPL was conducted on Days 52–58. The participant’s performance improved over the duration of training. During the last two weeks of training, with no computer assistance, the subject achieved an average success rate of 91·6±4·4%, compared to a median chance level of 6·2% calculated using simulations with random walk neural data as an input signal. Figure 3B shows the normalized performance index (success rate/chance level) over the duration of training. The participant’s performance increased over time even as the difficulty increased.
The participant also showed improvements in MPL control as evidenced by a decrease in block completion time (figure 3C) and an increase in path efficiency (figure 3D-E). Block completion time was reduced from a mean of 148 seconds to 112 seconds over the course of 7D training and also became less variable. Average path efficiency (figure 3D) increased as the participant moved along a straighter, more coordinated path. Mechanical constraints sometimes forced the MPL to deviate from an ideal linear trajectory, causing the maximum achievable efficiency under auto-control to be less than 1 (figure 3D). The greatest improvement in control appeared to be in the translation domain which increased linearly over time (figure 3E). An MPL kinematics sample of four consecutive trials is shown in figure 4. These plots show that the participant was able to rapidly reach the target position in the specified control domain (translation, orientation, or grasp) while maintaining her position in the other two domains (webvideo 1).
Neural changes were also observed during the 7D brain control testing period. The percentage of units tuned to 7D MPL velocity (Eq. 1) showed a linear increase (y = 0·002x + 0·100, R2 = 0·119, p=0·066) as shown in figure 5A. The percentage of units for R2 values >0·1 increased during the late training phase (Weeks 11–14) compared to the early phase (Weeks 5–8). The median R2 value was significantly larger during late training than during early training (p<0.001, Wilcoxon rank-sum test). The percentage of units with R2 values of 0·15–0·30 showed the greatest increase (figure 5B). This may suggest that the participant was learning to modulate her brain activity in order to better fit the neural encoding model to improve performance.19
Table 1 shows the completion time, as well as the total score, for each test item on each day that the ARAT was completed. If no time is listed, the score for that item was a 1 since the task was partially completed. For items with completion times listed, the score was a 2 because it took longer than 5 seconds (equivalent to the approximate amount of time it would take an unimpaired adult to complete the task18). The participant demonstrated consistent performance as her total score ranged from 15–17 across 7 testing days. The subject was unable to move her own upper limbs volitionally and therefore scored a 0 without the use of the BMI or other assistive technology. Webvideo 2 shows her ARAT performance on Day 87. The movements were generally smooth, direct, and fast. The subject reported that she was thinking about the goal of the action, such as “grab the block,” rather than issuing specific kinematic commands to the MPL endpoint, suggesting that the control was intuitive.
Table 2 shows the cone-stacking task completion time and the number of drops for each day of testing (See also webvideo 3). Dropped cones were repositioned by the experimenter. The fastest performance was achieved on the last day of testing, although the participant had more drops than on Day 87. In this case, she knocked the first cone off the base 6 times before quickly moving all three cones, with only one additional drop. The participant was able to complete this task consistently on all 4 days of testing.
This study shows that an individual with tetraplegia rapidly learned to reach and grasp using coordinated 7D control of a high performance anthropomorphic prosthetic limb. Performance on reaching and grasping tasks consistently improved over the course of 34 training sessions completed over 13 weeks. The steady improvement demonstrates a robust approach and that the participant was consistently learning. In this context, learning is defined as the modification of the correspondence between neural activity and action. Although the mechanism of this modification is unknown, the clear and consistent change in neural tuning is a hallmark of the BMI paradigm19–22 and our results show that the neural population became more tuned to MPL movement velocity over time. The use of neural extraction algorithms derived from natural movement along with an anthropomorphic effector likely promoted the acquisition of the skill necessary to achieve the performance demonstrated here. This type of BMI control is model-based, in contrast to classifiers which operate on pre-defined libraries of discrete input-output categories. With our approach, continuous 3D translation of the hand was achieved with a few hours of training, and 7D control was performed routinely in less than 4 months. The movements were smooth, coordinated and skillful. The subject had the flexibility to explore a spectrum of arm/hand configurations to complete tasks with different strategies (webvideo 4).
Non-human primate studies established the observation-based calibration, computer assist methods, and the extraction algorithms used in the current study.5, 6 Previous human studies have demonstrated that intracortical electrodes could be used to provide rudimentary control of computer cursors23 and robotic manipulators.12 Our study builds upon this work by adding degrees of freedom which allow for more natural and coordinated movements, a broader workspace, and the ability to interact with a variety of objects in different orientations (webvideo 4). Across multiple days of testing, the participant was able to improve her ARAT score from 0 to 15–17 (out of 27). This achievement far exceeds the threshold of 5·7 points for a clinically significant improvement in function21, demonstrating that use of a neuroprosthetic device has the potential to provide clinically significant functional benefit. As shown in webvideo 2, movements were smooth and coordinated, with speeds approaching that of able-bodied adults. Since we used a velocity-based decoding model, the MPL speed was controlled directly by the participant. The level of control achieved by the participant allowed for subtle corrective movements and object manipulation (webvideo 3). ARAT testing began once the participant had achieved consistent 7D control on the target-based task and her performance on clinical measures was consistent during the last three weeks of testing.
We expect a number of further developments in BMI technology in the near future. A wider range of tasks will become possible by adding hand shape to the repertoire of volitional control. Tactile feedback will be transduced by fingertip sensors and transmitted to sensory cortex. Telemetry will eliminate the need for transcutaneous leads and connectors. Paralyzed arms will be reanimated by activating the subject’s own muscles, which we know is a high priority for individuals with tetraplegia.6,7,24 Our current results and the very rapid developments in this field show the potential to restore much of the function lost to individuals with tetraplegia or upper limb amputation, allowing them to again perform natural behaviors to more fully interact with the world around them.
A literature review was conducted in PubMed without date or language restriction using the search terms “neuroprosthetics,” “brain-machine interface,” or “brain-computer interface.” Since our goal is to restore upper limb function, we restricted our survey to studies using robotic technology or functional electrical stimulation to achieve this same goal. In addition, the authors drew from their own substantive reference libraries. Two published reports of robotic arm12 and hand25 control in human subjects with motor impairments were identified. One non-human primate study reported 4D control of a robotic arm to perform a self-feeding task5, while two have investigated functional electrical stimulation of the upper limb.7, 8 For some individuals with amputation or residual muscle activity, alternative prosthetic control techniques including non-invasive approaches26, targeted-reinnervation27, or direct peripheral neural control28, 29 may be appropriate.
Previous work has shown that it is possible to record neural activity from motor cortex and translate that activity to movement of an external device or one’s own muscles. However, to date, no human studies have shown that natural and complex movements can be performed consistently for a variety of tasks. Here, we have demonstrated that a person with chronic tetraplegia can perform complex and coordinated movements freely in 7D space consistently over multiple weeks of testing. This is novel because previous studies have limited control to translation dimensions, used staged control schemes, or limited the workspace to complete very structured tasks. Increasing control dimensionality allows our participant to fully explore the workspace by placing the hand in the desired 3D location and orienting the palm in three dimensions. This is also the first time that performance has been quantified using functional clinical evaluations. While most human studies rely on only a few days of performance data, we have shown that the participant learned to improve her performance consistently over many days using a variety of metrics. By using training methods and algorithms validated in non-human primate work, individuals with long term paralysis can recover the natural and intuitive command signals for hand placement, orientation, and reaching to move freely in space and interact with the environment.
This material is based upon work supported by the Defense Advanced Research Projects Agency (DARPA) Revolutionizing Prosthetics program contract number N66001–10-C-4056, the National Institutes of Health (NIH) grant 8KL2TR000146–07, the Office of Research and Development, Rehabilitation Research & Development Service, Department of Veterans Affairs (Grants # B6789C, B7143R, and RX720), and the UPMC Rehabilitation Institute. The views expressed herein are those of the authors and do not reflect the official policy or position of the Department of Veterans Affairs, Department of Defense, or the United States government.
We thank Jan Scheuermann for her extraordinary commitment and effort related to this study as well as insightful discussions with the study team. We also thank Karina Palko for her participation as an honorary research team member and support of the study. We thank clinicians and researchers at the University of Pittsburgh and UPMC including Dr. Joseph Ricker (Physical Medicine and Rehabilitation, PM&R) for conducting presurgical neuropsychological screening, Dr. Richard Barbara (PM&R) for providing psychological monitoring throughout the study, Dr. Ferenc Gyulai (Anesthesiology) for directing general anesthesia for the implantation surgery, and Dr. Elizabeth Skidmore (Occupational Therapy) for advice related to functional assessments using the prosthetic limb. We thank Drs. Ramana Vinjamuri and Robin Ashmore (PM&R) for technical and software development efforts, Dr. Stephen Foldes (PM&R) for assistance with presurgical neuroimaging, Dr. Elke Brown (PM&R) for assistance with BMI testing sessions, Dr. Samuel Clanton (School of Medicine) who developed a prototype-system for high-dimensional decoding and subject training that was adapted for this experiment, Ms. Elizabeth Harchick (PM&R) for assistance with presurgical neuroimaging, recruitment, and regulatory compliance, and Ms. Alisha Shane-Cuniff (PM&R) for coordination of the Data and Safety Monitoring Board (DSMB). We thank the University of Pittsburgh Clinical and Translational Science Institute as well as the Office of Investigator-Sponsored IND and IDE support for assistance with protocol development and regulatory reporting and compliance. We thank the volunteer members of the DSMB for their ongoing monitoring of this study. Finally we thank the Johns Hopkins University Applied Physics Laboratory and Blackrock Microsystems, Inc. for coordination efforts and technical support related to this project.
ContributorsJC, WW, ET-K, DW, MB, and AS designed the study. JC, BW, and JD completed most of the data collection and analysis with all authors contributing to development of analysis techniques and interpretation of results. ET-K developed and performed the surgical implantation procedures. BW, JD, AM, and MV developed the software required for the real-time experiments. JC, MB, and AS wrote the manuscript and all authors provided critical review and approval of the manuscript.
Conflict of interest statement
MV and AS have a patent application pending that covers some of the methodology used in this study. We declare that we have no other conflicts of interest.
Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
Jennifer L Collinger, Department of Veterans Affairs Medical Center, Pittsburgh, PA, USA. Department of Physical Medicine and Rehabilitation, University of Pittsburgh, Pittsburgh, PA, USA. Department of Bioengineering, University of Pittsburgh, Pittsburgh, PA, USA.
Brian Wodlinger, Department of Physical Medicine and Rehabilitation, University of Pittsburgh, Pittsburgh, PA, USA. Center for the Neural Basis of Cognition, Pittsburgh, PA, USA.
John E Downey, Center for the Neural Basis of Cognition, Pittsburgh, PA, USA.
Wei Wang, Department of Physical Medicine and Rehabilitation, University of Pittsburgh, Pittsburgh, PA, USA. Department of Bioengineering, University of Pittsburgh, Pittsburgh, PA, USA. Center for the Neural Basis of Cognition, Pittsburgh, PA, USA. Clinical and Translational Neuroscience Institute, University of Pittsburgh, Pittsburgh, PA, USA.
Elizabeth C Tyler-Kabara, Department of Physical Medicine and Rehabilitation, University of Pittsburgh, Pittsburgh, PA, USA. Department of Bioengineering, University of Pittsburgh, Pittsburgh, PA, USA. Department of Neurological Surgery, University of Pittsburgh, Pittsburgh, PA, USA. McGowan Institute for Regenerative Medicine, University of Pittsburgh, Pittsburgh, PA, USA.
Douglas J Weber, Department of Veterans Affairs Medical Center, Pittsburgh, PA, USA. Department of Physical Medicine and Rehabilitation, University of Pittsburgh, Pittsburgh, PA, USA. Department of Bioengineering, University of Pittsburgh, Pittsburgh, PA, USA. Center for the Neural Basis of Cognition, Pittsburgh, PA, USA.
Angus JC McMorland, Systems Neuroscience Institute, University of Pittsburgh, Pittsburgh, PA, USA.
Meel Velliste, Systems Neuroscience Institute, University of Pittsburgh, Pittsburgh, PA, USA. Department of Neurobiology, University of Pittsburgh, Pittsburgh, PA.
Michael L Boninger, Department of Veterans Affairs Medical Center, Pittsburgh, PA, USA. Department of Physical Medicine and Rehabilitation, University of Pittsburgh, Pittsburgh, PA, USA. Department of Bioengineering, University of Pittsburgh, Pittsburgh, PA, USA. McGowan Institute for Regenerative Medicine, University of Pittsburgh, Pittsburgh, PA, USA.
Andrew B Schwartz, Department of Physical Medicine and Rehabilitation, University of Pittsburgh, Pittsburgh, PA, USA. Department of Bioengineering, University of Pittsburgh, Pittsburgh, PA, USA. Center for the Neural Basis of Cognition, Pittsburgh, PA, USA. McGowan Institute for Regenerative Medicine, University of Pittsburgh, Pittsburgh, PA, USA. Systems Neuroscience Institute, University of Pittsburgh, Pittsburgh, PA, USA. Department of Neurobiology, University of Pittsburgh, Pittsburgh, PA.