|Home | About | Journals | Submit | Contact Us | Français|
The brain typically utilizes a rich supply of feedback from multiple sensory modalities to control movement in healthy individuals. In many individuals, these afferent pathways, as well as their efferent counterparts, are compromised by disease or injury resulting in significant impairments and reduced quality of life. Brain-machine interfaces (BMI) offer the promise of recovered functionality to these individuals by allowing them to control a device using their thoughts. Most current BMI implantations use visual feedback for closed-loop control; however, it has been suggested that the inclusion of additional feedback modalities may lead to improvements in control. We demonstrate for the first time that kinesthetic feedback can be used together with vision to significantly improve control of a cursor driven by neural activity of the primary motor cortex (MI). Using an exoskeletal robot, the monkey's arm was moved to passively follow a cortically-controlled visual cursor, thereby providing the monkey with kinesthetic information about the cursor's motion. When visual and proprioceptive feedback were congruent, both the time to successfully reach a target decreased and the cursor paths became straighter, as compared with incongruent feedback conditions. This enhanced performance was accompanied by a significant increase in the amount of movement-related information contained in the spiking activity of neurons in MI. These findings suggest that BMI control can be significantly improved in paralyzed patients with residual kinesthetic sense and provide the groundwork for augmenting cortically-controlled BMIs with multiple forms of natural or surrogate sensory feedback.
Over the past decade, substantial advances have been made in cortically controlled brain machine interfaces (BMI) in both animals and humans (Serruya et al., 2002; Taylor et al., 2002; Carmena et al., 2003; Nicolelis et al., 2003; Kennedy et al., 2004; Musallam et al., 2004; Hochberg et al., 2006). Despite these early successes, first-generation clinical implementations suffered from acute and pervasive problems such as underlying instabilities and poor quality of control (Hochberg et al., 2006). Some of these control issues have been mitigated as current BMI implementations demonstrate robust, stable performance on significantly complex tasks requiring cursor stabilization (Kim et al., 2008; Mulliken et al., 2008), self-feeding using a robotic arm (Velliste et al., 2008), and the voluntary control of paralyzed muscles (Moritz et al., 2008; Pohlmeyer et al., 2009). Further control improvements may require next-generation BMIs to move beyond the current paradigm where feedback is only available through the subject's intact visual system to one that incorporates feedback both from vision and other relevant sensory modalities (i.e. proprioceptive, tactile and force feedback).
Multiple sensory modalities are frequently used to estimate the state of the body with respect to the external environment and are thus critical for normal motor control (Rossetti et al., 1995; van Beers et al., 1999; Sober and Sabes, 2005). Experimental evidence indicates that abilities for on-line control and error correction are highly-dependent on the proprioceptive system, especially when visual feedback is unreliable or unavailable (Gordon et al., 1995). Some shortcomings of current BMI control are mirrored in the movements of patients suffering from the loss of proprioceptive feedback due to large-fiber sensory neuropathies. These individuals can move by relying on vision, but their movements require great concentration and are slow compared to healthy individuals (Ghez et al., 1995; Sainburg et al., 1995).
The importance of proprioceptive feedback in the control of artificial limbs has been recognized since the early 20th century (Childress, 1980), and its utility for advanced BMI control has been widely acknowledged (Abbott, 2006; Hatsopoulos and Donoghue, 2009). However, its effects on BMI control remain largely unexplored. We addressed this gap in BMI research by conducting an experiment in which the presence and fidelity of proprioceptive feedback during BMI control were systematically varied. Using a robotic exoskeleton, we evaluated the hypothesis that congruent visual and proprioceptive feedback would improve the movement of a cortically-controlled visual cursor compared to a BMI relying solely on vision for feedback. Due to the movement of the arm and its interaction with the exoskeleton, the monkeys received a very rich feedback experience. In addition to visual feedback, these conditions included sensory information about limb position and velocity derived from receptors in the muscles, joints, and skin (Bosco and Poppele, 2001; Johnson et al., 2008) and force and tactile cues from the interaction of their arm with the exoskeleton. Within the scope of this study, we define the term ‘proprioceptive feedback’ to include all movement related sensory information originating from sensory receptors in the arm.
Two adult male rhesus macaques (Macaca mulatta) were operantly trained to control a cursor in a two–dimensional workspace using a two-link robotic exoskeleton (Scott, 1999). The animals sat in a primate chair with their arm in the exoskeleton. Their shoulder joint was abducted 90 degrees and supported by the manipulandum such that all movements were made within the horizontal plane. Direct vision of the limb was precluded by a horizontal projection screen above the monkey's arm. Visual feedback was available via a visual cursor projected onto the screen. The position of the cursor was controlled by one of two sources: either the position of the robot's end effector (i.e. the monkeys hand position) or the output of a BMI that decoded the position of the cursor based on recent neural activity. Cartesian coordinates of the visual cursor were determined by digitizing the shoulder and elbow angle along with angular velocity at 500 Hz and transforming these variables into a visual cursor position (cm) using the forward kinematic equations for the exoskeleton.
The random target pursuit task (RTP) required the monkeys to repetitively move a cursor (6 mm diameter circle) to a square target (2.25 cm2). The target appeared at a random location within the workspace (12 cm by 6 cm), and each time the monkey hit it, a new target appeared immediately in a new random location. In order to complete a successful trial and receive a juice reward, the monkey was required to sequentially acquire two to seven targets, depending on the experimental condition. Because of their increased difficulty, the number of successful hits per trial was reduced in the BMI conditions to encourage continuous play. Because each trial completion was followed by the immediate presentation of another target, the monkeys typically did not pause between trials resulting in continuous movement trajectories. A trial was aborted if any movement between targets took longer than 2500 ms or if the monkey removed his arm from the exoskeleton. After an aborted trial, a new sequence of randomly presented targets appeared.
Prior to the experiment, a monkey used its arm to perform a random target pursuit task designed to generate complex movements that thoroughly sampled the position and velocity space of the arm. We refer to this case as the “active movement” condition (AM; Figure 1a). Next, the animals observed the replay of target positions and cursor trajectories recorded during the AM phase while they voluntarily maintained a static arm posture in the robotic exoskeleton at a predetermined reference position. Since neurons in the motor cortex demonstrate congruent activity during active movement and observation of action (Tkach et al., 2007; Suminski et al., 2009), we used the neural discharge recorded during the playback to construct a neural decoder that predicted Cartesian position of the cursor based on recorded cell activity. It is important to note that this decoder was trained solely on the observed target positions and cursor trajectories while the monkey's arm remained motionless in an attempt to mimic a training paradigm that could be used in patients with severe motor dysfunction. A new neural decoder was trained during each experimental session.
During the experiment, the monkey used a real-time decoding (BMI) algorithm to move the cursor in the same task based on the activity of an ensemble of recorded motor cortical (MI) neurons under three different conditions. In the first condition, referred to as the visual feedback only condition, VBMI, the animal moved the cursor only via the neural decoder while maintaining a static arm posture in the robotic exoskeleton at a predetermined reference position (Figure 1b). If the monkey moved the manipulandum away from this reference position and outside an invisible “hold” region (1 cm radius around the reference position) or removed his arm from the manipulandum, the game was “turned off” (i.e. the visual cursor and target were extinguished) until the monkey resumed proper posture. A gentle elastic force was applied to the manipulandum to keep the animal's arm from drifting away from the reference position, but was not applied within the “hold” region. Therefore, within the “hold” region, the exoskeletal robot's small inherent friction served as the only resistance.
The second condition, referred to as the visual and proprioceptive feedback condition, V+PBMI, provided the animal with congruent visual and proprioceptive feedback about the position of the decoded cursor (Figure 1c). Here, the animal controlled the cursor with the BMI while his arm was moved by the exoskeleton to follow the visual cursor, thereby transforming the monkey's arm from an effector driving the movement into a transducer providing an accurate proprioceptive estimate of the cursor's position and velocity. Cursor trajectories were reproduced well by movements of the exoskeleton as the median (1st quartile, 3rd quartile) correlation coefficient between hand and cursor position was 0.92 (0.81, 0.97) and 0.89 (0.76, 0.95) for the X and Y direction, respectively.
Lastly, we included a control condition, visual and noisy proprioceptive feedback (V+NBMI), in which the monkey moved the cursor via the BMI while its arm was moved by the exoskeleton through a trajectory that was different than the cursor (Figure 1d). The movement of the arm was driven by the replay of cursor trajectories recorded during the visual only and the visual and proprioceptive feedback BMI conditions that were completed earlier in the same experimental session. Despite the discrepancy in their feedback signals, hand movements contained information about BMI cursor movements as the median (1st quartile, 3rd quartile) correlation coefficient between the position of the BMI cursor and the hand was 0.62 (0.48, 0.80) and 0.57 (0.46, 0.75) for the X and Y direction, respectively. The purpose of this condition was to confirm that any performance gains observed during the V+PBMI condition were due to information contained in the proprioceptive feedback and not task-irrelevant sensory stimulation related to the motion of the limb.
We implemented a PD controller to minimize the deviation between the commanded and realized position of the exoskeleton during the V+PBMI and V+NBMI decoding conditions. Due to the dynamics of the exoskeleton, we found that this controller introduced a time delay between the control signal and the realized position of the monkey's arm causing a delay in the proprioceptive feedback. To compensate for this delay, we imposed a 100 ms time delay on the visual feedback in all of the decoding conditions resulting in the temporal alignment of the cursor position and the monkey's hand during the congruent feedback BMI condition.
To assess the accuracy of the proprioceptive feedback given by the PD controller without voluntary control of the arm, we performed a separate control experiment where we measured the dynamics and average error between the commanded (i.e. the cursor) and actual positions of the robot (and arm) during replayed BMI trajectories with an anesthetized animal. This experiment provided a means to reject trials in the V+PBMI and V+NBMI conditions where the monkey may have made undesirable arm movements leading to large errors between hand and cursor direction above and beyond the error introduced by the passive dynamics of exoskeleton and monkey's arm.
In this control experiment, the animal was anesthetized (Ketamine, 2mg/kg; Dexmedatomidine, 75mcg/kg; Atropine, 0.04mg/kg) and then placed in the primate chair with its arm strapped into the manipulandum. Cursor position was digitized (500 Hz) and recorded independently while the monkey's relaxed arm was moved through the replayed cursor trajectories for approximately 5 minutes. Playback of each trajectory was repeated three times for a total exposure time of 15 minutes. We computed the cross-correlation between the X and Y cursor and hand positions during passive arm movements to measure the time delay between movement of the cursor and the manipulandum. As expected, a strong correlation (> 0.95) was observed between cursor and hand position at average time delays of 98ms and 52ms in the X and Y direction, respectively. To compute the error between cursor and hand positions, we first corrected for the dynamics of the position controller/manipulandum by shifting the hand position data by the appropriate time delay and then used the dot product to compute the angular difference in movement direction (error) between the cursor and hand on a sample-by-sample basis. In this control experiment, when the animal was anesthetized, the error between the cursor and hand direction averaged 41.6.± 7.5 (μ ± σ) degrees for monkey MK and 42.8 ± 6.7 degrees for monkey B.
Based on the values obtained from the control passive movement experiment, we computed the direction error between the hand and decoded cursor or replayed trajectory on a sample by sample basis for the V+PBMI and the V+NBMI conditions, respectively. Trials with an average error exceeding the mean error plus three standard deviations (as obtained from the control experiment described above) were excluded from further analysis. This threshold was 64.1 degrees for monkey MK and 62.9 degrees for monkey B. We also excluded trials where the monkey may have been voluntarily contributing to the movement of the exoskeleton. Here, a cross-correlation analysis between the commanded and actual hand position on each trial was used to determine those trials where the movement of the hand preceded (i.e. led) the movement of the command signal by greater than 0ms. As a result, across all experiments in both monkeys, 385 of 1210 trials and 335 of 561 trials from the V+PBMI and V+NBMI conditions, respectively, passed both tests and were analyzed further. We varied the correlation delay threshold in the V+PBMI to investigate the effect of the monkeys' voluntary contribution to movement of the exoskeleton on BMI performance. In this analysis we considered three additional thresholds for the elimination of trials: 1) hand led BMI cursor by > 25ms, 2) hand led cursor by > 50ms and 3) no lead/lag threshold (i.e. all data was considered). A total of 650 (53%), 860 (71%) and 1210 (100%) of V+PBMI trials were included in the analysis based on these exclusion criteria. A total of 708 and 756 trials were analyzed from the VBMI and AM conditions, respectively.
Each animal was chronically implanted with a 100-electrode (400 μm interelectrode separation) microelectrode array (Blackrock Microsystems, Inc., Salt Lake City, UT) in primary motor cortex (MI) contralateral to the arm used for the task (Maynard et al., 1999). The electrodes on each array were 1.5 mm in length. The tips of the electrodes were coated with iridium oxide. During a recording session, signals from up to 96 electrodes were amplified (gain of 5000), band-pass filtered between 0.3 Hz and 7.5 kHz, and recorded digitally (14-bit) at 30 kHz per channel using a Cerebus acquisition system (Blackrock Microsystems, Inc., Salt Lake City, UT). Only waveforms that crossed a user defined threshold were used for real-time decoding and stored for further analysis. The neural data used to train and drive the BMI during the real-time decoding conditions were comprised of single and multiunit spiking events that were spike-sorted online. On average, 43.67±1.43 and 44.80±2.35 (mean ± 1 standard error) neural channels were allocated for use with the decoder per experimental session for monkeys MK and B, respectively. In contrast, the stored spike waveforms that were used in our neural analyses were spike-sorted offline, using Offline Sorter (Plexon, Inc., Dallas, TX). In total, 609 individual cell samples were identified via Offline Sorter: 337 cell samples across six datasets from monkey MK and 272 cell samples across five datasets from monkey B. Because of the chronic nature of our recordings, we use the term “cell samples” when aggregating over data sets to make clear that these represent independent samples across different data sets but may or may not represent different neurons.
This study was conducted over an extended period of time and the monkeys were trained on multiple variants of the task described above. The 11 data sets (5 from monkey B and 6 from monkey MK) included in our analysis represent all data collected using this particular experimental design. A data set is defined as the simultaneously recorded neural activity during a single recording session. All of the surgical and behavioral procedures were approved by the University of Chicago Institutional Animal Care and Use Committee and conform to the principles outlined in the Guide for the Care and Use of Laboratory Animals.
The real-time decoder is implemented using a Wiener filter (Wessberg et al., 2000; Serruya et al., 2002; Fagg A. H., 2009). The spikes generated by a single neuron or multiunit activity during one second prior to prediction were binned into a total of 20 time bins, each of which was 50ms in duration. The predicted cursor position is a linear function of these binned spike counts. The coefficients for the model are selected using a ridge regression approach that trades prediction accuracy on the training set for a smoother prediction surface (Bjèorck, 1996). This approach helps to address the over-fitting problem that can occur with small training data sets in high-dimensional feature spaces.
In our approach, a time series of hand position data points is reconstructed from a linear combination of neural responses from many neurons at multiple times:
where S(t) is the signal to be reconstructed, N(i, j) denotes the activity of neuron i at time j, ai,j, is the corresponding coefficient, C is the number of cells, Tpre is the time before the current time t and dt is the width of the time bins (50 ms). The set of coefficients (ai,j) that minimize a cost function that includes the least mean squared difference between the actual and reconstructed signal can be solved for analytically:
where (NTN) is the correlation matrix that contains the pair-wise correlations between each pair of neurons at different time bins, α is a regularization constant required for the ridge regression, and I is the identity matrix. Since hand movements are restricted to a plane, independent estimates of the X and Y components of hand position were made in order to predict the motion of the hand.
We computed the chance percentage level of successful trials for each animal by randomly shuffling the binned firing rate time series for each unit during the VBMI and V+PBMI conditions. Cursor trajectories were then generated (offline) by applying the actual filters computed during the experiment to the shuffled neural data. Each simulated trajectory was required to hit the same sequence of targets as were presented during the real-time decoding conditions. Just as in the real experiment, cursor trajectories were allowed up to 2.5 seconds for the cursor to hit a target, and 2 consecutive target hits constituted a successful trial. We repeated this procedure 1000 times allowing us to compute a mean and standard deviation for the percentage of successful trials. The averaged chance success rate for monkey B was 29.9±0.8% and 36±0.4% (mean + 1 SE) for the VBMI and V+PBMI conditions, respectively. The averaged chance success rate for monkey MK was 38.7±2.2% and 32.6±0.8% for the VBMI and V+PBMI conditions, respectively.
Kinematic parameters (position and direction) of hand and cursor movement in each condition were binned in 50 ms bins and boxcar-smoothed using a 150 ms sliding window for most analyses. To assess performance differences between the active movement and real-time decoding conditions, we used three kinematic measures: 1) normalized time-to-target, 2) normalized path-length and 3) normalized path reversals. The normalized time-to-target metric is defined as the time difference between consecutive target hits divided by the Euclidian distance between the targets. The normalized path-length metric is defined as the path length of the cursor between consecutive targets divided by the Euclidian distance between consecutive targets. Normalized path-length is a unit-less ratio of distance measures, and captures the straightness of the cursor path. The normalized path reversal metric is defined as the number of positive to negative zero crossings in the movement speed directed along a line connecting consecutive targets. To compute the number of normalized path reversals we first rotated each trajectory such that it started at the origin and ended at a point on the positive x axis. We then computed the velocity of the trajectory in the x direction and counted the number of positive to negative zero crossings (i.e. the number of times the visual cursor started to move away from the target). Lastly, we divided the number of zero crossings for each target hit by the Euclidian distance between consecutive targets. Kinematic observations for each metric were pooled by monkey across datasets and a one-way ANOVA with post-hoc t-tests was used to determine if BMI performance differed as a function of experimental condition.
In our off-line sorting and analysis, we sought to isolate single units possessing signal-to-noise ratios of three or higher. Signal-to-noise ratios were defined for each sorted unit as the difference in mean peak-to-trough voltage divided by twice the mean standard deviation. The mean standard deviation was computed by averaging the standard deviation of the spike waveform over all acquired spikes at each of the 48 sample time points of the waveform. Unlike the preceding kinematic analyses, we considered all observations related to the spiking activity of single cells to be independent and pooled them across subjects for all population analyses.
We performed a principal components analysis (PCA) on the ensemble of neurons recorded in each dataset in order to understand how the different experimental conditions modulate spiking activity. Specifically, we were interested in exploring whether passive movement of the arm in the BMI conditions resulted in an increase in the “flexibility” of the neural activity (i.e. the ability to utilize more degrees of freedom). The methodology has been previously described (Yu et al., 2009). Briefly, spike trains (1ms resolution) for each neuron in the ensemble were binned in 20ms bins and the resulting spike counts were square root transformed to stabilize the spiking noise variance. A smoothed estimate of the firing rate for each neuron was then created by convolving the binned spike trains with a Gaussian kernel (σ = 60ms). We then used PCA to determine the number of orthogonal bases required to account for 90% of the variance in the ensemble activity. The number of bases in each condition was pooled across both datasets and subjects and a one-way ANOVA was used to determine if the number of bases varied across experimental condition.
Mutual information between binned neural data and kinematics (50 ms bins) was calculated at multiple time leads and lags (Paninski et al., 2004). This analysis captures both linear and nonlinear relationships between the two variables by means of signal entropy reduction. The computation yields a measure of the strength of the relationship between the two variables when they are shifted with respect to each other by different time lags. By examining the relative timing of the peak mutual information, we were able to determine at what time lag a neuron's modulation was most related to the cursor movement. The kinematic probability distributions (one-dimensional distribution of instantaneous movement direction) conditioned on the number of observed spikes were estimated by histograms of the empirical data. To account for biases in this estimation, the information calculated from shuffled kinematic bins (mean of one hundred shuffles) was subtracted from the values obtained from the actual data for each cell. Lastly, the lead/lag mutual information profiles were boxcar-smoothed with a 3-bin window (150 ms).
Electromyograms (EMGs) were recorded from the biceps and triceps using differential surface electrodes (Noraxon Myosystem 1200). EMG signals were band-pass filtered between 10-450 Hz, amplified (100×), and digitized at 2000 Hz using the Cerebus acquisition system. Residual offsets were subsequently removed from the digitized EMGs, which were then rectified and filtered at 4 Hz with a zero-phase low-pass filter (4th order Butterworth). Next, the resulting time series' of muscle EMG were normalized by the mean value of the rectified and filtered activity recorded during the active movement phase of the experiment. We then characterized the coordination between muscles at the elbow by estimating the degree of antagonist muscle co-contraction using a measure also known as ‘wasted contraction’ (Thoroughman and Shadmehr, 1999). For the pair of antagonist muscles (Biceps and Triceps) at each sampling instant, the minimum value of the two normalized EMG signals was selected to yield a time varying co-contraction signal which represents the magnitude of normalized EMG that is equal and opposite in the antagonist muscle pair. One-way ANOVAs were used to determine differences in muscle activity and co-contraction across conditions.
We used a cross-correlation technique (Suminski et al., 2007) to evaluate the temporal relationship between elbow angular velocity and individual muscle activities during each trial of the active movement and real-time decoding conditions. The resulting correlation time-series were averaged across trials within each experimental condition for display purposes. The peak correlation magnitude for successful trials in each condition was Fisher transformed and compared using a one-way ANOVA.
Eye movements were recorded with an infrared oculometer (R-HS-S6 Applied Science Laboratories, Bedford, MA) located in front of the animal. The eye position was calibrated using the ASL EyeTracker Software Suite™ while the monkey performed the RTP task with targets placed in a 3×3 grid that spanned the rectangular workspace. During the BMI conditions, the Cartesian positions of the monkey's gaze was digitized at 1 kHz, time-stamped with respect to the neural and kinematic data, and saved to disk. These values were regressed against the x and y target positions in the active movement condition in order to determine the location of gaze in the same coordinate frame as the cursor and target. Using the resulting regression coefficients, x and y gaze position was then correlated with the x and y target positions on a per trial bases during each of the BMI conditions. Correlation coefficients for each of the BMI conditions were Fisher transformed and the means of the resulting distributions were compared against each other using a one-way ANOVA statistical test.
Both monkeys achieved successful control of the cursor in all three BMI conditions as assessed by the percentage of trials in which two consecutive targets were hit within 2500ms. Monkey B achieved an average success rate of 75.0±6.2%, 75.4±5.9% and 74.4±6.4% (mean + 1 SE; across all recording sessions) for the VBMI, V+PBMI and V+NBMI conditions, respectively. Monkey MK achieved 81.7±4.5%, 82.4±4.1% and 77±6.5% success rates for the VBMI, V+PBMI and V+NBMI conditions, respectively. The success rates observed during the BMI conditions were less than those observed during active movement (97.2±0.5% and 92±2.5% for monkeys B and MK, respectively), but were substantially greater than the chance rate computed offline (See Chance Decoder Performance). Due to the daily retraining of the neural decoder, success rates varied from day to day. We found no evidence of day-to-day gains in success rate that have been reported by others using decoders that were fixed across experimental sessions (Ganguly and Carmena, 2009). We were unable to determine the cause of missed trials because the task did not require a specific behavior to initiate a trial. Missed trials were likely due to some combination of poor performance and sporadic periods of inattention. Success rates were computed based on all trials and thus do not reflect the trial selection criteria imposed in the subsequent analyses.
Because our monkeys were trained to relax their arm and no restraints were used during any of our experiments, it was important to select only those trials for our analyses during which the monkeys were receiving intended sensory feedback and to remove any trials during which the monkeys' voluntary movements interfered with the intended sensory information (see Trial Selection). Examination of the cursor trajectories in each condition revealed that on average the movements generated by the BMI incorporating both veridical visual and proprioceptive feedback were straighter and less variable compared to the other BMI conditions. We used three behavioral metrics to quantify the effects of multisensory feedback on the quality of BMI performance: (1) the normalized time to target (i.e. the time to reach a target divided by the distance to the target), (2) normalized path length (i.e. the actual path length divided by the target distance) and (3) the normalized number of path reversals (i.e. the number of times the transiently moved away from the target divided by the target distance). One-way ANOVAs with post-hoc t-tests were used to evaluate the condition dependent differences in these metrics for each animal separately.
Consistent with our hypothesis, both monkeys moved the visual cursor faster and straighter when using a BMI that provided veridical visual and proprioceptive feedback compared to a BMI with visual feedback alone (Figure 2a; Table 1). The mean normalized time to target was significantly shorter during the V+PBMI condition compared to both the VBMI and the V+NBMI conditions (p < 0.05, for each animal). Monkeys also generated straighter paths under the V+PBMI condition (p < 0.05, for each animal). Interestingly, BMI performance was significantly poorer according to the path length metric during the V+NBMI condition compared to the VBMI condition indicating a detrimental effect of incongruent feedback (p < 0.05, for each animal). Finally, we found that the BMI-generated paths having the least number of trajectory reversals occurred during the V+PBMI condition compared to the VBMI and V+NBMI conditions. In fact, the average percent difference between V+PBMI and VBMI for these three metrics show approximately a 40% improvement when the monkeys moved a visual cursor using a BMI incorporating congruent visual and proprioceptive feedback compared to visual feedback alone (Table 1). According to these metrics, cursor movements generated in the V+PBMI condition reasonably approximated natural reaching movements observed in the AM condition.
In addition to the presence of congruent multisensory feedback, the degree of congruence also contributed to the improvement in BMI movements we observed. In examining the relationship between task performance and direction error in the V+PBMI condition, we found that as the discrepancy between cursor and hand movements decreased both time-to-target and path length decreased (i.e. improved performance; Figure 2b, solid red curves). As the elimination criteria became less stringent, performance decreased (Figure 2b, dashed red, dashed black and solid black curves). As direction error became large performance degraded dramatically. The worst performance was observed in the V+NBMI condition, where the average direction error between the visual cursor and hand movement was greatest (Figure 2b, gray circle). Finally, in both V+PBMI and V+NBMI, we observed a significant decrease in performance when all of the data was included in the computations (Table 1; denoted as V+PALL,BMI and V+NALL,BMI).
Task performance in the VBMI condition was likely influenced by the 100ms delay we imposed to compensate for the dynamics of the exoskeleton. To investigate the magnitude of this effect, we compared the performance of the BMI incorporating congruent visual and proprioceptive feedback to the performance of a vision only BMI having no delay. We collected an additional 10 datasets (5 from monkey MK and 5 from monkey B) where we imposed no delay on the movement of the visual cursor in the VBMI condition (Table 1; denoted as V0,BMI). For each monkey separately, a two-sample t-test was used to evaluate the condition dependent differences for each performance metric. As expected, we found that performance improved when the 100ms visual delay was removed (2 sample t-test; p < 0.0005 for each metric). However, we found that the BMI incorporating congruent multisensory feedback (even with the imposed 100 ms delay) outperformed the vision only BMI with no delay in the time-to-target (p = 0.019 and p = 0.004, for MK and B) and path length (p = 0.007 and p < 0.0005, for MK and B) metrics by 10% and 15%, respectively (Table 1). There was no difference in the number of trajectory reversals between these two conditions.
We collected surface EMGs from the Biceps and Triceps in monkey MK to verify that the cursor movements observed during the BMI conditions were not influenced by the monkey's active arm movements. Figure 3a compares the arm kinematics and EMG activity during a representative experiment. Changes in the magnitude of Biceps and Triceps muscle activity are readily apparent in conditions where the arm is moved, either by the monkey (AM, blue bar) or the exoskeleton (V+PBMI and V+NBMI, red and gray bars, respectively). To quantify condition-related changes in muscle activity, we computed the trial-averaged EMG magnitude for each muscle and compared them with a one-way ANOVA. The test found a significant effect of experimental condition for both muscles, with EMG magnitude being significantly increased in the V+PBMI and V+NBMI conditions compared to AM and VBMI (Figure 3b, p < 0.05). In order to verify that the filtering algorithm was removing V+PBMI trials when muscle activity may have contributed to improper kinesthetic feedback, we compared EMG magnitude in the population of included and excluded trials for both muscles. We found that muscle activity was significantly reduced in the population of V+PBMI trials that survived the filtering process (two sample t-test; p = 0.04 and p = 0.03 for Biceps and Triceps, respectively).
We used two measures to assess whether this increase in muscle activity during the V+PBMI and V+NBMI conditions was related to the monkey's attempt to move the visual cursor by actively moving his arm. First, we computed a measure of the antagonist muscle co-contraction for the Biceps and Triceps. A one-way ANOVA found a significant increase in the magnitude of co-contraction in the V+PBMI and V+NBMI conditions compared to AM and VBMI (Figure 3c, p < 0.05). Next, we used a cross-correlation analysis to examine the relationship between muscle activity (the Biceps and the Triceps) and elbow angular velocity during the AM and BMI conditions. A one-way ANOVA and post-hoc t-test found no differences in the peak correlation magnitude between EMG activity and elbow joint velocity across the three BMI conditions (p > 0.05). Furthermore, the correlation magnitudes in the BMI conditions (at the time lag of peak correlation in AM) were not different from zero suggesting that the monkey was not actively attempting to move its limb during BMI control (Figure 3d). Peak correlation magnitude between EMG activity and elbow joint velocity significantly increased during the AM condition compared to the BMI conditions (p < 0.05) demonstrating the expected relationship between muscle activity and joint motion during reaching movements (Figure 3d).
In addition to collection of muscle activity, we recorded the eye movements in monkey MK to confirm that our results are not biased by the position of the monkeys' gaze. A one-way ANOVA found that there was no difference in cross-correlation magnitude between eye position and target location across the three BMI conditions (p = 0.27 and p = 0.61 for X and Y positions, respectively) indicating no effect of gaze position on our behavioral result.
We next investigated how the condition-dependent modulation in MI neurons contributed to the performance gains observed when congruent visual and veridical proprioceptive feedback were available to the monkeys during BMI control. We first examined neuronal spike rate modulation during the AM and the three BMI conditions. When viewed over the timescale of the entire experiment, the responses of individual neurons varied substantially across the experimental conditions (Figure 4a, colored bar). Some neurons seemed to prefer active movement, while others preferred individual decoding conditions or some combination of movement and the decoding conditions. This diverse, structured neural activity is illustrated Figure 4a which shows the normalized binned firing rate (50ms bins) as a function of time for each of the 61 neurons recorded during a single session. Changes in the experimental condition precisely correlate with substantial changes in the firing rate of individual neurons appearing as vertical striations in Figure 4a.
The modulation in the time series' of firing rates raised the possibility that the observed changes in BMI performance may be caused by simple changes in the firing properties of neurons and not the introduction of congruent multisensory feedback. We performed two separate analyses to determine if changes in the rate of spiking could explain the observed differences in BMI performance. First, we computed the average firing rate during included trials for all 609 neuron samples in each BMI condition. Figure 4b contains a set of scatter plots in which each black dot represents the average firing rate of a neuron in the indicated pair of conditions. A regression line was fit to the rate data for all 609 neuron samples (Figure 4b, red lines) and a one-way analysis of covariance (ANCOVA) with posthoc t-test was used to test for differences in the gain (slope) and/or baseline firing rate (intercept) between conditions. We observed a significant firing rate gain (i.e. significantly greater than 1) when comparing the rates measured in the AM condition to every other condition (p < 0.0005; gain = 1.08, 1.11 and 1.17 for VBMI, V+PBMI and V+NBMI, respectively). Furthermore, there were significant firing rate gains in the V+PBMI and V+NBMI conditions compared to VBMI (p < 0.0005, gain = 1.11 and 1.08, respectively). ANCOVA found no significant gain effect between firing rates in the V+PBMI and V+NBMI conditions (p = 0.63, gain = 1.01). Gain effects were not accompanied by similar changes in baseline firing rate.
Next, we performed principal components analysis on the ensemble of smoothed firing rate time series estimated for each data set. We sought to show that the improvement in BMI performance was not due to an increase in the “flexibility” of ensemble neural activity (i.e. the possibility that MI enters a higher dimensional space during some conditions thereby utilizing more degrees of freedom). We computed the number of orthogonal bases required to account for 90% of the variance in the ensemble activity for each data set. ANOVA found a significant reduction in the number of bases required to explain the data variance in the V+PBMI condition compared to the VBMI condition (p < 0.05, 25 and 31 bases, respectively). This finding demonstrates that the increase in BMI performance is accompanied by increased efficiency in the ensemble spiking activity. The AM and V+NBMI conditions required 30 and 27 bases to account for 90% of the data variance.
We computed the mutual information between the instantaneous firing rate of each cell and the instantaneous cursor direction for a range of time lags (Figure 5). We considered only those neurons whose mutual information peaked at a lag within the range of -600 to 600 ms (499 out of 609 neuron samples satisfied this criterion). The magnitude of mutual information varied from neuron to neuron as seen in the three examples shown in Figure 5. Some neurons expressed graded amounts information about movement direction across the experimental conditions (Figure 5a), while others contained information during a subset of the conditions. We found that the activity of many neurons carried information only during the real-time decoding conditions (Figure 5b). Mutual information often peaked at positive lags, indicating that cell activity carried information about the future direction of cursor movement (Figure 5a and b). This is consistent with the idea that these cells causally “drive” the cursor movement (Moran and Schwartz, 1999; Paninski et al., 2004; Suminski et al., 2009). However, during V+PBMI, there were some neurons in which the peak mutual information occurred at near zero or even at negative time lags, indicating that the neural response occurred after the cursor movement, suggestive of a sensory-type response (Figure 5c).
We were specifically interested in understanding how the strength and temporal relationship of mutual information was modulated by sensory feedback modality in cells whose activity contained significant information about direction in each of the experimental conditions. We included cells in the following analyses if they exhibited significant peak mutual information with the direction of cursor movement in at least one condition. Based on this criterion, 410 (82.2%), 218 (43.7%), 147 (29.5%) and 126 (25.3%) of the 499 neurons were analyzed in the AM, VBMI, V+PBMI, and V+NBMI conditions, respectively.
Summarizing the strength of the mutual information profiles pooled across both monkeys (Figure 6a), a one-way ANOVA with post-hoc t-tests found that neural activity carried the greatest amount of information about the direction of cursor movement during the condition that provided the monkeys with congruent visual and proprioceptive feedback about the decoded cursor movement (V+PBMI condition; 0.09±0.005 bits). The addition of congruent proprioceptive feedback yielded a 125% increase in mutual information in neural activity compared to the VBMI condition (0.04±0.002 bits; p < 0.05). This increase was not due simply due to movement of the arm, as the information magnitude was greater than that observed in the V+NBMI (0.057±0.003 bits, p < 0.05). We found no difference in the amount of information in the neural activity about the direction of cursor movement during the VBMI and V+NBMI conditions. The neural activity recorded in the AM condition carried 0.06±0.003 bits of information about movement direction. It should be noted that when all 499 neurons (regardless of their significance) or when only those neurons that exhibited a significant information peak in all conditions (97 neurons) were used, the relationship between the magnitudes of mutual information in all conditions remained unchanged to those described above.
Our results also indicated a shift in the lag at which the mutual information peaked with respect to cursor direction during the BMI conditions compared to the AM condition. The mean lag of peak mutual information during the AM condition was 67±6 ms (Figure 6b), but shifted to a longer lag time of 189±8 ms and 209±13 ms during VBMI and V+NBMI conditions, respectively (Figure 6c and e). This longer delay time during the BMI conditions (approximately 130ms) can be explained by the combination of the computational delay imposed by the neural decoder and the visual feedback delay we imposed to compensate for the dynamics of the exoskeleton. Unlike the other conditions, we found a bimodal distribution of peak mutual information lags during the V+PBMI condition (Figure 6d). We characterized the two modes using a mixture of Gaussians model and an Expectation-Maximization (EM) clustering algorithm (Hastie et al., 2001), which found two modes at lags of -63±14 ms and 242±10 ms. The actual distribution of lags fit the model with an R value of 0.83. To understand the condition-dependent relationship between the magnitude of information carried by the neural activity and the timing of peak information, we computed the mean mutual information profiles across the two populations of neurons categorized by the sensory and motor modes of the V+PBMI distribution (Figure 7a and 7b, respectively). Despite the very different timing of the two modes, we found that the magnitude of peak information in each mode was significantly enhanced in the conditions where visual and proprioceptive sensory feedback were congruent (AM and V+PBMI) relative to the VBMI and V+NBMI conditions.
Our results are the first to demonstrate the utility of feedback modalities other than vision in a cortically-controlled brain-machine interface. Congruence in sensory feedback was an important factor driving performance as we observed that both time-to-target and path length decreased (i.e. increased performance) as the error between the visual and proprioceptive estimates of hand position decreased. Improvements in behavioral performance were accompanied by an increase in the magnitude of direction-related mutual information in the spiking activity of MI at time lags indicative of both sensory and motor activity.
A 100 ms visual feedback delay was imposed in all BMI conditions in order to synchronize the visual and proprioceptive feedback modalities in the V+PBMI which, by definition, increased the time-to-target values in all conditions. However, even when the delay was removed in the vision only condition (V0,BMI), the performance in the V+PBMI condition (with the delay) was still improved by 10-15% relative to that in the V0,BMI condition. This smaller yet significant performance improvement (comparison of V+PBMI and V0,BMI in Table 1) is likely conservative and could be increased by providing feedback at physiological latencies through improvements in exoskeleton control or via a surrogate methodology (e.g. direct electrical stimulation of the nervous system).
We observed an increase in EMG activity in the conditions where the arm was moved by the exoskeleton. This muscle activity was not the driver of improved performance as our correlation analysis demonstrated it was unrelated to the monkey's volitional attempt to move the visual cursor with its arm (thereby effecting movement of the BMI cursor). Rather, this muscle activity caused an increase in co-contraction at the elbow. In some V+PBMI trials, the force generated by this undesired muscle activity was sufficient to overcome the force generated by the exoskeleton causing feedback to be incongruent. Therefore, we removed those trials where the sensory feedback was inappropriate or improperly timed (in the V+PBMI and V+NBMI conditions). This filtering procedure removed a significant number of trials in both conditions, but ensured that the sensory feedback was appropriate in each condition.
We chose to use a decoder based on a linear filter because of its computational simplicity and its strong presence in the literature (Serruya et al., 2002; Taylor et al., 2002; Carmena et al., 2003; Musallam et al., 2004; Hochberg et al., 2006; Kim et al., 2008; Mulliken et al., 2008; Ganguly and Carmena, 2009). Other decoding algorithms were not considered because the focus of our paper was on the effect of feedback on BMI control. However, we believe that any gains achieved through the addition of multisensory feedback would transfer, at least partially, to systems using other decoding algorithms.
Our results compare favorably with state of the art BMIs that rely on vision for feedback during closed-loop control. Ganguly and Carmena trained monkeys to use a linear filter based BMI that decoded the Cartesian position of a visual cursor using MI spiking activity (Ganguly and Carmena, 2009). After performance reached a steady state, the movement time for their monkeys averaged 3.76s for a 7cm cursor movement (0.054 s/mm in our normalized time-to-target metric). In contrast, the normalized time-to-target for V+PBMI averaged 0.021s/mm. This large performance difference may be due in part to the numbers of neurons included in the neural decoder. Because Ganguly and Carmena were interested in decoding from a stable population of neurons, they restricted their decoder to at most 15 neurons while our BMI averaged approximately 44 neural channels for monkeys MK and B. Performance in V+PBMI also surpassed that reported in a clinical experiment involving two human patients with tetraplegia (Kim et al., 2008). After a 500ms dwell time was removed from the movement times reported in the manuscript, the time-to-target averaged 2.58s (0.027 s/mm) and 2.8s (0.029 s/mm) for the 9.2cm cursor movements generated by either a position based linear filter decoder or velocity-based Kalman filter decoder, respectively.
The performance increase with congruent sensory feedback was accompanied by three significant changes in the spiking activity of MI neurons during the BMI conditions. First, we found a significant gain modulation in the firing rate of MI neurons during the V+PBMI condition compared to VBMI. This result was not surprising given the well documented effect of passive movement on the activity of neurons in MI (Fetz et al., 1980; Suminski et al., 2009). However, this firing rate gain between VBMI and V+PBMI does not appear to be the source of the performance increase because the firing rate gain between V+PBMI and V+NBMI were not statistically different from one another despite a significant difference in BMI performance between these conditions. We cannot determine whether the decreased performance observed in V+NBMI resulted from a sensory integration strategy in which visual information was weighted more heavily than proprioception or whether the disparity in visual and proprioceptive feedback was simply distracting. We believe the former to be the case given the similarity in performance in VBMI and V+NBMI, as well as the higher than expected correspondence between the cursor and hand direction in the V+NBMI condition.
Second, principal components analysis on the ensemble neural activity from each dataset found a reduction in the average dimensionality of the space spanned by the spiking activity in V+PBMI compared to VBMI. This is inconsistent with the hypothesis that performance improvements may result from greater “flexibility” of the neural activity (i.e. the ability to utilize more degrees of freedom) when the restrictions on arm movement were relaxed during BMI control (V+PBMI). Instead, the increase in BMI performance in V+PBMI was accompanied by greater efficiency of the neural ensemble possibly resulting from a better estimate of the state of the system due to the integration of visual and proprioceptive feedback (Wolpert and Ghahramani, 2000). In support of this view, we found a strong relationship between BMI performance and the magnitude of the error between cursor and hand movements. As the degree of congruence between the visual and proprioceptive feedback increased, the monkeys' ability to command the BMI cursor was significantly improved. Similarly, the poorest BMI performance was found in VBMI and V+NBMI, when the discrepancy between cursor and hand movements was greatest.
Third, our mutual information analysis provides the strongest evidence supporting the hypothesis that the performance gains observed in V+PBMI were the result of congruent visual and proprioceptive feedback. The bimodal distribution of peak mutual information lag times for the V+PBMI condition suggests two distinct neural populations. The population with positive lag times exhibits a substantial increase in information during V+PBMI relative to VBMI. This enhancement is particularly evident at the positive - “motor” - lag times. These are the only relevant lag times for the neural decoder because they imply a causal relationship between neural modulation and cursor movement. Therefore, we believe that this population is primarily responsible for the enhanced BMI performance observed in the congruent condition.
On the other hand, the population with negative lag times exhibits a short-latency “sensory” peak during V+PBMI which is not evident in the visual or incongruent feedback conditions. This population appears to be sensitive to limb state information via proprioception that is known to reach the cortex in as little as 10 ms (Conrad et al., 1975; Lemon et al., 1976; Evarts and Fromm, 1977; Fetz et al., 1980; Suminski et al., 2009). Although this population may not be directly responsible for the enhanced BMI performance in V+PBMI, it may be mediating the increase in motor information in the first population perhaps via horizontal connectivity between the two populations (Huntley and Jones, 1991).
This “sensory” information may contribute to improved BMI performance in two ways. First, the integration of visual and kinesthetic feedback may provide the monkey a more accurate estimate of the current state of the system. Second, the kinesthetic feedback generated by moving the arm was likely smoothed with respect to the visual feedback due to the dynamics of the arm/exoskeleton. This feedback smoothing may partially contribute to the improved performance that we report due to its effect on the activity of MI.
Our findings may have important practical implications for severely motor-disabled patients with residual proprioception. Incomplete spinal cord injury, such as anterior cord and central cord syndromes, results in severe loss of motor function but can leave a certain degree of proprioception intact. Moreover, other conditions such as ALS and locked-in syndrome have devastating motor consequences but may not affect proprioception. With the recent advent of light-weight wearable robotic exoskeletons (Kiguchi et al., 2005; Rocon et al., 2007), these patient populations may be served with a cortically-controlled BMI that drives these devices, and in turn, moves the affected but proprioception-intact limbs, thereby augmenting these systems with proprioception.
More importantly, these findings provide a key foundation for further research involving the integration of different forms of sensory feedback in BMI systems including bi-directional BMIs, in which surrogate feedback is provided via electrical stimulation. Direct stimulation of the primate somatosensory cortex via microelectrodes has been shown to elicit discernable sensory percepts for reach target cuing (Fitzsimmons et al., 2007; London et al., 2008). This paradigm offers the benefit of a bi-directional BMI to a larger group of patients who have lost both motor and sensory function due to complete spinal cord injuries or limb amputation.
The authors wish to thank Josh Coles for his assistance with the experiments. We would also like to thank Lee Miller for his assistance with the design of these experiments and his comments on an earlier version of the manuscript. The work was supported by funding from NIH NINDS R01 N545853-01 to NGH and AHF and a postdoctoral fellowship form the Paralyzed Veterans of America Research Foundation to AJS.