|Home | About | Journals | Submit | Contact Us | Français|
Brain-machine interfaces (BMIs)1,2 use neuronal activity recorded from the brain to establish direct communication with external actuators, such as prosthetic arms. While BMIs aim to restore the normal sensorimotor functions of the limbs, so far they have lacked tactile sensation. Here we demonstrate the operation of a brain-machine-brain interface (BMBI) that both controls the exploratory reaching movements of an actuator and enables the signalling of artificial tactile feedback through intracortical microstimulation (ICMS) of the primary somatosensory cortex (S1). Monkeys performed an active-exploration task in which an actuator (a computer cursor or a virtual-reality hand) was moved using a BMBI that derived motor commands from neuronal ensemble activity recorded in primary motor cortex (M1). ICMS feedback occurred whenever the actuator touched virtual objects. Temporal patterns of ICMS encoded the artificial tactile properties of each object. Neuronal recordings and ICMS epochs were temporally multiplexed to avoid interference. Two monkeys operated this BMBI to search and discriminate one out of three visually undistinguishable objects, using the virtual hand to identify the unique artificial texture (AT) associated with each. These results suggest that clinical motor neuroprostheses might benefit from the addition of ICMS feedback to generate artificial somatic perceptions associated with mechanical, robotic, or even virtual prostheses.
BMIs have evolved from one degree-of-freedom systems 3 to many-degree-of-freedom robotic arms 4 and muscle stimulators 5 that enact complex limb movements, such as reaching 6–8 and grasping 9. However, somatosensory feedback, which is essential for dexterous control 10–12, remains underdeveloped in BMIs. With the exception of a few studies combining BMIs with tactile stimuli applied to the body 13, existing systems rely almost exclusively on visual feedback. Prosthetic sensation has been studied in the context of sensory substitution 14 and targeted reinnervation 15, however these approaches have limited application range and channel capacity. To provide a proof-of-concept method for sensorizing neuroprostheses, we implemented a BMBI that extracts movement commands from the motor areas of the brain while delivering ICMS feedback in somatosensory areas 1,2,16 to evoke discriminable percepts 17–20. This idea received support from our pilot study 16 in which a monkey responded to ICMS cues with the movements of a BMI-controlled cursor. However the ICMS cue did not provide feedback of object-actuator interactions in this previous demonstration.
The BMBI developed here enabled active tactile exploration 21 during BMI control (Fig. 1a). Two monkeys (M and N) received multielectrode implants in M1 and S1 (Fig 1b). They explored virtual objects with either a computer cursor or a virtual image of an arm (Supplementary Fig. 1a,b). In hand control (HC), the monkeys moved a joystick with their left hands to position the actuator. They searched through a set of virtual objects, selected one with a particular artificial texture (AT) conveyed by ICMS, and held the actuator over that object to obtain reward (Fig. 1a; Supplementary Fig. 1c,d). During brain control (BC), the joystick was disconnected and the actuator was controlled by the activity of right-hemisphere M1 neurons 9,22,23. The behavioural tasks varied in the number of objects on the screen, ATs employed, and the actuator type (Fig. 2a) and were more difficult than previously reported BMI tasks because of the presence of multiple objects in the workspace, a prolonged object selection period, and the necessity of interpreting ICMS feedback.
ICMS was delivered through two pairs of microwires to the hand representation area of S1 in monkey M, (Fig. 1c) and through one pair of microwires to the leg representation in monkey N. Each AT consisted of a high-frequency pulse train presented in packets at a lower secondary frequency (Fig. 1d; Supplementary Fig. 2a). The rewarded AT (RAT) consisted of 200 Hz pulse trains delivered in 10 Hz packets. The comparison ATs were represented by 400 Hz pulse trains delivered in 5 Hz packets (unrewarded artificial texture, UAT) or by an absence of ICMS (null artificial texture, NAT).
The major challenge solved here was the real-time coupling of ICMS feedback with the BMI decoder. As ICMS artefacts masked neuronal activity for 5–10 ms after each pulse (Fig. 1d, e), we multiplexed neuronal recordings and ICMS with a 20 Hz clock rate (Supplementary Fig. 2a). The interleaved intervals proved adequate for online motor control and artificial sensation—a result that was not clear a priori since S1 stimulation could have affected M1 processing through the connections between these areas.
BMBI performance improved with training. In Task I (Fig. 2a–I), monkey M surpassed chance performance after nine sessions; monkey N after four (P < 0.001, one-sided binomial test). Improvement continued with more difficult tasks (Tasks II–V) (Fig 2a,b; Supplementary Fig. 3a). In particular, the time spent exploring unrewarded ATs decreased (Fig. 2c; Supplementary Fig. 3b). Additionally, performance improved within daily experimental sessions (Fig. 2d). Psychometric analysis of RAT stimulation amplitudes indicated that at least 8 nC per ICMS waveform phase (100 μs wide current pulses of 80 μA) was needed for the AT discrimination (P < 0.001, one-sided binomial test). Performance was at chance level for catch trials (in Task II), where ICMS was not delivered (P = 0.90, one-sided binomial test).
The statistics of object exploration intervals (total time spent over a particular object on a given trial) indicated that the monkeys uniquely discriminated each AT type (Fig. 2c; 3a,c) and interpreted ICMS within hundreds of milliseconds—a timescale comparable to the discrimination of peripheral tactile stimuli 24,25. Early in Task I, exploration intervals were equal for RAT and NAT (P > 0.5, Wilcoxon signed-rank test) and, with training, became longer for RAT and shorter for NAT (Tasks I–II) and UAT (Tasks III–V). During HC, the mean interval was longest for RAT (monkey M: 1,396 ± 21 ms; monkey N: 1,165 ± 15 ms; mean ± s.e.m.), shortest for NAT (304 ± 8 ms; 300 ± 10 ms), and intermediate for UAT (452 ± 13 ms; 402 ± 14 ms) (P < 0.01, ANOVA). During BC, intervals spent exploring NAT (498 ± 15 ms; 587 ± 25 ms) and UAT (685 ± 20 ms; 764 ± 32 ms) were longer as compared to HC, but still shorter as compared to RAT (1,420 ± 28 ms; 1,398 ± 55 ms) (P<0.01, ANOVA).
Additional hallmarks of active exploration were seen in the conditional probabilities of selecting different ATs (Figure 3b,d). During HC trials, the monkeys stayed over the first encountered AT (arrows that loop back to the same AT in Fig. 3b,d) with high probability if it was RAT (P=0.70 for monkey M and 0.76 for monkey N), but with low probability if it was UAT (0.05 and 0.01) or NAT (0.0 and 0.0) (Fig. 3b,d, left). After examining the second AT, the monkeys could identify the correct AT either by apprehending it directly or through a process of elimination. This follows from the increase in the probability of moving to RAT from NAT or UAT from chance to approximately 0.7 and the decrease in the probability of revisiting UAT or NAT to approximately 0.2 (Fig. 3b,d right). Similar effects were observed for BC (Fig. 3e, red text).
BC started in Task IV. During BC with hand movements (BCWH), the monkeys continued to hold the joystick although it was disconnected 16,22. During brain control without hand (BCWOH) 9,22, the joystick was removed. In monkey M, with more than 200-recorded neurons, performance was less accurate during BCWH (73.75 ± 3.00%, mean ± s.e.m.) than during HC (91.48 ± 1.20%). In Monkey N with 50-recorded neurons, performance dropped further (50.37 ± 3.74 % versus 91.45 ± 1.91%), but still significantly exceeded the 33% chance level. M1 neurons exhibited directionally tuned modulations (Supplementary Fig. 5; 6) that were retained across different interfering ICMS patterns both during HC (Supplementary Fig. 4a,b) and BC (Fig. 4a,b).
In BCWOH, task requirements were eased: the object selection period was reduced to 300–500 ms and monkeys were allowed to overstay at an incorrect object. Performance for monkey M, measured as the number of rewards per minute, steadily improved from 1.021 ± 0.007 to 2.962 ± 0.005 (mean ± s.e.m.) (Fig. 2d). Similar improvements were observed for HC and BCWH (inset in Fig. 2d). The average frequency of actuator displacements, calculated from power spectra, was correlated with the improvement in performance during BCWOH (R2=0.16 for the X-coordinate and R2=0.26 for the Y-coordinate, P<0.001, F-test), which indicated that the monkey modulated its brain activity to scan the targets faster. This behaviour was not random, as the exploration interval for NAT (3,620 ± 350 ms, mean ± s.e.m.) was significantly shorter (P<0.02, Wilcoxon rank sum test) than for UAT (4,270 ± 310 ms). The exploration of RAT (2,255 ± 94 ms) was the shortest due to the reduced selection period. For monkey N, BCWOH performance (2.084 ± 0.085 rewards per minute) did not change within sessions, and the differences in exploration intervals were not significant.
In agreement with others 26–30, we observed that M1 neurons represented the movements of the actuator even when it was passively observed by the monkey (Supplementary Fig. 7). Actuator movements (task V) replayed for the monkeys could be reconstructed from M1 activity, using a separately trained decoder (Fig. 4d), with similar accuracy to reconstructions made for HC (Fig. 4c). M1 representation of the passively viewed actuator is consistent with our suggestion that a neuroprosthetic limb might become incorporated in brain circuitry 1.
Our BMBI demonstrated direct bidirectional communication between a primate brain and an external actuator. As both the afferent and efferent channels bypassed the subject’s body, we propose that BMBIs can effectively liberate a brain from the physical constraints of the body. Accordingly, future BMBIs may not be limited to limb prostheses, but may include devices designed for reciprocal communication between and among neural structures and with a variety of external actuators.
All animal procedures were performed in accordance with the National Research Council’s Guide for the Care and Use of Laboratory Animals and were approved by the Duke University Institutional Animal Care and Use Committee. Two rhesus monkeys were implanted with micro-wire arrays in both hemispheres. These implants were used for both recordings and ICMS (symmetric, biphasic, charge-balanced pulse trains; 100–200 μs, 120–200 μA). Monkeys manipulated a joystick to produce reaches with an actuator (computer cursor or a virtual reality arm) towards up to three objects displayed on a computer monitor. The task required searching for the object with particular artificial tactile properties. Objects consisted of a central response zone and a peripheral feedback zone. Artificial tactile feedback was delivered when the actuator entered the feedback zone and continued in the response zone. Monkeys held the actuator over the correct object for 0.8–1.3 s to receive a fruit juice reward. Holding over an incorrect object cancelled the trial. In brain control trials, the actuator was controlled by cortical ensemble activity decoded using an Unscented Kalman Filter 23. An interleaved scheme of alternating recording and stimulation subintervals (50 ms each, 50% duty cycle) was implemented to achieve concurrent afferent and efferent operations. In all offline analyses, ICMS periods were excluded from calculations of neuronal firing rates. The virtual reality arm was animated using Motion Builder (Autodesk, Inc., San Rafael, CA).
Two adult rhesus macaque monkeys (Macaca mulatta) participated in this study. Each monkey was implanted with four 96-micro-wire arrays constructed of stainless steel 304. Each hemisphere received two arrays: one in the upper and one in the lower limb representation areas. These array sampled neurons in both primary motor (M1) and primary sensory (S1) cortex. We used recordings from the right hemisphere arm arrays in each monkey, since both manipulated the joystick with their left hands. Within each array, micro-wires were grouped in two 4-by-4 uniformly-spaced grids of 16-triplets of electrodes. The separation between triplets of electrodes was 1 mm. The electrodes of each triplet had three different lengths, staggered at 300 μm intervals. The penetration depth of each triplet was adjusted with a miniature screw. After adjustments during the month following the implantation surgery, the depth of the triplets was fixed. The longest electrodes in all triplets protruded to 2 mm in length measured from the cortical surface.
The monkeys were trained to manipulate a computer cursor or a virtual reality arm and to make reaches towards objects displayed on a computer monitor. The objects were visually identical, but exhibited different tactile properties as conveyed by ICMS of S1. In manual control, each trial commenced when the monkey held the joystick with their working hand. Then, a target appeared in the centre of the screen. The monkey had to hold the actuator (cursor or virtual reality monkey arm) within that centre target for a random hold time uniformly drawn from the interval 0 to 2 s. After this, the central target disappeared and was replaced by a set of virtual objects radially arranged about the centre of the screen. Each of these consisted of a central response zone and a peripheral feedback zone, distinguished by their shading (Supplementary Fig. 1c). Tactile feedback was delivered in the feedback zone or corresponding response zone. For monkey M, the radius of the response zone varied from 1.5 to 4.0 cm and the radius of the feedback zone varied from 4.5 to 7.25 cm, across all tasks and sessions. For monkey N, the radius of the response zone varied from 1.5 to 4.5 cm and the radius of the feedback zone varied from 4.75 to 9.5 cm, across all tasks and sessions. A trial was concluded when the monkey placed the actuator within the response zone for a hold interval (800 to 1300 ms for HC, depending on the session; 300 to 500 ms for BC) or the monkey released the joystick handle (in manual control trials). The next trial could commence after an inter-trial interval (500 ms). The sequence of events was the same during brain control trials. In some brain control sessions, the joystick was removed from the behavioural setup. For these, each new trial commenced following the previous inter-trial interval without the requirement for the monkey to hold the joystick. For Tasks I through III, monkeys chose from a set of two objects. For Task I, the monkeys had to choose between RAT and NAT for fixed object locations. For Task II, RAT and NAT were presented on the screen at different angular locations on each trial. For Task III, object number and spatial arrangement were the same as in Task II, but RAT and UAT were used. For Task IV, three objects were used (RAT, UAT and NAT), whose arrangement on the screen varied from trial to trial. Finally, for Task V, the virtual reality monkey arm replaced the computer cursor.
Psychometric measurements determined the minimum ICMS amplitude that the monkeys could discriminate. In these measurements, the ICMS amplitude was different on every trial. In each psychometric session a range of amplitudes was selected so that about half were in a range clearly above the monkeys’ threshold for discrimination and half were in a range of unknown discriminability.
In some sessions, a small percentage of trials (typically 1%) were designated for catch trials. For these trials, the microstimulator delivered pulse trains with zero amplitude, however all other aspects of the behavioural task remained the same. This allowed us to confirm that there were no unintentional sources of information that the monkeys could use to perform the tasks.
An Nth-order Unscented Kalman filter (UKF) 23 used for BC predictions. Up to a 10th-order UKF was used in some sessions, but in most sessions we found that the 3rd-order UKF was sufficient. The filter parameters were fit based on the hand movements of the monkeys while they performed the task using a joystick or based on passive observation of actuator movements while the monkeys’ arms were restrained.
Symmetric, biphasic, charge-balanced pulse trains were delivered in a bipolar fashion across pairs of microwires. The channels selected had clear sensory receptive fields in the upper limb (monkey M, two pairs of microwires with synchronous pulse trains) or lower limb (monkey N, one pair of microwires). For monkey M, the anodic and cathodic phases of stimulation had a pulse width of 105 μs; for monkey N the pulse width was 200 μs. The anodic and cathodic phases of the stimulation waveforms were separated by 25 μs.
We implemented an interleaved scheme of alternating recording and stimulation intervals (Supplementary Fig. 2a). Our BMI had a 10 Hz update rate. That is, 100 ms of past neural data are used to make predictions about the desired state of the actuator. We broke up each 100 ms interval into two 50 ms sub-intervals. In the first sub-interval (Rec), neural activity was recorded as usual and the measured spike count was used to estimate the firing rate for the whole 100 ms interval. The second sub-interval (Stim) was reserved exclusively for delivering ICMS; all spiking activity occurring in this subinterval was discarded. Whenever the actuator was in contact with a virtual object at the start of a Stim interval, an ICMS pulse train was delivered. For RAT, nine pulses of ICMS were delivered; For UAT, 18 pulses of ICMS were delivered; for NAT, no pulses of ICMS were delivered. The neural activity in the Stim interval was discarded even in the case of NAT, so that there would be no bias induced by ICMS-occluded neural data.
In Task V, we introduced a novel brain-controlled virtual reality arm with realistic kinematic movements and spatial interactions. The control loop rate was 50 Hz, with visual refreshing at 30 Hz. The arm model was designed to depict a rhesus conspecific. We presented a first person perspective of the virtual reality arm to the monkey, who controlled the position of the hand. Arm posture was controlled using a mixture of direct control of end effectors and inverse kinematics, constrained by the physical interdependencies of the joints.
We thank D. Dimitrov for conducting the animal neurosurgeries, G. Lehew and J. Meloy for building brain implants, J. Fruh for rendering the monkey arm, T. Phillips, L. Oliveira, and S. Halkiotis for technical support, and E. Thomson and Z. Li for comments. This research was supported by DARPA N66001-06-C-2019, TATRC W81XWH-08-2-0119, NIH NS073125, NICHD/OD RC1HD063390 and NIH Director’s Pioneer Award DP1OD006798 to MALN. The content is solely the responsibility of the authors and does not necessarily represent the official views of the Office of the NIH Director or the NIH.
Author ContributionsJEO, MAL and MALN designed experiments, analyzed data and wrote the paper, JEO, MAL, PJI and KZZ conducted experiments, SS and HB developed the virtual reality monkey arm.