PMCCPMCCPMCC

Search tips
Search criteria 

Advanced

 
Logo of nihpaAbout Author manuscriptsSubmit a manuscriptNIH Public Access; Author Manuscript; Accepted for publication in peer reviewed journal;
 
IEEE Trans Neural Syst Rehabil Eng. Author manuscript; available in PMC Mar 7, 2013.
Published in final edited form as:
PMCID: PMC3590844
NIHMSID: NIHMS372629
Virtual Active Touch Using Randomly Patterned Intracortical Microstimulation
Joseph E. O’Doherty, Mikhail A. Lebedev, Zheng Li, and Miguel A.L. Nicolelis
Joseph E. O’Doherty, Department of Neurobiology and the Center for Neuroengineering, Duke University, Durham, NC 27710 USA;
Joseph E. O’Doherty: joseph.odoherty/at/duke.edu; Mikhail A. Lebedev: lebedev/at/neuro.duke.edu; Zheng Li: zheng/at/cs.duke.edu; Miguel A.L. Nicolelis: nicoleli/at/neuro.duke.edu
Intracortical microstimulation (ICMS) has promise as a means for delivering somatosensory feedback in neuroprosthetic systems. Various tactile sensations could be encoded by temporal, spatial, or spatiotemporal patterns of ICMS. However, the applicability of temporal patterns of ICMS to artificial tactile sensation during active exploration is unknown, as is the minimum discriminable difference between temporally modulated ICMS patterns. We trained rhesus monkeys in an active exploration task in which they discriminated periodic pulse-trains of ICMS (200 Hz bursts at a 10 Hz secondary frequency) from pulse trains with the same average pulse rate, but distorted periodicity (200 Hz bursts at a variable instantaneous secondary frequency). The statistics of the aperiodic pulse trains were drawn from a gamma distribution with mean inter-burst intervals equal to those of the periodic pulse trains. The monkeys distinguished periodic pulse trains from aperiodic pulse trains with coefficients of variation 0.25 or greater. Reconstruction of movement kinematics, extracted from the activity of neuronal populations recorded in the sensorimotor cortex concurrent with the delivery of ICMS feedback, improved when the recording intervals affected by ICMS artifacts were removed from analysis. These results add to the growing evidence that temporally patterned ICMS can be used to simulate a tactile sense for neuroprosthetic devices.
Index Terms: bidirectional interface, brain-machine interface, intracortical microstimulation, neural prosthesis
Sensory neuroprostheses and sensory substitution systems for the restoration of hearing [1], [2] and vision [3]–[8] have been investigated for several decades. Interest in neuroprosthetic devices that combine both motor and sensory components has developed more recently [9]–[14]. One example of a bidirectional neuroprosthesis is a robotic limb controlled by brain activity while sensory information from prosthetic sensors is delivered to somatosensory areas of the brain [15], [16]. Other possible implementations include sensorized neuroprostheses for the restoration of bipedal walking [17] and putative systems combining both speech production [18], [19] and hearing [1], [2], [20].
In recent years we have been studying intracortical microstimulation (ICMS) delivered through microelectrode arrays chronically implanted in the primary somatosensory cortex (S1) as a means of adding a somatosensory feedback loop to a brain-machine interface (BMI) [9], [10], [21]. Taken together with previous work showing that primates [22]–[24] and rodents [25]–[28] can discriminate ICMS patterns, there is growing evidence that ICMS of S1 could equip neuroprosthetic limbs with the sense of touch.
One of the neuroprosthetic devices that we envision in the future is a BMI-operated robotic arm that is equipped with touch sensors [9], [15], [16]. In such a sensorized neuroprosthesis, the touch sensors would detect instances when the arm interacts with external objects sending signals to the brain in the form of ICMS. We have suggested that long-term operation of such a system, which we call a brain-machine-brain interface (BMBI), could result in the incorporation of the prosthesis into the brain’s representation of the body, so that the artificial limb starts to act and feel as belonging to the subject [15]. Notwithstanding initial encouraging results [9], [10], it is unclear whether ICMS would be sufficient to reproduce the rich sensory information of the world of touch [29].
In particular, it is not well understood which kinds of ICMS patterns are most useful for virtual active touch. Previously, we have shown that both New World [21] and Old World monkeys [10] can discriminate temporal ICMS patterns applied to S1 that consist of short (50–300 ms) high-frequency (100–400 Hz) pulse-trains presented at a lower secondary frequency (2–10 Hz). In these experiments, ICMS served as a cue that instructed the direction of reach. These patterns of ICMS could, in principle, mimic a wide variety of tactile inputs, especially when combined with spatial encoding [21]. Modulations of sensory inputs in this frequency range correspond to the sensation of flutter [30]–[32]. These timescales are also similar to neuronal modulations involved in texture encoding in the somatosensory system [33]–[36], which makes such ICMS patterns worthy candidates for exploration.
In this study, we examined the ability of rhesus monkeys to discriminate a range of temporal ICMS patterns applied to S1 in the context of an active exploration task in which ICMS mimicked the tactile properties of virtual objects. We manipulated the ICMS patterns in a graded fashion, modulating the degree of periodicity of the pulse-trains while maintaining a constant average pulse rate. We sought to determine the minimal perturbation of the periodic pattern that the monkeys could discriminate. The degree of randomness (as quantified by the coefficient of variation, CV) was varied from trial to trial, which allowed us to quantify the monkeys' sensitivity to ICMS frequency modulations. Concurrently with ICMS delivery, we recorded from large populations of cortical neurons using multielectrode implants. Kinematics of reach movements were extracted from this large-scale activity offline to estimate the accuracy of a BMBI with a somatosensory feedback loop that transmits aperiodic ICMS patterns.
This investigation of sensitivity to ICMS periodicity in S1 was motivated by a possible application in neuroprosthetic limbs. We expect that the patterns of ICMS triggered by the interaction of an upper-limb neuroprosthesis with objects in the environment could be highly irregular. The precise temporal structure of such patterns would depend on the interaction of touch sensors in the robotic prosthesis with the specific surface structure of the manipulated objects and on the specific exploratory movements used by the individual to interact with the objects. Therefore, by knowing the limits of the nervous system in discriminating aperiodic ICMS patterns, we can infer a principled upper bound on the maximum fidelity touch sensor that could be used in a neuroprosthesis, beyond which no additional function would be restored.
This study builds on the results obtained by Romo et al. about ICMS of S1 [22], [23], [31] and our own previous work [9], [21]. One notable difference between the temporal ICMS patterns implemented here and those used by Romo et al. is that the aperiodic patterns of ICMS that we used had the same mean pulse interpulse intervals as the periodic comparison ICMS pulse trains. Thus the average number of pulses in a pulse train was the same for both periodic and aperiodic patterns. This allowed us to probe S1 sensitivity to the temporal structure of ICMS without the confound of average stimulus intensity. Romo et al. used periodic pulse trains with different frequencies [22], which left open the possibility that some of their results could be explained by differences in average ICMS intensity.
Another major difference between this study and previous studies of ICMS-evoked S1 sensations in primates is that the ICMS patterns employed here were used in an active-exploration paradigm in which ICMS was used to simulate the tactile properties of virtual objects. Our monkeys explored the virtual objects, making self-paced exploratory movements, and decided which objects to explore, in what order, and for how long. This is a more realistic model of a clinical somatosensory neuroprosthesis than previous designs.
A. Implants
The experiments were conducted in two rhesus monkeys (M and N) chronically implanted with multielectrode arrays in several cortical areas following our implantation methods [37]. We used this same electrode array design for both large-scale neural recordings and ICMS delivery [9], [10]. Each monkey received four 96-channel microelectrode arrays placed in the arm and leg representation areas in sensorimotor cortex (Fig. 1A). Each hemisphere was implanted with two arrays: one in the arm representation and one in the leg representation. Within each array, electrodes were grouped in two 4-by-4 uniformly spaced grids of electrode triplets. The electrodes within each triplet had different lengths, staggered at 300 µm intervals. One grid was aligned over primary motor cortex (M1) and the other over S1. The monkeys were implanted for a series of studies beyond those described here. For the purpose of this study, we recorded neuronal activity from the right hemisphere arm arrays while the monkeys performed a manual task with their left hands. Stimulation was applied to the right hemisphere arm subdivision of S1 in monkey M and the right hemisphere leg subdivision of S1 in monkey N. All animal procedures were performed in accordance with the National Research Council’s Guide for the Care and Use of Laboratory Animals and were approved by the Duke University Institutional Animal Care and Use Committee.
Fig. 1
Fig. 1
Implants and task paradigm. (a) The monkeys were implanted with microwire arrays targeting M1 and S1 of the upper and lower limbs. (b) Channels used for stimulation with monkey M are accented in red. (c) Objects on the screen consisted of a central response (more ...)
B. Behavioral Task
The monkeys were trained in a reaching task in which they manipulated a hand-held joystick to move a virtual reality arm (avatar) displayed on a computer screen (Fig. 1C,D). The monkeys reached with the avatar arm towards screen objects and searched for an object with a particular artificial texture indicated by ICMS of S1. The objects were circular in shape and visually identical. Each monkey was previously trained in other variants of this task. In this study, the monkeys were shown two objects, one of which was associated with a periodic ICMS pattern and the other with an aperiodic pattern (Fig. 2A). The monkeys were rewarded for selecting the object paired with periodic ICMS. The objects appeared at different locations on the screen, with the constraint that the distance from the screen center to each object was fixed, and the angle between the objects was 180 degrees (i.e. centrally symmetric). If the correct object was selected, the monkeys were rewarded with a drop of fruit juice.
Fig. 2
Fig. 2
Example aperiodic ICMS pulse trains. (a) Raster indicates the range of variability of inter-burst intervals (CV=0.8, cyan; CV=0.5, blue; CV=0.25, green; CV=0.05, red; CV=0, black). Note that each vertical line indicates a short burst of ICMS, not a single (more ...)
Each trial commenced when the monkey grasped the joystick with its left hand. At this point, a circular target appeared in the center of the screen. The monkey placed the avatar arm on that center target for 0.5–1.0 s. Then, the central target disappeared and two peripheral objects appeared. Each consisted of a central response zone and a peripheral feedback zone (Fig. 1C). When the avatar hand entered the feedback or response zones, ICMS pulse trains were delivered to S1. A trial was concluded with a reward when the monkey placed the avatar hand within the response zone of the correct object for 2 s; no reward was delivered if the incorrect object was selected. The monkeys were permitted to explore the virtual objects in any sequence, but the trial ended when they stayed over an object’s response zone longer than the hold period of 2 s. Then a 0.5 s delay was issued before the next trial began.
C. ICMS Patterns
ICMS trains consisted of symmetric [38], biphasic, charge-balanced pulses of ICMS delivered in a bipolar fashion through adjacent pairs of microwires [9], [21]. For monkey M, the anodic and cathodic phases of stimulation each had amplitudes of 150 µA and pulse widths of 105 µs; for monkey N, 150 µA and 200 µs, respectively. The anodic and cathodic phases were separated by 25 µs.
ICMS was delivered to different subdivisions of S1 for each monkey. For monkey M, the hand representation area of S1 was used as the target for ICMS, so that the monkey experienced putative sensations in its hand (Fig. 1B). For monkey N, ICMS was applied to the thigh representation area of S1. Two electrode pairs were used for each animal.
The temporal pattern of ICMS consisted of 200 Hz pulse trains delivered for 50 ms and presented at a lower secondary frequency. The secondary frequency for the rewarded artificial texture was a constant 10 Hz. For the unrewarded textures, the timing of the ICMS bursts was aperiodic. The interval between each aperiodic burst was a random variable drawn from a gamma distribution of instantaneous inter-burst intervals [39]:
equation M1
(1)
where f is the probability density function, x is the inter-burst interval, k is the shape parameter, θ is the scale parameter, and Γ is the Gamma function. We computed the shape and scale parameters as a function of the mean inter-pulse interval, μ, and the coefficient of variation, CV, the ratio of the standard deviation to the mean:
equation M2
equation M3
(2)
This allowed the construction of aperiodic pulse trains with inter-burst intervals equal in expectation to the periodic pulse trains while giving control over the degree of aperiodicity: the higher the CV, the more aperiodic the pulse-train. A pulse train with a CV of zero was equivalent to the periodic, rewarded pattern. The average number of ICMS pulses per unit time was the same for the periodic and aperiodic patterns. Examples of pulse trains with different CVs are shown in Fig. 2.
D. Artifact Suppression
An important question arising from the use of ICMS for sensory feedback is whether the stimulation causes artifacts in cortical neural ensemble recordings and how these artifacts can be dealt with to minimize their impact on BMI operations. To address this question, we processed the neural recordings by removing (blanking) a window of neural activity immediately subsequent to each pulse of ICMS and then performed decoding with the processed data. By systematically varying the length of the blanking intervals we could determine the amount of artifact removal that produced the most accurate movement reconstructions.
Artifact removal was implemented as follows. The stimulation artifacts had stereotypical shapes when recorded by the spike acquisition system, and we could reliably detect artifacts by using spike-sorting templates that matched the artifact shapes, allowing us to determine the precise time of each stimulation pulse. We then ignored all spiking on every channel for x milliseconds after each stimulation pulse, where x varied in value from 0 to 10, in integer steps. This was done by first counting spikes in 1 ms non-overlapping time windows (i.e. binning at 1 ms resolution). We then zeroed the spike counts in bins that were equal to or less than x ms after the stimulation pulse. For example, for x=3, we zeroed the bins at t, t+1, t+2, and t+3, where t is the 1 ms bin of the stimulation pulse (See Fig. 6A). For x=0, we zeroed the bin at the stimulation pulse only. Then, we summed adjacent bins to produce spike counts in 100 ms non-overlapping bins for our decoders. For this last step, we adjusted the spike counts by multiplying by the quantity 100 / (100 − nblanked), where nblanked is the number of zeroed 1 ms bins in the 100 ms bin. This operation preserves, in expectation, the number of spikes in each 100 ms bin, by performing extrapolation.
Fig. 6
Fig. 6
Exploration of blanking intervals. (a) Schematic of the blanking procedure. Spikes and ICMS pulses were categorized into 1 ms bins. For each bin containing an ICMS pulse, that bin and a variable number of bins (three shown here) were blanked subsequently. (more ...)
E. Kinematics Extraction
The X and Y position of the avatar was extracted from cortical activity using a 5th order unscented Kalman filter [40] and Wiener filter [37]. For both algorithms, we evaluated the decoding accuracy after artifact blanking. We performed 2-fold cross-validation with each algorithm on 26 sessions, 13 from each monkey. For the unscented Kalman filter, we used a tuning model with linear weights for position, velocity, distance from center of workspace, and magnitude of velocity. The unscented Kalman filter had three future taps, two past taps, and one tap in the movement model (see [40] for details). The tuning model weights were fit with adaptive ridge regression [41], with the ridge parameter found by cross-validation on the training data. For the Wiener filter, we used 10 taps of spiking history and predicted the position only. The Wiener coefficients were fit using ridge regression with the ridge parameter found by cross-validation on the training data.
A. Learning
Initially, both monkeys were required to discriminate between the periodic (rewarded) ICMS pattern and an aperiodic pattern with a CV of 0.8. Each monkey learned this discrimination task in approximately 8 daily sessions (Fig. 3). Monkey N stabilized at a performance level of approximately 90% correctly executed trials, monkey M at an 85% level. These learning curves are consistent with our previous results on rhesus monkey learning with ICMS-instructed tasks [9].
Fig. 3
Fig. 3
Summary of the behavioral performance of monkeys M (circles) and N (diamonds) for 10 sessions as they learned the task. Each symbol shows the mean performance for the session. Filled symbols depict sessions with performance significantly different from (more ...)
B. Psychometrics
After both monkeys learned to discriminate periodic ICMS from aperiodic with a CV of 0.8, we began to vary the CV of the aperiodic ICMS pattern on every trial. In these sessions, the distribution of CVs was picked so that for half of the trials the CV of the unrewarded object was greater or equal to 0.6. These sessions continued for 2 weeks, yielding a database for psychometric analysis.
Psychometric curves (i.e., graphs showing the proportion of correctly performed trials as the function of CV, Fig. 4) indicated a clear dependency of discrimination accuracy on the degree of randomness of the comparison ICMS pattern. Performance stabilized for CVs higher than 0.8 and gradually decreased for CVs lower than that value. The threshold CV for discrimination for both monkeys was 0.25. Below this value, the monkeys performed at chance levels.
Fig. 4
Fig. 4
Psychometric curves for different coefficients of variation on the aperiodic pulse trains. (a) Mean performance at differentiating periodic versus aperiodic ICMS pulse trains as a function of CV for monkey N. Each symbol represents the mean performance (more ...)
C. Active Exploration
Discrimination of ICMS patterns was performed through active exploration: a monkey would probe the feedback zone of an object with the avatar to acquire an ICMS pattern and then either select that object if it perceived the ICMS pattern as periodic or explore the other object if the pattern was judged as different from periodic. This active exploration was evident from an analysis of object exploration intervals (Fig. 5A,B). We designated intervals during which the avatar hand continuously stayed over a given object as “visits”. For very low CVs, the statistics of visit durations were the same for periodic and aperiodic patterns (Fig. 5A).
Fig. 5
Fig. 5
Active exploration of the virtual objects as a function of CV. (a) Histograms of visit durations to the rewarded object (CV=0, black trace) versus the unrewarded object (CV=0.05, red trace) for trials with that CV combination. Visits are quantified as (more ...)
The distribution of these intervals indicated short (less than 2 s) exploratory visits and a prominent peak at 2 s that corresponded to selecting an object. This is because the monkey was required to hold the avatar hand over the response zone of the object for 2 s to obtain a reward (or to get a trial cancellation if the object was selected incorrectly). Accordingly, the peak at 2 s corresponded to visits for which the monkey selected a given object. Intervals longer than 2s were possible because visits comprised the portion spent over the feedback zone (but outside of the response zone) as well as the time spent over the response zone. We called visits with durations less than 2 s short visits, and those with durations of 2 s or longer long visits. Both the short-visit portion of the distribution and the long-visit part were preserved for periodic ICMS (CV=0, Fig. 5A black line) versus weakly aperiodic ICMS e.g. CV of 0.05 (Fig. 5A red line).
The distributions of visit-durations were markedly different for higher CVs (Fig. 5B). The distribution of visit-durations for aperiodic ICMS with a CV of 0.8 (cyan line) revealed a predominance of short visits with an average duration of 0.8 s and a small proportion of long visits. For the periodic pattern (black line), the distribution showed the predominance of long visits. These data indicate that it took the monkey on average 0.8 s to recognize the unrewarded aperiodic ICMS pattern and to switch to the correct object (periodic pattern) when sufficiently aperiodic ICMS patterns were used.
The change in monkey exploratory behavior for different degrees of ICMS-pattern aperiodicity is clear from the statistics of visits, expressed as the proportion of short visits normalized by the total number of visits (Fig. 5C). When a monkey touched an object associated with an aperiodic pattern (Fig. 5C triangles), it tended to make more short visits than when the monkey touched an object associated with a periodic pattern (squares). For high CVs (CV greater than 0.7), the proportion of short visits constituted approximately 80% of the total number of visits. For lower CVs, this value decreased, indicating that the monkey made a decision to stay on the unrewarded object more often.
D. Kinematics Extraction
Figure 6 shows the average accuracy for the extraction of avatar position from cortical ensemble activity for different lengths of artifact blanking intervals. Consistent with our previous results [40], the unscented Kalman filter consistently outperformed the Wiener filter. For monkey M, the peak accuracy was 5.4 ± 0.17 dB (mean ± standard error) for the unscented Kalman filter and 4.0 ± 0.14 dB for the Wiener filter. For monkey N, these values were 2.9 ± 0.16 dB and 2.5 ± 0.14 dB, respectively. These accuracy values are within the range that we typically observe for BMI predictions [9], [17], [41].
Both algorithms benefited somewhat from artifact blanking, more so for monkey M. For monkey M, maximum accuracy was achieved with 5 ms of artifact blanking for both the Weiner filter and the unscented Kalman filter. For monkey N, maximum accuracy was achieved with 2 ms of artifact blanking for both decoders. These values reflect the difference in artifact duration and amplitude in two monkeys. The artifacts were more prominent and of longer duration in monkey M because of the close proximity of the stimulation site (hand representation of S1) to the area where neuronal activity was collected (arm representation of M1 and S1). The artifacts were smaller and of shorter duration for monkey N, which received stimulation in the leg representation area of S1 with recordings performed in the arm representation area.
Curiously, the performance of the unscented Kalman filter was slightly better for the no blanking condition than for 0 ms of blanking. This was because the recording channels that detected ICMS artifacts occasionally recorded additional mechanical artifacts related to monkey head movements. Apparently, the filter could utilize these mechanical artifacts that influenced the spike recording channels (blanked them or introduced erroneous spikes) to improve predictions, and its performance was very slightly reduced when these artifacts were removed. This underscores the importance of registering the artifacts and removing them to minimize their influence on the filter performance.
To quantify the decrease in predictions caused by the presence of ICMS artifacts, we recorded from both monkeys as they performed a center-out task without any ICMS. For this task, they had to move the avatar from the center of the screen to a single peripheral object and hold for 2 s. For monkey M, accuracy in this task was 23% higher than during the ICMS sessions with the unscented Kalman filter and 32% higher with the Wiener filter. For monkey N, these values were 2.7% and 27%, respectively. Thus, the artifacts worsened the predictions, even after optimal blanking, but still within a tolerable range.
This study continued our work on the development of an artificial somatosensory channel for BMIs [9], [10], [21]. Monkeys scanned virtual objects with an avatar hand and discriminated their artificial textures as represented by temporal patterns of ICMS. This paradigm models the requirements of a clinically relevant neuroprosthetic arm sensorized with an artificial tactile channel. Such a neuroprosthetic arm could be used to touch external objects and estimate their tactile properties (roughness/smoothness, hardness/softness, wetness/dryness, temperature) using sensors on the prosthetic hand. The transmission of this information to the nervous system is a difficult problem because of the artificial nature of the stimulation methods. We explored the capability of temporally patterned ICMS as a way to deliver somatosensory feedback to the brain by parametrically varying the degree of randomness of ICMS trains. Monkeys learned to distinguish regular ICMS patterns from irregular ones, a result which suggests that they were able to discriminate the fine temporal structure of ICMS trains. Irregular bursts of sensory discharges are expected to occur in practical neuroprostheses, when the prosthesis interacts with realistically textured objects. A neuroprosthetic hand used to scan a ridged surface, for example, would generate an ICMS burst each time a ridge interacts with the prosthetic sensor. In this setting, the degree of periodicity of ICMS pulse trains could inform the prosthesis user about the regularities or irregularities of an object’s material or shape.
Our results complement previous work on ICMS frequency discrimination conducted by Romo and colleagues who trained their monkeys to discriminate periodic ICMS pulse trains [22]–[24] and to discriminate the mean rate of aperiodic pulse trains [22], [42]. Our study expanded the range of temporal patterns that could be represented by ICMS of S1 by changing the regularity of the secondary frequency. Moreover, ICMS in our experiments served as somatosensory feedback during virtual active touch, rather than merely a cue in a forced choice task as in the majority of previous studies. The animals actively explored virtual objects with an avatar hand, spending similar times over these objects as would be needed for normal interaction with the environment. Additionally, chronically implanted electrodes were used for ICMS delivery, which allowed us to monitor long-term learning to utilize ICMS as sensory feedback. In previous studies, stimulating electrodes were often inserted in the brain anew during each daily session. Long-term usage of ICMS in the present experiments (as well as in previous experiments with the same monkeys) did not result in deterioration of performance, which indicates that the charge-balanced ICMS used here did not damage the electrodes or brain tissue, or that any such damage was below a threshold where it would begin to impact task performance.
Our results show that monkeys detect distortions in the 10 Hz ICMS secondary frequency after random variations of that frequency exceeded 25%, that is, instantaneous frequency fluctuated from 7.5 to 12.5 Hz. This estimate of the detection threshold can be used in future neuroprosthetic designs as a characteristic sensitivity value. Future studies should probe the sensitivity of discrimination to different primary and secondary frequencies. Additionally, spatiotemporal ICMS [21] and ICMS of different durations should be explored as ways to encode information in BMBI sensory channels.
The interaction of a neuroprosthesis with realistically textured objects in the natural world will inevitably result in a stream of temporally patterned sensory information. Either the user or the neuroprosthesis (or some combination thereof) will therefore need to deal with these signals. Texture analysis could be delegated, in part, to a shared control algorithm [43]. In this mode of operation, a sensation processor would analyze raw signals from sensors on the prosthesis and interpret them in the context of how the neuroprosthetic device “skin” moved against the surface of textured objects. Simplified ICMS patterns—representing different classes of textures—could then be sent to the brain. Alternatively, ICMS could directly encode a signal representing both the spatiotemporal movements of the prosthetic limb and the intrinsic microstructure of the material being touched. In this case, the temporal patterns of ICMS would have to be interpreted by the user in the context of the particular exploration pattern used [44]. The choice of the encoding scheme will likely be dictated by the requirements of the specific neuroprosthetic application.
It would be of interest for future studies to explore the optimal temporal properties of ICMS modulations at different S1 sites. Romo et al. reported best results when they applied ICMS to rapidly adapting neurons in area 3b [22]. We stimulated in area 1, where the distinction between rapidly adapting and slowly adapting categories of neurons is less clear. Additionally, we used multi-session training periods with chronically implanted electrodes, in contrast to Romo et al. who used independent stimulation sessions with acute electrodes. It is possible that that our longer training interval facilitated the discrimination capacity of the monkeys. The distinction between different S1 locations and the role of learning will need to be studied in more detail in future studies.
BMBIs equipped with afferent ICMS feedback loops need to compensate for electrical artifacts produced by ICMS pulses that may interfere with neuronal recordings. In our previous BMBI designs, we either discounted the entire period of ICMS application [9] or used interleaved recording and ICMS delivery intervals [10]. These previous approaches limited the flexibility of ICMS delivery. In this study, we did not impose limitations on the timing of ICMS delivery and treated ICMS artifacts as they occurred. We found that blanking the periods after ICMS delivery by short intervals (2–5 ms) improved the accuracy of extraction of limb kinematics from neuronal activity. Overall accuracy of predictions was 20–30% less as compared to sessions in which ICMS was not used. Nonetheless, the predictions were still acceptable and within range of previously reported accuracy of BMI decoding. This result suggests that artifact blanking is practical for bidirectional neuroprostheses using irregular ICMS pulse trains.
The precise character of perceptions evoked by periodic versus aperiodic patterns of ICMS will have be evaluated in human subjects [45]. There is a suggestion by Fridman et al. that ICMS amplitude, pulse-width, and frequency all interact to contribute to a unitary perception of “perceived intensity” [46]. Therefore, one might argue that our monkeys discriminated the periodic and aperiodic pulse trains on the basis of their instantaneous peak intensities rather than their temporal patterns. However, this simple explanation is unlikely because the peak instantaneous frequency of ICMS was 200 Hz for both the periodic as well as the aperiodic artificial textures. Therefore, our results indicate that the monkeys must have been using a strategy beyond simply detecting the maximum instantaneous frequency. One possible neural implementation could employ a leaky integrator mechanism that detected variability of the ICMS secondary frequency by integrating neural responses to ICMS within an optimal time window, thus detecting transient increases in ICMS frequency. This and other alternative mechanisms will need to be elucidated with future studies.
Our current and previous [9], [10], [21] results suggest that new perceptions may evolve as subjects practice with ICMS. We observed that it took monkeys 1–2 weeks to start to understand ICMS, even if they were previously overtrained with a vibrotactile variant in the same task. However, once they learned the first ICMS task, learning subsequent tasks took much less time. A virtual active touch setting where subjects evoke ICMS and associated sensations through their own actions [47]–[49] may contribute to shaping the artificial perception and lead to the development of anticipatory cortical modulations similar to corollary discharge [50].
The problem of artifacts will be compounded as multiple stimulation channels are employed with asynchronously delivered pulses. Excessive masking of the recordings by ICMS artifacts should be avoided in the design of such systems. In the future, the problem of artifacts [51], as well as the unreliable spatial extent of ICMS [52] could be mitigated by optogenetic stimulation [53]–[55].
Acknowledgment
The authors thank D. Dimitrov for assistance with the animal surgeries, S. Shokur for design and programming of the monkey avatar, and G. Lehew, J. Meloy, T. Phillips, L. Oliveira, and S. Halkiotis for invaluable technical support.
This work was supported by DARPA grant number N66001-06-C-2019, TATRC grant number W81XWH-08-2-0119, the National Institutes of Health through NICHD/OD, grant number RC1HD063390 and the NIH Director’s Pioneer Award Program, grant number DP1OD006798, to MALN.
Biographies
An external file that holds a picture, illustration, etc.
Object name is nihms372629b1.gif Object name is nihms372629b1.gif
Joseph E. O’Doherty received the B.S. degree in physics from East Carolina University, Greenville, North Carolina, USA, in 2001 and the Ph.D. degree in biomedical engineering from Duke University, Durham, North Carolina, USA, in 2011.
He is currently a Research Scholar at the Duke University Center for Neuroengineering, in Durham, North Carolina, USA. His research interests include methods for providing artificial somatic sensation and proprioception for neural prostheses.
An external file that holds a picture, illustration, etc.
Object name is nihms372629b2.gif Object name is nihms372629b2.gif
Mikhail A. Lebedev received the M.S. degree in physics from the Moscow Institute of Physics and Technology, Moscow, Russia, in 1986 and the Ph.D. degree in neurobiology from the University of Tennessee, Memphis, USA, in 1995.
He is a Senior Research Scientist at the Duke University Center for Neuroengineering, in Durham, North Carolina, USA. He has held research appointments at the Institute for the Problems of Information Transmission, Moscow, (1986–1991), the International School for Advanced Studies, Trieste, Italy, (1995–1997) and the US National Institute of Mental Health (1997–2002). His research interests include primate neurophysiology and brain–machine interfaces.
An external file that holds a picture, illustration, etc.
Object name is nihms372629b3.gif Object name is nihms372629b3.gif
Zheng Li received the B.S. degree in computer science and mathematics from Purdue University, West Lafayette, Indiana, USA, in 2004 and the Ph.D. degree in computer science from Duke University, Durham, North Carolina, USA, in 2010.
He is a Postdoctoral Associate at the Duke University Center for Neuroengineering, Durham, North Carolina, USA. His research interests are in the computational aspects of brain-machine interfaces.
An external file that holds a picture, illustration, etc.
Object name is nihms372629b4.gif Object name is nihms372629b4.gif
Miguel A. L. Nicolelis received the M.D. and Ph.D. degrees from the University of Sao Paulo, Sao Paulo, Brazil, in 1984 and 1988, respectively.
He is the Anne W. Deane Professor of Neuroscience with the departments of Neurobiology, Biomedical Engineering and Psychology at Duke University, Durham, North Carolina, USA. He is the Co-Director of Duke's Center for Neuroengineering. He is also Founder and President of the Edmond and Lily Safra International Institute for Neuroscience of Natal, Brazil and a Fellow of the Brain and Mind Institute at the École Polytechnique Fédérale de Lausanne, Switzerland.
Dr. Nicolelis' research was highlighted in MIT Review's Top Emerging Technologies, and he was named one of Scientific American's Top 50 Technology Leaders in America. Other honors include the Whitehead Scholar Award; Whitehall Foundation Award; McDonnell-Pew Foundation Award; the Ramon y Cajal Chair at the University of Mexico and the Santiago Grisolia Chair at Catedra Santiago Grisolia. He was awarded the International Blaise Pascal Research Chair from the Fondation de l'Ecole Normale Supérieure and the 2009 Fondation IPSEN Neuronal Plasticity Prize. Dr. Nicolelis is a member of the French and Brazilian Academies of Science and has authored over 170 manuscripts, edited numerous books and special journal issues, and holds three US patents.
Contributor Information
Joseph E. O’Doherty, Department of Neurobiology and the Center for Neuroengineering, Duke University, Durham, NC 27710 USA.
Mikhail A. Lebedev, Department of Neurobiology and the Center for Neuroengineering, Duke University, Durham, NC 27710 USA.
Zheng Li, Department of Neurobiology and the Center for Neuroengineering, Duke University, Durham, NC 27710 USA.
Miguel A.L. Nicolelis, Departments of Neurobiology, Biomedical Engineering, Psychology, and the Center for Neuroengineering, Duke University, Durham, NC 27710 USA.
1. Merzenich MM, Schindler DN, White MW. Feasibility of multichannel scala tympani stimulation. Laryngoscope. 1974 Nov;vol. 84:1887–1893. [PubMed]
2. Fallon JB, Irvine DRF, Shepherd RK. Cochlear implants and brain plasticity. Hear. Res. 2008 Apr;vol. 238:110–117. [PMC free article] [PubMed]
3. Bach-y-Rita P, Colins CC, Saunders FA, White B, Scadden L. Vision substitution by tactile image projection. Nature. 1969 Mar;vol. 221:963–964. [PubMed]
4. Bach-y-Rita P, Kaczmarek KA, Tyler ME, Garcia-Lara J. Form perception with a 49-point electrotactile stimulus array on the tongue: a technical note. J. Rehabil. Res. Dev. 1998 Oct;vol. 35:427–430. [PubMed]
5. Bach-y-Rita P, Kercel SW. Sensory substitution and the human-machine interface. Trends Cogn. Sci. 2003 Dec;vol. 7:541–546. [PubMed]
6. Dobelle WH, Mladejovsky MG, Girvin JP. Artificial vision for the blind: electrical stimulation of visual cortex offers hope for a functional prosthesis. Science. 1974 Feb;vol. 183:440–444. [PubMed]
7. Dagnelie G. Psychophysical evaluation for visual prosthesis. Annu. Rev. Biomed. Eng. 2008;vol. 10:339–368. [PubMed]
8. Cohen ED. Prosthetic interfaces with the visual system: biological issues. J. Neural Eng. 2007 Jun;vol. 4:R14–R31. [PubMed]
9. O'Doherty JE, Lebedev MA, Hanson TL, Fitzsimmons NA, Nicolelis MAL. A brain-machine interface instructed by direct intracortical microstimulation. Front. Integr. Neurosci. 2009;vol. 3:20. [PMC free article] [PubMed]
10. O'Doherty JE, et al. Active tactile exploration enabled by a brain-machine-brain interface. Nature. to be published.
11. Mussa-Ivaldi FA, et al. New Perspectives on the Dialogue between Brains and Machines. Front. Neurosci. 2010;vol. 4:44. [PMC free article] [PubMed]
12. Stanslaski S, et al. An implantable bi-directional brain-machine interface system for chronic neuroprosthesis research. Conf. Proc. IEEE Eng. Med. Biol. Soc. 2009;vol. 2009:5494–5497. [PubMed]
13. Fagg AH, et al. Toward a biomimetic, bidirectional, brain machine interface. Conf. Proc. IEEE Eng. Med. Biol. Soc. 2009;vol. 2009:3376–3380. [PubMed]
14. Marzullo TC, Lehmkuhle MJ, Gage GJ, Kipke DR. Development of closed-loop neural interface technology in a rat model: combining motor cortex operant conditioning with visual cortex microstimulation. IEEE Trans. Neural Syst. Rehabil. Eng. 2010 Apr;vol. 18:117–126. [PMC free article] [PubMed]
15. Lebedev MA, Nicolelis MAL. Brain-machine interfaces: past, present and future. Trends Neurosci. 2006 Sep;vol. 29:536–546. [PubMed]
16. Nicolelis MAL, Lebedev MA. Principles of neural ensemble physiology underlying the operation of brain-machine interfaces. Nat. Rev. Neurosci. 2009 Jul;vol. 10:530–540. [PubMed]
17. Fitzsimmons NA, Lebedev MA, Peikon ID, Nicolelis MAL. Extracting kinematic parameters for monkey bipedal walking from cortical neuronal ensemble activity. Front. Integr. Neurosci. 2009;vol. 3:3. [PMC free article] [PubMed]
18. Brumberg JS, Nieto-Castanon A, Kennedy PR, Guenther FH. Brain-Computer Interfaces for Speech Communication. Speech Commun. 2010 Apr;vol. 52:367–379. [PMC free article] [PubMed]
19. Guenther FH, et al. A wireless brain-machine interface for real-time speech synthesis. PLoS One. 2009;vol. 4:e8218. [PMC free article] [PubMed]
20. Peterson NR, Pisoni DB, Miyamoto RT. Cochlear implants and spoken language processing abilities: review and assessment of the literature. Restor. Neurol. Neurosci. 2010;vol. 28:237–250. [PMC free article] [PubMed]
21. Fitzsimmons NA, Drake W, Hanson TL, Lebedev MA, L. Nicolelis MA. Primate reaching cued by multichannel spatiotemporal cortical microstimulation. J. Neurosci. 2007 May;vol. 27:5593–5602. [PubMed]
22. Romo R, Hernández A, Zainos A, Salinas E. Somatosensory discrimination based on cortical microstimulation. Nature. 1998 Mar;vol. 392:387–390. [PubMed]
23. Romo R, Hernández A, Zainos A, Brody CD, Lemus L. Sensing without touching: psychophysical performance based on cortical microstimulation. Neuron. 2000 Apr;vol. 26:273–278. [PubMed]
24. de Lafuente V, Romo R. Neuronal correlates of subjective sensory experience. Nat. Neurosci. 2005 Dec;vol. 8:1698–1703. [PubMed]
25. Butovas S, Schwarz C. Detection psychophysics of intracortical microstimulation in rat primary somatosensory cortex. Eur. J. Neurosci. 2007 Apr;vol. 25:2161–2169. [PubMed]
26. Houweling AR, Brecht M. Behavioural report of single neuron stimulation in somatosensory cortex. Nature. 2008 Jan;vol. 451:65–68. [PubMed]
27. Talwar SK, et al. Rat navigation guided by remote control. Nature. 2002 May;vol. 417:37–38. [PubMed]
28. Venkatraman S, Carmena JM. Active Sensing of Target Location Encoded by Cortical Microstimulation. IEEE Trans. Neural Syst. Rehabil. Eng. 2011 Jun;vol. 19:317–324. [PubMed]
29. Katz D, Krueger LE. The world of touch. Hillsdale, N.J.: L. Erlbaum Associates; 1989.
30. Mountcastle VB, Steinmetz MA, Romo R. Frequency discrimination in the sense of flutter: psychophysical measurements correlated with postcentral events in behaving monkeys. J. Neurosci. 1990 Sep;vol. 10:3032–3044. [PubMed]
31. Salinas E, Hernández A, Zainos A, Romo R. Periodicity and firing rate as candidate neural codes for the frequency of vibrotactile stimuli. J. Neurosci. 2000 Jul;vol. 20:5503–5515. [PubMed]
32. Lebedev MA, Denton JM, Nelson RJ. Vibration-entrained and premovement activity in monkey primary somatosensory cortex. J. Neurophysiol. 1994 Oct;vol. 72:1654–1673. [PubMed]
33. Sinclair RJ, Burton H. Tactile discrimination of gratings: psychophysical and neural correlates in human and monkey. Somatosens. Mot. Res. 1991;vol. 8:241–248. [PubMed]
34. Sinclair RJ, Pruett JR, Burton H. Responses in primary somatosensory cortex of rhesus monkey to controlled application of embossed grating and bar patterns. Somatosens. Mot. Res. 1996;vol. 13:287–306. [PubMed]
35. Gamzu E, Ahissar E. Importance of temporal cues for tactile spatial-frequency discrimination. J. Neurosci. 2001 Sep;vol. 21:7416–7427. [PubMed]
36. Phillips JR, Johnson KO. Tactile spatial resolution. II. Neural representation of bars, edges, and gratings in monkey primary afferents. J. Neurophysiol. 1981 Dec;vol. 46:1192–1203. [PubMed]
37. Carmena JM, et al. Learning to control a brain-machine interface for reaching and grasping by primates. PLoS Biol. 2003 Nov;vol. 1:E42. [PMC free article] [PubMed]
38. Koivuniemi A, Otto K. Asymmetric vs. Symmetric Electric Pulses for Intracortical Microstimulation. IEEE Trans. Neural Syst. Rehabil. Eng. to be published.
39. Dorval AD, Kuncel AM, Birdno MJ, Turner DA, Grill WM. Deep brain stimulation alleviates parkinsonian bradykinesia by regularizing pallidal activity. J. Neurophysiol. 2010 Aug;vol. 104:911–921. [PubMed]
40. Li Z, et al. Unscented Kalman filter for brain-machine interfaces. PLoS One. 2009;vol. 4:e6243. [PMC free article] [PubMed]
41. Grandvalet Y. Least absolute shrinkage is equivalent to quadratic penalization. In: Niklasson L, et al., editors. Perspectives in Neural Computing. Springer Verlag; 1998. pp. 201–206.
42. Hernández A, Zainos A, Romo R. Neuronal correlates of sensory discrimination in the somatosensory cortex. Proc. Natl. Acad. Sci. USA. 2000 May;vol. 97:6191–6196. [PubMed]
43. Kim HK, et al. Continuous shared control for stabilizing reaching and grasping with brain-machine interfaces. IEEE Trans. Biomed. Eng. 2006 Jun;vol. 53:1164–1173. [PubMed]
44. Lederman SJ, Klatzky RL. Hand movements: a window into haptic object recognition. Cogn. Psychol. 1987 Jul;vol. 19:342–368. [PubMed]
45. Heming E, Choo R, Davies J, Kiss Z. Designing a thalamic somatosensory neural prosthesis: Consistency and persistence of percepts evoked by electrical stimulation. IEEE Trans. Neural Syst. Rehabil. Eng. to be published. [PubMed]
46. Fridman GY, Blair HT, Blaisdell AP, Judy JW. Perceived intensity of somatosensory cortical electrical stimulation. Exp. Brain Res. 2010 Jun;vol. 203:499–515. [PMC free article] [PubMed]
47. Sinclair R, Burton H. Responses from area 3b of somatosensory cortex to textured surfaces during active touch in primate. Somatosens. Res. 1988;vol. 5:283–310. [PubMed]
48. Simões-Franklin C, Whitaker TA, Newell FN. Active and passive touch differentially activate somatosensory cortex in texture perception. Hum. Brain Mapp. 2011;vol. 32:1067–1080. [PubMed]
49. Bolanowski SJ, Verrillo RT, McGlone F. Passive, active and intra-active (self) touch. Behav. Brain. Res. 2004 Jan 5;vol. 148:41–45. [PubMed]
50. Crapse TB, Sommer MA. Corollary discharge across the animal kingdom. Nat. Rev. Neurosci. 2008 Aug;vol. 9:587–600. [PubMed]
51. Rolston JD, Gross RE, Potter SM. A low-cost multielectrode system for data acquisition enabling real-time closed-loop processing with rapid recovery from stimulation artifacts. Front. Neuroengineering. 2009;vol. 2:12. [PMC free article] [PubMed]
52. Histed MH, Bonin V, Reid RC. Direct activation of sparse, distributed populations of cortical neurons by electrical microstimulation. Neuron. 2009 Aug;vol. 63:508–522. [PMC free article] [PubMed]
53. Zhang F, Aravanis AM, Adamantidis A, de Lecea L, Deisseroth K. Circuit-breakers: optical technologies for probing neural signals and systems. Nat. Rev. Neurosci. 2007 Aug;vol. 8:577–581. [PubMed]
54. Boyden ES, Zhang F, Bamberg E, Nagel G, Deisseroth K. Millisecond-timescale, genetically targeted optical control of neural activity. Nat. Neurosci. 2005 Sep;vol. 8:1263–1268. [PubMed]
55. Zhang F, Wang LP, Boyden ES, Deisseroth K. Channelrhodopsin-2 and optical control of excitable cells. Nat. Methods. 2006 Oct;vol. 3:785–792. [PubMed]