|Home | About | Journals | Submit | Contact Us | Français|
The cognitive neural prosthetic (CNP) is a very versatile method for assisting paralyzed patients and patients with amputations. The CNP records the cognitive state of the subject, rather than signals strictly related to motor execution or sensation. We review a number of high-level cortical signals and their application for CNPs, including intention, motor imagery, decision making, forward estimation, executive function, attention, learning, and multi-effector movement planning. CNPs are defined by the cognitive function they extract, not the cortical region from which the signals are recorded. However, some cortical areas may be better than others for particular applications. Signals can also be extracted in parallel from multiple cortical areas using multiple implants, which in many circumstances can increase the range of applications of CNPs. The CNP approach relies on scientific understanding of the neural processes involved in cognition, and many of the decoding algorithms it uses also have parallels to underlying neural circuit functions.
The number of patients suffering from some form of paralysis in the United States alone has been estimated to be from 1.7 million (U.S. Dept. Health Human Serv. 1995) to 5.6 million (Christopher & Dana Reeve Found. 2009). Paralysis can result from spinal cord lesion and other traumatic accidents, peripheral neuropathies, amyotrophic lateral sclerosis, multiple sclerosis, and stroke. Another 1.4 million patients have motor disabilities due to limb amputation (U.S. Dept. Health Human Serv. 1995). A majority of these patients still have sufficiently intact cortex to plan movements, but they are unable to execute them. Thus they are candidates for assistance using cortical neural prosthetics.
Figure 1 shows the concept of cortical neural prosthetics generally, and cognitive neural prosthetics (CNPs) more specifically. In this particular case, the patient is shown to have a spinal cord lesion, but a similar logic applies to other forms of paralysis or to amputation. The patient can still plan movements but cannot execute them. Recordings can be made from microelectrode arrays in cortex. The implants not only record the activity of populations of nerve cells but also transmit these signals wirelessly to external assistive devices. Implants can be placed in a variety of areas, and they record the intent or other cognitive variables of the subject. Decoding algorithms interpret the meaning of the recorded signals. These algorithms can be incorporated into hardware in the implant or in the external devices. The decoded neural signals are further transformed to provide control signals to operate assistive devices. In the example in Figure 1, these devices can include robotic limbs, functional electrical stimulation of otherwise paralyzed limbs for reanimation, wheelchair navigation, Internet access, email, telephone and other forms of computer-assisted communication, and the control of the patient's environment including television, temperature control, and calls for assistance. Elements of a cortical neural prosthetic, including the electrodes, decoding algorithms, and associated electronics, are often referred to collectively as a brain-machine interface (BMI).
In research applications, healthy monkeys are used to test cortical prosthetics. Typically, the animals have a permanently implanted array of electrodes, similar to the case for human patients. The animals control an output device with their thoughts and can do this without eliciting movements. This process is often referred to as a brain-control task. It is also called a closed-loop task since the animals receive feedback about their performance, for instance, the movement of a cursor on a computer screen controlled by their neural activity.
Many studies have involved extracting motor execution signals from motor cortex (Carmena et al. 2003, Fetz 1969, Serruya et al. 2002, Taylor et al. 2002). It was often observed that the monkeys did not need to actually move the limb to bring a cursor under brain control. In terms of the current topic, this would constitute cognitive control, in which brain signals not directly related to executing a movement can nonetheless be harnessed for the task. This control can be derived from motor imagery, planning, attention, decision making, or executive control, to name just a few of the cognitive signals that are potentially useful for neuroprosthetics. The distinction is not the brain location of the recording but rather the type of signal that is being extracted (Andersen et al. 2004a). This being said, some brain areas will no doubt be better sources of signals for particular neuroprosthetic applications. The specialization of different cortical areas is an advantage for CNPs. For example, for mute patients speech can potentially be decoded directly from speech areas rather than using a letter board and controlling a cursor from motor cortex. CNPs can also take advantage of parallel decoding, in which implants are placed in multiple cortical areas and different signals are decoded simultaneously. A prime example of this parallel decoding is the application of CNPs to complex, multi-effector movements, discussed below.
One central element of neuroprosthetics is engineering. Advanced statistical and signal-processing techniques are commonly used to optimize decoding algorithms as well as develop algorithms that are adaptive. Such approaches are essential to CNPs. However, CNPs also benefit from the additional component of a scientific understanding of the brain processes being performed by the region(s) of recording. This understanding extends to functional neuroanatomy and network/circuit properties that can guide the selection of recording sites and the design and implementation of decoding algorithms. The following sections highlight examples that match cortical function, and in some cases decoding algorithms, to particular cognitive neuroprosthetic applications.
Neural prosthetic applications have often used trajectory signals to bring a cursor or a robotic hand to a goal (Carmena et al. 2003, Serruya et al. 2002, Taylor et al. 2002). This approach, especially for cursor control, is similar to using a mouse to drag a cursor to a location on a computer screen. However, many applications would benefit from being able to rapidly indicate a series of goals. One example would be the rapid indication of letters on a letter board for communication. Another would be to provide a sequence of movements for a robotic limb to ensure a more fluid programming of a string of movements.
The primary motor cortex (M1) contains some goal information, but goal information is strongly represented in the premotor and posterior parietal cortex (Hatsopoulos et al. 2004, Snyder et al. 1997) (Figure 2). This goal information reflects the intent of the animal to make a movement to the goal.
Gnadt & Andersen (1988) first demonstrated intended eye movement signals in the lateral intraparietal area (LIP) of the posterior parietal cortex (PPC). Subsequent studies showed that neurons in the parietal reach region (PRR) of PPC encode the intent to make reach movements (Snyder et al. 1997). A variety of experiments have demonstrated intention-related activity that is not spatial attention (Cui & Andersen 2007, Gail & Andersen 2006, Quiroga et al. 2006, Scherberger & Andersen 2007, Scherberger et al. 2005, Snyder et al. 1998). These movement plans can be formed but then cancelled without executing a movement; therefore, they do not reflect motor execution but rather the higher-level plan to move (Andersen & Buneo 2002, Bracewell et al. 1996, Snyder et al. 1998). The more high-level, abstract nature of the intention signals is also evident from the finding that the intended reach activity in PRR is coded in visual rather than limb coordinates (Batista et al. 1999).
Putative human homologues of LIP and PRR have been identified in humans using functional magnetic resonance imaging (fMRI) experiments (Astafiev et al. 2003, Connolly et al. 2003, Filimon et al. 2009). Interestingly, electrical stimulation of the posterior parietal cortex in human patients invoked the conscious intention to move various body parts even though no movements resulted from the stimulation (Desmurget et al. 2009). Whereas it was possible to demonstrate intention-related activity in monkey PPC neurons, this study shows that the conscious awareness of intention also arises with increased PPC activity.
Musallam et al. (2004) demonstrated decoding of four or eight goal locations from array recordings from the PRR of PPC and the dorsal premotor cortex (PMd) in frontal cortex (Figure 3A). To emphasize the cognitive nature of the signal, they decoded the persistent activity that results when monkeys plan a movement to a briefly cued location in space but while withholding the execution of the movement. This goal signal is decoded in the dark, with no stimulus present and no movement being executed; it is endogenously generated and represents the movement thought or intent of the animal. This intent can be decoded very rapidly. If a 100 ms time segment was used for decoding, it was nearly as accurate as using a 900 ms time segment. Subsequent studies in PMd showed that three goals could be decoded in rapid succession (Santhanam et al. 2006).
Many natural movements are highly coordinated concatenations of movement sequences rather than single reaches as discussed above. Frontal areas encode the parts of sequences including the directions and order of movements (Averbeck et al. 2006, Fujii & Graybiel 2003, Histed & Miller 2006, Lu & Ashe 2005, Mushiake et al. 2006, Ninokura et al. 2003, Ohbayashi et al. 2003, Tanji & Shima 1994). An early study of PRR found that only the next movement of a sequence is encoded, but the task was complex and involved the canceling of old plans and formation of new ones (Batista & Andersen 2001). In a more direct test of sequential planning in PRR, it was found that the area represents simultaneously and in parallel the first and second goals in a sequence of two movements (Baldauf et al. 2008). This dual representation was present regardless of whether the two movements were made rapidly or slowly. A nearest-neighbor decoding algorithm applied to the data revealed that the sequence of planned movements could be decoded. This decoding was done during a delay period in which there was no stimulus or movement, again emphasizing the cognitive nature of the signals encoding the sequence. Thus PRR activity encodes simultaneously and in parallel a sequence of movements, and this feature could be utilized to provide for a more fluid operation of output devices for prosthetic applications.
A basis of neural prosthetic control may be motor imagery. The fact that primary motor cortex cells can be trained to respond without evoking a movement suggests that movements can be imagined even from an area very close to the final motor output (Fetz 2007, Hochberg et al. 2006). Noninvasive studies using fMRI have provided a picture of the circuits involved in imagined movement. Motor imagery activates a subset of the areas that are also active during real movements, particularly premotor areas in the frontal lobe and areas in the posterior parietal cortex (Decety 1996, Gerardin et al. 2000, Stephan et al. 1995) (Glidden et al. 2005), and this activation can be as large as that seen for real movements [with the exception of motor cortex, which shows much less activation for imagined compared to real movements (Glidden et al. 2005)].
Another potential source of motor imagery for prosthetic control is the mirror neuron. Mirror neurons respond when a monkey makes a movement and also when the monkey observes the experimenter making the same movement (di Pellegrino et al. 1992, Fogassi et al. 2005, Gallese et al. 1996, Tkach et al. 2007). It has been proposed that mirror neurons form the basis of action understanding (Fogassi et al. 2005). It is possible that mirror neurons may also be activated during internally generated, imagined movements. If this is the case, and these cells also exist in human cortex, they may provide a source of control of complex and meaningful movements for prosthetics applications.
Many of the cortical areas that can produce control signals are also involved in action selection. These areas represent the expected value or utility of an action. Decision making is based on choosing the alternative with the highest value. In monkey experiments, the value is generally appetitive and includes the type, amount, and probability of reward. Many areas in the parietal and frontal cortex which represent movement plans also represent the expected value of the planned action (Barraclough et al. 2004, Campos et al. 2005, Hikosaka & Watanabe 2000, Kobayashi et al. 2002, Leon & Shadlen 1999, Matsumoto et al. 2003, Platt & Glimcher 1999, Schultz 2000, Shidara & Richmond 2002, Sugrue et al. 2004, Tremblay & Schultz 2000).
Expected value can be decoded from PRR recordings in both delayed reach and brain-control tasks (Musallam et al. 2004). In the latter, the monkey plans a movement but does not execute it and instead uses the planning activity to move a cursor to a goal on a computer screen. During a session, one reward variable (type, size, or probability) changed from trial to trial. The cue size indicated on each trial whether the animal would receive the preferred or less-preferred reward for successful completion of the trial. The cue size was varied across sessions so that a large cue represented a more desirable or less desirable reward on different days. In general, the anticipation of a preferred reward led to a larger response and improved spatial tuning. The increase in activity was unlikely to be due to increased attention given that no increase in activity was seen when the non-preferred reward was aversive (saline solution). Overall, the cells carried more information for preferred reward expectation. Parallel decoding showed that expected reward and spatial location could be decoded simultaneously. Moreover, since the cells carried more information about spatial location when higher reward was expected, the decoding performance for target location was better in high-reward trials.
A practical advantage of the reward expectation decoding results is that they provide insight into the preferences and potentially the mood of the patient. The first thing a doctor asks a patient is, “How are you feeling?” On a more general level, this study was the first to show that a very high level cognitive signal, expected value, could be decoded in brain-control trials. These results open the door to decoding many complex cognitive signals including speech, attention, executive control, and emotion for prosthetics applications.
Numerous studies support the idea that the brain constructs internal forward and inverse models to control movement (Atkeson 1989, Jordan & Rumelhart 1992, Kawato et al. 1987, Wolpert et al. 1995). The forward model predicts the sensory consequences of a movement by incorporating recent motor commands into a model of the movement dynamics, thereby predicting the upcoming state of the effector (e.g., the limb). The inverse model produces the motor commands necessary to achieve the desired movement.
Efference copy, and by extension a forward model, can be used to cancel the sensory effects of one's own movements (Andersen et al. 1987, Bradley et al. 1996, Claxton 1975, Crowell et al. 1998, Diedrichsen et al. 2005, Duhamel et al. 1992, Haarmeier et al. 2001, Roy & Cullen 2004, Royden et al. 1992, Shadmehr & Krakauer 2008, Weiskran et al. 1971). Another important feature of forward models is that they remove the delays that are present between movements and the resulting sensory feedback. When sensory feedback alone is used to correct movements online, these delays would normally lead to overcompensation and instability. For example, the execution of a goal-directed arm movement will result in visual signals that will take approximately 90 ms (Raiguel et al. 1999) and somatosensory signals that will take 20 to 40 ms (Allison et al. 1991) to reach sensorimotor cortex. Subsequent processing delays for sensorimotor integration, motor command generation, and execution result in delays of more than 100 ms for somatosensory control (Flanders & Cordo 1989) and over 200 ms for visualmotor control (Georgopoulos et al. 1981, Miall et al. 1993). However, by monitoring the movement commands through an efference copy of the command, the current state of the arm can be estimated internally well in advance of the late-arriving sensory information.
The forward model can also be incorporated into an observer framework (Goodwin & Sin 1984, Miall & Wolpert 1996) (see Figure 4). The forward model derives an estimate of the upcoming or current state of the limb. Sensory events arriving later are integrated with the forward model to update and refine the estimate. This combination of the forward model and sensory feedback is called the “ob-server,” and for linear systems with additive and Gaussian noise, the optimal observer is known as a Kalman filter (Kalman 1960).
Studies in humans have suggested that the observer may, at least in part, be located in the PPC. Lesions of the PPC produce optic ataxia in which patients have difficulty in locating and reaching to targets (Balint 1909, Perenin & Vighetto 1998, Rondot et al. 1977), in making corrective movements (Grea et al. 2002, Pisella et al. 2000), and in maintaining an estimate of the internal state of the arm (Wolpert et al. 1998). Transcranial magnetic stimulation (TMS) over the PPC interferes with the correction of trajectories or adaptation to novel force-fields (Della-Maggiore et al. 2004, Desmurget et al. 1999). PPC lesions disrupt the mental simulation of movement (Sirigu et al. 1996), and a disruption is not seen with lesions of M1 (Sirigu et al. 1995), cerebellum (Kagerer et al. 1998), or basal ganglia (Dominey et al. 1995). The movement durations are generally underestimated during simulation by these patients, suggesting a disruption of the forward model.
Neurophysiological recording experiments also suggest that a forward model may include PPC. This part of cortex receives massive feedback projections from motor structures and input from visual and somatosensory cortex and thus is ideally situated for integrating efference copy and sensory signals (Andersen et al. 1985a, 1990; Goldman-Rakic 1998; Johnson et al. 1996; Jones & Powell 1970). In an experiment designed to investigate forward models in PPC, monkeys were trained in a joystick task to move a cursor to targets either directly or around obstacles (Mulliken et al. 2008b). The obstacles afforded a more curved trajectory (Figure 5A). Neurons in the PPC not only encoded the static goal of the movement endpoint, but also the dynamic heading angle of the moving cursor. The timing of the dynamic component was centered on zero lag (Figure 5B ). Thus it was too late to represent the motor command and too early to be derived from sensory input. Rather, it appears to represent the current direction of cursor movement consistent with a forward estimate. This direction was approximately linear in space-time, indicating that it encodes a mostly linear instantaneous trajectory. Similar dynamics have recently been observed between hand kinematics and neural activity in area 5 (Archambalut et al. 2009).
A subsequent experiment showed that this forward estimate could be harnessed for neuroprosthetic applications (Mulliken et al. 2008a). The trajectories of a cursor could be decoded in joystick and brain-control tasks (Figure 5C and 5D). In the latter, the neural activity moved the cursor on the computer screen in real time. A goal-based Kalman filter was also applied for decoding, which used both the forward estimate and the goal component of neural activity. This decoding method was superior to other methods that did not use a combination of the goal and trajectory information.
So far, neural prosthetics applications have focused on single effector movements, for instance a robotic limb or a cursor. However, natural movements often involve several body parts, especially for bimanual operations, hand-eye coordination, and reach-to-grasp.
Cells in parietal and premotor areas show response specificity for effectors (Figure 3B). For instance, in the PPC there are cells specific for reaching, eye movements, and grasp (Andersen et al. 1985a, 1990; Andersen & Buneo 2002; Sakata et al. 1997; Snyder et al. 1997). These cells tend to be clustered into cortical areas—saccade selectivity in lateral intraparietal (LIP) area, reach in PRR, and grasp in the anterior intraparietal (AIP) area. A similar clustering of specificity has been shown in the frontal lobe, with saccades for the frontal eye fields (FEFs) (Bizzi 1967, Bruce & Goldberg 1985, Bruce et al. 1985), reach for PMd (Wise 1985), and grasp for the ventral premotor cortex (PMv) (Rizzollatti et al. 1994). To date, closed-loop brain control for reach has been shown in PRR and PMd (Carmena et al. 2003, Mulliken et al. 2008a, Musallam et al. 2004, Santhanam et al. 2006) and online decoding for grasp in AIP and PMv (Townsend et al. 2007), but not for eye movements from FEF or LIP.
A most natural extension of brain-control reach is reach-to-grasp. One method of approaching this problem would be to record from the limb area and hand area of M1 (Velliste et al. 2008). However, an alternative would be to record from reach (PRR/PMd) and grasp areas (AIP/PMv) in parietal and premotor cortex, where the movements are more abstractly represented (Baumann et al. 2009). For instance, cells in AIP and PMv represent the shapes of objects and the hand shape needed to grasp them (Baumann et al. 2009, Rizzollatti et al. 1994, Sakata et al. 1997). Thus, single cells can indicate the configuration of the hand and would not require a large number of cells for the different digits (as would perhaps be the case for M1 recordings).
There are very few investigations of the neural mechanisms for bimanual movement. Most early studies considered M1 to be involved only in the control of the contralateral limb. However, experiments in which monkeys made bimanual movements showed that a significant number of M1 cells responded to ipsilateral movements, although less than the contralateral limb (Donchin et al. 1998). When comparing bimanual movements to single-limb movements, most M1 cells showed significant differences in activity, indicating that bimanual interactions are extremely common. These effects could not be accounted for by postural differences between the single- and two-limb tasks. The interactions were often quite complex and included facilitation, suppression, and even changes in preferred direction tuning (Donchin et al. 1998, Rokni et al. 2003). The supplementary motor area (SMA) contains a large number of bimanual-responding neurons and bimanual interactions (Donchin et al. 1998). In PRR, a recent study found a continuum of representations of the limb from pure contralateral representation to bimanual representation (Chang et al. 2008). These studies indicate a high degree of coordination between the limbs in parietal-frontal circuits and open the possibility of being able to control two limbs effectively in bimanual operations.
Recordings from eye movement areas may be used for improving the decoding of reaches. This combination of recording from eye and reach areas utilizes the fact that eye and hand movements are coordinated and we look to where we reach. Using eye position information recorded from an external eye tracker or estimated from neural activity, the success for decoding reach targets can be improved (Batista et al. 2008). Similarly, activity in parietal and frontal areas indicates the focus of attention. Attention is automatically attracted to the target of a reach (Baldauf et al. 2006, Deubel et al. 1998) and could also be used to facilitate decoding.
Cells in LIP and PRR encode visual targets mostly in eye coordinates (Andersen et al. 1985b, Batista et al. 1999). That is, they signal the location of a target with respect to the eyes, and if the gaze direction changes, the location in space for which the cells are sensitive shifts with the gaze. The spatial locations of sounds are initially extracted with respect to the head. However, when auditory stimuli are the targets of saccades or reaches, the encoding of the saccade targets in LIP and reach targets in PRR are often represented in eye coordinates (Cohen & Andersen 2000, 2002; Stricanne et al. 1996). Common coordinate frames between these areas may facilitate decoding during hand-eye coordination.
Brain-control trials from PRR have been performed with the eyes fixating straight ahead to compensate for the eye-centered encoding of stimuli. However, when the eyes are free to move, the efficiency of spatial decoding is about the same as with eyes fixed (Musallam et al. 2004). This curious observation raises several possibilities. It may be that compensations are made through updating or gain fields (Andersen et al. 1985b, Duhamel et al. 1992, Gnadt & Andersen 1988), the decoding algorithms may extract the regularities of hand-eye coordination, or PRR may change coordinate frames depending on the constraints of the task.
PMd uses a different coordinate frame for encoding reach targets from PRR. Whereas PRR uses predominantly an eye-centered coordinate frame (Batista et al. 1999, Cisek & Kalaska 2002, Pesaran et al. 2006b), PMd encodes simultaneously the target with respect to the eye (eye-centered), the target with respect to the hand (hand-centered), and the hand with respect to the eye (hand-in-eye) (Pesaran et al. 2006b). Rather than encoding the three variables in absolute spatial coordinates, it represents all three with respect to one another. This relative coding may be tailored to coordinating different body parts invariant of particular locations in the workspace. Area 5 may use a similar relative coordinate frame since it encodes reach targets with respect to the hand and the eye (Buneo et al. 2002), although the hand-ineye coding has yet to be tested.
This relative coordinate frame encoding has potential advantages for neuroprosthetic applications. It defines a “work space,” as mentioned above, which can be used for multi-effector movement tasks. Since the three relative frames are in extrinsic coordinates, it also allows inversions/transformations between coordinate frames (Pesaran et al. 2006b). Relative codes can reduce the accumulation of errors that may result from maintaining absolute encodings of spatial location (Csorba & Durrant-Whyte 1997, Dissanayake et al. 2001, Newman 1999, Olfati & Murray 2002).
Sensorimotor context determines movement goals. For instance, one may wish to reach to the location of a cookie, but reach away from the location of a bee. Most neural prosthetics research has used straightforward goal-directed movements toward a stimulus.
One method for studying context is the antimovement task (Figure 6A). The animal is cued to either move toward or away from a target (Boussaoud et al. 1993, Crammond & Kalaska 1994, di Pellegrino and Wise 1993, Everling et al. 1999, Gail & Andersen 2006, Georgopoulos et al. 1989, Gottlieb & Goldberg 1999, Kalaska 1996, Schlag-Rey et al. 1997, Zhang & Barash 2000). This task has typically been used for saccades and reaches to dissociate sensory signals from movement signals. For antimovement trials, if a neuron only codes the stimulus location, it is considered sensory; if it only encodes the movement direction, it is considered movement-related; and if it codes both, it is considered sensorimotor.
The pro-/antimovement task can be structured as a sensorimotor transformation with two opposing stimulus-response mappings. The executive function for applying the abstract rule for transformation may reside in the prefrontal cortex, premotor cortex, and basal ganglia (Boettiger & D'Esposito 2005, Nixon et al. 2004, Pasupathy & Miller 2005, Petrides 1982, Toni & Passingham 1999, Wallis et al. 2001, White & Wise 1999), although rule-based activity has also been reported in the PPC (Grol et al. 2006, Stoet & Snyder 2004). This rule can then act on the sensorimotor transformation process in PPC, premotor, and motor areas. We present here a recent example from PRR, since in this case neural decoding techniques were used as part of the analysis and shed light on how a CNP could determine the abstract rule and the appropriate stimulus-response mapping (Gail & Andersen 2006).
A pro-/antireach task was used for recordings from PRR (Figure 6A) that had three advantages: (a) Four different directions for pro and anti movements were used so the spatial tuning of the cells for both rules could be determined; (b) briefly flashed targets were used, and delays were interposed before the “go” signal to highlight cognitive-related activity; and (c) the task rule to be applied was provided each time at the beginning of the trial prior to the presentation of the target cue. This last feature of the task is important for examining whether the rule can be decoded, since often the features of the target dictate the rule and lead to a confounding in time of rule-based activity with other variables such as a sensory response to the target and the possible cancelling of an automatic plan toward the target in the anti-movement trials. Using this paradigm, it was found that the number of cells tuned only to the target was statistically insignificant, the number tuned only to the movement direction predominated (45%), and a small number were sensorimotor (7%). The sensorimotor cells were initially tuned to the stimulus location and then, during the delay period, were tuned to the direction of the planned movement. The fact that most cells were only tuned for movement direction rules out spatial attention as a contributing factor for those neurons.
Decode performance for task rule and direction (2 × 4, 12.5% chance) rose steeply after the brief (200 ms) presentation of the target cue, reaching 90% peak in 150 ms, and remained high during the variable delay (1–1.5 s) before the “go” signal. A transconditional decode revealed that the cue location was only weakly coded during the brief cue presentation, through the sensorimotor cells. PRR represented the movement plan from the end of the cue throughout the delay period (Figure 6C). These dynamics indicate that PRR immediately transforms the sensory representation into a movement representation without any residual memory of the location of the sensory signal. Interestingly, the rule could already be predicted above chance a short time before the presentation of the target, indicating that the rule was already explicitly represented in PRR before the appearance of the target cue (Figure 6B). In a separate neural network modeling study based on the above experiment, it was found that the context-based information could be integrated with the sensory target location through a classic gain field mechanism (Brozovic et al. 2007). It was suggested that this context modulation may result from top-down information originating from the frontal or parietal lobe.
Recent experiments show that rule-based sensorimotor transformations can also be extended to brain-control experiments (E.J. Hwang and R.A. Andersen, personal observation). The monkeys were able to move the cursor on a computer screen in the opposite direction to a cue using cell recordings from PRR and without any overt reaches. In a second experiment, the monkeys were trained to associate an arrow presented at the straight-ahead position on a computer screen with brain-control cursor movements in the direction the arrow was pointing (Hwang & Andersen 2008).
In the above reach tasks and brain-control tasks, the rules are applied to sensorimotor transformations. However, the fact that rules and their effect on neural transformations can both be decoded from the same population of cells in PPC suggests that other types of executive functions can be decoded in other brain areas. Executive rules that lie outside sensorimotor transformation include categorization, direction of spatial attention, and the formation of abstract concepts and thoughts.
Typically, neural prosthetic applications have relied on spiking activity of neurons as a signal source. Information from spikes is very precise and represents fundamental building blocks of the brain. Another signal that is interesting, particularly from the viewpoint of CNPs, is the local field potential (LFP) (Andersen et al. 2004b, Pesaran et al. 2006a). The LFP is determined by a number of factors, including the geometry and alignment of the sources, and thus can vary from region to region due to changes in local architecture. Also, the LFP is derived from multiple sources including synaptic potentials and action potentials and is summed over a volume of tissue that contains hundreds or thousands of cells. Still there are features of the LFPs that make them generally useful for prosthetic applications. They are often tuned, for instance, to the direction of planned reaches (Scherberger et al. 2005). Thus they can provide additional information to improve decoding when used in combination with spikes. They have a larger “listening sphere” than that of single cells. Electrode array implants generally have fixed geometries. The sampling of cells is hit or miss, and many electrodes will not be near neurons and will not yield recordings. Typically, to increase yield, lower impedance electrodes are used. This approach increases the listening sphere but also lowers the signal-to-noise ratio and makes single-cell isolation difficult. LFPs, on the other hand, sample from a large listening sphere and so the yield is much higher. Over time, the reliability of recording spikes often goes down, although the LFP signal remains largely unaffected. The basis of this decrease in performance is not completely clear, but it may include long-term encapsulation of the electrodes by glial scarring, which would be expected to have a larger effect on the more local signals of spikes compared to LFPs.
From the viewpoint of CNPs, LFPs have two primary contributions. The first is that it is actually easier to decode cognitive state from LFPs than from spikes. Recordings made from single electrodes in LIP during a memory saccade task showed that, on a single-trial basis, the direction of a planned saccade could be equally well decoded from single-cell spike activity and the LFP (Pesaran et al. 2002). On the other hand, the time of transition from planning to executing a saccade could be decoded from the LFP but not the spike recording. Interestingly, the decoding of direction was obtained from the higher-frequency (30–100 Hz) LFP spectrum and the cognitive state transition from the lower (0–20 Hz) spectrum. A similar result was found in PRR decoding from a population of PRR recording sites obtained on different days of recording (Scherberger et al. 2005). In this particular task, the monkeys made saccades or reaches on different trials so there were a variety of cognitive states that included baseline (fixating only), reach planning, saccade planning, reaching, and saccading. Whereas state could be decoded from LFPs and spikes, it took many more recording sites using spikes compared to LFPs to reach a similar performance. On the other hand, although direction could be decoded from both spikes and LFPs, the spike decodes were slightly better.
The second contribution of LFPs to CNPs is that they may provide complementary information due to some differences in source and as a result allow a broader view of overall network activity. In cortex, the largest class of neurons is the pyramidal (output) cell. Also, single-cell recording is biased toward larger cells, again the pyramidal neurons. Thus, spiking activity tends to represent the output of a cortical region (Figure 7). The LFP is not simply a sum of averaged spike activity but rather reflects, to a considerable degree, the mean synaptic activity that derives from inputs and intracortical processing (Buzsaki & Draguhn 2004, Logothetis et al. 2001). Thus, a component of LFP activity in the PPC may derive from inputs into the area (Figure 7). An efference copy from motor areas to PPC may explain why the LFP in PPC is so sensitive to the transition from planning to executing, since the PPC output is not thought to contribute to the execution of movements (Andersen & Buneo 2002). Recently, monkeys have been trained to generate a self-paced “go” LFP signal in PPC for brain-control experiments (Hwang & Andersen 2007). The animals were trained to generate the signature LFP signal associated with the state change from planning to executing actual reaches, an increase in lower frequencies and decrease in higher frequencies, without making any movements. The feasibility of a hybrid BMI system was also demonstrated in brain-control experiments in which the direction was decoded from the PRR spike activity while the state was decoded from the LFP recorded at the same site (Hwang & Andersen 2007). As mentioned above, the direction can be decoded from both spikes and LFPs in PRR. Additional brain-control experiments showed that using spikes and LFPs together increased decode performance compared to using either alone, demonstrating that combining information from the two sources can lead to better decodes (Hwang & Andersen 2009).
Most applications using brain-control tasks have shown a high degree of learning. This learning can be very rapid, over a few minutes to hours of training (Fetz 1969, Jarosiewicz et al. 2008, Moritz et al. 2008), over a period of days (Mulliken et al. 2008a), or even over a period of weeks (Carmena et al. 2003, Musallam et al. 2004, Taylor et al. 2002). This learning extends to LFPs (Hwang & Andersen 2007). Learning effects have been seen in many brain areas including the motor cortex, premotor cortex, and parietal cortex. To our knowledge, it is currently not understood which areas may show more plasticity or be better for learning particular categories of tasks.
A number of classes of decoding algorithms have been applied in brain-control experiments. Interestingly, many of these algorithms have parallels with brain function and may be successful, in part, because of these similarities. Bayesian decoding, which calculates maximum likelihood of an intended movement (Bokil et al. 2006, Gao et al. 2002, Scherberger et al. 2005, Shenoy et al. 2003), is one example. Recent modeling studies have suggested that cortical areas represent probability distributions and may use Bayesian inference for decision making (Beck et al. 2008). Population vector decoding has also been used in brain-control experiments (Taylor et al. 2002, Velliste et al. 2008). This algorithm was originally developed to explain how the direction of reaches is represented by populations of neurons in motor cortex (Georgopoulos et al. 1986). As mentioned above, Kalman filter decoding works well in PPC, and this cortical region has properties of state estimation that can be modeled effectively with Kalman filters (Mulliken et al. 2008a, Wu et al. 2004). Transitions between cognitive states show changes in brain activity, particularly reflected in the spectrum of LFPs but also in spike firing rates. State-space models such as finite state machines and Markov models have been successful in decoding state changes and may reflect the states and transitions between states in brain activity (Shenoy et al. 2003). Decoding has been improved by taking into account correlations between spike trains (Abbott & Dayan 1999, Averbeck et al. 2006, Brown et al. 2004, Nirenberg & Latham 2003) and the temporal regularities in responses (Musallam et al. 2004). These decoding methods that take into account correlations and dynamics may be exploiting underlying temporal coding strategies used by the brain.
The applications of CNPs are very wide ranging and rely on the decoding of signals that are neither motor execution nor sensory signals but rather rely on internal cognitive states. Although several specific examples are given in this review, in principle all cognitive states likely can be decoded with the appropriate recording technologies, placement of electrodes, and decoding algorithms. Spike activity is a major source of signal, but LFPs are also particularly useful for determining cognitive state. Extensions of signal analysis may include the measure of spike-field coherence between cortical areas, which may provide additional insights into the processing of cognitive functions, and their decoding, across cortical circuits (see Future Issues, below). CNPs may also be extended to patients' volitional control of brain stimulation, which can be applied to movement disorders, depression, epilepsy, and other brain diseases that may benefit from neural stimulation. Clearly, the future is bright for CNPs and their future application to assisting patients with brain disorders. An added benefit is that research in CNPs will continue to uncover the neural basis of cognitive functions through the basic research that forms the foundations of CNPs as well as the insights that are afforded by the operation and performance of CNPs.
We thank the National Institutes of Health, the Defense Advanced Research Projects Agency, the Boswell Foundation, and the McKnight Foundation for supporting this research. We also thank Drs. Eb Fetz and Michael Campos for discussions during the preparation of this review. We thank Dr. Viktor Shcherbatyuk, Tessa Yao, Kelsíe Pejsa, and Nicole Sammons for technical and editorial assistance.
The authors are not aware of any biases that might be perceived as affecting the objectivity of this review.