|Home | About | Journals | Submit | Contact Us | Français|
In tasks of associative learning, animals establish new links between unrelated items by using information about trial outcome to strengthen correct/rewarded associations and modify incorrect/unrewarded ones. To study how hippocampal neurons convey information about reward and trial outcome during new associative learning, we recorded hippocampal neurons as monkeys learned novel object-place associations. A large population of hippocampal neurons (50%) signaled trial outcome by differentiating between correct and error trials during the period after the behavioral response. About half these cells increased their activity following correct trials (correct-up cells) while the remaining half fired more following error trials (error up cells). Moreover, correct up cells but not error up cells conveyed information about learning by increasing their stimulus-selective response properties with behavioral learning. These findings suggest that information about successful trial outcome conveyed by correct up cells may influence new associative learning through changes in the cell’s stimulus-selective response properties.
Findings from a convergence of human and animal studies show that the hippocampus is important for the successful acquisition of new declarative memories for facts, events and relationships (Eichenbaum and Cohen, 2001; Squire et al., 2004). Consistent with this idea, numerous previous studies have shown that hippocampal neurons signal the acquisition of new associations with striking changes in their firing rate (Berger et al., 1976; Cahusac et al., 1993; Frank et al., 2004; Fyhn et al., 2002; McEchron and Disterhoft, 1999; Wirth et al., 2003). While these studies focused on changes in neural activity during the stimulus or delay periods of the tasks, a key requirement during the acquisition of new associations is the ability to use information about behavioral outcome typically signaled by the delivery or withholding of reward to strengthen correct performance or modify errors on subsequent trials. Consistent with this idea, previous studies have described medial temporal lobe neurons that signal reward-related as well as outcome- related information.
Some of the earliest descriptions of rat hippocampal activity described cells that responded to either reward delivery or lack of reward (O’Keefe, 1976; Rank, 1973). Neurons that respond to reward delivery have also been reported in the monkey hippocampus (Watanabe and Niki, 1985) and entorhinal cortex (Sugase-Miyamoto and Richmond, 2007). Smith and Mizumori (2006) showed that some rat hippocampal neuron signaled reward delivery in one task but not another related task performed in the same environment. Other studies showed that reward can modulate various stimulus-evoked or spatial signals in the medial temporal lobe. For example, several studies have reported that reward can change the spatial selectivity of place cells in the hippocampus (Breese et al., 1989; Hollup et al., 2001; Holscher et al., 2003; Kobayashi et al., 1997; Lee et al., 2006). Other studies in rats, in contrast, failed to find a relationship between reward magnitude and place cell activity (Tabuchi et al., 2003; Wiener et al., 1989). Studies in monkeys reported that sensory responses in the perirhinal cortex can come to signal the reward value of the stimulus (Liu and Richmond, 2000; Mogami and Tanaka, 2006) and monkey hippocampal cells have been reported to signal spatial-reward associations (Rolls and Xiang, 2005).
Other studies suggest that hippocampal neurons signal information about trial outcome. For example, Watanabe and Niki described a small number of cells in the monkey hippocampus that differentiated between correct and error trials during and after the response period of a spatial delayed response task. Some cells responded to the delivery of juice following a correct response, but not to a random drop of reward. Other cells showed differential firing following an error response or when the juice reward was omitted. Similar responses have been reported in the prefrontal cortex (Niki and Watanabe, 1979; Rosenkilde et al., 1981; Tsujimoto and Sawaguchi, 2004; Tsujimoto and Sawaguchi, 2005; Watanabe, 1990), orbitofrontal cortex (Thorpe et al., 1983) and habenula (Matsumoto and Hikosaka, 2007). These outcome-selective cells have generally been interpreted as signaling the consequence of the animal’s response, though their contribution to learning has not been examined. To further explore both reward-related and outcome-related responses in the monkey hippocampus, we recorded neural activity during an associative learning task where information about trial outcome plays a key role in the learning process. We used a variant of an object-place associative learning task known to be highly sensitive MTL damage both in humans (Cave and Squire, 1991; Milner et al., 1997) as well as in monkeys (Bachevalier and Nemanic, 2008; Gaffan and Parker, 1996; Malkova and Mishkin, 2003). Here, we report that fifty percent of the isolated hippocampal cells differentiated between correct and error responses during the time period following the behavioral response (outcome-selective cells). To further evaluate the information carried by these hippocampal outcome selective cells, we characterized their sensitivity to various reward manipulations. We also asked how this sensitivity to trial outcome might contribute to the learning process.
Two macaque monkeys (one rhesus and one bonnet macaque) performed an object-place associative learning task (Figure 1A) in which they learned to associate different object-place combinations with either an early or late bar release response. Animals initiated each trial by holding a bar and fixating a central fixation point for 500 ms. They were then shown one of 4 possible object place combinations for 500 ms. Each combination was composed of one of two possible visual objects in one of two possible spatial locations (Figure 1C). Both the objects and the spatial locations changed daily. Following a 700 ms delay interval when nothing but the fixation point was shown on the screen, the animals could make a bar release response during the presentation of either an orange circle presented for 500 ms (early response), or during the 500 ms presentation of a green circle shown immediately after the orange circle (late response). If the response was correct, a “positive” auditory feedback tone was played 15 ms (+/−1ms) after the bar release. Except for “omit reward” trials (see below) the auditory tone served as a reliable signal of the future delivery of reward. For the vast majority of trials, correct responses were rewarded with between two to four juice drops starting 30 ms (+/−1ms) after bar release and ending 933 ms (+/−1ms) after bar release if 4 drops were given. With trial and error, animals learned to associate each object-place combination with either the early or a late bar release response. For correct trials, juice reward was followed by a 2000 ms inter-trial interval before the initiation of the next trial. For error trials, a 2000 ms inter-trial interval started immediately after an incorrect bar release response. Fixation was required from the time the animal initiated the trial until the time of the bar release response.
To characterize behavioral learning of individual object-place combinations, we used a Bayesian state-space model for analyzing learning of multiple problems simultaneously (Smith et al., 2007; Supplementary Methods 1 and Supplementary Figures 1 and 2). Of the 156 sessions, there were 55 sessions during which animals did not learn any association to criterion. For the remaining sessions, animals learned 1 of 4 (17 sessions), 2 of 4 (20 sessions), 3 of 4 (17 sessions) or 4 of 4 (17 sessions) associations to criterion. Learning criterion was defined as the trial number when the estimated probability correct performance was significantly above chance (Smith et al., 2007; Supplementary Methods 1). Monkey M required an average of 68.60 +/− 4.34 total (i.e., consecutive) interleaved trials to learn an individual association. This corresponds to and average of 14 +/− 0.75 trials for each individual association. Monkey E required an average of 82.4 +/− 5.08 total interleaved trials to learn an individual association, and an average of 17 +/− 4.02 trials for each individual association.
In order to administer several key control experiments, animals also performed a “fixation only” task given in blocks of trials at the end of the recording sessions (Figure 1B). The fixation only task was identical to the object-place task except that animals were only required to maintain fixation on the central fixation cross through the object-place presentation and delay periods of the trial to receive a reward (i.e., no associative learning required). Animals successfully completed an average of 73.57%, +/− 1.52 of the fixation only trials (i.e. completed trial without break of fixation). In a subset of 30% of the fixation only trials, we examined the effect of omitting reward following successful fixation (omit reward trials; the auditory tone was played on these omit reward trials). On other fixation only trials, we tested the effect of giving random, unexpected reward at variable time points throughout the fixation only trials (random reward control trials).
We recorded the activity of 165 hippocampal neurons from 2 monkeys (109 from monkey M and 56 from monkey E) during learning of novel object-place-response associations. Recordings were made throughout the full anterior-posterior extent of the hippocampus and based on MRI reconstructions, appeared to include neurons from all hippocampal subdivisions (Figure 1D). We did not attempt to select cells based on their firing properties and instead recorded from the first well-isolated hippocampal cells encountered. To examine how cells signaled information about trial outcome, we focused on neural activity during the 2000 ms following the bar release response. We chose this time period because it was the longest common post-response time period available for analysis for both correct and error trials. For correct trials, this period included both the time period when reward is being given (from 30 ms following bar release to 933 ms following bar release if 4 drops of reward were given) as well as the initial part of the inter-trial interval. For error trials, this 2000 ms period included the entire inter-trial interval period. Our initial analysis revealed a heterogeneous mix of both sustained and more transient responses during this 2000 ms period. In order to characterize these responses with higher temporal resolution, we split the post-release period into two consecutive 1000 ms time periods. Compared to the baseline firing rate period, defined as activity during the 500 ms fixation period at the start of each trial, we found 77% (127 of 165) of the hippocampal neurons responded with significant activity in either one or both of the 2 consecutive 1000 ms periods analyzed (t-test, p<0.05). Moreover, 83 of the 127 responsive neurons (65%, of the responsive cells or 50% of the total population) were outcome-selective in that they differentiated between correct and error trials (t-test comparing responses on correct vs. error trials, p<0.05).
Further analysis revealed two distinct subpopulations of outcome-selective neurons. The first population (30 of 83 responsive neurons), termed correct up cells increased their activity on correct trials compared to error trials in the first (n=11), second (n=7) or both 1000 ms periods (n=12) following bar release (Figure 2A, and Supplementary Figures 4A and C). A sliding window calculation (See Experimental Procedures) revealed that the correct up cells started differentiating correct from error trials 342 +/− 75 ms after bar release (p<0.05, t-test). A second subpopulation termed “error up” cells (38 of 83 responsive neurons) increased their firing rate on error trials relative to correct trials during the first (n=3), second (n=15) or both 1000 ms periods (n=20) after bar release. Among error up cells, 14 cells exhibited a significant decrease in response following a correct trial, though the response following errors was significantly above baseline. Error up cells were also characterized by clear motor-related activity that peaked at the time of bar release (Figure 2B, Supplementary Figures 4B and 4D; See Supplementary Information 2 and Supplementary Figure 5 for a further discussion of the motor-related activity of the error up cells). Error up cells differentiated between correct and error trials starting 489 +/− 63 ms after the animal’s bar release response (p<0.05, t-test). Smaller populations of cells decreased their activity on correct trials (“correct down” cells; n=5) or on error trials (“error down” cells; n=7) relative to baseline, or responded significantly to both correct and error trials relative to baseline activity (“mixed” outcome cells”; n=3). Because of the relatively small number of cells in these latter three categories, they were not examined further. An analysis of the ratio of spike height vs. spike width showed that the majority of our isolated neurons were putative pyramidal cells and that correct up and error up cell categories included both putative principle cells as well as putative fast spiking neurons (Supplementary Information 1 and Supplementary Figure 3).
Given that the correct up signal is not expressed until well after the first drop of juice is delivered 30 ms after the bar release (Figure 2A), one possibility is that these cells may simply provide a delayed signal of reward delivery (Ranck, 1973; Smith and Mizumori, 2006). To test the dependence of the correct up signal on aspects of reward delivery, we examined the effect of several different reward manipulations. To test whether correct up cells respond to delivery of reward per se, we examined responses on a subset of both standard and fixation only trials in which random rewards were given. Specifically, we compared neural activity during the 500 ms following the first reward drop in correct trials, to the activity during the 500 ms following a random reward drop using a t-test (p<0.05). None of the correct up cells tested in these conditions (n = 8) showed a reliable response to random reward delivery (Figure 3A).
While the correct up cells did not respond to random drops of juice, we next tested the possibility that they might be sensitive to the timing of reward delivery. To address this possibility, we examined neural activity on standard trials in which the delivery of reward was delayed from 30 ms to 518 ms after the bar release response but the auditory feedback signal continued to be given immediately (i.e., 15 ms +/−1 ms) following a correct response as in standard conditions (See Experimental Procedures). Following several days of habituation to the delayed reward delivery, neural activity was recorded. We found that correct up cells continued to respond with the same latency to the auditory feedback sound even if the reward itself was delayed by 488 ms (Figure 3B). Next, we eliminated the feedback sound for these delayed reward trials and we saw a significant increase in the latency of the differential correct/error signal relative to the standard condition (Figure 3B). These findings suggest that the response latency of the correct up cells signals is not controlled by the timing of reward delivery. Instead, correct up cells appear to be sensitive to information about trial outcome whether it’s signaled by an auditory feedback signal or the delivery of reward.
To determine if correct up cells differentiate between correct and error trials for other tasks, we examined the responses of these cells on the fixation only task. Unlike the differentiation observed during the object-place trials, the correct up cells tested in fixation trials (n =12) did not discriminate between correctly executed fixation only trials and un-rewarded break fixation trials (two sample t-test, p>0.05). This suggests that the correct up cells signal correct/rewarded outcome more specifically during a learning context and do not convey general information about successful trial completion.
Because our previous studies showed that a subset of hippocampal neurons could change their firing rate correlated with new associative learning (Wirth et al., 2003; Yanike et al., 2008), we next asked if the magnitude of the correct up signal might change over the course of the learning session. To address this question, we used a one-way ANOVA with time period (i.e., early middle and late periods of the session) as a main factor to examine the amplitude of the correct up signal over time. We analyzed the first 1000 ms following bar release and the second 1000 ms following bar release separately and found no difference in the amplitude of the correct up responses during either time bin over the course of the learning session. For the first 1000 ms, the average rate for each consecutive third of the session was: 14.41+/−3.26; 14.18 +/− 3.06; 14.90 +/−3.06 (F[2,87]=0.01, p=0.98). For the second 1000 ms time bin following bar release the average rate for each consecutive third of the session was 14.70 +/−2.87; 15.09 +/− 2.79; 15.85 +/−2.85 (F[2,87]=0.04, p=0.95). Thus, the response of the correct up cells during the reward and ITI periods of correct trials remained stable over the course of learning.
Error up cells (38/83 outcome selective cells) increased their activity on error trials relative to correct trials in the 2000 ms following bar release for the object-place association task. To test the hypothesis that the error up cells signal the absence of a possible reward, we examined neural activity during correctly executed fixation only trials where reward was occasionally omitted (n = 13, omit reward trials). We hypothesized that if the error up cells provided a general signal of the absence of a possible reward, we should see a similar increase in activity following the omit trials as we saw on the error trials. Consistent with this prediction, we found that the error up cells increased their activity on omit reward trials relative to rewarded fixation only trials (n=13, t-test, p<0.05 Figure 4A).
If the error up cells signal the absence of a possible reward then we predicted that they might also be sensitive to manipulations of the timing of the reward delivery. To address this question, we examined the response of error up cells on trials in which we delayed the delivery of reward from 30 ms to 518 ms following bar release (no auditory feedback signal given; n = 19 error up cells; no trials were available with the auditory feedback together with delayed reward). Following habituation to the delayed reward signal, we recorded the activity of error up cells and found that delaying the reward resulted in a significant increase in the latency of the differential correct/error signal from the standard object-place trials with no delay in reward delivery (mean latency with no delay = 489 ms +/− 63, n = 19; mean latency with delay = 800 +/−66, n =19; two sample t-test, p <0.05). Thus, errors up cells are sensitive to the latency of reward delivery and may use this information as a cue to signal the absence of a possible reward.
To determine if error up cells differentiate between correct and error trials on other tasks, we compared activity during the object-place associative task to activity during the fixation only task. Unlike the correct up cells, the error up cells exhibited a similar response in both tasks, increasing their response following erroneous break fixation trials relative to correctly executed responses (t-test, p<0.05). The cells discriminated between break fixation trials and correctly completed trials 426 +/− 86 ms after the end of the trial when reward was not delayed (n= 9, shown in Figure 4B) and 783 +/− 81ms (n=13) after the end of the trial when reward was delayed (n=13, data not shown). These findings support the idea that error up cells provide a general signal of the absence of a possible reward in multiple task situations.
Similar to the correct up cells, we also asked if the magnitude of the neural responses of the error up cells changed over the course of the session. To address this question, we used a one way ANOVA with the time period as a main factor to compare the amplitude of the error up signal averaged over two or three time periods of the session (i.e. early, middle and late for 29 sessions, and early and middle for 7 sessions in which there were no more error trials during the last third of the session) for the population data. We analyzed the first 1000 ms following bar release and the second 1000 ms following bar release separately and found no difference in the amplitude of the correct up responses during either time bin over the course of the learning session. The average firing rate for each consecutive third of the session for the first 1000 ms following bar release was: 10.87 +/−1.81; 11.7 +/− 2.32; 9.69 +/− 1.95 (F[2,98]=0.32; p=0.72). During the second 1000 ms time bin following bar release the values were: 11.88 +/− 2.15; 11.93 +/− 2.29; 11.11 +/− 2.23 (F[2,98]=0.15; p=0.85). Thus, the response of error up cells during the reward and ITI periods of error trials remained stable over the course of the learning session.
While correct up and error up cells both convey information about trial outcome, another important question concerns how these populations of cells might use this information about trial outcome to influence new learning of object-place associations. Numerous pervious studies have shown that neurons in both the medial temporal lobe (changing cells: Wirth et al., 2003) as well as in cortex (Baker et al., 2002; Kobatake et al., 1998; Sigala et al., 2002) change their stimulus-selective responses in parallel with learning. Given these previous data, we tested the hypothesis that the correct up or error up cell populations might also convey information about learning with shifts in their stimulus-selective response properties. To address this question, for sessions during which significant behavioral learning was seen (21 correct up cells and 24 error up cells), we calculated neural selectivity of the population of correct up and error up cells during the cue and delay periods of the task both before and after behavioral learning was achieved. We also examined the selectivity of a control population of 82 non-outcome selective cells (including 38 non responsive cells and 44 responsive but not outcome selective cells) in the same manner. To determine whether the shifts in selectivity were specific to learning, we calculated selectivity on sessions during which no learning occurred (9 correct up cells, 14 error up cells and 32 non-outcome selective cells) for the first 60 trials and the remaining trials (60 corresponds to the average number of trials to learn). We used a two-way ANOVA applied to the selectivity measures during the cue and delay periods of the task before and after learning (two levels of repeated measures) using cell category and learning as the main 2 factors. The ANOVA revealed a significant interaction between the cell category and learning status of the selectivity measures before and after learning (F[2,144]=4.5. p=0.0012; Figure 5A and 5B). Post hoc comparisons showed there was a significant increase in selectivity after learning relative to before learning in correct up cells only in sessions where significant learning was found (Neuman-Keuls; p<0.001). In contrast, no change in selectivity was seen in either the error up cells or the control non-outcome-selective cells. Differences in excitability between learning and no learning sessions could not explain the striking increase in selectivity seen in the correct up cells (Supplementary Information 3). We also asked if there were learning-related changing cells in this hippocampal population (Wirth et al., 2003; Yanike et al., 2008). We identified a subset of hippocampal changing cells during object-place associative learning task, but showed that the changing cells (that also exhibit increased selectivity with learning) were not driving this increase in selectivity exhibited by the correct up cells (Supplementary Methods 2 and Supplementary Information 4). To better illustrate the distribution of selectivity in these different populations of cells, we calculated the differential selectivity between trials before and after learning (Figure 5C and 5D). The population of correct up cells also exhibits a wider distribution of the selectivity differences with more cells showing increases relative to the other populations of cells (F-test, p<0.01). Thus, these findings suggests that the correct up cells but not the error up cells or the control cells convey information about learning by shifting their stimulus-selective response properties during the cue and delay periods of the task in parallel with behavioral learning.
In this study we examined the outcome-related signals of hippocampal neurons as monkeys learned new object-place associations. We identified two distinct populations of outcome selective cells. Correct-up cells signalled correctly executed object-place trials but not correctly executed fixation only trials and did not respond to random rewards. Thus correct up cells appear to provide a task-specific signal of correct trial outcome. One caveat is that because we did not examine the responses of correct up cells in object-place learning sessions with no reward delivery, we cannot rule out the possibility that the correct up cells respond to the combination of successful outcome coupled with liquid reward. Correct up cells also convey information about learning by increasing their stimulus-selective response properties in parallel with behavioral learning. Error up cells, by contrast, increased their activity following error trials relative to correct trials. In contrast to the task-selective responses of correct up cells, error up cells appeared to provide a more general signal for the absence of a possible reward, responding similarly on incorrect object-place trials, correctly executed omit reward trials as well as for break fixation trials. In contrast to the correct up cells, error up cells did not change their stimulus-selective response properties with learning. Both the correct up and error up signals remained stable throughout the recording session. During associative learning, monitoring information about both failure and success is highly informative. Our findings suggest that hippocampal correct up and error up cells serve distinct kinds of monitoring functions. These findings also provide insight into how information about trial outcome may influence new associative learning.
Outcome-selective cells made up 50% of the isolated hippocampal cells, representing one of the strongest selective and task-related signals observed in the primate hippocampus. For example, most previous studies have reported only modest proportions of selectively responding hippocampal cells on comparable computer-based visual memory tasks (Typically between 4%–22% of the population; Cahusac et al., 1989; Miyashita et al., 1989; Rolls et al., 1989; Rolls et al., 1993; Rolls and Xiang, 2005; Xiang and Brown, 1999) with few studies reporting >30% selectively responding hippocampal cells (Riches et al., 1991; Wilson et al., 1990). These findings suggest that episodic-like information about trial outcome is much more prominently expressed in the primate hippocampus than previously appreciated.
Our analysis of correct-up cells confirms and extends previous reports from the literature. For example, early studies reported that hippocampal cells can respond to reward in situations without explicit task requirements (O’Keefe, 1976; Rank, 1973) or respond to reward in certain tasks but not others in the same environment (Smith and Mizumori, 2006). Niki and Watanabe (1985) described small numbers (<6% of the responsive cells) of hippocampal cells that responded to the presentation of juice per se as well as cells that responded selectively to the delivery of juice following a completed delayed response trial but not to random juice delivery. Similar signals to both reward per se as well as task-selective reward signals have also been described in the dorsolateral prefrontal cortex (Niki and Watanabe, 1979; Rosenkilde et al., 1981; Watanabe, 1990), the cingulate cortex (Niki and Watanabe, 1979) as well as the orbitofrontal cortex (Thorpe et al., 1983). Recent studies report prefrontal cells can also signal reward outcome together with a variety of other measures including the direction of an immediately preceding saccade (Tsujimoto and Sawaguchi, 2004), whether the target signal was mnemonic or sensory (Tsujimoto and Sawaguchi, 2005) or the timing of the reward delivery (Tsujimoto and Sawaguchi, 2005; Tsujimoto and Sawaguchi, 2007). The large proportion of correct up cells in our population relative to the original report by Niki and Watanabe (1985) may reflect either the more prominent role of the hippocampus in object-place associative learning compared to the spatial working memory task used by Niki and Watanabe (1985), the stronger requirement for outcome monitoring in the former relative to the latter task, or both elements in combination.
Perhaps the most striking feature of the correct up cells is that they increase their stimulus-selective response properties with learning. These findings taken together with our analysis of the changing cells in this hippocampal population (Supplementary Information 4) provide new insight into the range of hippocampal learning-related signals seen during associative learning tasks. We not only showed that correct up cells increase their stimulus-selective responses with learning, but a robust changing cell population also increased their stimulus selective responses with learning and partially overlapped with both the correct up and error up populations. These findings suggest that changing cells and correct up cells may be part of a continuum of learning-related hippocampal cells that use changes in their stimulus-selectivity as a common signal to indicate that learning has taken place. These changes in stimulus-selectivity may participate in learning by making hippocampal cells more sensitive to the stimuli being used to solve the task. Moreover, the correct up cell population taken together with the changing cell population increase substantially the proportion of identified hippocampal associative learning related signals seen in this task (18% changing cells alone, 32% changing cells and correct up cells). Because the correct up cells are sensitive to both trial outcome and convey information about learning, they may also provide a way for information about successful trial outcome to influence learning.
The second major hippocampal cell category identified in this study was the error up cells. Error-related signals have also been described in several other brain areas as well as in the hippocampus. Watanabe and Niki (1985), described a small number of error up-like cells that increased activity following error trials as well as on trials when reward was omitted during a spatial delayed response task. In rodents, Deadwyler and colleagues (1996) used canonical discriminant analysis to extract the sources of variance from hippocampal neurons during a spatial delayed non-matching to sample task. One of the significant sources of variance was the presence of errors associated with the longer delay intervals of the task. Thus, this signal resembles the error up cells reported in the monkey, though the error signal in the rodent was mainly associated with the reward period itself and disappeared during the inter-trial interval period which is different from the error up signal that started relatively late in the reward period and continued into the inter-trial interval period of the task. Error up-like cells have also been described in the monkey dorsolateral prefrontal cortex (Niki and Watanabe, 1979; Rosenkilde et al., 1981), the orbitofrontal cortex (Thorpe et al., 1983) and the habenula in both monkeys (Matsumoto and Hikosaka, 2007) and humans (Ullsperger and von Cramon, 2003).
A convergence of studies from EEG studies (Ullsperger and von Cramon, 2006), fMRI studies (Carter et al., 1998; Holroyd et al., 2004; Mars et al., 2005; Ullsperger and von Cramon, 2003), as well as single unit physiologoical recordings in humans (Williams et al., 2004) have implicated the anterior cingulate cortex (Ito et al., 2003; Shima and Tanji, 1998) in monitoring of performance. The striking error up signals in the monkey hippocampus suggest that the hippocampus may be part of a large network of brain areas involved in monitoring negative outcome. However, these different areas may use the information about outcome/error monitoring for different purposes. For example, while evidence suggests that the anterior cingulate may use the negative outcome information to shift motor strategies (Shima and Tanji, 1998), prefrontal cortex may use error information in its role in mediating arbitrary cue-response associations including reversal learning (Asaad et al., 1998; Pasupathy and Miller, 2005). The orbitofrontal cortex may be involved in coding the emotional or hedonic aspects of negative outcome for use in decision making (Kringelbach, 2005; Petrides, 2007). The error up signal in the hippocampus may provide episodic information about the unfolding of the relevant features of the associative learning task (Eichenbaum and Cohen, 2001; Schacter and Tulving, 1982). Thus, while many areas signal information about error/negative outcome, this information may be used in different ways by different brain systems.
One possible source of reward/outcome-related information that may inform both the correct up and error up cells is the direct projections from the dopaminergic cells of the ventral tegmental area (VTA) that project directly to the hippocampus (Amaral and Cowan, 1980; Lewis et al., 2001; Samson et al., 1990). Indeed, a growing body of studies in animals and humans emphasize the importance of the functional interaction between the VTA and medial temporal lobe in learning and memory. In animals, damage to the hippocampal dopaminergic system causes spatial memory impairments (Gasbarri et al., 1996). Induction of long term potentiation in area CA1 of the hippocampus is modulated by dopaminergic inputs from midbrain neurons (Frey et al., 1990; Frey et al., 1991; Frey and Morris, 1998; Li et al., 2003) as well as by local stimulation of dopamine receptors within the CA1 area (Swanson-Park et al., 1999). fMRI studies have shown that joint substantia nigra/VTA and hippocampal activation is associated with successful long-term memory formation (Adcock et al., 2006; Schott et al., 2006; Wittmann et al., 2005). In our learning paradigm, reward for correct responses early in the session is likely to elicit VTA firing because of the unexpected nature of the reward. But later in the session when a reward can be predicted by a learned object-place cue, it is likely that the VTA would no longer respond to primary reward, but instead, would become responsive to the predictive object-place stimulus. Thus, while information about reward and reward prediction error from the VTA may influence the correct up and error up cells early in learning, it is likely that other reward/outcome sensitive afferents to the medial temporal lobe including possibly orbitofrontal cortex (Thorpe et al., 1983) dorsolateral prefrontal areas (Niki and Watanabe, 1979; Rosenkilde et al., 1981; Tsujimoto and Sawaguchi, 2005) cingulate cortex (Ito et al., 2003; Shima and Tanji, 1998) and the habenula (Matsumoto and Hikosaka, 2007; Ullsperger and von Cramon, 2003) also influence the hippocampal outcome-selective cells, particularly later in the learning process.
The findings reported above lead to two main conclusions. First, we show that during object-place associative learning, a surprisingly large proportion of hippocampal cells are outcome selective. These findings suggest that episodic-like representation of trial outcome may be a much more prominent signal in the hippocampus than previously appreciated. Second, we showed that correct up cells, like changing cells previously reported in the hippocampus (Wirth et al., 2003; Yanike et al., 2008) change their stimulus-selective response properties in parallel with learning. These findings not only expand the categories of associative-learning related signals in the hippocampus, but they suggest a way that information about successful trial outcome conveyed by correct up cells may influence new associative learning. Further studies will be needed to explore the specific mechanisms underlying this relationship.
All procedures and treatments were done in accordance with NIH guidelines. One male rhesus macaque monkey (14.7 kg, monkey 1) and one male bonnet macaque (8.1 kg, monkey 2) were used for these experiments. The position of the recording chamber for each animal was calculated using stereotaxic coordinates derived from the animals’ individual MRI images (Fig. 1C). Once the chamber was in place, the MRI images allowed us to estimate the anterior-posterior, medial-lateral and dorsal-ventral recording locations within the hippocampus in each animal for the entire experiment.
During training, animals were seated in a primate testing chair (Crist Instruments) 53.96 centimeters away from a 19 inch CRT monitor. We monitored and controlled the behavioral task experiments using CORTEX software (NIMH Laboratory of Neuropsychology, Bethesda, MD, USA). Eye movements were monitored using a non-invasive infrared eye tracking system (Iscan Inc., Burlington, MA, USA). Animals were trained on the object-place association task with fixation control described in the text and illustrated in Figure 1A. Each day animals learned which object-place combination was associated with one of two possible bar release responses (i.e., early or late). The rewarded response (i.e., early or late) was counterbalanced across the object-place combinations such that if object A in place 1 is early, then object A in place 2 is late, object B in place 1 is late and object B in place 2 is early (Figure 1C). Animals were required to hold their gaze within a 5 × 5 degrees virtual window around the fixation point from trial initiation until the bar release response. For most trials during both training and the recording sessions, between 2 and 4 drops of juice were given on a random reward schedule with the first drop given 30 ms after bar release and the 4nd drop given 933 ms after bar release. For a small number of trials during the recording session only 1 to 2 reward drops were given such that they occurred 30 ms and 331 ms respectively after bar release. Recording experiments started once animals could consistently learn one set of new object-place associations each day though learning during recording session varied from one to four new associations learned per session.
In addition to the standard object-place trials shown in Figure 1A, we also used 4 different control experiments to test various aspects of the neural response. First, to probe the timing of the outcome-related activity, we delayed the delivery of the reward after the animal’s bar release from 30 ms to 518 ms following bar release on the standard object-place learning trials (delayed reward control trials). The auditory tone played 15 ms (+/−1ms) after bar release was maintained in these trials. We first “trained” animals on this delayed reward version of the task so they would know what to expect and then used this version of the task in a continuous block of 14–16 sessions of recording. Thus, animals could anticipate exactly when the delayed reward would be delivered. In another manipulation we removed the auditory tone while maintaining the delayed reward delivery. Second, to test how sensitive the outcome-related activity was to particular behavioral tasks, in a second control experiment, animals performed a “fixation only” task in which monkeys were required to maintain fixation throughout the cue and delay periods of the task to receive reward (Figure 1B). Third, to test the effect on unexpected omissions of reward on neural activity, in an average of 30% of the fixation only trials, we omitted reward. Fourth, to test the effect of random, unexpected reward on the reward-ITI related activity, in some of the fixation only control trials, juice reward was sometimes given randomly at variable time points during the object-place presentation or the delay periods of the task. Unexpected reward could also be given during the standard object-place trials during either the object-place presentation or the delay period of the task. Overall, random reward was given in 25% of the fixation only or standard trials.
To record single unit activity, we used individual tungsten microelectrodes (Catalog #UEWLEFSM4N1E, FHC, Bowdoinham ME, USA) advanced into the brain with a hydraulic microdrive (Naguchi, JP). The microelectrodes were inserted through a stainless steel guide tube positioned in a Crist grid system (Crist Instruments, Damascus, MD; USA) on the recording chamber. Twenty one recording sessions used an online spike-sorting system (MSD, Nazareth, Israel) to isolate the activity of individual neurons throughout the recording areas. The MSD system uses a template matching system to isolate individual spikes. The remaining 135 sessions used a Plexon recording system with on-line spike sorting system (Plexon Inc., Dallas, Texas). All the neurons collected with the Plexon system, were later resorted using the offline sorter software (Plexon Inc., Dallas, Texas). We used a semi-manual sorting method such that 3 selected parameters (Principal components, height, and time) allowed us to separate the units from background activity and yield well isolated clusters. The stability of these clusters was carefully monitored over time and recording was terminated if the neuron’s activity merged with background due to recording instability. We monitored for instability of isolation manually, using the general rule that if the cell showed less than a three to one signal to noise ratio, we considered it unstable.
All neuronal and behavioral data were analyzed with custom written Matlab (MathWorks, Natick, MA, USA) programs. Statistical analyses were done using either the statistics toolbox in Matlab or Stastistica (StatSoft, Tulsa, OK, USA). Baseline activity was defined as the average activity during the 500 ms fixation period. In this study, we focused on neural activity for the 2000 ms immediately after the bar release response. For all of the neural analyses, the responses were normalized by subtracting the baseline activity from either the response or the ITI interval of the task.
We used the activity averaged for 50ms time bins slid by 5ms steps for correct and error trials, starting after the bar release (0 to 50; 5 to 55; etc). Using two sample t-tests (p<0.05), we defined the outcome response latency as of the beginning of the first time bin of a series of 10 consecutive bins for which activity was significantly different on correct trials and error trials.
We separated each recording session into the trials before and after behavioral learning of the first object-place association occurred. When no behavioral learning occurred, we subdivided the session into the first 60 trials and all subsequent trials. Sixty trials correspond to the average number of trials for learning the first association. We used the activity collected during the cue (0 to 500ms following cue onset) and the delay (0 to 700ms following cue offset) periods of the task for each of the 4 object place associations. For each cell, we calculated a selectivity index (Moody et al., 1998; Wirth et al., 2003) both before and after learning. The selectivity index measures the cell’s stimulus selectivity to the 4 possible object-place combinations and was based on the following formula:
where n is the total number of object-place combinations, λi is the firing rate of the neuron to the ith object-place combination and λmax is the neuron’s firing rate to the object-place combination that elicited the maximum firing rate. Thus, if a neuron responds to only one object-place combination and not to any the others, the SI would be 1. If the neuron responded identically to all object-place combinations, the SI would be 0.
Using the selectivity indexes obtained for each cell before and after learning for the cue and delay periods, we performed a two way ANOVA with two levels of repeated measures. We used cell group and learning as the main factors and before versus after learned and trial period as the two levels of repeated measures. We then performed Newmann-Keuls test for post-hoc two by two comparisons.
This work was supported by NIH grants MH48847, DA015644, MH59733 and MH071847, and a grant from Fondation pour la Recherche Médicale to S.W.
Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.