|Home | About | Journals | Submit | Contact Us | Français|
The amygdala is critical for associating predictive cues with primary rewarding and aversive outcomes. This is particularly evident in tasks in which information about expected outcomes is required for normal responding. Here we used a Pavlovian over-expectation task to test whether outcome signaling by amygdala might also be necessary for changing those representations in the face of unexpected outcomes. Rats were trained to associate several different cues with a food reward. After learning, two of the cues were presented together, in compound, followed by the same reward. Prior to each compound training session, rats received infusions of NBQX or saline into either the basolateral or central nucleus of amygdala. We found that infusions into central nucleus abolished the normal decline in responding to the compounded cue in a later probe test, whereas infusions into basolateral amygdala had no effect. These results are inconsistent with the proposal that signaling of information about expected outcomes by basolateral amygdala contributes to learning, at least in this setting, and instead implicate the central nucleus in this process, perhaps due to this area’s hypothesized involvement in attention and variations in stimulus processing.
The amygdala is critical for associating predictive cues with primary rewarding and aversive outcomes (Maren and Quirk, 2004; Murray, 2007). This is particularly evident in tasks in which information about expected outcomes is required for normal responding (Hatfield et al., 1996; Malkova et al., 1997; Blundell et al., 2001; Corbit and Balleine, 2005; Wellmann et al., 2005; Machado and Bachevalier, 2007; Johnson et al., 2009). This function depends on basolateral amygdala (ABL), which is required for reinforcer-specific devaluation and Pavlovian-to-instrumental transfer (Corbit and Balleine, 2005; Johnson et al., 2009). While such studies reveal a role for ABL in the acquisition and use of value representations during initial learning, they do not address whether signaling of outcome information by ABL is necessary for changing associative representations when outcomes differ from expectations (Belova et al., 2008).
Pavlovian over-expectation is well-suited to address this question. In this task, rats learn that several cues are independent predictors of reward. After learning, two cues are presented together, in compound, followed by the same reward. Normal rats exhibit a spontaneous decline in responding to these cues when they are presented separately later during a probe test. This decline is thought to reflect learning induced by violations of the summed expectations for reward in the compound phase (Rescorla, 1999, 2006, 2007). By dissociating learning in the face of unexpected outcomes outcomes in compound training, from the use of the new information, in the probe test, this task allows one to test whether a given brain region has a specific role in learning driven by changes in expected outcomes. For example, inactivation of orbitofrontal cortex (OFC) during the compound phase prevents the later decline in responding (Takahashi et al., 2009), suggesting that information about expected outcomes signaled by neurons in OFC facilitates learning in the face of unexpected outcomes (Schoenbaum et al., 2009). Notably, the OFC has close anatomical and functional relationships to ABL (Krettek and Price, 1977; Shi and Cassell, 1998; Ongur and Price, 2000; Ghashghaei and Barbas, 2002); like ABL, OFC is critical for normal behavior when information about expected outcomes is required (Gallagher et al., 1999; Izquierdo et al., 2004; McDannald et al., 2005b; Machado and Bachevalier, 2007; Ostlund and Balleine, 2007).
Here we used Pavlovian over-expectation to test whether outcome signaling by amygdala neurons might also be necessary for learning when actual outcomes differ from expectations. Rats were trained as before; prior to compound training rats received infusions of saline or NBQX, an AMPA receptor antagonist, into either ABL or the central nucleus (CeN) of amygdala. We found that inactivation of AMPA receptors in ABL had no effect on the normal decline in responding to the compounded cue in the later probe test, whereas inactivation in CeN abolished this decline. These results suggest that signaling of outcome expectancies by ABL neurons is not required for learning, at least in this setting, and instead implicate the CeN in this process, perhaps due to its role in attention and variations in stimulus processing (Holland and Gallagher, 1999).
Experiments used male Long-Evans rats (Charles Rivers), tested at the University of Maryland School of Medicine in accordance with University and NIH guidelines.
Cannulae (23g; Plastics One inc., Roanoke, VA) were implanted bilaterally in CeN (2.3 mm posterior to bregma, 4.0 mm lateral, and 6.0 mm ventral) in 18 rats or in ABL (2.7 mm posterior to bregma, 5.0 mm lateral and 6.5mm ventral) in 20 rats in order to allow infusion of inactivating agents or saline vehicle prior to some testing sessions. Actual infusions were made at 2.3 mm posterior, 4.0 mm lateral, and 8.0 mm ventral in CeN and 2.7 mm posterior, 5.0 mm lateral and 8.5mm ventral in ABL.
Training was done in 16 standard behavioral chambers from Coulbourn Instruments (Allentown, PA), each enclosed in a sound-resistant shell. A food cup was recessed in the center of one end wall. Entries were monitored by photobeam. A food dispenser containing 45 mg sucrose pellets (plain, banana-flavored, or grape-flavored; Bio-serv, Frenchtown, NJ) allowed delivery of pellets into the food cup. White noise or a tone, each measuring approximately 76 dB, was delivered via a wall speaker. Also mounted on that wall were a clicker (2 Hz) and a 6-W bulb that could be illuminated to provide a light stimulus during the otherwise dark session.
Rats were shaped to retrieve food pellets, then they underwent 10 conditioning sessions. In each session, the rats received eight 30-sec presentations of three different auditory stimuli (A1, A2, and A3) and one visual stimulus (V1), in a blocked design in which the order of cue-blocks was counterbalanced. For all conditioning, V1 consisted of a cue light, and A1, A2 and A3 consisted of a tone, clicker, or white noise (counterbalanced). Two differently flavored sucrose pellets (banana and grape, designated O1 and O2, counterbalanced) were used as rewards. V1 and A1 terminated with delivery of three pellets of O1, and A2 terminated with delivery three pellets of O2. A3 was paired with no food. After completion of the 10 days of simple conditioning, rats received four consecutive days of compound conditioning in which A1 and V1 were presented together as a 30-sec compound cue terminating with three pellets of O1, and V1, A2, and A3 continued to be presented as in simple conditioning. Cues were again presented in a blocked design, with order counterbalanced. For each cue, there were 12 trials on the first three days of compound conditioning and six trials on the last day of compound conditioning. One day after the last compound conditioning session, rats received a probe test session consisting of eight non-reinforced presentations of A1, A2 and A3 stimuli, with the order mixed and counterbalanced.
On each compound conditioning day, cannulated rats received bilateral infusions of inactivating agents (CeN: n = 11; ABL: n = 10) or the same amount of PBS (CeN: n = 7; ABL: n = 6) immediately before the cue-block in which A1 and V1 were presented in compound. Procedures were identical to those used previously (Takahashi et al., 2009), except with volumes and a concentration of NBQX (2,3-dihydroxy-6-nitro-7-sulfamoyl-benzo[f]quinoxaline-2,3-dione), an AMPA receptor antagonist, based on prior work by Holland and colleagues (McDannald et al., 2005a). Briefly, dummy cannulae were removed and 30G injector cannulae extending 2.0 mm beyond the end of the guide cannulae were inserted. Each injector cannula was connected with PE20 tubing (Thermo Fisher Scientific, Inc., Waltham, MA) to a Hamilton syringe (Hamilton, Reno, NV) placed in an infusion pump (Orion M361, Thermo Fisher Scientific, Inc., Waltham, MA). Each infusion consisted of 4 μg NBQX (Sigma, St Louis, MO). The drug was dissolved in 0.2 μl PBS and infused at a flow rate of 0.2 μl/min. At the end of each infusion, the injector cannulae were left in place for another two to three minutes to allow diffusion of the drugs away from the injector. Approximately 10 min after removal of the injector cannulae, rats underwent compound conditioning.
The primary measure of conditioning to cues was the percentage of time that each rat spent with its head in the food cup during the 30-sec CS presentation, as indicated by disruption of the photocell beam. We also measured the percentage of time that each rat showed rearing behavior during the 30-sec CS period. To correct for time spent rearing, the percentage of responding during the 30-sec CS was calculated as follows: % of responding = 100 * ((% of time in food cup) / (100 − (% of time of rearing)). There was neither a significant main effect nor any interactions with group in rearing behavior in any phase of training (F’s < 1.7; p’s > 0.2).
Thirty-four rats were trained as illustrated by the experimental timeline in Figure 1. Prior to training, all rats underwent surgery to implant cannulae bilaterally in either ABL (n = 16) or CeN (n = 18). These rats were divided into three groups: ABLi rats (n = 10) and CeNi rats (n = 11) that later received NBQX infusions and controls that later received infusions of PBS vehicle (7 CeN and 6 ABL). Cannulae location is illustrated in Figure 1. The controls for CeN and ABL infusions exhibited no differences in any of the subsequent measures of conditioning, and thus they were combined into a single group.
After surgery and a two week recovery period, all rats underwent 10 days of conditioning, during which cues were paired with flavored sucrose pellets (banana and grape, designated as O1 and O2, counterbalanced). Three unique auditory cues (tone, white noise and clicker, designated A1, A2, and A3, counterbalanced) were the primary cues of interest. A1 served as the “over-expected cue” and was associated with three pellets of O1. A2 served as a control cue and was associated with three pellets of O2. A3 was associated with no reward and thus served as a CS-.
Acquisition of conditioned responding to the critical auditory cues is shown in Figure 1A. Rats in all three groups developed equally elevated responding to A1 and A2, compared to A3, across the 10 sessions. In accord with this impression, ANOVA (group X cue X session) revealed significant main effects of cue and session (cue: F(2,62)=75.3, p<0.0001; session: F(9,279)=9.1, p<0.0001), and a significant interaction between cue and session (F(18,558)=12.0, p<0.0001); however there were no main effects nor any interactions with group (F’s < 0.81, p’s > 0.52). A step-down ANOVA comparing responding to A1 and A2 revealed no statistical effects of either cue or group (F’s < 1.7, p’s > 0.205), indicating that rats learned to respond to these two cues equally. Furthermore, there were no effects of session in the final two days of training (F’s < 1.4, p’s > 0.26), indicating that responding was at ceiling.
Rats were also trained to associate a visual cue (cue light, V1) with three pellets of O1. V1 was to be paired with A1 in the compound phase to induce over-expectation; therefore a non-auditory cue was used in order to discourage the formation of compound representations. All three groups of rats responded identically to the V1 cue used to induce over-expectation during compound training (data not shown). An ANOVA (group X day) during initial and compound training revealed a significant main effect of day during initial conditioning (F(9,279)=16.0, p<0.0001). There were no other significant effects or interactions in other analyses of the V1 cue (F’s < 0.8, p’s > 0.5).
After initial conditioning, the rats underwent four days of compound training. These sessions were the same as preceding sessions, except that V1 was delivered simultaneously with A1. V1 also continued to be presented separately to support its associative strength and thereby maximize the effect of over-expectation on A1. A2 and A3 continued to be presented as before. Immediately prior to each session, rats in the two experimental groups received bilateral infusions of NBQX in order to inactivate ABL or CeN; rats in the control groups received PBS vehicle infusions.
Responding during compound training is shown in Figure 1B. Rats in all three groups maintained elevated responding to A1/V1 and A2, compared to A3, across the four sessions. ANOVA (group X cue X session) revealed a significant main effect of cue (cue: F(2,62)=94.1, p<0.0001) and a significant interaction between cue and session (F(6,186)=5.19, p< 0.0001); however there was no main effect nor any interactions with group (F’s < 0.9, p’s > 0.5). A step-down ANOVA comparing responding to A1 and A2 showed neither main effects nor any interactions of cue or group during compound training (F’s < 1.9, p’s > 0.18).
Finally all rats received a probe test in which A1, A2 and A3 were presented alone without reinforcement. Conditioned responding in this session is shown in Figure 1C. Rats in each group showed elevated responding to A1 and A2, compared to A3, and this responding extinguished across the session. A 3-factor ANOVA (group X cue X trial) revealed significant main effects of cue and trial (cue: F(2,62)=35.463, p<0.0001; trial: F(7,217)=10.616, p<0.0001).
In addition, rats in both the control and ABLi groups exhibited a significant and sudden decline in responding to the A1 cue in the probe test. Reduced responding was evident throughout the extinction session. Accordingly an ANOVA comparing responding to A1 and A2 in ABLi and control rats revealed significant main effects of cue (F(1,21)=14.9, p < 0.001) and trial (F(7,147)=4.2, p<0.001), but no significant main effect nor any interactions with group (F’s < 0.75, p’s > 0.6). Post-hoc comparisons showed that both groups responded significantly more to A2 than to A1 (p’s < 0.05). Importantly there were no interactions between cue and trial, thus the difference in responding between A1 and A2 in each group was consistent across the session, consistent with the proposal that it resulted from prior compound training rather than new learning in the session.
By contrast, rats in the CeNi group extinguished normally in the probe test but failed to show any selective decline in responding to the A1 cue. Consistent with this interpretation, an ANOVA comparing responding to A1 and A2 in the CeNi and control rats revealed a significant main effect of trial (F (7,154) = 8.41, p < 0.0001), a near significant main effect of cue (F (1,22) = 3.49, p < 0.07), and a significant interaction between cue and group (F (1,22) = 4.62, p < 0.04). Post-hoc comparisons showed that, unlike controls, CeNi rats responded similarly to A1 and A2 (p = 0.85).
Here we show that AMPA receptor blockade in CeN but not ABL disrupts learning in response to over-expectation. Rats that received NBQX infusions into CeN failed to learn in response to the compound training, exhibiting similar levels of responding to both the compounded and control cues in the probe text.
By contrast, rats that received NBQX infusions into ABL prior to the critical compound training learned as well as controls that received saline infusions. This was evident in the subsequent probe test, in which both groups showed a decline in responding to the cue that had been presented in compound, while maintaining responding to the control cue. Importantly other labs have found behavioral effects of NBQX infusions in ABL (Walker et al., 2005; Rabinak et al., 2009), and we have recently used an identical approach in another study to disrupt behavioral changes correlated with changes in single-unit activity in ABL (Roesch et al., 2010). As a result, we do not believe this negative result is due to any lack of effect of our manipulation, though of course it is possible that some other manipulation, such as dopamine blockade or neurotoxic lesions could reveal a role for ABL in over-expectation.
The failure of ABL inactivation to affect learning in this setting is inconsistent with proposals that neural correlates of cue value or outcome expectancies or even putative error signals reported in ABL neurons play a general role in the recognition and calculation of reward prediction errors by neurons in other brain regions. Although such firing patterns may explain the role of ABL in guiding behavior, they do not appear to be necessary for learning driven by negative errors in reward prediction.
Of course, these results do not directly address whether ABL is involved in learning driven by errors in punishment prediction or by positive errors in reward prediction. Potential evidence for a role in learning from positive reward prediction errors comes from Pavlovian reinforcer devaluation studies, in which damage to basolateral amygdala appears to cause deficits in the acquisition of the original associations (Pickens et al., 2003). However these deficits seem more likely to reflect a direct role in storage rather than any contribution to error signaling, particularly since signaling of outcome expectancies should retard rather than accelerate learning in this setting. Similarly while ABL is important for learning in an aversive over-expectation task (Cole and McNally, 2008), this was shown using NMDA antagonists, again suggesting a role in storage of the new information rather than any contributions to error signaling.
To the extent over-expectation is comparable to extinction learning, showing spontaneous recovery and extinction for example (Rescorla, 1999, 2006, 2007), the failure of ABL inactivation to affect learning in this setting is in accord with reports showing that ABL is typically not necessary for extinction (or reversal) of responding (Lindgren et al., 2003; Schoenbaum et al., 2003; Izquierdo and Murray, 2005, 2007; but see also McLaughlin and Floresco, 2007). Our results build on these reports by using a paradigm that requires summation of reward expectancies for normal learning and by manipulating ABL function only at the time of learning.
The failure of ABL inactivation to affect this process stands in contrast to the effects of OFC inactivation, which impairs learning in response to over-expectation (Takahashi et al., 2009). This dissociation is notable since single-unit activity and BOLD response in both OFC and ABL represent information about expected outcomes (Schoenbaum et al., 1998; Tremblay and Schultz, 1999; O’Doherty et al., 2002; Gottfried et al., 2003; Padoa-Schioppa and Assad, 2006; Patton et al., 2006; Tye and Janak, 2007; Belova et al., 2008; Tye et al., 2008), and both areas have been associated with comparable behavioral deficits in tasks that require the use of outcome expectancies for normal behavior responding (Hatfield et al., 1996; Malkova et al., 1997; Blundell et al., 2001; Corbit and Balleine, 2005; Wellmann et al., 2005; Machado and Bachevalier, 2007; Johnson et al., 2009).
It is only recently that a limited number of dissociations have begun to appear. The dissociations are generally consistent with the proposal that ABL is preferentially involved in the acquisition of the original associative information, while OFC plays an ongoing role in integrating that simple retrospective information with additional inputs to drive behavior and new learning. For example, while both OFC and ABL are involved in changes in responding to cues after reinforcer devaluation, OFC is particularly critical for using the information at the time of the final probe test (Pickens et al., 2003; Pickens et al., 2005). This function requires the real-time integration of the original association with information regarding the new value of the outcome. Likewise while both OFC and ABL are implicated in temporal discounting, the role of ABL appears consistent with encoding the original associative strengths of the different responses (or cues), whereas OFC seems to play a more variable role consistent with the use of that information at the time a decision must be made (Mobini et al., 2002; Kheramin et al., 2003; Winstanley et al., 2004; Rudebeck et al., 2006). The current data is consistent with this dichotomy, since over-expectation provides a situation in which new learning requires the integration of existing simple associative representations. This is because the reward given in the compound phase is appropriate for either individual cue; it requires the real-time integration of the individual expectancies for an error in reward prediction to be perceived. OFC is required for this integration, whereas ABL is not.
Perhaps more surprising is the involvement of CeN in over-expectation, particularly in the absence of effects of ABL inactivation. CeN is often viewed as a relay station, operating in serial with ABL in the expression of learned behavior (LeDoux, 2000). This reflects the primarily unidirectional flow of information from ABL to CeN, and the role of CeN in the expression of fear-associated behaviors. However growing evidence suggests that CeN can also function in parallel with the ABL, independently encoding critical aspects of associative relationships in the environment (Balleine and Killcross, 2006; Wilensky et al., 2006). For example, ABL is critical for normal performance in a variety of settings that do not require CeN, including second-order conditioning, outcome-specific transfer, and also Pavlovian devaluation tasks (Hatfield et al., 1996; Killcross et al., 1997; Corbit and Balleine, 2005). Likewise, CeN is involved in more general learning and attention-related behaviors, including learning in unblocking settings, which are not dependent on ABL (Gallagher et al., 1990; Holland and Gallagher, 1993a, b; Hatfield et al., 1996; Killcross et al., 1997; Corbit and Balleine, 2005; Lu et al., 2005; McDannald et al., 2005a). Our results provide another example of a behavioral setting in which ABL and CeN function in parallel rather than in serial.
We would suggest however that in this case CeN’s role in over-expectation likely reflects this area’s involvement in supporting variations in the processing of environmental stimuli rather than any selective role in representing certain types of associative information. Variations in stimulus processing are a key component of a class of learning theories exemplified by the Pearce-Hall theory (1980). Unlike Rescorla-Wagner (1972) or temporal difference reinforcement learning (Sutton and Barto, 1998) that treat the associability or salience of conditioned (and perhaps unconditioned) stimuli as fixed quantities, Pearce-Hall and related theories propose that associability or salience changes in response to errors in reward prediction, increasing when errors are large and decreasing when errors are small. Highly associable or salient cues drive learning, thus when errors are large and associability is high, learning occurs, whereas when errors are small and associability is low, learning is retarded.
Variations in stimulus processing can explain learning in a variety of settings, including some that cannot be easily accounted for by other models. One key example is increased responding induced by declines in the value of a predicted reward, as occurs in unblocking tasks when the reward predicted by a cue is reduced in conjunction with the addition of a novel second cue. On subsequent testing, rats will show increased responding to this novel second cue, despite the fact that it essentially predicts less reward. This increased responding cannot be easily explained by Rescorla-Wagner or temporal difference theories, since these theories posit a negative prediction error and declines in associative strength for the cues that are present in this situation. However this result is readily explained by a mechanism in which associability increases – and learning can occur - whenever rewards are poorly predicted.
Holland and colleagues have shown that a circuit of structures, including CeN is critical for learning in this setting (Holland and Gallagher, 1993a, b). Thus rats with CeN lesions do not show unblocking when reward is reduced. This effect was interpreted as showing a role for CeN in incrementing attention and cue processing, as posited by Pearce and Hall, in the face of decrements in expected reward. Like over-expectation (Takahashi et al., 2009), learning in this setting also depends on midbrain dopamine neurons (Han et al., 1999; Lee et al., 2008). More recent work has suggested this may also reflect variations in processing of the unconditioned stimulus (Holland and Kenmuir, 2005). This deficit was not observed when reward was increased, either because increased responding could be equally well-supported by simple error learning mechanisms or because CeN is only involved in processing decrements in reward value.
The effects of CeN inactivation in Pavlovian over-expectation, reported here, could be explained by a similar mechanism. Pearce and Hall originally proposed that incrementing attention and processing of available stimuli could lead to inhibition of behavior by pairing with an explicit “no reward” representation, explicitly defined as the absence of reward when it was expected (Pearce and Hall, 1980). While such learning might be difficult to isolate in an unblocking task, in which the putative associations with no-reward would have to out-compete new associations with the remaining reward (Holland and Kenmuir, 2005), learning capable of inhibiting behavior should be readily apparent in the over-expectation task, in which the cues are already heavily trained. According to this model, CeN may promote changes in behavior as a result of over-expectation by increasing the associability of the compounded cues when the expected rewards fail to materialize, thereby facilitating acquisition of an association with the imaginary missing reward. We would suggest this is a more parsimonious account of these data than proposing a novel role for CeN in either signaling negative prediction errors or in contributing information - expectancies - necessary for their calculation, although we cannot directly rule out these alternative accounts.
In addition to extending the current account of CeN’s role in attentional processing, our results are noteworthy for two other reasons. First in using a temporally-specific manipulation - inactivation during learning - rather than pre-training lesions, our results provide stronger evidence that CeN is not critical to the storage of the original associative information. Further, since CeN was back online during probe testing, this also rules out a critical role in the final use of the new information. Also by inducing changes in expected reward by manipulating abstract expectations rather than by changing the timing, amount or quantity of reward actually delivered, our results avoid potential confounds whereby the omission of one reward might inflate the value of what remains (Holland and Kenmuir, 2005). As a result, these findings provide strong support for proposals implicating CeN as an important part of a circuit of structures that support learning via attentional mechanisms validated through empirical studies of animal learning.
This work was supported by grants from NIDA to GS, KB, and MR, from the HFSP to YT, and from NIMH to GS.