|Home | About | Journals | Submit | Contact Us | Français|
A neurally oriented conceptual and computational model of fear conditioning manifested by freezing behavior (FRAT), which accounts for many aspects of delay and context conditioning, has been constructed. Conditioning and extinction are the result of neuromodulation-controlled LTP at synapses of thalamic, cortical, and hippocampal afferents on principal cells and inhibitory interneurons of lateral and basal amygdala. The phenomena accounted for by the model (and simulated by the computational version) include conditioning, secondary reinforcement, blocking, the immediate shock deficit, extinction, renewal, and a range of empirically valid effects of pre- and post-training ablation or inactivation of hippocampus or amygdala nuclei.
Cues that predict the imminent onset of pain or danger as well as the situational contexts in which such events occur come to evoke fear. In laboratory animals this has a variety of manifestations including freezing, cardiac responses, and analgesia (Bolles and Fanselow, 1980). Such fear conditioning provides one of neuroscience's most promising and active arenas for analyzing neural mechanisms of learning, generally. There is now considerable evidence that the acquisition of conditional fear can be substantially attributed to forms of long-term potentiation at synapses of cortical, thalamic, and hippocampal afferents on cells of the baso-lateral region of the amygdala (BLA; Blair et al., 2001). Synaptic plasticity within the central nucleus, which contains the amygdala's fear-producing output neurons and which receives input derived from BLA, also seems to contribute (Pare et al., 2004; Wilensky et al., 2006; Ciocchi et al., 2010). And although there is not yet consensus on the locus of extinction, a variety of lines of work, both behavioral and neurophysiological, have established that under most circumstances extinction is at least in part due to learned inhibition of activity in fear-producing pathways and not primarily to erasure of the changes that caused the original conditioning (Bouton, 1993, 2004). While these are major advances, there are many conspicuous properties of fear learning for which we neither know the mechanisms nor have hypotheses of a kind that invite physiological testing.
The well-known phenomenon of “renewal” provides an excellent example (Bouton, 1993, 2004). It is generally agreed that once fear is conditioned to a cue, fear will be expressed when that cue is presented in most any context. However, if fear is then extinguished, either in the context of original training or elsewhere, the extinction will be quite specific to the context in which it occurred, and the CS will still evoke fear in other contexts; i.e., fear will be “renewed.” Renewal provides some of the best evidence of the inhibitory nature of extinction, but hypotheses as to its mechanisms are not usually neural in character. Discussions of mechanism are generally in terms of “occasion setting,” “disambiguation,” “storage and retrieval,” “gating” of inhibition, etc. These are essentially psychological constructs and do not provide much guidance for implementation at a neural level.
The goal of the present work was therefore to construct a conceptual and computational neural model of fear conditioning that can account for what we consider some of the most salient aspects of fear conditioning and that can be justified by known neurophysiology. This model is referred to as “FRAT” (for “Fraidy Rat” after an earlier, simpler version of the model used for teaching). While for modeling purposes we focused on freezing, the model is expected to generalize to other commonly used measures such as fear-potentiated startle, autonomic arousal, conditioned suppression, and inhibitory avoidance, with the greatest differences being in response generation circuitry downstream from the BLA (Hitchcock and Davis, 1986; LeDoux et al., 1988; Tomaz et al., 1993; Amorapanth et al., 1999). The phenomena that the model was explicitly designed to account for are listed in Table Table1.1. Models of this kind, which are constructed to provide possible explanations of certain phenomena, are sometimes referred to as “top-down” models. This is to be contrasted with so-called “bottom-up” models in which low-level properties of a system are comprehensively incorporated into a model and the emergent behavior of the running system then examined. A bottom-up model of fear conditioning that shares features in common with the present top-down model has recently been described and analyzed (Li et al., 2009). It incorporated much-more detailed information on synaptic and cellular properties than does the present model. It simulated both conditioning and extinction, but its scope did not permit examination of many of the phenomena of Table Table1.1. The two types of models have complementary utility. Bottom-up models allow one to explore the completeness of one's understanding of the properties of a system that has been extensively analyzed; if one really understood it all and incorporated it into the model, the model ought to successfully emulate the functioning system. Top-down models allow one to explore what sorts of lower-level properties and inter-relations could explain the known behavior of a complex system; they can guide one in the search for actual mechanisms and the interpretation of existing observations.
The present model can account for many properties of Pavlovian fear conditioning. Most of the ideas upon which it was based have been separately expressed by various authors. However, the inclusion of a computational implementation makes it possible to determine whether and how independently plausible mechanisms can work together in a comprehensive model. It is to be expected that the attempt to engineer a system that can successfully emulate the behavior produced by biological fear-learning circuitry will help us to make sensible interpretations of experimental findings on actual fear-learning circuitry and will aid efforts to determine the actual neural mechanisms of fear conditioning and extinction.
FRAT is an attempt to construct a relatively simple neurally oriented model of context conditioning and of “delay” cue conditioning, in which CS and US offset coincide or the US begins contiguously with CS offset. FRAT is not presently subject to trace conditioning.
The model has been made computational because we believe informal reasoning does not provide a reliable way of determining how the fairly large number of mechanisms needed to account for observed phenomenology will interact. Accurate qualitative predictions are the model's primary goal. Although some ordinal quantitative predictions of the model may be useful, available information is insufficient to guide production of a model that could be expected to make comprehensive quantitative predictions.
The phenomena that FRAT was constructed to emulate are those listed in Table Table1.1. There are certainly some phenomena, such as latent inhibition and trace conditioning, that the model does not presently simulate, nor intend to. Although we believe that small additions would have allowed simulation of these and other phenomena, we have wished to limit the models complexity. However even as it stands, the model makes a substantial number of empirically valid predictions that go beyond the design-criteria of Table Table11 and many predictions that remain to be tested.
At present FRAT deals with only two CSs (CS1 and CS2) and three contexts (A, B, and C), which are considered to have completely non-overlapping neural representations. This has allowed us to defer issues of storage capacity and related complexities that are better initially considered independently of the matters listed in Table Table11 and that we hope to address in later work.
The resting potential of FRAT neurons is, for convenience, taken as zero. Membrane potential (V) is specified as deviation from rest. Active synapses open postsynaptic ion channels (cause conductances) for particular ions. Excitatory input causes ion movements that depolarize the membrane potential toward an excitatory reversal potential E (taken as 100mV above rest). Inhibitory input opens ion channels that have an equilibrium potential near the resting level and thus result in current flows that move the membrane toward the resting level (as with GABA-mediated chloride conductance in real neurons in cases where the neurons are not persistently depolarized by tonic excitatory input). Therefore, inhibitory input, rather than working by hyperpolarizing the cell, is effective mainly because it allows excitatory currents to pass out of the cell instead of depolarizing it, thus attenuating EPSPs (so-called “divisive” inhibition).
When a FRAT neuron becomes depolarized beyond its firing threshold, its firing rate (also referred to as “activity” or “activation”) increases. Maximum activity, taken as unity, is reached at a depolarization level that varies according to neuron type. Individual spikes are not represented in FRAT, only firing rates.
Synapses can vary in strength or synaptic “weight” (W) according to their type and history of stimulation. The greater a synapse's weight, the more postsynaptic conductance G (conductances are expressed relative to leakage conductance) will be produced by a given level of presynaptic activity A according to the relationship,
The neurons of this model treat the summation of excitatory inputs as though all excitatory input synapses were located close together on their postsynaptic targets; in computational terms, they all innervate a single “dendritic compartment.” Inhibition has quite different consequences depending on whether inhibitory synapses are located electrically close to excitatory synapses (i.e., in the same electrical compartment) or at a separate location more proximal to the soma (for modeling purposes, a second electrical compartment); we thus distinguish here between “distal” and “proximal” inhibition (Vu and Krasne, 1992). When inhibition is distal, it competes more or less symmetrically with excitation; an increase in one can always be offset by an increase in the other. However this is not the case for proximal inhibition, which for reasons explained below, was used here.
Denoting the sum of all the conductances generated by excitatory input as Ge, the equilibrium potential for excitation as E, and the sum of inhibitory conductances by Gi, the depolarization produced by summed inputs is of the form
In the absence of inhibition, membrane potential is given by the ratio in the left brackets, which is graphed in Figure Figure11 (bold solid line). When Ge is much smaller than unity, the depolarizations produced by each active, effective synapse sum approximately linearly. However, as Ge grows, each increment of input adds less to V, and the total depolarization becomes asymptotic to E. This non-linearity of summation is used to advantage in the model. As seen in the right bracket term of Eq. 2, proximal inhibition acts as a divisor for excitation. It is effect is shown in the thin solid lines of Figure Figure11 (for comparison the effect of distal inhibition is shown by the dashed lines).
A detailed technical account of FRAT is provided in Section “Supplementary On-line Materials.” However a few basic features that will be needed for what follows must be mentioned here: The state of FRATs variables is updated every 1s of real time, and all graph abscissas are in seconds. Values of variables should be thought of as averages or peak values for the interval in which they are determined. Thus phenomena that occur on millisecond time scales are not emulated as such. Events occurring in a given, 1s time interval cause plastic changes that are expressed behaviorally in the next.
Finding a single set of parameters (thresholds, change rates, etc.) that will allow FRAT to behave so as to meet the specifications of Table Table11 required considerable experimentation. Search routines such as Simulated Annealing might have been useful, but we chose informal experimentation because it was helpful in providing insight into the interactions of various processes. The parameters that must be set include potentiation and depression rates for BLA principal cells and inhibitors in the presence of reinforcement and extinction modulator, the relative numbers of neurons carrying each elementary and configural representation of hippocampus and cortex, the thresholds and saturation depolarizations of all the neuron types indicated in Figure Figure5A,5A, the rate of formation of hippocampal and cortical configural representations, and depolarization and “calcium” levels at which potentiation and depression began and ended in the curves of Figure Figure5C.5C. The parameters eventually settled on (specified in section Supplementary On-line Materials) are certainly not claimed to be in any way a “best” set; however, they do establish that a model with the structure employed can generate the desired set of phenomena, which is the purpose of this paper.
Neuron type names are in bold font; thus, “CEm” refers to neuron type CEm, which may be thought of as a single neuron or as a population of neurons all of which behave similarly. “CEm,” not in bold font, refers to the central nucleus CEm.
A full mathematical description of the model and presentation of simulations for which there is not space here can be found at http://escholarship.org/uc/item/3hk369c4. A document providing Matlab code for FRAT and instructions for using the program is in preparation; when completed it will be placed at this URL.
In order to create a model, specific assumptions about fear learning and its circuitry had to be adopted. Many of the assumptions made are widely accepted as valid. However, some remain matters of debate. In these cases we selected particular possibilities as working hypotheses; as new facts come to light, revisions will surely be required.
Here we consider the broad BLA as composed of lateral amygdala (LA), the lateral basal area (BL), and the medial basal region (BM). These regions project via a variety of routes to medial central nucleus (CEm), which is believed to be the portal to fear-expression circuitry that lies downstream of the amygdala itself. Many of the synapses within the BLA and its pathways to CEm are plastic (starred in Figure Figure2A),2A), and most have at one time or another been suggested to play some role in fear conditioning or extinction. Although it is generally agreed that changes mediating cue and context fear conditioning occur within BLA, the roles of the different sub-regions of the BLA are still being resolved, as is the route of information flow from LA and BL to CEm. Figure Figure2A,2A, modeled after a figure of Pare et al. (2003), summarizes aspects of the known circuitry; Figure Figure2B2B portrays the circuit as implemented in FRAT.
Based on available data (see Figure Figure2A;2A; McDonald, 1998; Goosens and Maren, 2001; Nader et al., 2001; Calandreau et al., 2005), the LA of FRAT receives input from cortical sensory and sensory association areas (hereafter referred to collectively as “stimulus representation” or “representation” cortex) and from thalamus (McDonald, 1998; Doron and Ledoux, 1999), and LA mediates a major portion of cued fear conditioning. Current belief is that LA output might reach CEm via BL or BM (Pare et al., 1995, 2003; Pitkanen et al., 1995; Savander et al., 1995; Royer et al., 1999) or by disinhibitory mechanisms via intercalated cells (ITC) or the lateral part of the central nucleus (CEl; Pare et al., 2004; Ciocchi et al., 2010; Haubensak et al., 2010), but since post-training lesions of BL appear to abolish conditional responses to discrete cues (Anglada-Figueroa and Quirk, 2005), FRAT LA was made to ordinarily access CEm via BL. However, to reflect the fact that CS fear can be acquired and extinguished normally after pre-training lesions of BL (Nader et al., 2001; Yaniv et al., 2004; Anglada-Figueroa and Quirk, 2005; Calandreau et al., 2005; Onishi and Xavier, 2010; but see Goosens and Maren, 2001), conditioning of FRAT in the absence of BL was made to cause development of an alternate pathway from LA-to-CEm.
Synaptic changes within LA are widely regarded as essential for development of cued fear. There is limited evidence that has been taken to suggest that most LA principal cells receive cortical and thalamic innervation that would allow conditioning to almost any CS (Repa et al., 2001; Rumpel et al., 2005; Han et al., 2007; Won and Silva, 2008; Zhou et al., 2009). Although far from established, we take this as justification for the initial, simplifying assumption that each LA principal cell is innervated by all afferents to LA and that after conditioning all LA principal cells respond similarly (referred to here as “uniform innervation”). Under this assumption, any synaptic changes downstream of LA would alter responses to all stimuli indiscriminately, which would lead to arbitrary and unwanted stimulus generalization. Therefore conditioning to a CS in FRAT has been made entirely due to plasticity at the level of BLA. However, in reality central nucleus plasticity appears to contribute to conditioning (Wilensky et al., 2006; Ciocchi et al., 2010). In fact, if one lifts the assumptions of uniform innervation at the level of LA, plasticity downstream of BLA becomes a possible asset, as addressed in the Section “Discussion.”
Since pre-training inactivation of BL as well as suppression of activity or plasticity of hippocampus, which projects richly to BL, substantially reduces context fear conditioning (Fanselow and Kim, 1994; Young et al., 1994; Stiedl et al., 2000; Bast et al., 2003; Yaniv et al., 2004; Calandreau et al., 2005; Quinn et al., 2005; Parsons and Otto, 2008; Schenberg and Oliveira, 2008; Onishi and Xavier, 2010; Raineki et al., 2010; but see Goosens and Maren, 2001; Matus-Amat et al., 2004), the hippocampus–BL pathway was taken to mediate fear conditioning in normal (unlesioned) FRAT, and BL was assumed to directly excite CEm. The actual pathway to BL in rodents may be dorsal hippocampus–ventral hippocampus–BL (see Fanselow and Dong, 2010), but dorsal and ventral hippocampus were not distinguished in the model.
The role commonly attributed to the hippocampus in context fear learning is the rapid exposure-dependent establishment of neural representations of elemental stimulus conjunctions, including contexts, independent of actual training with a US (Kiernan and Cranney, 1992; Fanselow, 2000; Rudy and O'Reilly, 2001; Rudy, 2009). It is presumed to be because of this that at least some tens of seconds of exposure to a context must precede US onset for context fear to be learned (the so-called “immediate shock deficit”; Blanchard et al., 1976; Fanselow, 1986, 1990; Landeira-Fernandez et al., 2006). This essential rapid learning process is prevented if the activity or plasticity of the dorsal hippocampus is suppressed (Matus-Amat et al., 2004; Stote and Fanselow, 2004). These characteristics were design-criteria for FRAT.
It is often postulated that hippocampus re-establishes in cortex representations of the full set of elemental cues that compose a context (e.g., Marr, 1971; McClelland et al., 1995). FRAT contains circuitry designed to implement such reinstatement. However, if reinstated representations of elementary contextual cues provide sufficient input to amygdala to support context fear conditioning, then according to the anatomy proposed in Figure Figure2B,2B, context fear learning would remain possible despite BL lesions. Since we decided above to accept those data which suggest that such learning does not occur in rats, projection to amygdala of reinstated elementary cues has been made of negligible strength in the simulations of this paper. Nevertheless it must be acknowledged that BL lesion studies do typically find some residual context conditioning; therefore the consequences of increasing the effect of reinstated contextual cues should at some point be explored.
It is well-known that whereas context fear is lost if hippocampus is ablated soon after conditioning (retrograde amnesia), this does not happen if sufficient time passes before ablation (Kim and Fanselow, 1992). It is believed that conjunctive representations established in the hippocampus are gradually shifted to cortex, the inferred process of “consolidation” (our references here will be only to this slow, “system” consolidation as opposed to the much faster, protein synthesis-dependent stabilization of synaptic change sometimes referred to as “cellular” consolidation). Moreover, if sufficient time for consolidation is allowed following simple familiarization, before context conditioning is attempted, no retrograde amnesia results from post-training hippocampal ablation (Anagnostaras et al., 2001). Thus, hippocampal representations, once formed by incidental learning, seem to become re-established in cortex even in the absence of conditioning. These characteristics were design requirements for FRAT (Table (Table1).1). Consolidation was assumed to cause replacement of hippocampal representations of familiarized conjunctive stimuli with representations located in sensory-representation cortex. Since hippocampus innervates BL, whereas representation cortex innervates LA, post-conditioning consolidation must not only transfer conjunctive representations from hippocampus to cortex but must also replace hippocampus–BL synaptic changes with cortex-LA changes that will produce similar behavior.
Whereas suppression of hippocampal function greatly impairs context conditioning, actual removal of hippocampus does not (Maren et al., 1997; Frankland et al., 1998; Cho et al., 1999; Richmond et al., 1999; Rudy et al., 2002; Wiltgen et al., 2006). One interpretation of this has been that intact hippocampus suppresses conditioning of fear to cortical representations of the elemental cues that compose contexts (Matus-Amat et al., 2004; Biedenkapp and Rudy, 2009; Rudy, 2009), and thus, once hippocampus is removed, conditioning to elemental cues of context becomes possible, whereas before it was not. However, if cortically represented elements of context that were present but suppressed before hippocampal removal, and operative after hippocampal removal, were capable of supporting context conditioning, then they should be able to support context conditioning even without any pre-exposure to context. However, the fact appears to be that even hippocampus-ablated rats require tens of seconds of experience with a context before they can be conditioned to it, just as do normal rats (Wiltgen et al., 2006). This suggests that conditioning in such animals is not to pre-existing elemental cues, but rather it is to conjunctive representations that, in the absence of hippocampus, form cortically over tens of seconds as they do within the hippocampus in normal animals (Fanselow, 2010). We refer to this hypothetical process as cortical “compensation.” In FRAT such compensation occurs and creates conjunctive representations in sensory-representation cortex; under the parameter values used for the simulations of this paper, elemental cues of context are never the basis for context conditioning.
It is not surprising that more signs of cortical compensation should be seen after hippocampal ablation than during disruptions of function, since ablation causes degeneration as well as dysfunction. However, whether some degree of compensation also occurs when only function is disrupted, is unclear. Limited context fear is often, though not always learned despite hippocampal APV or muscimol infusion (Young et al., 1994; Stiedl et al., 2000; Bast et al., 2003; Quinn et al., 2005; Parsons and Otto, 2008; Schenberg and Oliveira, 2008; Raineki et al., 2010), but it is not known whether such partial effects should be attributed to compensation or to manipulations that were incomplete. As a working hypothesis for the design of FRAT, we assumed the latter; in FRAT no compensation is produced by suppression of hippocampal activity or plasticity unless hippocampus is actually removed. This choice led to an unexpected ability to correctly predict the outcome of several experiments on hippocampal involvement in extinction described below.
Assumptions about the development and locus of conjunctive representations were important in the construction of FRAT not only because of the relevance of such representations to learning of context fear, but also because inhibition associated with such cues can contribute to the context-specificity of extinction that is revealed by renewal. Therefore, if as just discussed, hippocampus-ablated animals can form conjunctive representations cortically, it might be expected that they would also be subject to renewal. The literature on renewal in hippocampus-ablated rats is somewhat conflicting (Wilson et al., 1995; Frohardt et al., 2000; Ji and Maren, 2005; Zelikowsky et al., 2011); however we felt that there was sufficient evidence that such rats are subject to fairly normal renewal to make this capacity one of the design-criteria of Table Table11.
The pre-frontal cortex (PFC), which is reciprocally well connected to amygdala and hippocampus (Figure (Figure2),2), affects fear conditioning in several ways:
The PFC is commonly postulated to play some sort of executive role that enables or facilitates complex information processing tasks that extend over time (Miller and Cohen, 2001). We think the potent, but somewhat complex and ill-defined effects of PFC on fear are consistent with its acting as a modulator of fear learning, expression, and extinction without it actually mediating these processes or being the final repository of contextual representations or fear engrams. Moreover, given the significance of context in extinction learning, we believe that the role of PFC in extinction may well be, at least in part, secondary to the role in context learning suggested by points 1–3 above (see Morgan et al., 2003; Zelikowsky et al., 2010). Indeed, if the normal functions of the hippocampus are in fact somewhat PFC-dependent, then one possible interpretation of the loss of extinction during the first 24h after its establishment in PFC-lesioned rats is that hippocampal pattern-completion mechanisms are compromised. These rats may fail to compensate for small changes in context due to the passage of time, with the result that extinction, which is extremely context-specific, suffers more renewal due to the passage of time than would normally be the case. What appears as “forgetting” might merely be such renewal.
Given the above observations, and considering the presumed executive function of PFC, both the processes of compensation and consolidation and the activation of cortical conjunctive representations in FRAT have been made PFC-dependent. Some precedent for a role of PFC in the consolidation process is provided by recent evidence that off-line PFC activity is required for post-training stabilization of motor skill learning (Kantak et al., 2010).
The destination within amygdala of cortical neurons that, through compensation or consolidation, have come to represent cue conjunctions have not been studied. Sensory association areas preferentially innervate LA, but very high level association cortex, and especially polymodal cortex, may also innervate BL (McDonald, 1998; Pitkanen et al., 2000; Lindquist et al., 2004; Suzuki, 2010). In FRAT we made these cortical representation neurons project exclusively to LA.
One can only guess whether the same or different cortical neurons contribute to the representations formed as the result of compensation and consolidation. It is commonly supposed that consolidation causes the development of cortical representations that are extensively integrated with other pre-existing cortical representations. Representations formed by compensation are perhaps more akin to temporary working memories or hippocampal “snapshots” of cortical activity patterns (as in Marr, 1971). Representations formed by consolidation and compensation might therefore be expected to be separate and have different properties. However, pending evidence of this, we provisionally opted for simplicity and used the same cortical neurons for both types of representation in FRAT.
Table Table22 lists the main assumptions discussed above.
There are a great many anatomical routes via which information about footshock USs might reach the baso-lateral and central complexes of the amygdala. Nociceptive information reaches the brain at via spino-thalamic, spino-reticular, spino-mesencephalic, spino-parabrachial, and perhaps still other pathways, and each of these can then project nociceptive information to the amygdala via both direct and indirect routes (see, e.g., Willis and Westlund, 2004). Given this abundance, it has been difficult to define which pathways are actually necessary or sufficient to reinforce fear learning (Shi and Davis, 1999; Brunzell and Kim, 2001). However, a recent study (Johansen et al., 2010b) has reported that muscimol inactivation of periaqueductal gray (PAG) of the spino-mesencephalic pathway, electrical stimulation of which can act as a US for fear conditioning (Di Scala et al., 1987), abolishes at least cued fear conditioning. PAG output can itself reach amygdala via intralaminar thalamus, anterior cingulate, hypothalamus, locus coeruleus, and ventral tegmental area, and, again, which pathways are most important has not be determined (see, e.g., Herrero et al., 1991; Shi and Davis, 1999; Brunzell and Kim, 2001; Lanuza et al., 2004; Johansen et al., 2010b). For the purposes of designing FRAT, we accept the likely importance of portions of the PAG as an ascending route for the US, in addition to its being on the output pathway mediating freezing (see Carrive, 1993), but we do not try to specify further.
The operation of the model once learning has occurred is described first:
Conditioning is due to potentiation of synapses made by cortical and thalamic afferents on LA principal cells and hippocampal afferents on BL principal cells. LA input mostly codes simple cues used as CSs, but may also code for complex configurations (as in visual configuration-sensitive cells of inferotemporal cortex (e.g., Miyashita and Chang, 1988; Perrett et al., 1992) if these have developed as the result of consolidation or compensation (see previous section), or other processes. BL principal cells are innervated by LA output cells via non-plastic synapses as well as by conjunction-representing neurons of hippocampus.
The facts of renewal demand that extinction be at least in part due to active inhibition of responses whose ability to be excited is not erased. Given the assumption of uniform innervation (see above), extinction can only be stimulus-specific if inhibition is recruited via synapses made by LA and BL afferents. In FRAT these are synapses of LA and BL afferents on feed-forward inhibitory interneurons (Hestrin and Galarreta, 2005; Woodruff and Sah, 2007).
A recent paper by Herry et al. (2008) has reported a class of projection neurons in BL (referred to there as “extinction” neurons) that increase their responses during extinction, while activity of another class that previously became active during conditioning (“fear” neurons), declines. This suggests that in BL, afferents may at least in part recruit inhibitory interneurons via a special class of projection neurons. Computationally this would be equivalent to what was assumed here, though the downstream effects of the projecting extinction neuron activity, which are unknown, presumably have consequences of their own.
The phenomena of renewal require that the inhibition that mediates extinction of a CS be preferentially operative in the context where extinction occurred. In order to achieve this it was assumed that hippocampus forms representations of conjunctions of context and CSs that have been experienced in the context. Inhibition conditioned to such representations at the level of BL is the mechanism used in FRAT to produce context and CS-specific inhibition of fear responses. Blair and colleagues (Moita et al., 2003) have in fact seen hippocampal units that respond to a conditioned CS only when the rat is in a particular location.
Stimuli such as contextual cues that are continuously present would be expected to be less salient and have less impact than stimuli that have sudden onset and are present only briefly (Phillips and LeDoux, 1992). This is captured in FRAT by partially discounting the activations of contextual inputs to LA and BL by an “Habituation Factor” when calculating the activity of amygdala neurons; habituation in FRAT affects only performance, not synaptic weight alterations.
Figure Figure44 shows the sort of output that a FRAT CEm cell (Figure (Figure3)3) would produce in an experiment where training occurred in context A, extinction in B, and a test was then given in C (“ABC renewal”). After conditioning the CS in context A, potentiation would have developed at synapses of CS-representing neurons on LA principal cells, context A-representing neurons on BL principal cells, and context A/CS conjunction-representing neurons on BL principal cells. As a result, CEm neurons would respond to context A itself and to CS in context A as shown in the first two bars on the left of Figure Figure4.4. When moved to context B there would be no response either to the new context by itself or to the context B/CS conjunction. However, responding to the CS itself would remain, with the result that some CEm firing in the new context would still occur, as shown in the third bar on the left. After extinction in context B, inhibition would be conditioned in varying degrees to the CS itself, to context B itself, and to the context B/CS conjunction. This would result in greatly diminished CEm response to the CS when tested in context B, as shown in the fourth bar on the left of Figure Figure4.4. After the post-extinction move to context C the only inhibition would be to the CS itself, with the result that responding to the CS would be greater than it was in B as shown in the last bar on the left. If manifestations of fear were a direct reflection of this profile of activity, there would be two deviations from reality: (1). Although the fear produced by the CS would transfer substantially from context A to context B, it would be much stronger in A than B. (2). Extinction, though asymptotic, would be incomplete. The model deals with such difficulties by interposing between CEm and the activation of fear behavior a fear-expression neuron F whose firing threshold and dynamic range are such that F fails to fire when CEm activity is below level a and fires at its maximum rate when CEm activity is above b. The firing rate of F would then be as shown in the right panel; the result of the limited dynamic range of F is that expression of conditional fear prior to extinction is the same in B as in A, and behavioral extinction in B is complete even though some activity of CEm remains.
Some forms of LTP have properties that seem to make them ideal as a mechanism for associative conditioning. Thus, it is natural to imagine that fear conditioning might be simply a matter of establishing LTP at CA1-like synapses of CS-representing cells on CR-producing cells by pairing presynaptic activity with strong US-caused postsynaptic depolarization. However, this simple Hebbian scenario by itself seems to us to be an insufficient basis for simulating many of the properties of real fear learning.
Three considerations have caused us to make potentiation of presynaptically active synapses on FRAT LA and BL principal cells dependent on something other than input which directly represents the US: (1) We do not want a CS that begins after the onset of the US to cause conditioning; but it would do so if LTP on principal cells were determined by direct US-caused depolarization. Although some procedures, demonstrate the possibility of simultaneous or backward pairing or even the superiority of the latter for producing associations (Mowrer and Aiken, 1954; Heth and Rescorla, 1973; Mahoney and Ayres, 1976; Rescorla, 1980; Ayres et al., 1987; Matzel et al., 1988; Albert and Ayres, 1997; Esmoris-Arranz et al., 2003; Leising et al., 2007), it appears to us that for the sorts of experiments modeled by FRAT, forward pairing produces considerably more substantial and reliable conditioned fear than do non-forward procedures when fear is measured by CS-produced suppression of an ongoing operant, aversion, or freezing (Mowrer and Aiken, 1954; Heth and Rescorla, 1973; Mahoney and Ayres, 1976; Ayres et al., 1987; Albert and Ayres, 1997; Esmoris-Arranz et al., 2003). We have chosen to approximate this situation by making the model produce freezing only after forward pairing. We have also chosen to make this rule apply to the formation of associative change per se, despite the possibility that the relative ineffectiveness of non-forward conditioning could be due to a performance rather than a learning deficit (Rescorla, 1980; Matzel et al., 1988). (2). A well-conditioned CS should block conditioning of a new stimulus presented along with it (Kamin, 1969). Furthermore, blocking should occur even if the representation for one of the stimuli innervates LA while that of the other innervates BL. (3) Second-order conditioning must be possible. Moreover, a CS that causes CEm activity and fear due to potentiation that has occurred in LA (e.g., a well-conditioned cue) should be able to reinforce new conditioning mediated by potentiation in BL (e.g., to a contextual stimulus), and conversely (Marlin, 1983; Helmstetter and Fanselow, 1989).
The first requirement, was met by having the depolarization needed for LTP induction in LA and BL principal cells produced, not by input derived directly from the US, but rather by input from a “reinforcing” neuron R that fires only transiently and only at US onset. R fires briefly when its input increases, which it does when the activity of the US-representing neuron U increases (Figure (Figure5A).5A). The transience of US-evoked depolarizing input to principal cells is consistent with the observation in rats that BLA principal cells fire only briefly at the onset of a US (Johansen et al., 2010b).
The second requirement was met by letting CEm inhibit the U neurons, which they do via inhibitory neuron Ω. Consistent with this, responses of rat BLA principal cells to shock are markedly attenuated when an established CS precedes them (Johansen et al., 2010b) and the apparent aversiveness of shock, as measured by post-shock freezing, is diminished if the shock comes during a previously well-conditioned CS (Fanselow and Bolles, 1979b). Moreover, opiate receptor antagonists reduce both blocking and the CS-caused diminution in post-shock freezing (Fanselow and Bolles, 1979b; Cole and McNally, 2007), and the locus of these effects appears to be the PAG (Hammer and Kapp, 1986; Helmstetter and Landeira-Fernandez, 1990; Cole and McNally, 2007). For this reason, the inhibition produced by FRAT's Ω neuron is taken to be mediated by an opioid transmitter. The quantitative effect of ongoing CEm activity on responses of R to USs is shown in Figure Figure55B1. It should be noted that these mechanisms of blocking are very similar to those which appear to operate in the cerebellar circuit that mediates eye-blink conditioning (Fanselow, 1998; Kim et al., 1998).
Although introduction of the reinforcing neuron R and its inhibition by Ω may appear to allow the satisfaction of requirements (1) and (2), problems remain if R exerts its effect only by depolarizing BLA principal cells. Although, as described, onset of a new CS during a continuing unconditional stimulus would not cause conditioning to the CS, occurrence of a new CS during continued exposure to a previously well-conditioned CS would cause conditioning because the well-conditioned CS would cause a maintained depolarization of principal cells due its direct input; this would violate requirement (1). Other complexities would arise during blocking experiments, especially if the US were replaced by a previously conditioned CS. Such problems were avoided in FRAT by making potentiation depend not only on principal cell depolarization but also on a neuromodulatory input provided by R. Such an assumption is far from lacking empirical precedent. Development of virtually all forms of LTP and LTD as studied in slices appear to be sensitive to most neuromodulators. The VTA dopamine system provides an obvious precedent for neuromodulatory control of learning (Schultz and Dickinson, 2000). And there is increasing evidence in rats that reinforcement of fear conditioning is substantially dependent on dopaminergic or noradrenergic modulation (Johnston et al., 1999; Bissiere et al., 2003; Bush et al., 2010; Johansen et al., 2010a).
The third requirement was met by establishing a connection between CEm neurons and R-cells via a “secondary reinforcement” neuron S (Figure (Figure5A).5A). This connection allows sufficiently strong conditional responses to drive R at their onset and thereby both depolarize and provide reinforcing neuromodulator to LA and BL principal cells.
Blocking of second-order conditioning (Leyland and Mackintosh, 1978) could have been achieved by letting Ω inhibit S as well as U. However, if CEm is already very active due to the presence of a previously conditioned stimulus when a secondary reinforcement occurs, S will be able to increase its activity very little and thus will cause very little activity of R. Consequently, blocking of higher-order conditioning should occur even without input to S from Ω, and such a connection was not included in the model. The quantitative effect of prior CEm activity on responses of R to secondary reinforcers is shown in Figure Figure55B1.
Extinction in FRAT is due to LTP at synapses made on feed-forward inhibitory interneurons by the same afferents that feed LA and BL principal cells. Thus, LTP of synapses on inhibitors must develop when non-reinforced CSs occur. Inhibition should be sufficient to reduce CEm activity below the threshold a of the freezing neurons F, but it need not totally prevent CEm activation. Indeed, if inhibition at the level of LA is too great, renewal will not occur when shifts of context reduce inhibition at the level of BL. On the other hand, in order to simulate the observation that extinction remains approximately normal in BL-lesioned rats, sufficient inhibition must be able to develop at the level of LA to allow extinction. By making development of LTP at synapses on inhibitors contingent on a sufficient amount of activity of both CEm cells and the principal cell targets of the inhibitors, it can be assured that extinction will occur under appropriate conditions without inhibition at the level of LA growing too great to allow renewal.
Control of inhibitor LTP by CEm activity is mediated by “extinction” neurons X (Figure (Figure5A),5A), which distribute a neuromodulator that enables LTP development on inhibitory neurons and has other extinction-promoting effects. Although it would have been simplest to let CEm neurons directly excite X-cells, it is the case that in rats extinction is prevented by PAG opiate antagonists (McNally et al., 2004b; Parsons et al., 2010); therefore in FRAT CEm output was made to recruit X neurons through disinhibition driven by Ω neurons, as indicated in Figures Figures5A5A and B2.
The control of LTP at synapses on inhibitors by principal cell targets is exerted via recurrent inputs to the inhibitors from their targets (Figure (Figure5A).5A). These inputs have ligand-gated synaptic effects that are somewhat parallel to those causing LTP and LTD of afferents to cerebellar Purkinje cells, which are by far the best studied of inhibitory neurons with plastic inputs. Recurrent input is taken to produce LTP by mechanisms similar to those which foster LTP at afferent synapses on Purkinje cells. There, moderate input is thought to initiate LTP by causing modest Ca2+ elevations (see Lev-Ram et al., 2003; Coesmans et al., 2004; Jorntell and Hansel, 2006). Since extinction in FRAT is mostly due to growth of inhibition rather than erasure of potentiation of principal cell synapses, reconditioning of a previously extinguished FRAT is achieved by erasing the inhibition that was established by extinction training. This is effected by strong input to inhibitors from R-cells. Such strong input causes large “Ca” elevations, which in the presence of reinforcing neuromodulator, initiates LTD of synapses made by active afferents on the inhibitors. This is analogous to the LTD of Purkinje cells synapses produced by climbing fiber input.
The graphs of Figure Figure5C5C quantify the effects and interactions of ordinary and neuromodulatory input to principal cell and inhibitory interneurons, as described in the preceding paragraphs. In the presence of R-cell neuromodulator, active synapses on principal cells potentiate in proportion to postsynaptic depolarization, above some threshold. In the presence of X-cell neuromodulator they may undergo slight LTD, but the parameter ζ (Figure (Figure5C,5C, left), which controls the extent of such depression is set to zero in the simulations of this paper. In the presence of X-cell neuromodulator (Figure (Figure5C,5C, right), active afferent synapses on inhibitory interneurons undergo LTP to an extent determined by the amount of “Ca” that enters the interneuron due to recurrent input or input from R. The increment of LTD increases with “Ca” concentration above a rather low threshold and then rapidly saturates with increasing concentrations. In the presence of R-cell modulator, LTD develops as a function of “Ca” concentration to an extent that mirrors the relationship between increments of LTP in principal cells and depolarization.
R-cell effects always dominate those of X-cells because R inhibits X absolutely (Figure (Figure5A).5A). Extinction never develops during a protracted US because US activity also inhibits X absolutely.
The changes in synaptic strength just described occur only at active synapses. However, activity per se must be neither necessary nor sufficient, because learning fails if CS onset does not actually precede US onset, and USs that occur just as CSs terminate do cause conditioning. This caused Sutton and Barto (1981) to introduce the concept of eligibility for change. In FRAT eligibility is established by receipt of transmitter from active CS and context-representing neurons. However, as with NMDA receptors, which cannot open in response to postsynaptic depolarization until some milliseconds after glutamate binds to them and which continue to be responsive to depolarization for some milliseconds after unbinding (Magleby, 2004; Clarke and Johnson, 2006), eligibility slightly lags transmitter binding. In FRAT synapses become eligible for change one computational time step after the onset of presynaptic activity (i.e., the variable eligibility goes from zero toward unity) and retain eligibility for one time step after cessation of activity. Thus, onset of presynaptic activity must precede the occurrence of other change-promoting signals. Moreover, because eligibility is continued during the time step after CS offsets, cue conditioning is successful even if the US, instead of co-terminating with the CS, begins contiguously with CS offset. Eligibility in FRAT is a continuous variable that, above a threshold is proportional to activation of the input but that saturates at unity when activation reaches a certain level, which is a parameter of the model.
The three factors of eligibility, R or X activity, and the effects of postsynaptic depolarization in principal (P) cells or calcium elevation in inhibitory interneurons (I) combine as the product of three corresponding numerical factors:
Eligibility is as defined above. AR and AX are the activations of R and X neurons, as discussed above. RP, XP, RI, and XI are the relative change factors for principal cells and inhibitors as given in the graphs of Figure Figure55C.
If and only if BL is suppressed, coincident activity of LA principal cells and US-evoked activity of CEm neurons causes an effective pathway from BL to CEm to become established. This pathway is inhibited (but remains established) if suppression of BL is discontinued.
Although there has been discussion in the literature as to how hippocampal circuitry might work to establish conjunctive representations (Marr, 1971; Skaggs and McNaughton, 1992; O'Reilly and McClelland, 1994), in FRAT this was simply assumed to occur. The numbers of hippocampal neurons (or in absence of hippocampus, of cortical conjunction-representing neurons) that can be activated by a stimulus and their levels of activation grow with negative acceleration whenever the stimulus is present (see Figures Figures6A6A and and10).10). About 20s of exposure are needed before FRAT hippocampal representations develop to a point where they can have any impact on behavior, and it requires several minutes of exposure before activity is maximal. In the absence of hippocampus (Figure (Figure6C),6C), cortical representations grow similarly but a little slower (see simulations of Figure Figure10,10, which are roughly consistent with experimental observation Wiltgen et al., 2006). Once established, hippocampal representations are projected to BL, and cortical representations to LA, to provide bases for excitatory and inhibitory conditioning as described above.
Consolidation requires the gradual and presumably experience-independent transfer of hippocampal conjunctive representations to representation cortex. Since representation cortex innervates LA rather than BL, it is also necessary to establish new synaptic weights that allow the newly functional circuitry to mimic that which it replaces (Figure (Figure6B).6B). Algorithms for doing this biologically have not been proposed, but the process is believed to depend on off-line neural activity, perhaps during sleep (Marr, 1971; Wilson and McNaughton, 1994; McClelland et al., 1995; Ji and Wilson, 2007). In FRAT the replacement was done one BL input channel (A, B, C, A1, A2, etc.) at a time: First the CEm output produced by a controlled BL input with inhibition disabled was determined and the synaptic weight on LA principal cells that would be needed in order for a similar LA input to produce the same output was calculated. Then BL principal cells were provided just enough excitation to bring CEm to its maximum firing rate, the reduction produced by BL input to the inhibitor was determined, and the synaptic weight on LA inhibitors needed to produce a similar reduction in CEm activity was calculated. Once the new weights were determined, the hippocampus–BL circuit was replaced by its cortex-LA correspondent (see Supplementary On-line Materials for details). We believe that plausible neural algorithms that involve off-line activity and synaptic plasticity could be invented.
Once representations of context and context-CS conjunctions have become cortical so that all input is to LA, AAB renewal cannot occur if excitation and inhibition are directed to a common electrical compartment, because the switch to context B removes comparable amounts of inhibition and excitation. However, if inhibition is proximal and Ge is large, the value of the left bracket of Eq. 2 decreases little, while that of the right bracket increases substantially, and renewal occurs. This is why we have used two-compartment models of principal cell dendrites and directed extinction-causing inhibition to the proximal compartment.
In the next section the freezing responses of FRAT to simulations of various ablations, inactivations, and pharmacological manipulations are discussed. The manipulations were done as follows:
Suppression of hippocampus (Hs) is emulated by reversibly setting the activations of all hippocampal neurons to zero and disabling exposure-dependent formation of hippocampal representations, thus disabling incidental learning.
Ablation of hippocampus (Hx) is emulated by additionally enabling the exposure-dependent development of cortical representations of configural stimuli, as described above. Hx is irreversible.
Suppression of LA (LAs) is emulated by reversibly setting all LA neuron responses (depolarizations, activity, etc.) to zero. Given the rules described above, this prevents all new synaptic change, but existing change is retained and can again be expressed if suppression is discontinued.
Suppression of BL (BLs) is emulated in the same manner as was LAs, but additionally the plasticity of the LA–CEm pathway (as described above) is enabled. LA and BL suppression disable the part of the consolidation algorithm that adjusts amygdala synaptic weights, but permit the establishment of cortical configural representations.
Suppression of CEm (CEMs) is emulated by reversibly setting the activity of CEm neurons to zero.
Suppression of PFC (PFCs) is emulated by reversibly disabling development of consolidated or compensatory cortical responsivity to configural cues. During the period of suppression any previously developed responses of cortical configuration-representing neurons is also attenuated. This last feature is considered especially provisional.
The effects of opiate receptor antagonism were simulated by attenuating the activity of Ω.
The effect of PAG GABA-agonist (antagonist) was simulated by attenuating (augmenting) activation of X by CEm.
FRAT also has a number of behavioral characteristics that are consistent with available data but that were not considered in designing the model, and still others that constitute untested predictions. Some of these are listed in Tables Tables33 and and4,4, and a few are illustrated in this section. Unless otherwise indicated, all simulated experiments were preceded by a session in which FRAT was given sufficient exposure to all three contexts to establish full representations for them.
Figure Figure77 shows some of the basic properties of the behavior produced by FRAT.
Figures Figures7A–C7A–C illustrate the effects of five CS conditioning trials in context A followed by extinction of the CS in either context A or in new context B. Conditioning of fear to both the CS and the context develop rapidly and extinguish relatively slowly. As with rats, CS fear acquired in context A is also well expressed in context B.
Also, as with rats a precise evaluation of CS fear in the conditioning context is complicated by the presence of contextual fear. To provide a less contaminated evaluation of CS fear in A, Figure Figure7B7B shows a simulation in which extinction of the CS in context A was preceded by 50min of contextual fear extinction without any CS occurrences. The Figure shows that even in the absence of background contextual fear, freezing to the CS was strong, but it then itself extinguished about as it did in Figure Figure7A.7A. Also note in Figure Figure7B7B that occurrence of the CS at the start of extinction caused some second-order conditioning of fear to the context.
After extinction in any context, FRAT shows renewal of fear when the extinction context is changed. This is seen in Figure Figure7D7D for AAB, ABA, and ABC renewal. Evaluation of ABA renewal is complicated by the return of context fear when FRAT is returned to its training context (A); thus a test of renewal after extinguishing fear of A was also done. Renewal occurs with all three experimental paradigms, but it tends to be greatest for ABA, next for ABC, and least for AAB. This order seems to be consistent with available data for the behavior of rats (Bouton, 2002; Thomas et al., 2003). In the model, responsiveness during the renewal test is less for AAB than for ABC renewal because during tests in a new context (here, B or C) the only thing suppressing fear is inhibition conditioned to the CS itself. And more inhibition becomes associated to the CS during extinction in A than elsewhere because FRAT is more fearful during extinction in A (where CS, A, and the A/CS compound all promote CEm, and hence X, activity) than elsewhere (where only the CS itself promotes CEm and X activity). Responsiveness during a renewal test is greater for ABA than ABC paradigms because after extinction in B, return to A reinstates fear of both A and the A/CS compound whereas a shift to C is movement to a neutral context. Note, however, that the picture is complicated somewhat by the occurrence of secondary reinforcement during extinction, which is in fact a biological reality (Helmstetter and Fanselow, 1989).
Figure Figure7E7E illustrates the effect of contextual fear training when the US is presented without any predictive CS. A much greater level of context fear develops than was seen in Figures Figures7A–C,7A–C, because there is no response to the CS to drive strong opioid inhibition of US-representing neurons. Animals show a similar enhancement of context fear when there are no predictive CSs (Fanselow, 1980; Fanselow et al., 1994).
As discussed above, cued fear established in one context seems to be fully expressed in other contexts. However, underlying fear, as reflected in the activity of CEm neurons, is considerably greater in the conditioning context than elsewhere because fear conditioned to CS/context conjunctive representations, as well as possibly behaviorally subthreshold contextual fear, are absent outside the training environment. Fear-expression in FRAT is not usually diminished away from the training context, because the dynamic range of downstream fear-expression-producing neurons is limited; very strong CEm activity in the conditioning context and less strong activity elsewhere both commonly produce maximal freezing scores. However differences can become evident if conditioning is weak. This is shown in Figure Figure8,8, in which training consisted of a single conditioning trial with a very weak US. A similar result is found in real rats (Hall and Honey, 1990).
Evidence of context-specificity of conditioning can also be seen in experiments where rats that had been conditioned to several CSs, each in different context, have all CSs extinguished in a new context. In such experiments (Harris et al., 2000), and in FRAT, renewal is much greater when a CS is subsequently tested in the context where it, rather than some other cue was originally conditioned (see simulations in on-line materials). Context-dependence of cued fear is also suggested by the observation that hippocampal ablation, which would be expected only to cause retrograde losses of responses conditioned to conjunctive cues, in fact causes losses in cue conditioning if the conditioning was weak (Quinn et al., 2008b); this is what would be predicted if some of the conditioning were to cue/context stimulus compounds.
As discussed above, at least at initial learning, the hippocampus is the FRAT amygdala's primary source of information about configural stimuli, and in particular of contextual information. The hippocampal representations of configural stimuli that project to amygdala are formed fairly rapidly during exposure to the stimuli, and once these are formed, they provide input to BL where conditioning of fear to them, as well as conditioning of inhibitory activity that contributes to context-specific extinction, become possible. From this alone it would be expected that (a). Contextual fear conditioning would be entirely due to potentiation of synapses within BL, which is where FRAT hippocampus projects. (b). Contextual fear could not be established until time has been allowed for the formation of a hippocampal representation of the context (c). Context fear conditioning should be prevented by hippocampal ablation (anterograde amnesia). (d). Post-training hippocampal removal should abolish previously learned contextual fear (retrograde amnesia). (e). The context-specificity of extinction and possibly extinction itself should be dependent on hippocampus.
However, there are several (empirically motivated) aspects of FRATs construction that complicate this picture: (1). If hippocampus is ablated, a PFC-dependent compensation process of unspecified mechanism allows sensory-representation cortex to form configural representations, though the formation of configural representations by this process is a little slower than normal. (2). If hippocampal plasticity or activity are suppressed pharmacologically but hippocampus remains intact, the PFC-dependent compensation process does not occur. (3). Over extended periods of time, systems-type consolidation occurs: hippocampal representations are replaced by ones located in sensory-representation cortex, and if hippocampus-to-BL principal cell or inhibitory interneuron synapses had become potentiated, potentiation that has similar consequences for downstream activity becomes established at the LA targets of the new cortical representations. In the following sections we discuss some of the behavioral consequences of these considerations.
Ablation of hippocampus soon after conditioning in FRAT has little effect on CS-evoked freezing but abolishes freezing to context (compare Figures Figures9A,B).9A,B). If time is allowed for consolidation before ablation, context fear survives (Figure (Figure9C).9C). This phenomenology replicates standard empirical findings.
Extinction occurs in a normal manner after early post-training ablation (Figure (Figure9B),9B), but it is slow if hippocampus is suppressed rather than removed (Figure (Figure9D).9D). The slowing of extinction in hippocampus-suppressed FRAT was an unanticipated consequence of programming FRAT to meet the conditions of Table Table1.1. It occurs because hippocampal representations of the context and context/CS conjunctions are missing with hippocampus-suppressed, so the only representations present during extinction are those of cortex. Consequently, less total inhibition per trial is acquired than with hippocampus intact (with ablation, as opposed to suppression, cortical representations of the extinction context and of context/CS conjunctions are rapidly established at the start of extinction). In fact, Corcoran and Maren (Corcoran et al., 2005) have observed slow extinction in rats under just these circumstances.
After extinction of hippocampectomized FRAT, renewal also occurs normally (Figures (Figures9B,C).9B,C). This is the case because cortical conjunctive representations to which inhibition gets conditioned develop rapidly at the start of extinction. However, if hippocampus is suppressed rather than ablated, AAB and ABC fail though ABA renewal persists. In rats all three types of renewal have been reported to fail under similar circumstances (Corcoran et al., 2005). This may represent a discrepancy between FRAT and reality; however, we consider it is possible that apparent lack of ABA renewal in the rats was due to ceiling effects imposed by the measurement of CS fear on a high contextual fear baseline in the experiments reported.
Table Table33 summarizes a number of effects of post-conditioning, hippocampal ablations and suppressions in FRAT. It should be noted that if consolidation has been allowed to occur, extinction in the conditioning context during hippocampal suppression is normal and renewal occurs if FRAT is then tested in a different context (i.e., AAB extinction and renewal are normal). However, if extinction with hippocampus-suppressed occurs elsewhere, this is not the case. This is because consolidation establishes representations of the conditioning context and context/CS compound in cortex; but such representations for different contexts are lacking and cannot be established during extinction if hippocampus is suppressed. These would be interesting predictions to test.
If FRAT hippocampus is either ablated or suppressed soon after extinction is complete, there is partial recovery from extinction, and responsiveness becomes independent of both conditioning and extinction context (not illustrated). This occurs because all of the conjunctive representations to which inhibition has become conditioned become unavailable. Exactly this outcome was seen when rats were given hippocampal muscimol injections soon after extinction (Corcoran and Maren, 2001) although results of a later study gave somewhat different results (Corcoran and Maren, 2004); FRAT simulates the findings of both reports if one assumes that suppression of hippocampal activity by muscimol was not quite complete (see Supplementary On-line Materials). If post-extinction hippocampal ablation or suppression are performed after FRAT consolidation mechanisms have been allowed to operate, conjunctive representations have moved to cortex and synaptic potentiation has occurred that replaces the hippocampal pathway to principal cell inhibitors (as in Figure Figure6A).6A). Thus loss of hippocampus causes no loss of extinction, and renewal remains normal. We are not aware of comparable animal experiments.
The immediate shock deficit as seen in FRAT is illustrated in Figure Figure10.10. In the normal case shocks given after less than about 13s of exposure to an unfamiliar context cause development of no manifest context fear, but fear develops if there has been more cumulative exposure prior to the shock. Freezing has become nearly asymptotic by around 800s of exposure.
If FRAT hippocampus is suppressed prior to exposure to an unfamiliar context, no fear can be conditioned to that context (Figure (Figure10B).10B). However, cue conditioning appears to occur normally (Figure (Figure10C),10C), though careful evaluation of strength of conditioning, context-dependence of conditioning, etc., discloses abnormalities (not illustrated). These characteristics follow from a total absence of any contextual input to amygdala if hippocampus has been suppressed prior to any stimulus exposure. If hippocampus is ablated rather than suppressed, compensation processes allow the formation of configural representations in cortex, so the properties of both cue and context conditioning and extinction appear fairly normal, though quantitative differences can be detected. In particular, a little more context exposure is needed prior to conditioning to achieve a given level of context fear (Figure (Figure10B);10B); in this, FRAT is very similar (by design) to real animals (Wiltgen et al., 2006).
As discussed above and as prescribed in the design-criteria of Table Table1,1, PAG opiate receptor inhibitors attenuate both blocking (Figure (Figure11A)11A) and extinction (Figure (Figure11B)11B) in FRAT, as in rats (McNally et al., 2004a,b; Cole and McNally, 2007).
The discovery in rats of attenuation of extinction by opiate inhibitors and enhancement of extinction by an opiate enhancing agent (McNally et al., 2004b; McNally, 2005) would seem to imply that extinction is dependent on some signal that is itself PAG-dependent-in-effect an “extinction” signal that is extrinsic to the amygdala.
There are a number of minor variations of the circuitry controlling R and X in FRAT that would have the same ultimate effects on behavior. However, in the circuit actually employed (Figure (Figure11)11) opiate antagonists, while attenuating blocking for primary reinforcement, would not do so for secondary reinforcement. Moreover, PAG GABA inhibitors would facilitate extinction and interfere with conditioning by causing continual firing of the X-cells. It would be interesting if such effects were found in animals.
The goal of this project was to try to create a neurally plausible model that reproduces a number of known empirical features of fear learning (Table (Table1).1). If the neural plausibility of the model is accepted, the goal has been met. Moreover, the neural circuitry created to meet the criteria of Table Table11 generated a number of additional behavioral properties which appear to be empirically valid (Table (Table4;4; simulations in Supplementary On-line Materials). For example, the model predicts that renewal should be only partial and that the procedure ABA should be most and AAB least effective (Table (Table4,4, D); this conforms with common findings (Bouton, 2002; Thomas et al., 2003). The model also correctly predicts that suppression of hippocampus during extinction should slow extinction and prevent what extinction occurs from being context-dependent, whereas suppression following extinction, should itself cause partial, context-independent recovery from extinction (Table (Table4,4, E–G; Corcoran et al., 2005).
The model additionally makes many untested predictions, some of which are listed at the bottom of Table Table4.4. For example, it predicts that suppression of BLA, or CEm during the consolidation period over which contextual fear becomes hippocampus-independent should prevent such consolidation (Table (Table4,4, J). It also predicts that extinction should be prevented by CEm suppression during extinction training (Table (Table4,4, S).
Two aspects of the model especially deserve further discussion.
Control, of plasticity by R and X-cell neuromodulators was used in two ways in FRAT: (i).To make synaptic change contingent on the ultimate output of the amygdala circuit (as assessed by CEm activity). This was crucial to the handling of both secondary reinforcement and blocking. Reliable renewal also depended on control of extinction by the circuit's output. In the absence of such control, inhibition at the level of LA could grow so strong during extinction training that not enough (context-independent) input from LA to BL would remain after extinction to renew fear when context shifts lead to reduced inhibition at the level of BL. (ii). To make conditioning contingent specifically on onset of USs (or in the case of secondary reinforcement, of well-trained CSs). This was crucial to prevent a new CS that comes on during a protracted US or established CS from becoming conditioned.
These matters could perhaps have been dealt with without the use of neuromodulators. However, changes in synaptic strength must in one way or another be made subject to control by the current output of the circuit, by the level of US activity, and by increments in each of these quantities. It is possible that increments could be computed by properties of the target cells themselves, but in FRAT the computation was implicit in the properties of the R-cells. In order for external plasticity-controlling information to be utilized, it must be represented within plastic cells in a way that does not interfere with other sorts of processing. It would seem to be difficult to mediate such control via target cell membrane potential without unwanted interactions between normal processing and plasticity control signals. Indeed, prevention of unwanted secondary reinforcement by CS-caused depolarization of principal cells was the immediate impetus to positing a neuromodulatory component to the R-cell signal in FRAT. It might have been possible to avoid this, but designing the circuit was made easier by assuming that external plasticity-controlling signal work via the cell's internal chemical signaling systems, i.e., that they are neuromodulatory.
Neuromodulatory control of conditioning of course has precedent in VTA dopamine system-dependent learning (Schultz and Dickinson, 2000). Fear learning or BLA plasticity also seems to be controlled, at least in some circumstances, by dopamine and/or norepinephrine (Selden et al., 1991; Johnston et al., 1999; Bissiere et al., 2003; Pezze and Feldon, 2004; Tully et al., 2007; Fadok et al., 2009, 2010; Bush et al., 2010; Iordanova, 2010; Johansen et al., 2010a).
The presumption that an external signal influences extinction has somewhat less precedent. However, discussions of the midbrain dopamine system often imagine a role for negative error signals in extinction, and there is evidence for the importance of such signals in extinction of cerebellar eye-blink conditioning (Medina et al., 2002). The discovery that extinction of fear conditioning in rats can be attenuated by opiate inhibitors and enhanced by an opiate enhancing agent (McNally et al., 2004b; McNally, 2005) would seem to imply that extinction is dependent on some signal that is itself PAG-dependent, and thus dependent on events external to the amygdala.
It is known that dopamine neurons, whose increased activity may mediate reinforcement, slow their ongoing rates of firing when expected reinforcements are omitted and that extinction of cerebellar eye-blink responses is at least in part due to slowed spontaneous firing of climbing fibers when unreinforced eye-blink responses occur (Medina et al., 2002). This suggests a single external signal whose increase promotes reinforcement and decrease promotes extinction. However, such bipolar control requires neurons with substantial baseline firing rates, and we elected instead to use separate reinforcing and extinction-promoting signals, carried by the R and the X-cells, respectively. One consequence of this, given the effects in rats of opiate transmitter system alterations, is that CEm had to recruit X by disinhibition, whereas if the global extinction signal had been reduction in spontaneous R firing, this complexity would have been avoided. The matter is probably subject to experimental test: If extinction-promoting signals are actually recruited via disinhibition, then it should be possible to prevent extinction with PAG GABA-agonists, and to augment extinction by use of antagonists (see Supplementary On-line Materials).
Learned representations of context/cue conjunctions are crucial to the return of extinguished CS fear when FRAT is moved from contexts in which extinction was carried out (i.e., renewal). The removal of inhibition conditioned to the context alone can cause some renewal, but in real rats renewal of responding to a CS is at least partially dependent on removing the animal from the particular context in which that particular CS was extinguished (e.g., Harris et al., 2000). Related observations are what led Bouton to suggest that the CS-driven inhibition postulated to mediate extinction of a CS is “gated” by context (Bouton and Ricker, 1994), and it is what caused us to introduce context/cue conjunctive representations in FRAT. In FRAT these representations are established in hippocampus. But as pointed out below, it might also be possible to have them arise by convergence on BL cells of CS information from LA and context information from hippocampus.
Given other assumptions, the existence of context/cue conjunctive representations and the requirement that they develop within hippocampus have several implications, some of which have been partially verified: (1). If cue conditioning is weak, cued fear should be greater in the context of conditioning (where conditioned context/cue representations are present) than elsewhere (see Hall and Honey, 1990; Harris et al., 2000; and Figure Figure8).8). (2). After LA lesions, cued fear learning to context/CS conjunctive representations, which are effective in BL, should still be possible, but such cued fear should be manifest only in the context of conditioning (Table (Table4,4, M). (3). Post-training hippocampal ablation should cause some retrograde amnesia for weak cued fear if the there is a degree of overlap between training and test contexts (see Quinn et al., 2008b). (4). Manifestations of conditioning to context/cue compounds should require hippocampus-dependent pre-conditioning experience with the to-be-conditioned CS in the context where conditioning will occur (Table (Table4,4, R).
Our goal in constructing FRAT was to create an “automaton” that would emulate properties of biological fear conditioning in a biologically plausible way, but biological plausibility and parsimony sometimes conflicted. For example, LTP on inhibitory interneurons must not be allowed to go too far, or renewal will be liable to fail. Two mechanisms served to limit such LTP: (1) X-cells were made to have a sufficiently high threshold so that the neuromodulator required for inhibitory neuron synapse LTP is not released once CEm activity falls below a certain point (see Figure Figure55B2). (2) LTP at inhibitor synapses cannot occur if “Ca” level produced in inhibitors by recurrent input from principal cells falls too low (see Figure Figure5C,5C, right). It may well be that the first mechanism, which is probably the more important, is sufficient to ensure failures of renewal. But the second mechanism conforms to the biological reality that LTP and LTD commonly depend on postsynaptic Ca2+ levels. Of course there is undoubtedly considerable redundancy in biological systems. Nevertheless, one important piece of unfinished business is to determine how important individual aspects of the model are to successful simulation of various attributes of fear conditioning.
There were several points during the construction of FRAT where problems were solved by fiat without creating neural circuitry that could accomplish the needed computations. Thus: (i) Hippocampal representations of configural cues were assumed to form at defined rates during stimulus exposure. (ii) A slightly slower process with similar outcome was assumed to lead to creation of cortical conjunctive representations if hippocampus was ablated, and the formation of these cortical representations was PFC-dependent. No mechanisms for either process were postulated, nor was any mechanism for involving PFC in the latter process suggested. Nor was any mechanism postulated for detecting the absence, as opposed to the silence of the hippocampus after its ablation. (iii) Configural cue representations that had been formed in hippocampus were assumed to gradually shift to representation cortex, and their projections to LA were adjusted so that they could functionally replace the hippocampal–BL connections they replaced (consolidation). Although a detailed algorithm for accomplishing this was postulated, no neural mechanisms for implementing the algorithm was offered. (iv) Utilization of cortical representations of configural cues was made somewhat PFC-dependent, but no theory was offered as to why this might be so.
There has been considerable thinking about the formation of hippocampal representations (Marr, 1971; Skaggs and McNaughton, 1992; O'Reilly and McClelland, 1994), and this could provide guidance in creating a model of that process, but, as far as we know, the other matters have not previously been considered at a theoretical or computational level. Ever since Marr (1971) first presented his theory of hippocampal function, it has been conjectured that consolidation requires neural activity. Although experimental evidence for a role of off-line activity associated with consolidation has been sought (Marr, 1971; Wilson and McNaughton, 1994; McClelland et al., 1995; Ji and Wilson, 2007), we know of no concrete algorithms using such activity that have been proposed. In FRAT we found it necessary to set the weights of consolidated pathway synapses so that the output of the amygdala circuit in response to input under various conditions would be the same before and after consolidation. Thus, in-effect we found it necessary to use an activity-dependent mechanism. Neurally plausible algorithms for accomplishing this are needed.
Some kinds of conditional responses are well-timed events, with the CR beginning slightly before the expected onset time of the US, and there have been theoretical attempts to deal with this, such as the concept of the serial compound stimulus (Sutton and Barto, 1990) used in discussions of primate VTA dopamine neuron responses (Schultz et al., 1997). Time has been less a focus in fear conditioning studies, and the representation of time plays no role in FRAT. In models where CS duration is explicitly coded, it is possible to condition expectation of the US at a particular time after CS onset, and the failure of the US to occur at the expected time during extinction can generate an error signal that helps trigger changes responsible for extinction. In FRAT, which lacks such a signal, incremental changes responsible for extinction simply occur in each time interval during which there is fear but no reinforcement. A consequence of this is that extinction rate is affected rather strongly by CS duration during extinction; short CSs cause much less extinction per trial than longer ones (not illustrated). There is some evidence that this prediction is empirically valid (Shipley, 1974).
We made the initial simplifying assumption that LA and BL cells of FRAT are uniform populations of neurons, each innervated by all the afferents to the region via plastic synapses, and that learning and extinction of fear responses do not involve plasticity of synapses downstream of BLA. However, if there were such uniform innervation of LA and BL neurons, and if, as is likely, stimuli are coded by groups of neurons each of which also contributes to the representation of other stimuli, then after conditioning to numerous stimuli, many synapses of the circuit would be potentiated, and over-generalization to novel stimuli would be likely.
This problem can probably be greatly reduced by abandoning the assumption of uniform innervation and assuming that each BLA principal cell is innervated by different, though partially overlapping populations of afferents, or that competitive mechanisms of the kind recently discussed by Silva and colleagues (Zhou et al., 2009) force different BLA cells to mediate responsiveness to different stimuli. Under such circumstances downstream plasticity should help to make conditioning stimulus-specific (see, e.g., Brindley, 1969). It is in fact the case that plasticity downstream of BLA has been found to be important for both learning and extinction (Wilensky et al., 2006; Amano et al., 2010).
If the cells of BLA are heterogeneously innervated, then different cells of BL will be innervated by different combinations of LA neurons and by different contextual representation neuron from hippocampus. This alone should lead to at least some degree of context-specificity of extinction even without the use of context/CS conjunctive input from hippocampus. A related approach was taken in a recent computational model of BL (Vlachos et al., 2011) aimed at explaining the “extinction” neurons found there (Herry et al., 2008). Whether context/CS conjunctive representations that develop within hippocampus would in this case still be needed to produce realistic renewal behavior will depend on quantitative considerations that cannot be pre-judged. However, whether essential or not, it seems plausible that context/CS conjunctive representations should in fact form in hippocampus and should be among the representations to which fear becomes conditioned.
Although FRAT simulates many known attributes of fear conditioning (Tables (Tables11 and and4),4), there are others that the model as it now stands does not simulate. Some of these could be incorporated by small elaborations of features already present. For example, spontaneous recovery would automatically emerge if passage of time were made to alter context (Bouton, 1993). Other phenomena would require somewhat greater additions. Of particular importance in this regard are a number of effects that are thought to be due to shifts of attention to or from particular CSs as a result of past experience (McNally et al., 2004a; Iordanova et al., 2006; McNally and Westbrook, 2006). Such phenomena have motivated or supported major psychological theories of conditioning such as the Mackintosh or Pearce–Hall models (Mackintosh, 1975; Pearce and Hall, 1980). The addition of attentional mechanisms is thus an important topic for development.
It should be clear from the present paper that computational models of the kind employed here can help to make sense of findings that previously lacked explanation and can also be a rich source of untested hypotheses. It is also clear from the preceding discussion that there is considerable scope for future work.
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
This work was supported in part by NIH Grant R01 MH602122 to Michael S. Fanselow.