Search tips
Search criteria 


Logo of nihpaAbout Author manuscriptsSubmit a manuscriptHHS Public Access; Author Manuscript; Accepted for publication in peer reviewed journal;
J Neurosci. Author manuscript; available in PMC 2009 September 22.
Published in final edited form as:
PMCID: PMC2748754

Working memory for social cues recruits orbitofrontal cortex and amygdala: A functional magnetic resonance imaging study of delayed matching to sample for emotional expressions


During everyday interactions, we continuously monitor and maintain information about different individuals and their changing emotions in memory. Yet to date, working memory (WM) studies have primarily focused on mechanisms for maintaining face identity but not emotional expression, and studies investigating the neural basis of emotion have focused on transient activity, not delay related activity. The goal of this functional magnetic resonance imaging study was to investigate WM for two critical social cues: identity and emotion. Subjects performed a delayed match-to-sample task that required them to match either the emotional expression or the identity of a face after a 10 s delay. Neuroanatomically, our predictions focused on the orbitofrontal cortex (OFC) and the amygdala, as these regions have previously been implicated in emotional processing and long-term memory, and studies have demonstrated sustained OFC and medial temporal lobe activity during visual WM. Consistent with previous studies, transient activity during the sample period representing emotion and identity was found in the superior temporal sulcus and inferior occipital cortex, respectively. Sustained delay-period activity was evident in OFC, amygdala, and hippocampus, for both emotion and identity trials. These results suggest that although initial processing of emotion and identity is accomplished in anatomically segregated temporal and occipital regions, sustained delay related memory for these two critical features is held by the OFC, amygdala and hippocampus. These regions share rich connections, and have been shown previously to be necessary for binding features together in long-term memory. Our results suggest a role for these regions in active maintenance as well.

Keywords: social cognition, prefrontal cortex, hippocampus, parahippocampal cortex, active maintenance, faces


Everyday, we process and maintain information about the identity and changes in emotional expression of those with whom we interact. This study focuses on how this continuously changing social information is maintained in working memory (WM). Early WM studies explored how objects and locations, initially processed separately by dorsal and ventral visual streams, were maintained in WM. It was initially postulated that object “what” and location “where” data were maintained separately in prefrontal cortex (PFC) (Wilson et al., 1993), but a seminal study challenged this idea of anatomical segregation, suggesting instead that individual PFC neurons can hold both spatial and object information across a delay (Rao et al., 1997). In the present study, instead of examining two non-social cues (e.g. what and where), we used a delayed match-to-sample task to examine two salient social cues: Emotion and Identity. We contrasted two competing predictions. The first, based on evidence that emotion and identity are processed by different anatomical regions (Hasselmo et al., 1989), was that delay related activity would be anatomically segregated for emotion and identity. The second prediction, based on the demonstration that PFC neurons bind together information in WM, was that both emotion and identity are maintained by the same neuroanatomical system. A summary of neuroanatomical regions that we predicted would be critical for social WM follows.

Studies exploring face perception have demonstrated that attending to emotional expression activates superior temporal cortex, whereas attending to face identity activates inferior temporal and fusiform cortex (Hasselmo et al., 1989; Narumoto et al., 2001; Vuilleumier et al., 2001; Andrews and Ewbank, 2004; Winston et al., 2004). Both areas have strong anatomical connections with the orbitofrontal cortex (OFC) (Barbas, 1988; Seltzer and Pandya, 1989; Morecraft et al., 1992). A strong functional connection also exists between the amygdala and fusiform gyrus during face processing (Fairhall and Ishai, 2007).

The OFC is implicated in processing emotional information (Iidaka et al., 2001; Beer et al., 2006; Barbas, 2007; Eippert et al., 2007; Wright et al., 2008), and may maintain visual information in WM when task demands are high (Schon et al., 2008). Consistent with its role in social reinforcement learning (Kringelbach and Rolls, 2003; Rolls, 2004), the OFC contains face-selective neurons (Thorpe et al., 1983; Rolls et al., 2006) and damage to the OFC can impair face identification (Hornak et al., 2003).

Facial expressions consistently activate the amygdala (Schneider et al., 1997; Morris et al., 1998), and the amygdala enhances long-term memory for emotional stimuli (Hamann et al., 1999; Kilpatrick and Cahill, 2003; Dolcos et al., 2004; Kensinger and Schacter, 2006). Face-selective neurons exist in the amygdala (Leonard et al., 1985), and patients with amygdala damage have impaired recognition of expressions (Adolphs et al., 2002; Adolphs and Tranel, 2003).

Strong anatomical connections exist between amygdala, OFC, and additional medial temporal lobe (MTL) structures (Saunders et al., 1988; Barbas, 2000; Ghashghaei and Barbas, 2002; Barbas and Zikopoulos, 2006; Ishikawa and Nakamura, 2006; Furtak et al., 2007) suggesting that these regions play a critical role in WM for social cues.

Materials and Methods


Nineteen healthy individuals (six male; mean age = 23 ± 3) from the Boston community, with no history of neurological or psychiatric illness, were recruited for this study. Vision was either normal or corrected to normal. All subjects were screened to ensure magnetic resonance imaging (MRI) environment compatibility. Eligible individuals who agreed to participate gave signed informed consent in accordance with the Partners Human Research Committee and the Institutional Review Board of Boston University.



Stimuli for the task were selected from the University of Pennsylvania database of facial expressions (Gur et al., 2002). We selected three moderate positive, negative, and neutral valenced expressions from photographs of 24 individual actors of varying age, sex, and race, for a total set of 72 unique stimuli. All stimuli were cropped to 350×467 pixels at 28.35 pixels/cm resolution (12.35cm × 16.47cm), put on a grey background, and converted to grey-scale. An oval mask was used to isolate the central facial features (see Fig. 1). All peripheral features (e.g. hair, clothes, eye color) were removed. Control stimuli were Fourier phase scrambled versions of the original face stimuli.

Figure 1
DMS and CON tasks. In all tasks, a trial consisted of three time-locked components: a 2 s face presentation (Sample), followed by a 10 s Delay period, and by a 2 s face presentation (Test). Trials were separated by a variable length ITI. During EMO_DMS ...

Delayed-Match-to-Sample Task

During MRI scanning, subjects performed a Delayed-Match-to-Sample Task (DMS) consisting of 8 runs. The DMS task required subjects to make match or nonmatch judgments by comparing a test stimulus to a sample stimulus separated by a short delay period (see Fig. 1). Subjects performed two types of the DMS task that differed only in instruction. The first type required the subject to match the emotional expression (EMO) of the face, and the second type required the subject to match the identity (ID) of the face. For the EMO_DMS task, the test face was always a different identity than the sample face, and for the ID_DMS task, the test face was always a different emotion than the sample face (Fig. 1). This prevented the subject from seeing an identical stimulus for the Sample and Test phases of a trial. Emotional expressions and identities were counterbalanced throughout the 8 functional MRI (fMRI) runs, and the two tasks were equated in terms of the emotional and identity information that was presented. The emotional expressions were characterized as positive, negative, and neutral for counterbalancing and analysis purposes only, however the subjects were not provided with these verbal labels. Behaviorally, subjects were simply told to match the emotion or the identity, and were not informed of the number of individuals or types of emotions presented. Labeling the emotion was not a factor in selecting a response. Each trial consisted of 2 s presentations of a sample face and a test face with an intervening 10 s delay period. The length of the delay period was kept fixed because previous work has shown that different delay lengths can result in different neural response patterns (Elliott and Dolan, 1999). A variable-length inter-trial interval (ITI) (6 – 26 s, counterbalanced using Optseq2) (Dale, 1999) separated all trials. The variable ITI provided temporal “jitter” for selective averaging across trials (Postle et al., 2000a; Schon et al., 2004).

Visuomotor Control Task

Half of all trials consisted of a visuomotor control task (CON). During these trials, subjects were presented with scrambled versions of the face stimuli and responded to a “1” or “2” presented in the center of the test stimulus. The CON task acted as an effective baseline because it was designed to account for activity that was generalized to low-level visual and motor responses. These trials required no active maintenance, but maintained the subjects’ attention to the stimuli by requiring an arbitrary response.

Functional runs either consisted of EMO trials or ID trials to minimize brain activation because of task switching. Before each functional scan, the subjects received verbal instruction that the upcoming run was an EMO or ID task. In addition, the color of the central fixation cross present during delays and ITIs reminded the subject of the task type. Subjects performed 12 trials in each of the 8 functional runs (6 DMS and 6 CON trials per run; 4 EMO and 4 ID runs, randomly ordered), for a total of 96 trials (24 EMO_DMS, 24 ID_DMS, 48 CON).

Before scanning, subjects viewed task instructions and performed a practice version of the task. Responses and reaction times (RTs) were recorded from an MRI compatible button box. We created 4 different randomized versions of the task so that no greater than 5 subjects performed identical versions. Tasks were designed and presented and behavioral data were recorded using Psyscope 1.2.5 (Cohen et al., 1993).

fMRI Data Acquisition

Data were acquired on a 3.0 Tesla Siemens (Erlangen, Germany) Trio scanner at the Athinoula A. Martinos Center for Biomedical Imaging (Massachusetts General Hospital, Charlestown, MA). Two high-resolution T1-weighted anatomical images were acquired for each subject. Cognitive tasks were performed during 8 functional T2*-weighted gradient-echo, echo-planar blood oxygen level dependent (BOLD) scans [Matrix = 64×64, Field of View = 200mm, repetition time (TR) = 2 s, echo time (TE) = 30 ms, Flip Angle = 90°, 26 interleaved axial-oblique slices, voxel size = 3.125 × 3.125 × 5 mm, no interslice gap]. Slices were aligned parallel to the line connecting the anterior and posterior commisures and 144 images per slice were acquired during each scan.

fMRI Data Analysis


fMRI data was preprocessed using the SPM2 software package (Statistical Parametric Mapping, Wellcome Department of Cognitive Neurology, London, UK). BOLD images were corrected for differences in slice timing and realigned to the first image within a series. Realignment and unwarping was performed to correct for image distortions caused by susceptibility-by-movement interactions. The images were then normalized into MNI (Montreal Neurological Institute) stereotactic space, resampled to 2 × 2 × 2 mm isotropic voxels, and spatially smoothed using a 4 mm full-width half-maximum Gaussian filter.

fMRI Statistical analysis

Analysis of BOLD activity during the DMS task was assessed with multiple regression with near orthogonal regressors using the SPM2 software package. This approach has been successfully used to assess delay period activity in fMRI studies with delayed match-to-sample tasks (Courtney et al., 1997; Courtney et al., 1998; Postle and D'Esposito, 1999; Schon et al., 2004; Schon et al., 2005). As originally proposed by Courtney et al. (1997), and similar to Postle et al. (2000) and previous work in our laboratory (Schon et al., 2004; Schon et al., 2005), we created 6 regressors that reflected the comparisons of interest and convolved them with a Gamma hemodynamic response function (HRF) (Boynton et al., 1996) in MATLAB 6.5 (Mathworks, Natick, MA). The regressors consisted of the following comparisons: regressor 1 assessed activation caused by nonspecific visual stimulation, regressor 2 assessed the difference between DMS stimuli and CON stimuli, regressor 3 assessed the difference between the sample stimulus and the test stimulus for CON trials, regressor 4 assessed the difference between the sample stimulus and the test stimulus for DMS trials, regressor 5 assessed differences in activity during task delays (across tasks) versus during the ITI, and regressor 6 assessed the difference between activity during the DMS Delay versus activity during the CON Delay (see Fig. 2).

Figure 2
Multiple regression analysis. A) Stick functions were used to create six contrasts for comparison of DMS and CON task components. S = Sample; D = Delay; T = Test; ITI = Inter-trial Interval. Regressors 2 and 6 were used to assess activity related to face ...

This procedure generated a statistical parametric t-map (SPM{T}) for each regressor and for each subject. Group analyses were performed on the regressors of interest (regressors 2 and 6) by entering the SPM{T} maps from each subject into second-level random-effects one-sample t-tests, treating subjects as a random factor. For regressor 2, the significance level to identify suprathreshold voxels was p ≤ 0.001 with false discovery rate (FDR) correction (Genovese et al., 2002) across the whole brain. For regressor 6, the significance level was p ≤ 0.01 with FDR correction across the whole brain. The statistical threshold was more stringent for regressor 2 than for regressor 6 because more robust activity was expected during face presentations than during the intervening delay period. In both cases, the threshold extent was 5 contiguous voxels.

Our regressors were designed to isolate brain regions that contribute specifically to the processing of the sample and test stimuli (see Fig. 2, regressor 2) and the maintenance of the sample stimulus during the delay period (see Fig. 2, regressor 6) by contrasting DMS trials to CON trials. These regressors were constructed to account for the slow onset of the BOLD signal (HRF) relative to stimulus presentation. In addition, regressor 6 was separated into five 1/5th size stick functions spread across the 5 TRs of the delay period to account for the sustained time course and expected weaker signal during this phase of the task (Schluppeck et al., 2006) (see Fig. 2). We compensated for this expectation by using a more lenient significance threshold (described above) for analysis of delay period activity. This regression model allowed us to differentiate neural activity that was sustained across the delay period of the task from transient activity that occurred only during presentation of the face stimuli. However, because of residual collinearity between regressors 2 and 6 caused by overlap of the HRFs, the distinct contribution of a region of interest to the Sample, Delay, and Test components of the task can be determined most accurately by analyzing the time course of activation within that region. Therefore, regions that were initially activated by the sample face presentation and sustained activity across the delay were revealed by both regressors. Further exploration of the contribution of brain regions to the DMS task was accomplished by analysis of the time course of activation within a given region.

In addition, a subsequent analysis was also performed using a 16-regressor model that allowed for all possible comparisons of task conditions and trial components. This analysis searched for regions showing differential EMO and ID task effects that may not have been captured by the 6-regressor model. Data from this analysis are provided as supplemental text (available at as supplemental material).

Regions-of-interest analysis

The Regions of Interest (ROIs) chosen for our time-series analysis were the orbitofrontal cortex, the amygdala, the hippocampus, parahippocampal cortex, inferior occipital/fusiform cortex, and superior temporal cortex. ROI peak activation coordinates were derived from second-level one-sample t-tests on regressors 2 and 6. In all cases, time-series analysis was conducted on suprathreshold voxels within a 5 mm sphere surrounding the peak coordinate within each ROI (Table 1).

Table 1
Significant areas of activation from the active maintenance analysis (regressor 6; threshold at p ≤ 0.01FDR) and the face processing analysis (regressor 2; threshold at p ≤ 0.001FDR). Peak coordinates in MNI space are listed using the ...

Time-series analysis

The VOI (voxel-of-interest) tool in SPM2 was used to extract signal intensity time courses for each subject from suprathreshold voxels within each ROI. Signal intensities (Y) were adjusted for effects of interest and were extracted for each time point (144 per BOLD scan) and selectively averaged across subjects. Group level time course line charts were generated by plotting Y values across the 13 TRs (26 s) of each trial for the DMS and CON tasks (see Fig. 3Fig 5). Because the grand mean of the data was scaled to 100, the signal intensity values represent percent signal change with respect to the global mean intensity of the scaled images. The tasks were designed such that the CON task serves as the baseline. Thus, significant positive activations are evident for events where DMS > CON. To contrast activity from DMS and CON trials, signal intensity values for CON trials were subtracted from corresponding values for DMS trials to compute a signal difference (see Fig. 3Fig 5). Signal difference values were selectively averaged by task type (EMO and ID) and emotion condition (Positive, Negative, and Neutral). In order to differentiate activity related to the individual components of the task, time points were averaged together to compute separate signal amplitudes that represent the Sample (4th and 5th TRs), Delay (6th–9th TRs), and Test (10th and 11th TRs) phases of the DMS trial (see Fig. 3Fig 5). TR bins were selected in order to capture the peak of the HRF for the sample and test face responses and to account for an initial temporal lag of 6 seconds (Buckner, 1998). These values were statistically analyzed with t-tests and a three-factor repeated-measures ANOVA with the within-subject factors task type (EMO vs. ID), emotion condition (Positive vs. Negative vs. Neutral), and task component (Sample vs. Delay vs. Test). All statistics were corrected for unequal variance among factor levels whenever necessary.

Figure 3
Sustained delay period activity in orbitofrontal cortex. A) fMRI results from the active maintenance analysis (regressor 6) demonstrating sustained activity in left OFC, x = −46, y = 26, z = −6 (arrows). Statistical parametric maps are ...
Figure 5
fMRI results from face processing analysis (regressor 2) demonstrating transient activity that is modulated by task. A) Right inferior occipital cortex, x = 50, y = −74, z = −14 (arrows). B) Right superior temporal sulcus, x = 50, y = ...

Behavioral Data Analysis

Match/nonmatch responses and reaction times were recorded for each trial to ensure the subjects were performing the task as instructed. Accuracy and average RT’s were analyzed with t-tests.


Behavioral Results

Subjects performed DMS tasks at close to 90% accuracy and showed no significant difference in accuracy between the EMO task (mean ± SE: 85.9 ± 1.7%) and ID task (89.6 ± 2.8%). Reaction times for the two tasks (EMO: 1210 ± 35 ms; ID: 1189 ± 34 ms) also did not differ significantly. Identity trials were excluded for two subjects because of poor task performance. The CON task was performed at close to 100% accuracy (mean ± SE: 99.4 ± 0.6%) and with significantly faster reaction times (mean ± SE: 822 ± 15 ms) than both DMS tasks (EMO vs. CON, t26 = 10.328, p < 0.001; ID vs. CON, t21 = 10.000, p < 0.001).

fMRI Results

Sustained delay related activity in the orbitofrontal cortex, amygdala, and hippocampus

Active maintenance of information in working memory was assessed by contrasting DMS delay period activity with CON delay period activity (Fig. 2, regressor 6). This analysis revealed significant delay period activity in left orbitofrontal cortex (OFC; peak at x = −46, y = 26, z = −6; T = 7.84, Z = 5.92, p < 0.01FDR) (Fig. 3). This cluster of activation extended dorsally and rostrally into inferior frontal gyrus (triangularis) and laterally into the insula. A repeated-measures ANOVA on the signal difference (DMS - CON) time-series in this ROI revealed a significant main effect of task (EMO vs. ID) (F1,16 = 9.76, p = 0.007) and a significant main effect of component (Sample vs. Delay vs. Test) (F2,32 = 4.20, p = 0.032). t-tests revealed significantly greater activation for EMO trials than ID trials (t99 = 2.262, p = 0.026) (Fig. 3C) during the Sample phase in this ROI. t-tests on the individual emotion conditions (Positive vs. Negative vs. Neutral) in the OFC revealed significantly greater activation for negative faces than positive faces (t36 = 2.613, p = 0.013) (Fig. 3D) during the Test phase for EMO trials.

Significant delay period activity was also found in medial temporal lobe structures including left amygdala (peak at x = −18, y = −4, z = −20; T = 3.63, Z = 3.32, p < 0.01FDR) (Fig. 4A) and left mid-hippocampal body (peak at x = −32, y = −30, z = −8; T = 5.23, Z = 4.46, p < 0.01FDR) (Fig. 4B). A repeated-measures ANOVA on the signal difference (DMS - CON) time-series in the left amygdala ROI revealed a significant main effect of component (Sample vs. Delay vs. Test) (F2,32 = 4.92, p = 0.016). A repeated-measures ANOVA and t-tests did not reveal any significant differences between any of the conditions in the left hippocampal ROI. In both ROIs, activation appeared to be sustained early in the delay period (Fig. 4A,B, line graphs). Significant activation was also present in left insula, left inferior frontal gyrus (triangularis), the supplementary motor area (SMA), anterior parahippocampal gyrus, right posterior MTL (including the hippocampus and parahippocampal cortex), and left inferior parietal cortex.

Figure 4
Sustained delay period activity in medial temporal lobe structures. A) Left amygdala, x = −18, y = −4, z = −20 (arrows). B) Left mid-hippocampal body, x = −32, y = −30, z = −8 (arrows). Statistical parametric ...

In summary, robust activity in the OFC was sustained across the delay period. In OFC, during the Sample phase, responses were larger for EMO trials than ID trials, and during the Test phase of EMO trials, negative faces elicited a greater response than positive faces. Sustained activity was also found in the hippocampus and amygdala early in the delay period, but activity in these MTL areas was not modulated by task. See Table 1 for a complete list of areas of activation revealed by regressor 6.

Transient activity in temporal and occipital cortices and posterior parahippocampal cortex

Transient responses to the presentation of face stimuli were assessed by contrasting DMS Sample and Test phase activity with CON Sample and Test phase activity (Fig. 2, regressor 2). This analysis revealed significant transient responses to the sample and test face presentations in right inferior occipital cortex (peak at x = 50, y = −74, z = −14; T = 9.87, Z = 6.78, p < 0.001FDR) (Fig. 5A). The activation extended anteriorly into inferior temporal gyrus and medially into the fusiform gyrus. A repeated-measures ANOVA on the signal difference (DMS - CON) time-series in this ROI revealed a significant main effect of component (Sample vs. Delay vs. Test) (F2,32 = 47.72, p < 0.001) and a significant task (EMO vs. ID) by component interaction (F2,32 = 3.82, p = 0.042). t-tests revealed significantly greater activation for ID trials than EMO trials (t105 = 2.300, p = 0.023) (Fig. 5A) during the Sample phase in this ROI.

This analysis also revealed significant transient responses during Sample and Test periods in right superior temporal sulcus (STS, peak at x = 50, y = −36, z = 0; T = 6.50, Z = 5.23, p < 0.001FDR) (Fig. 5B). This activation extended along the rostral/caudal axis of the STS. A repeated-measures ANOVA on the signal difference (DMS - CON) time-series in this ROI revealed a significant emotion (Positive vs. Negative vs. Neutral) by component (Sample vs. Delay vs. Test) interaction (F4,64 = 4.59, p = 0.009). t-tests revealed significantly greater activation for EMO trials than ID trials (t98 = 1.987, p = 0.050) (Fig. 5B) during the Sample phase in this ROI.

Transient responses were also found in right posterior parahippocampal cortex (peak at x = 16, y = −36, z = −6; T = 8.16, Z = 6.07, p < 0.001FDR) (Fig. 5C). A repeated-measures ANOVA on the signal difference (DMS - CON) time-series did not reveal any significant effects, however t-tests did show significantly greater activation for EMO trials than ID trials (t105 = 2.177, p = 0.032) (Fig. 5C) during the Test phase in this ROI. Significant activation was also present in left inferior occipital cortex, left fusiform gyrus, left STS, bilaterally in the insula, bilaterally in orbitofrontal and inferior frontal cortex, left posterior parahippocampal cortex, bilaterally in the anterior MTL (including the amygdala), and in the SMA spanning ventrally along the medial wall into anterior cingulate cortex.

In summary, transient activity related to the Sample and Test phases of each trial that returned to baseline during the delay period was found in temporal and occipital cortices, including the right inferior occipital cortex and right STS, and in right posterior parahippocampal cortex. The most robust activity was found in the inferior occipital cortex. During the Sample phase, responses in inferior occipital cortex were larger for ID trials than EMO trials, whereas responses in STS were larger for EMO trials than ID trials. During the Test phase, responses in posterior parahippocampal cortex were larger for EMO trials than ID trials. See Table 1 for a complete list of areas of activation revealed by regressor 2.

Results from the 16-regressor model comparing EMO and ID tasks can be found in the supplemental text (available at as supplemental material).


Our goal was to explore working memory (WM) for two critical social cues, emotion (EMO) and identity (ID). We examined two competing predictions. The first, based on the fact that emotion and identity are processed perceptually by separate systems, was that sustained delay related activity in EMO and ID trials would be anatomically segregated and actively maintained within ventral visual areas. The second, based on the demonstration that prefrontal cortex (PFC) neurons can actively maintain disparate stimulus types, such as what and where, was that emotion and identity would be maintained by a single neuroanatomical system. Our results support this second prediction. Neuroanatomically, our predictions and analysis focused on (1) a perceptual system within the temporal and occipital cortices (TOC) that is known to separately process identity and emotion and (2) a neuroanatomically interconnected memory system comprised of the orbitofrontal cortex (OFC), amygdala, and hippocampus, that has been implicated in long-term emotional memory and more recently in visual WM paradigms.

A discussion of these two neuroanatomical systems, one critical for active maintenance and the other critical for transient perceptual dissociation, follows.

Active maintenance of social cues

Our results demonstrate sustained delay related activity within the OFC, the amygdala, and hippocampus for both emotion and identity (Fig. 3,Fig 4, Table 1). Neuroanatomically, strong connections exist between these three structures (Saunders et al., 1988; Barbas, 2000; Ghashghaei and Barbas, 2002; Barbas and Zikopoulos, 2006; Ishikawa and Nakamura, 2006; Furtak et al., 2007). Additional connections from TOC to OFC provide a critical link between the regions we found that were transiently active, and those that demonstrated sustained activity (Barbas, 2007).

Our results extend previous studies of WM demonstrating contributions of prefrontal structures, including the OFC, to the active maintenance of faces (Courtney et al., 1996; Haxby et al., 2000; Sala et al., 2003), and differ from several studies demonstrating active maintenance in TOC during face working memory tasks (Druzgal and D'Esposito, 2003; Postle et al., 2003; Xu and Chun, 2006; Yoon et al., 2006). However, it should be noted that fMRI studies examining WM maintenance in TOC do not consistently find delay related activity, and our results may help explain these inconsistencies. Critically, the present study differs from these earlier studies in that two features, identity and emotion, were varied across trials, rather than simply varying a single stimulus feature, identity. Therefore, it may be that the TOC is sufficient for the active maintenance of simple stimuli, including objects and faces (Druzgal and D'Esposito, 2003; Postle et al., 2003; Xu and Chun, 2006; Yoon et al., 2006), but when multiple stimulus dimensions such as emotion and identity are held across a delay, these TOC regions may no longer be sufficient. Therefore, although one might have predicted sustained activity in the superior temporal sulcus for maintaining emotion and sustained activity in inferior occipital/fusiform cortex for maintaining identity, our data do not support this. Instead, our data suggest that these separate streams of information are relayed from the TOC to the OFC, and to the amygdala (Barbas, 2007). These structures, together with the hippocampus, act to bind the two features together and hold this unified information in WM. A similar system for maintaining conjunctions of object and spatial information in WM has been proposed in the lateral PFC in the monkey (Rao et al., 1997; Rainer et al., 1998) and human (Owen et al., 1998; Postle et al., 2000b).

Our data are consistent with the role of the OFC in social reinforcement learning (Kringelbach and Rolls, 2003; Rolls, 2004, 2007), which, in turn, is consistent with its prominent role in rapid stimulus-reinforcement learning (Rolls, 2000; O'Doherty, 2004; Rolls, 2004; Rolls et al., 2006; Dolan, 2007). A recent review suggests the OFC may be critical for promoting flexible behaviors and guiding response selection, and in addition, is critical for reflecting changes in reinforcement values, and in modifying behavioral responses when predicted outcomes (such as reinforcement) change (for review, see Murray and Izquierdo, 2007). In our task, the subjects need to flexibly switch between selecting their responses based on emotion and identity. This might be accomplished by inhibiting one stimulus dimension and maintaining the other. Or, as we suggest, it might be accomplished by binding together and maintaining both stimulus dimensions, emotion and identity, across the delay, and then flexibly selecting the correct response.

We predicted that our task would engage the amygdala. Face responsive neurons have been identified in the amygdala (Leonard et al., 1985), and imaging studies have demonstrated amygdala activation in response to expressions (Schneider et al., 1997; Morris et al., 1998; Blair et al., 1999; Iidaka et al., 2001; Killgore and Yurgelun-Todd, 2004). Memory studies have implicated the amygdala in the enhancement of long-term encoding of emotionally charged stimuli (Cahill et al., 1996; Canli et al., 2000; Kensinger and Schacter, 2006), and the amygdala exerts an influence on activity in prefrontal cortex and medial temporal lobe (MTL) structures during long-term memory encoding (Hamann et al., 1999; Kilpatrick and Cahill, 2003). In addition, amygdala activity has been demonstrated during implicit processing of emotional stimuli (Critchley et al., 2000; Phelps and LeDoux, 2005) or when these stimuli are task-irrelevant distracters (Siegle et al., 2002; Dolcos and McCarthy, 2006; Siegle et al., 2007), which is consistent with the fact that this limbic region showed sustained activity in our task regardless of trial type. In addition to its known role in long-term emotional memory and consolidation (McGaugh, 2004), the sustained activity noted in the present study suggests an additional role for the amygdala in the active maintenance of emotional and socially relevant material in WM.

In addition to sustained activity in the OFC and amygdala, we also found sustained activity within the hippocampus. Recent fMRI data has demonstrated a role for MTL structures, including the hippocampus and surrounding parahippocampal cortex, in the short term maintenance of novel information (Ranganath and D'Esposito, 2001; Stern et al., 2001; Ranganath et al., 2003; Schon et al., 2004; Ranganath et al., 2005; Schon et al., 2005). Based on the rich connections between the hippocampus, amygdala, and OFC (Barbas, 2000; Insausti and Munoz, 2001; Munoz and Insausti, 2005), we suggest that together these areas form a circuit that binds together and actively maintains emotion and identity information in WM.

Transient responses to face presentations in temporal and occipital regions

Our results demonstrated transient activity during the Sample period in the inferior occipital cortex, extending into the fusiform gyrus, that was significantly greater during ID trials than during EMO trials, and transient activity in the superior temporal sulcus that was significantly greater during EMO trials than ID trials (Figs. 5A,B). This result is consistent with data showing that inferior occipito-temporal regions are involved in perception of facial identity (Kanwisher et al., 1997; Haxby et al., 2001), whereas superior temporal regions are involved in perception of emotional cues and may be important for social perception in general (Hasselmo et al., 1989; Narumoto et al., 2001; Vuilleumier et al., 2001; Andrews and Ewbank, 2004; Winston et al., 2004). Neuroanatomically, the superior temporal sulcus and inferior temporal cortex have strong connections with the OFC (Barbas, 1988; Seltzer and Pandya, 1989; Morecraft et al., 1992), and an effective connectivity study has demonstrated a strong functional connection between amygdala and fusiform gyrus during processing of faces (Fairhall and Ishai, 2007).

In addition to regions within the TOC, posterior parahippocampal cortex only responded transiently to face presentations (Fig. 5C). This result is consistent with a study that found sustained activity in the hippocampus, but transient activity in parahippocampal and anterior fusiform gyri, during delayed matching of faces (Ranganath and D'Esposito, 2001).

Linking the Two Systems

The active maintenance of social cues requires the cooperation between systems critical for perceptually identifying and processing emotion and identity, and systems critical for actively maintaining this information in memory. In studies where only one dimension, such as identity, is varied, it may be possible for posterior TOC to actively maintain this information in the same system where the information is initially perceptually processed. But in more natural social situations, such as when both identity and emotional expressions can vary, then information from these perceptual TOC regions may be relayed to a memory system that is capable of flexibly binding and maintaining this information together. We suggest this system is comprised of the orbitofrontal cortex, the amygdala, and the hippocampus. We conclude that this system, which is already known to play a critical role in binding together emotional information in long-term memory (McGaugh, 2004), is also critical for actively maintaining, and binding together, salient social cues in working memory.

Supplementary Material



This work was supported by National Institutes of Health Grant P50 MH071702-02, National Center for Research Resources Grant P41RR14075, National Science Foundation Science of Learning Center Grant SBE-0354378, and by the Program in Neuroscience and the Department of Psychology at Boston University. We thank Courtney Horwitz and Matthew Grace for assistance with stimulus preparation and data analysis, and Michael Hasselmo for discussions about the study. We also thank Dr. Ruben Gur’s Brain Behavior Lab at the University of Pennsylvania for supplying the face stimuli used in this study. Research was conducted at the Cognitive Neuroimaging Lab, Center for Memory and Brain, Boston University (Boston, MA), and the Athinoula A. Martinos Center for Biomedical Imaging, Massachusetts General Hospital, Harvard Medical School (Charlestown, MA).


  • Adolphs R, Tranel D. Amygdala damage impairs emotion recognition from scenes only when they contain facial expressions. Neuropsychologia. 2003;41:1281–1289. [PubMed]
  • Adolphs R, Baron-Cohen S, Tranel D. Impaired recognition of social emotions following amygdala damage. J Cogn Neurosci. 2002;14:1264–1274. [PubMed]
  • Andrews TJ, Ewbank MP. Distinct representations for facial identity and changeable aspects of faces in the human temporal lobe. Neuroimage. 2004;23:905–913. [PubMed]
  • Barbas H. Anatomic organization of basoventral and mediodorsal visual recipient prefrontal regions in the rhesus monkey. J Comp Neurol. 1988;276:313–342. [PubMed]
  • Barbas H. Connections underlying the synthesis of cognition, memory, and emotion in primate prefrontal cortices. Brain Res Bull. 2000;52:319–330. [PubMed]
  • Barbas H. Flow of information for emotions through temporal and orbitofrontal pathways. J Anat. 2007;211:237–249. [PubMed]
  • Barbas H, Zikopoulos B. Sequential and parallel circuits for emotional processing in primate orbitofrontal cortex. In: Zald DH, Rauch SL, editors. The orbitofrontal cortex. Oxford; New York: Oxford University Press; 2006. pp. 57–91.
  • Beer JS, John OP, Scabini D, Knight RT. Orbitofrontal cortex and social behavior: integrating self-monitoring and emotion-cognition interactions. J Cogn Neurosci. 2006;18:871–879. [PubMed]
  • Blair RJ, Morris JS, Frith CD, Perrett DI, Dolan RJ. Dissociable neural responses to facial expressions of sadness and anger. Brain. 1999;122(Pt 5):883–893. [PubMed]
  • Boynton GM, Engel SA, Glover GH, Heeger DJ. Linear systems analysis of functional magnetic resonance imaging in human V1. J Neurosci. 1996;16:4207–4221. [PubMed]
  • Buckner RL. Event-related fMRI and the hemodynamic response. Hum Brain Mapp. 1998;6:373–377. [PubMed]
  • Cahill L, Haier RJ, Fallon J, Alkire MT, Tang C, Keator D, Wu J, McGaugh JL. Amygdala activity at encoding correlated with long-term, free recall of emotional information. Proc Natl Acad Sci U S A. 1996;93:8016–8021. [PubMed]
  • Canli T, Zhao Z, Brewer J, Gabrieli JD, Cahill L. Event-related activation in the human amygdala associates with later memory for individual emotional experience. J Neurosci. 2000;20:RC99. [PubMed]
  • Cohen JD, MacWhinney B, Flatt M, Provost J. PsyScope: A new graphic interactive environment for designing psychology experiments. Behavioral Research Methods, Instruments, and Computers. 1993;25:257–271.
  • Courtney SM, Ungerleider LG, Keil K, Haxby JV. Object and spatial visual working memory activate separate neural systems in human cortex. Cereb Cortex. 1996;6:39–49. [PubMed]
  • Courtney SM, Ungerleider LG, Keil K, Haxby JV. Transient and sustained activity in a distributed neural system for human working memory. Nature. 1997;386:608–611. [PubMed]
  • Courtney SM, Petit L, Maisog JM, Ungerleider LG, Haxby JV. An area specialized for spatial working memory in human frontal cortex. Science. 1998;279:1347–1351. [PubMed]
  • Critchley H, Daly E, Phillips M, Brammer M, Bullmore E, Williams S, Van Amelsvoort T, Robertson D, David A, Murphy D. Explicit and implicit neural mechanisms for processing of social information from facial expressions: a functional magnetic resonance imaging study. Hum Brain Mapp. 2000;9:93–105. [PubMed]
  • Dale AM. Optimal experimental design for event-related fMRI. Hum Brain Mapp. 1999;8:109–114. [PubMed]
  • Dolan RJ. The human amygdala and orbital prefrontal cortex in behavioural regulation. Philos Trans R Soc Lond B Biol Sci. 2007;362:787–799. [PMC free article] [PubMed]
  • Dolcos F, McCarthy G. Brain systems mediating cognitive interference by emotional distraction. J Neurosci. 2006;26:2072–2079. [PubMed]
  • Dolcos F, LaBar KS, Cabeza R. Interaction between the amygdala and the medial temporal lobe memory system predicts better memory for emotional events. Neuron. 2004;42:855–863. [PubMed]
  • Druzgal TJ, D'Esposito M. Dissecting contributions of prefrontal cortex and fusiform face area to face working memory. J Cogn Neurosci. 2003;15:771–784. [PubMed]
  • Eippert F, Veit R, Weiskopf N, Erb M, Birbaumer N, Anders S. Regulation of emotional responses elicited by threat-related stimuli. Hum Brain Mapp. 2007;28:409–423. [PubMed]
  • Elliott R, Dolan RJ. Differential neural responses during performance of matching and nonmatching to sample tasks at two delay intervals. J Neurosci. 1999;19:5066–5073. [PubMed]
  • Fairhall SL, Ishai A. Effective connectivity within the distributed cortical network for face perception. Cereb Cortex. 2007;17:2400–2406. [PubMed]
  • Furtak SC, Wei SM, Agster KL, Burwell RD. Functional neuroanatomy of the parahippocampal region in the rat: the perirhinal and postrhinal cortices. Hippocampus. 2007;17:709–722. [PubMed]
  • Genovese CR, Lazar NA, Nichols T. Thresholding of statistical maps in functional neuroimaging using the false discovery rate. Neuroimage. 2002;15:870–878. [PubMed]
  • Ghashghaei HT, Barbas H. Pathways for emotion: interactions of prefrontal and anterior temporal pathways in the amygdala of the rhesus monkey. Neuroscience. 2002;115:1261–1279. [PubMed]
  • Gur RC, Sara R, Hagendoorn M, Marom O, Hughett P, Macy L, Turner T, Bajcsy R, Posner A, Gur RE. A method for obtaining 3-dimensional facial expressions and its standardization for use in neurocognitive studies. J Neurosci Methods. 2002;115:137–143. [PubMed]
  • Hamann SB, Ely TD, Grafton ST, Kilts CD. Amygdala activity related to enhanced memory for pleasant and aversive stimuli. Nat Neurosci. 1999;2:289–293. [PubMed]
  • Hasselmo ME, Rolls ET, Baylis GC. The role of expression and identity in the faceselective responses of neurons in the temporal visual cortex of the monkey. Behav Brain Res. 1989;32:203–218. [PubMed]
  • Haxby JV, Petit L, Ungerleider LG, Courtney SM. Distinguishing the functional roles of multiple regions in distributed neural systems for visual working memory. Neuroimage. 2000;11:380–391. [PubMed]
  • Haxby JV, Gobbini MI, Furey ML, Ishai A, Schouten JL, Pietrini P. Distributed and overlapping representations of faces and objects in ventral temporal cortex. Science. 2001;293:2425–2430. [PubMed]
  • Hornak J, Bramham J, Rolls ET, Morris RG, O'Doherty J, Bullock PR, Polkey CE. Changes in emotion after circumscribed surgical lesions of the orbitofrontal and cingulate cortices. Brain. 2003;126:1691–1712. [PubMed]
  • Iidaka T, Omori M, Murata T, Kosaka H, Yonekura Y, Okada T, Sadato N. Neural interaction of the amygdala with the prefrontal and temporal cortices in the processing of facial expressions as revealed by fMRI. J Cogn Neurosci. 2001;13:1035–1047. [PubMed]
  • Insausti R, Munoz M. Cortical projections of the non-entorhinal hippocampal formation in the cynomolgus monkey (Macaca fascicularis) Eur J Neurosci. 2001;14:435–451. [PubMed]
  • Ishikawa A, Nakamura S. Ventral hippocampal neurons project axons simultaneously to the medial prefrontal cortex and amygdala in the rat. J Neurophysiol. 2006;96:2134–2138. [PubMed]
  • Kanwisher N, McDermott J, Chun MM. The fusiform face area: a module in human extrastriate cortex specialized for face perception. J Neurosci. 1997;17:4302–4311. [PubMed]
  • Kensinger EA, Schacter DL. Amygdala activity is associated with the successful encoding of item, but not source, information for positive and negative stimuli. J Neurosci. 2006;26:2564–2570. [PubMed]
  • Killgore WD, Yurgelun-Todd DA. Activation of the amygdala and anterior cingulate during nonconscious processing of sad versus happy faces. Neuroimage. 2004;21:1215–1223. [PubMed]
  • Kilpatrick L, Cahill L. Amygdala modulation of parahippocampal and frontal regions during emotionally influenced memory storage. Neuroimage. 2003;20:2091–2099. [PubMed]
  • Kringelbach ML, Rolls ET. Neural correlates of rapid reversal learning in a simple model of human social interaction. Neuroimage. 2003;20:1371–1383. [PubMed]
  • Leonard CM, Rolls ET, Wilson FA, Baylis GC. Neurons in the amygdala of the monkey with responses selective for faces. Behav Brain Res. 1985;15:159–176. [PubMed]
  • Maldjian JA, Laurienti PJ, Kraft RA, Burdette JH. An automated method for neuroanatomic and cytoarchitectonic atlas-based interrogation of fMRI data sets. Neuroimage. 2003;19:1233–1239. [PubMed]
  • McGaugh JL. The amygdala modulates the consolidation of memories of emotionally arousing experiences. Annu Rev Neurosci. 2004;27:1–28. [PubMed]
  • Morecraft RJ, Geula C, Mesulam MM. Cytoarchitecture and neural afferents of orbitofrontal cortex in the brain of the monkey. J Comp Neurol. 1992;323:341–358. [PubMed]
  • Morris JS, Friston KJ, Buchel C, Frith CD, Young AW, Calder AJ, Dolan RJ. A neuromodulatory role for the human amygdala in processing emotional facial expressions. Brain. 1998;121(Pt 1):47–57. [PubMed]
  • Munoz M, Insausti R. Cortical efferents of the entorhinal cortex and the adjacent parahippocampal region in the monkey (Macaca fascicularis) Eur J Neurosci. 2005;22:1368–1388. [PubMed]
  • Murray EA, Izquierdo A. Orbitofrontal cortex and amygdala contributions to affect and action in primates. Ann N Y Acad Sci. 2007;1121:273–296. [PubMed]
  • Narumoto J, Okada T, Sadato N, Fukui K, Yonekura Y. Attention to emotion modulates fMRI activity in human right superior temporal sulcus. Brain Res Cogn Brain Res. 2001;12:225–231. [PubMed]
  • O'Doherty JP. Reward representations and reward-related learning in the human brain: insights from neuroimaging. Curr Opin Neurobiol. 2004;14:769–776. [PubMed]
  • Owen AM, Stern CE, Look RB, Tracey I, Rosen BR, Petrides M. Functional organization of spatial and nonspatial working memory processing within the human lateral frontal cortex. Proc Natl Acad Sci U S A. 1998;95:7721–7726. [PubMed]
  • Phelps EA, LeDoux JE. Contributions of the amygdala to emotion processing: from animal models to human behavior. Neuron. 2005;48:175–187. [PubMed]
  • Postle BR, D'Esposito M. "What"-Then-Where" in visual working memory: an eventrelated fMRI study. J Cogn Neurosci. 1999;11:585–597. [PubMed]
  • Postle BR, Zarahn E, D'Esposito M. Using event-related fMRI to assess delay-period activity during performance of spatial and nonspatial working memory tasks. Brain Res Brain Res Protoc. 2000a;5:57–66. [PubMed]
  • Postle BR, Druzgal TJ, D'Esposito M. Seeking the neural substrates of visual working memory storage. Cortex. 2003;39:927–946. [PubMed]
  • Postle BR, Stern CE, Rosen BR, Corkin S. An fMRI investigation of cortical contributions to spatial and nonspatial visual working memory. Neuroimage. 2000b;11:409–423. [PubMed]
  • Rainer G, Asaad WF, Miller EK. Memory fields of neurons in the primate prefrontal cortex. Proc Natl Acad Sci U S A. 1998;95:15008–15013. [PubMed]
  • Ranganath C, D'Esposito M. Medial temporal lobe activity associated with active maintenance of novel information. Neuron. 2001;31:865–873. [PubMed]
  • Ranganath C, Johnson MK, D'Esposito M. Prefrontal activity associated with working memory and episodic long-term memory. Neuropsychologia. 2003;41:378–389. [PubMed]
  • Ranganath C, Cohen MX, Brozinsky CJ. Working memory maintenance contributes to long-term memory formation: neural and behavioral evidence. J Cogn Neurosci. 2005;17:994–1010. [PubMed]
  • Rao SC, Rainer G, Miller EK. Integration of what and where in the primate prefrontal cortex. Science. 1997;276:821–824. [PubMed]
  • Rolls ET. The orbitofrontal cortex and reward. Cereb Cortex. 2000;10:284–294. [PubMed]
  • Rolls ET. The functions of the orbitofrontal cortex. Brain Cogn. 2004;55:11–29. [PubMed]
  • Rolls ET. The representation of information about faces in the temporal and frontal lobes. Neuropsychologia. 2007;45:124–143. [PubMed]
  • Rolls ET, Critchley HD, Browning AS, Inoue K. Face-selective and auditory neurons in the primate orbitofrontal cortex. Exp Brain Res. 2006;170:74–87. [PubMed]
  • Rorden C, Brett M. Stereotaxic display of brain lesions. Behav Neurol. 2000;12:191–200. [PubMed]
  • Sala JB, Rama P, Courtney SM. Functional topography of a distributed neural system for spatial and nonspatial information maintenance in working memory. Neuropsychologia. 2003;41:341–356. [PubMed]
  • Saunders RC, Rosene DL, Van Hoesen GW. Comparison of the efferents of the amygdala and the hippocampal formation in the rhesus monkey: II. Reciprocal and non-reciprocal connections. J Comp Neurol. 1988;271:185–207. [PubMed]
  • Schluppeck D, Curtis CE, Glimcher PW, Heeger DJ. Sustained activity in topographic areas of human posterior parietal cortex during memory-guided saccades. J Neurosci. 2006;26:5098–5108. [PMC free article] [PubMed]
  • Schneider F, Grodd W, Weiss U, Klose U, Mayer KR, Nagele T, Gur RC. Functional MRI reveals left amygdala activation during emotion. Psychiatry Res. 1997;76:75–82. [PubMed]
  • Schon K, Tinaz S, Somers DC, Stern CE. Delayed match to object or place: An eventrelated fMRI study of short-term stimulus maintenance and the role of stimulus pre-exposure. Neuroimage. 2008;39:857–872. [PMC free article] [PubMed]
  • Schon K, Hasselmo ME, Lopresti ML, Tricarico MD, Stern CE. Persistence of parahippocampal representation in the absence of stimulus input enhances long-term encoding: a functional magnetic resonance imaging study of subsequent memory after a delayed match-tosample task. J Neurosci. 2004;24:11088–11097. [PubMed]
  • Schon K, Atri A, Hasselmo ME, Tricarico MD, LoPresti ML, Stern CE. Scopolamine reduces persistent activity related to long-term encoding in the parahippocampal gyrus during delayed matching in humans. J Neurosci. 2005;25:9112–9123. [PubMed]
  • Seltzer B, Pandya DN. Frontal lobe connections of the superior temporal sulcus in the rhesus monkey. J Comp Neurol. 1989;281:97–113. [PubMed]
  • Siegle GJ, Steinhauer SR, Thase ME, Stenger VA, Carter CS. Can't shake that feeling: event-related fMRI assessment of sustained amygdala activity in response to emotional information in depressed individuals. Biol Psychiatry. 2002;51:693–707. [PubMed]
  • Siegle GJ, Thompson W, Carter CS, Steinhauer SR, Thase ME. Increased amygdala and decreased dorsolateral prefrontal BOLD responses in unipolar depression: related and independent features. Biol Psychiatry. 2007;61:198–209. [PubMed]
  • Stern CE, Sherman SJ, Kirchhoff BA, Hasselmo ME. Medial temporal and prefrontal contributions to working memory tasks with novel and familiar stimuli. Hippocampus. 2001;11:337–346. [PubMed]
  • Thorpe SJ, Rolls ET, Maddison S. The orbitofrontal cortex: neuronal activity in the behaving monkey. Exp Brain Res. 1983;49:93–115. [PubMed]
  • Vuilleumier P, Armony JL, Driver J, Dolan RJ. Effects of attention and emotion on face processing in the human brain: an event-related fMRI study. Neuron. 2001;30:829–841. [PubMed]
  • Wilson FA, Scalaidhe SP, Goldman-Rakic PS. Dissociation of object and spatial processing domains in primate prefrontal cortex. Science. 1993;260:1955–1958. [PubMed]
  • Winston JS, Henson RN, Fine-Goulden MR, Dolan RJ. fMRI-adaptation reveals dissociable neural representations of identity and expression in face perception. J Neurophysiol. 2004;92:1830–1839. [PubMed]
  • Wright P, Albarracin D, Brown RD, Li H, He G, Liu Y. Dissociated responses in the amygdala and orbitofrontal cortex to bottom-up and top-down components of emotional evaluation. Neuroimage. 2008;39:894–902. [PMC free article] [PubMed]
  • Xu Y, Chun MM. Dissociable neural mechanisms supporting visual short-term memory for objects. Nature. 2006;440:91–95. [PubMed]
  • Yoon JH, Curtis CE, D'Esposito M. Differential effects of distraction during working memory on delay-period activity in the prefrontal cortex and the visual association cortex. Neuroimage. 2006;29:1117–1126. [PubMed]