PMCCPMCCPMCC

Search tips
Search criteria 

Advanced

 
Logo of scanLink to Publisher's site
 
Soc Cogn Affect Neurosci. 2006 September; 1(2): 95–106.
Prepublished online 2006 August 14. doi:  10.1093/scan/nsl014
PMCID: PMC1810228
EMSID: UKMS167

The Kuleshov Effect: the influence of contextual framing on emotional attributions

Abstract

Filmmakers have long recognized the importance of editing techniques to guide the audiences' perceptions and enhance the impact of a scene. We demonstrate behaviorally that pairing identical faces with either neutral or emotionally salient contextual movies, an editing technique referred to as the ‘Kuleshov Effect’, results in both altered attributions of facial expression and mental-state. Using functional neuroimaging (fMRI), we show that faces paired with emotional movies enhance BOLD responses in the bilateral temporal pole, anterior cingulate cortices, amygdala and bilateral superior temporal sulcus relative to identical faces juxtaposed with neutral movies. An interaction was observed in the right amygdala when subtle happy and fear faces were juxtaposed with positive and negative movies, respectively. An interaction between happy faces and negative context was also observed in bilateral amygdala suggesting that the amygdala may act to prime or tag affective value to faces. A parametric modulation of BOLD signal by attribution ratings indicated a dissociation between ventrolateral and the ventromedial prefrontal cortex for negative and positive contextually evoked attributions, respectively. These prefrontal regions may act to guide appropriate choices across altering contexts. Together, these findings offer a neurobiological basis for contextual framing effects on social attributions.

Keywords: fMRI, kuleshov effect, context, affect, mental-state

Nearly a century ago, Soviet filmmaker Lev Kuleshov demonstrated that the manipulation of context can alter an audiences’ perception of an actor's facial expressions, thoughts and feelings. For example, juxtaposition of identical archived clips of actor Ivan Mozzhukhin's face with either a scene of a funeral or a child playing led the audience to infer Mozzhukhin's emotional disposition as subtly melancholic or happy, respectively (Kawin, 1992). Despite the somewhat anecdotal nature of Kuleshov's observations, subsequent empirical work has confirmed that appropriate contextual framing will cause an observer to perceive neutral faces as happy or sad (Wallbott, 1988), angry faces as fearful (Carroll and Russell, 1996) and screams as joyful (Goldberg, 1951). Importantly, while this phenomenon is a ubiquitous tool of filmmakers it also speaks to the highly complex way in which context influences social attributions.

Contextual frames are presumably built up through real-world experiences whereby particular scenarios are experienced and subsequently influence how we perceive and predict the social world (Levanthal and Scherer, 1987; Bar, 2004). Social psychological studies have shown that when context is not taken into consideration, gross errors in attribution judgments can often be made (i.e. the correspondence bias) (Gilbert and Malone, 1995; Ross, 1977). The power of context has been demonstrated in studies of boundary extension and false memory experiments where contextually relevant information not present in, for example, a picture or story, is incorporated into subsequent judgments (Gottesman and Intraub, 1999; Loftus, 1997). Context also facilitates our ability to recognize stimuli otherwise imperceptible (Bar et al., 2006; Cox et al., 2004). According to theorists, context acts to alter our perceptions through expectations, presumably in a top–down manner (Bar, 2004). Contexts may also operate as nodes by which common events are organized in memory (Anderson and Bower, 1972; Bar and Aminoff, 2003). From an evolutionary standpoint, framing effects would improve fitness by optimizing predictions of imminent threat through, for example, constraining search within memory systems (Sahakyan and Kelley, 2002).

Neurophysiological studies of face perception in the primate brain have demonstrated a distributed network of interconnected regions, which contain face-selective cells (Rolls et al., 2005). Most prominently, this network includes core face analysis regions in the fusiform gyrus (FFG) and superior temporal sulcus (STS), but also extends to higher-visual areas in the temporal pole and ventral prefrontal cortex (vPFC) (Haxby et al., 2000; Rolls et al., 2005; Tsao et al., 2006). To date, the role of the extended face systems including the temporal pole and vPFC in face perception remain speculative, particularly in humans. One theory is that these regions are involved in higher-order face perception, including social and contextual adjustments (cf. Haxby et al., 2000). Although functional imaging studies have begun to demonstrate how context exerts powerful top–down control over the neural circuitry mediating memory and perceptual operations (Cox et al., 2004; Maratos et al., 2001; Bar et al., 2006), little attention has been paid to how context influences socially relevant attributions.

We adapted the ‘Kuleshov Effect’ paradigm to elucidate the neural signature of contextual influences on face expression and mental-state attributions. While undergoing functional MRI, fourteen healthy volunteers were asked to rate emotional expression and mental-state (i.e. what the actor is thinking and feeling) from identical faces presented in negative, neutral and positively valanced contexts. As emotional context is enduring, and is not amenable to conventional event-related design, we used an epoch based event-related paradigm (Figure 1a) allowing us to statistically model phasic event-related activations. Since behavioral studies show that context is most effective when the clarity of the facial expression is low, but the clarity of the context is high (Ekman et al., 1982; Trope, 1986), we used neutral faces and faces displaying subtly fearful and happy facial expressions (Figure 1b). To emphasize a link between the actors’ faces and the contextual movie and reduce demand characteristics we chose to use a pseudo-candid photo manipulation, with subjects being led to believe that the actors’ expressions were in response to viewing a juxtaposed movie (Figure 1c).

Fig. 1
(a) Schematic illustration of the paradigm. Participants were presented with 24 randomly allocated contextual epochs in a 3 × 3 factorial design with 8 negative, 8 positive and 8 neutral valance contexts counterbalanced across subjects. Each epoch ...

METHODS

Participants

We scanned 17 healthy volunteers. Three subjects were later excluded due to a history of neurological problems (i.e. pediatric meningitis) and missing behavioral data. Post-scan questioning also revealed that one of the subjects ‘guessed’ the premise of the paradigm. The remaining 14 subjects (8 females: mean age and s.d. 27.5 ± 8.6) were all naïve to the premise of the study, English speaking, right-handed (Oldfield, 1971), had normal or correct vision and screened for history of psychiatric or neurological problems. All subjects gave informed consent and the study was approved by the joint Ethics committee of the National Hospital for Neurology and Neurosurgery (UCLH NHS Trust) and Institute of Neurology.

Experimental stimuli

We used 130 images taken from the international affective picture system (IAPS) (Lang et al., 1999). In addition, 14 supplementary images rated for valance by 10 volunteers were introduced to balance for content (e.g. humans, animals, objects, etc.) and visual complexity. Using the self-assessment manikin rating scale (Lang et al., 1999), three of the images were rated as positive (mean and s.d. 2.7 ± 0.5) and 11 were rated as neutral (4.9 ± 0.8). Based on their standard scores (1 = high positive/9 = high negative)(Lang, et al., 1999), and together with our additional set of images, 48 of the images were rated as positive (2.7 ± 0.6), 48 as neutral (4.7 ± 0.9) and 48 as negative valance (7.7 ± 0.9). Modeled on Kuleshov's original stimuli, our images were given a dynamic ‘movie’ effect by zooming in or out of the image. The use of dynamic, rather than static images provides a more vivid context (Wallbott, 1988). Both positive and negative contextual movies were significantly different for valance and arousal from the neutral movies (P < 0.0001).

The face stimuli were neutral, happy and fearful faces derived from the standardized NIM-STIM set. Thirty-six faces were presented in each context with 12 null events. An equal number of male and females faces balanced for ethnicity were used. Based on their standardized ratings, all faces were rated as significantly different for valence (F2,22 = 23.9: P < 0.0005). To facilitate the influence of context (Ekman et al., 1982), we used FantaMorph 2.5 (http://www.fantamorph.com) to generate morphs between neutral and happy or neutral and fearful facial expressions. This process created two subtle facial emotions (i.e. 25% happy and 25% fearful). Post-hoc ratings of subtle face affect by the subjects showed significant differences between facial expression ratings (F1.3,17.4 = 20.8: P < 0.0005; Greenhouse–Geisser corrected for non-sphericity; Figure 2a).

Fig. 2
(a) Mean percentage ratings and mean standard error (±s.e.m) ratings for face expression were significantly influenced by both positive and negative compared with neutral context and post-scan rating of faces. No significant differences were found ...

Experimental paradigm

Before each scan, the subject was told that the candid facial expressions depicted were captured in response to seeing the associated emotional or non-emotional movie. To facilitate this framing, we showed a pseudo-candid photograph illustrating the process in which face images were acquired (Figure 1c). Demand characteristics were reduced by telling the subjects that the study was concerned with the evaluation of everyday facial emotions in response to emotional material. In addition, subjects were told that all faces were different (i.e. no face was presented more than once), although this was not overtly evident due to the subtlety of the ‘real’ everyday emotions. Confounds such as familiarity or repetition effects were minimized by counterbalancing the stimuli across subjects. Subjects were given a practice session and told to rate each face for expression and mental-state (i.e. whether the actor is thinking positive or negative thoughts), emphasizing that the task was not to rate the movies.

Twenty-four randomly allocated epochs (contexts) were presented in a 3 × 3 factorial design with eight negative, eight positive, and eight neutral contexts, each lasting up to 18 min. Before each epoch began, a 17 s rest period was presented. Each epoch began with a contextual movie for 4 s. A jittered interstimulus interval (ISI) followed varying between 4 s, 6 s and 8 s. Following this, a face with a neutral or subtle facial expression was presented for 750 ms. After a short ISI of 650 ms, subjects were required to use a keypad to rate the face (at a self-paced rate, but timing out after 10 s) for emotional expression and mental state using an orthogonal two-dimensional rating scale based on Russell's Circumplex Model where values are recorded in Cartesian space (Russell, 1980). Each epoch lasted up to 2 min showing six contextual movies, and six events with up to five faces and no more than two null events. Following the scan session, each subject was asked to rate each face out of context (i.e. without the juxtaposed movie) for emotional intensity and mental state, again using the orthogonal rating scale. Stimuli were programmed using Cogent 1.24 (www.vislab.ac.uk).

fMRI acquisition

A Sonata 1.5T scanner (Siemans, Erlangen, Germany) was used to obtain gradient-echo, T2*-weighted echo-planar images (EPI) using a customized sequence to reduce signal dropout in the orbital frontal cortex (OFC) and amygdalar (Deichmann et al., 2005). To prevent head motion, a head brace attached to the head-coil was used. We acquired 24 volumes (2 mm thick, 1 mm slice gap, providing approx 65% brain coverage) and tilted at −25° to capture several a priori regions of interest [i.e. FFG, OFC, anterior cingulate cortex (ACC), temporal pole, insula, STS, ventrolateral and ventromedial prefrontal cortex and amygdala]. This constrained approach was used for two main reasons: (i) to maximize the temporal resolution and because (ii) we had no a priori reason to acquire parietal and superior occipital regions. Imaging parameters were as follows: repetition time (TR) = 2.16 s; echo time (TE) = 50 m; z shim prepulse = −1 mT/m* ms 3.0 mm; refocusing correction +5%; flip angle 84°; field of view (FOV) = 192 × 192 mm. We collected an average of 1347.1 ± 62.9 volumes per subject across three counterbalanced runs, including 15 dummy volumes (five in each run) to permit T1 equilibration (later discarded from the analysis).

Statistical analysis

Statistical parametric mapping (SPM2; Wellcome Department of Imaging Neuroscience, www.fil.ion.ucl.ac.uk/spm) was used to preprocess all fMRI data and included spatial realignment, slicetime correction, normalization and smoothing. To control for motion, all functional volumes were realigned to the mean volume, spatially normalized (Ashburner and Friston, 1999) to standard space Montreal Neurological Institute (MNI) template (Mazziotta et al., 1995) with a resample voxel size of 3 × 3 × 3 mm and smoothed using a Gaussian kernal with an isotropic full width at half maximum of 10 mm. In addition, high-pass temporal filtering with a cut-off of 128 s was applied to remove low-frequency drifts in signal and global changes were removed by proportional scaling. Following preprocessing, statistical analysis was conducted using the general linear model.

Analysis was performed to determine each subject's voxel-wise activation during emotional events compared with neutral events. State-related neural activity was modeled with boxcar functions representing activity sustained throughout contextual epochs. This model was one of the regressors within the general linear model used to decompose the variance in BOLD signal. Event-related neural activity was modeled with delta (stick) functions representing face onsets convolved with the canonical hemodynamic response function and time derivative to provide a regressor for event-related BOLD-signal. Random effects analysis (Penny and Friston, 2003) was used for group statistics. A statistical threshold of P < 0.05 corrected for multiple spatial comparisons across the whole-brain was used, except for a priori hypothesized regions which were thresholded at P < 0.005 uncorrected (only clusters involving five or more contiguous voxels are reported). These a priori regions of interest included the temporal pole, FFG, STS, amygdalar, ventrolateral PFC, (vlPFC), ventromedial PFC (vmPFC), insula and ACC. Interactions between emotional context and faces were examined using t-tests. This approach is standard in imaging and has the advantage of specifying clearly the direction of interaction in every contrast (e.g. happy face being ‘positively’ modulated by happy context), which allows straightforward interpretations of the images.

RESULTS

Behavioral data

Our behavioral results replicated the main findings from previous behavioral studies using the ‘Kuleshov Effect’ (Carroll and Russell, 1996; Goldberg, 1951; Wallbott, 1988). Despite faces being identical across contexts, on average, both positive and negative contexts resulted in significantly different ratings of faces compared with those presented in neutral contexts and to subsequent post-scan ‘out of context’ ratings (F1.4,18.2 = 17.68, P < 0.0005: repeated-measure ANOVA: Greenhouse–Geisser corrected for non-sphericity; Figure 2a). Similarly, mental-state attribution for faces presented in positive and negative contexts significantly altered ratings compared with those presented in neutral context and post hoc ratings (F1.2,16,1 = 7.07, P < 0.013; Figure 2b). Post hoc Fisher's LSD test confirmed a significant main effect when faces were juxtaposed with either positive and negative context compared with neutral context. Faces juxtaposed with neutral context and post-scan ratings of expression and mental-state were statistically indistinguishable (α = 05). Moreover, the significant correlations between facial expression and mental-state ratings (Pearson's R = 0.884; P < 0.001 one-tailed), neutral (R = −0.579; P < 0.03), negative contexts (R = 0.780; P < 0.001) and post-scan ratings (R = 0.754; P < 0.002) suggests a strong correspondence between the attributions of the actors’ facial expression and mental-state.

We next conducted a 3 × 3 repeated measures ANOVA (3 face expressions × 3 contextual categories) for behavioral responses in the scanner only. A main-effect was found for facial expression (F2,26 = 21.3, P < 0.001; corrected for non-sphericity) and context (F1.1,26 = 22.5, P < 0.001; corrected for non-sphericity). We also found a small, but significant interaction between face expression and context (F4,76 = 3.2, P < 0.040), suggesting that each context had a different effect on each type of face (Figure 3a). We next examined the mental-state ratings associated with each type of facial expression (face expression × contextual category) which showed a main-effect found for context (F1.1,26 = 24.4, P < 0.0005: repeated-measure ANOVA; corrected for non-sphericity) and facial expression (F2,26 = 18.4, P < 0.0005). A marginally significant interaction between mental-state attributions and context was also observed (F4,76 = 2.4, P < 0.091) (Figure 3a, b).

Fig. 3
Mean percentage ratings and ±s.e.m ratings for (a) each face expression and (b) mental-state ratings in each context. (c) Interaction between fearful faces and negative context and betas for fearful faces presented in the positive (black bars), ...

fMRI data

Direct comparison between emotionally salient and non-emotionally salient context

We first examined how brain activity elicited by faces was altered by emotional contexts (negative and positive minus neutral). The expectation here was that salient emotional contexts would result in differential brain activity for otherwise similar faces (Ekman et al., 1982). In keeping with this prediction, we observed that otherwise identical faces presented in the negative vs a neutral context (i.e. NegativeContext − NeutralContext) led to enhanced activity in a priori regions of interest including the ACC, left STS, right STS, right amygdala and the bilateral temporal pole. A similar pattern was observed for the contrast of positive vs neutral contexts (i.e. PositiveContext − NeutralContext) with enhanced activation in the bilateral STS, bilateral temporal pole and the right ACC (Figures 2a,b and Table 1). These regions have been implicated in contextual processing of objects (Smith et al., 2004) and in the processing of socially relevant stimuli (Brothers, 1990).

Table 1
Coordinates in MNI space and associated z-scores showing the BOLD differences for main effects of emotional minus neutral contexts

Interaction between subtle emotional faces and emotional contexts

We further investigated the effect of context on face processing by examining the interaction between emotional faces in congruent and incongruent contexts (Figure 3 and Table 2). We conducted four sets of analysis looking at the interaction between (i) fearful faces and negative contexts (NegativeContextFearFace − NegativeContextNeutralFace) − (NeutralContextFearFace − NeutralContextNeutralFace); (ii) interaction between happy faces and positive context (PositiveContextHappyFace − PositiveContextNeutralFace) − (NeutralContextHappyFace − NeutralContextNeutralFace); (iii) interaction between fearful faces and happy context (NeutralContextFearFace − NeutralContextNeutralFace) − (PositiveContextFearFace − PositiveContextNeutralFace) and (iv) interaction between happy faces and fearful context (NeutralContextHappyFace − NeutralContextNeutralFace) − (NegativeContextHappyFace − NegativeContextNeutralFace). We acknowledge that this analysis is under power (i.e. only 12 events per contrast), however, significant differences did emerge.]

Table 2
Coordinates in MNI space and associated z-scores showing the BOLD differences for the interaction between congruent and incongruent facial expressions and contexts

The interaction between subtle fearful faces and negative context revealed activity in the right amygdala, fusiform gyrus and bilateral temporal pole and insula. Activity was also observed in the left hippocampus and vlPFC and vmPFC. The interaction between happy faces and the congruent positive context showed increased activity in the right amygdala, right temporal pole, right FFG, left hippocampus and bilateral insula. An interaction between fearful faces and the incongruent positive contexts was found in the right vmPFC and left insula. The interaction between happy faces and the incongruent negative context also elicited increased activity in the right vmPFC and bilateral amygdala.

Parametric modulation of BOLD signal by attribution ratings

We also estimated the effects of contextually driven attributions more directly by conducting a parametric analysis that examined the relationship between brain activity and subjective ratings of facial expression across contexts for each category of face expression. Theoretically, because more ambiguous facial expressions produce the greatest contextual influence (Ekman et al., 1982), we were particularly interested in examining the neutral faces across contexts. The correlation between brain activity and positive attributions showed significant effects in the vmPFC, bilateral temporal pole and right anterior fusiform gyrus. The negative face expression attributions were associated with concomitant activity in the bilateral vlPFC, bilateral temporal pole and right FFG. These correlations were consistent with the hypothesis that the vmPFC and vlPFC mediate positive and negative attributions, respectively (Kim et al., 2004) (Figure 4a, b). Although not as significant, we also confirmed this ventral medial-lateral pattern by conducting the same parametric analysis with happy and fearful faces in respective contexts (Table 3). This dissociation was somewhat less consistent for the correlation with mental-state ratings. This finding is not surprising given that mental-state attributions would presumably be less associated with valance increases.

Table 3
Coordinates in MNI space and associated z-scores showing the BOLD differences correlations for face expression category and contexts
Fig. 4
Correlation between BOLD signal and (e) positive (vmPFC: 0, 44, −6, Z = 3.61) and (f) negative (vlPFC: −38, 46, −6, Z = 3.08) ratings of face expression associated with the neutral faces (Table 3).

DISCUSSION

Our environment conveys a rich array of contextual information which influences how the brain encodes, categorizes and recognizes objects and people (Bar, 2004; Smith et al., 2004; Cox et al., 2004). In relation to social cognition, context indicates appropriate behavior and influences how we perceive and appraise others (Gilbert and Malone, 1995; Ross, 1977). Using Kuleshov's paradigm, we provide behavioral evidence that when identical faces are juxtaposed with contextual movies of different valance, attributions of facial expression and mental-state are altered. fMRI data collected simultaneously revealed that, by contrasting faces juxtaposed with either positive or negative movies with those juxtaposed with the neutral movies, activity elicited by faces was altered in several regions of a priori interest including the bilateral temporal pole, STS insula and ACC. An interaction was observed in the right amygdala when subtle happy and fear faces were juxtaposed with positive and negative movies, respectively. An interaction between happy faces and negative context was also observed in bilateral amygdala suggesting that the amygdala may act to prime or tag affective value to faces. Parametric modulation of BOLD signal by attribution ratings supported a role for the bilateral temporal pole for both positive and negative ratings and further illustrates a dissociation between vlPFC and vmPFC for negative and positive contextually evoked attributions, respectively.

Our results confirm the importance of contextual framing in social attributions. A multitude of social psychological studies support the hypothesis that contextual cues influence social attributions (Trope, 1986; Ross, 1977). Social attributions are likely to rely on activation of stored contextual schemata derived from experience (Levanthal and Scherer, 1987; Bar, 2004). Contextual frames may be integrated into social attributions through expectations where, based on experience, the attributor's judgments can be shifted from one category to another (Trope, 1986). Although the way affect is incorporated into attributions is likely to be a multiprocess (cf. Forgas, 1995), at the simplest level, these contextual shifts may occur via linear-weighted computations where the context and face are analyzed separately and the emotion perceived in the face is based on the weighted average from the two sources of emotion (Carroll and Russell, 1996). Therefore, the extent to which context shifts attributions may depend upon the source clarity of the stimulus (Ekman et al., 1982; Trope, 1986). For example, in the present study, the subtle facial expressions are likely to be less salient than the highly valanced contextual movies. Contextual framing effects may be particularly evident when the context confirms, or is congruent with, expectations (Trope, 1996). Our behavioral results support such an effect showing that when congruent facial emotion and context are paired, subjects are most shifted in their attributions (cf. Ganis and Kutas, 2003; Davenport and Potter, 2004).

There is evidence that the temporal pole plays a key role in contextual framing (Smith et al., 2004). It has been theorized that the temporal pole serves as a repository for contextual frames or schemata (Ganis and Kutas, 2003; Smith et al., 2004; Cox et al., 2004). Acquired lesions in the vicinity of the right temporal pole can result in the loss of recognition of famous scenes, loss of memory for events and loss of person related knowledge (Tranel et al., 1997; Kitchener et al., 1999; Gorno-Tempini et al., 2004). In some cases, right temporal pole damage results in face processing deficits (i.e. prosopagnosia; Lough et al., 2006). This has led theorists to posit that the right temporal pole is part of an extended face system (Haxby et al., 2000; Calder and Young, 2005). Given the visual properties of this region, activity may also reflect the holding or recall of the contextual image in mind (Nakamura et al., 1994; Reiman et al., 1997; Lane et al., 1997). This theory seems unlikely given that examination of the null events did not reveal temporal pole activity (Table 4) and therefore supports a role for the temporal pole in face processing (Haxby et al., 2000; Calder and Young, 2005).

Table 4
Coordinates in MNI space and associated z-scores showing the BOLD differences for main effects of null events placed in emotional minus neutral contexts

While the left temporal pole has been implicated in increased familiarity (Rotshtein et al., 2005), possibly reflecting retrieval of semantic information about the person, this region could equally be involved in placing the person into the correct context (e.g. butcher on the bus phenomenon). Therefore, it might be expected that congruence would evoke more temporal pole activity than the incongruent conditions. This may happen because the congruent condition fits with previous experiences and expectancies. Further, the unexpectedness of the incongruent condition would presumably result in more cognitive effort or search. This may account for the increased vmPFC when fearful and happy faces were presented in incongruent context. Taken together, our findings are consistent with previous suggestions that the temporal poles are involved in the storage and recall of contextual information, particularly when affectively salient (Smith et al., 2004; Lane et al., 1997). This proposal is anatomically plausible given that the temporal pole is well-connected to structures important in the processing of emotional and social information including the STS, amygdala and ventral PFC (Chabardes et al., 2002; Kondo et al., 2003).

The mechanism by which the temporal pole uses contextual information to modulate social attributions may depend upon the interaction with the STS with which it is strongly connected (Chabardes et al., 2002; Eifuku et al., 2004). The STS contains face-selective cells (Tsao et al., 2006) and has a particular role in the perception of changeable aspects of the face, such as emotional expression and eye direction (Haxby et al., 2000). Human imaging studies have implicated the STS in the explicit judgment of socially relevant information including trust, intentions, mental-state and diagnostic person information (den Ouden et al., 2005; Pelphrey et al., 2004; Winston et al., 2002; Mitchell et al., 2006) and may reflect more abstract aspects of mental-state attributions. One possibility is that activity in STS, a region of multisensory integration (King and Calvert, 2001), reflects the integration of contextual and facial cues. Alternatively, because STS activity was increased when faces were juxtaposed with positive and negative contexts, this region may be engaged when more detailed and explicit judgment of facial emotion is required (Narumoto et al., 2001). Such a theory is plausible given the strong connection between the STS and amygdala (Eifuku et al., 2004). Presumably, the emotional movie would drive expectation and attention towards a more explicit analysis of facial affect (Smith et al., 1996; Mitchell et al., 2006).

Intuitively, contextual framing relies on the integration of extrinsic contextual cues and the retrieval of stored knowledge about the likely emotional disposition of an actor in this context. Fitting with this theory, research supports a role for the vPFC in top–down retrieval of information, particularly in relation to contextual guidance of social behavior (Bar, 2004). The vPFC has previously been associated with guessing and expectation, consistent with its role in top–down prediction (Frith and Dolan, 1997; Bar et al., 2006; Kao et al., 2005). The vPFC activity resulting from the parametric modulation of valance could be linked with top–down effects on a network of regions associated with contextually appropriate responses. The parametric increase in vPFC activity fits with its role in representing valance rather than intensity (Small et al., 2003; Anderson et al., 2003). This may occur in a Baysian fashion by constantly updating and integrating evidence about the present context with previously stored knowledge (Bar, 2004). Such an account is plausible given that both vlPFC and vmPFC project to discrete amygdala nuclei (Lissek and Gunturkun, 2005) and to other core and extended face processing regions including the STS, temporal pole and FFG (Dolan et al., 2001a; Haxby et al., 2000; Maratos et al., 2001). Whether directly or indirectly, the cytoarchitectonically distinct vlPFC and vmPFC may modulate the affective value of stimuli and act to guide choices across altering contexts (Kringelbach and Rolls 2004; O'Doherty et al., 2001; Kim et al., 2004). Such a theory fits with how affective states alter social judgments (Forgas, 1995), and studies showing that lesions to the vPFC can result in the inability to utilize previous experience to guide behavior (Damasio, 1994).

Greater activation in the vmPFC to null events for both positive and negative contexts (Table 4) may reflect insufficient information to direct behavior or a violation of expectation (Elliot et al., 2000). The vmPFC was also observed when faces were incongruent with the context, supporting fMRI data that this region is involved in the extent to which information is contextually appropriate (Kao et al., 2005). Moreover, ablation of primate vPFC suggests that this region is involved in integration of sensory signals which aid in choosing between competing responses (Izquierdo et al., 2005). Studies examining the effect of emotional context on objects (Smith et al., 2004) and words (Maratos et al., 2001) also suggest a role for the ACC in schema-based appraisals. In their study of context effects on surprised faces, Kim et al. (2004) showed the ACC to be functionally connected with both vmPFC and vlPFC when surprised faces were juxtaposed with positive and negative emotional verbal contexts, respectively. This suggests that these putative ACC (Smith et al., 1996) and vPFC appraisal systems interact. Together, our results suggest that the vPFC may act to guide appropriate responses across contexts. Other regions important in social cognition, including the dorsomedial PFC, may also be important in contextual framing and should be examined in future studies of context on socially relevant attributions.

Given the emotional nature of our faces and contextual movies, it is important to acknowledge the increased amygdala activity observed in this study. The amygdala has been implicated in the processing of socially relevant information (Brothers, 1990; Lieberman et al., 2005; Winston et al., 2002). In the current study, an interaction effect was observed in the right amygdala when subtle happy and fear faces were juxtaposed with positive and negative movies, respectively. An interaction between happy faces and negative context was observed in bilateral amygdala. Several mechanisms might account for this increased amygdala activity including affective priming (Dolan et al., 2001b), reinforcement history (Kim et al., 2004), intensity (Anderson et al., 2003) or tagging affective value to stimuli. Fitting with the latter hypothesis, amygdala damage impairs memory for gist, particularly in emotional contexts (Adolphs et al., 2005), and decreases perceptions of emotional salient events (Anderson and Phelps, 2001). A similar operation could take place during social attributions by facilitating attention towards an emotion expression (Bermpohl et al., 2006). Such an effect fits with our behavioral findings and studies of patients with focal amygdala lesions who show impairments in the recognition of facial emotions (Adolphs et al., 1999). Moreover, the amygdala is thought to influence visual areas including the FFG (Vuilleumier et al., 2004). This fits well with the increased FFG activity for the interaction between happy and positive context, and interaction between fearful faces and negative contexts. More broadly, our findings build on previous imaging studies demonstrating increased amygdala activity during the encoding and retrieval of neutral objects (Smith et al., 2004), surprised faces (Kim et al., 2004), and words (Maratos et al., 2001) juxtaposed with emotionally loaded contexts.

In summary, our results build on existing data showing contextual influences on behavioral and neurobiological systems underpinning socially relevant attributions. Our results suggest that a network of brain regions are involved in the storage and coordination of contextual framing. Although more research is needed, these regions may involve anterior temporal regions in the storage of contextual frames, the STS in the attention towards facial affects, and the amygdala in affective tagging of stimuli. Activity in these regions is likely to be influenced by top–down signals from the vPFC, which fits with its role in the contextual guidance of behavior. A better understanding of these complex systems will enable us to forge important links between affective neuroscience and social cognition.

Acknowledgments

We thank Klaas Stephan, Predrag Petrovic, Rik Henson, Cindy C. Hagan and Luen Otten for helpful comments on the study design and analysis. This research was supported by the Wellcome Trust. D. Mobbs is supported by Brain Research Trust Prize Studentship. Stimuli were taken from: Tottenham, N., Borscheid, A., Ellersten, K., Markus, D.J., and Nelson, C.A. (2002). Categorization of facial expressions in children and adults: Establishing a larger stimulus set. Poster presented at the Cognitive Neuroscience Society annual meeting, San Francisco.

REFERENCES

  • Adolphs R, Tranel D, Hamann S, et al. Recognition of facial emotion in nine individuals with bilateral amygdala damage. Neuropsychologia. 1999;37:1111–7. [PubMed]
  • Adolphs R, Tranel D, Buchanan TW. Amygdala damage impairs emotional memory for gist but not details of complex stimuli. Nature Neuroscience. 2005;8:512–8. [PubMed]
  • Anderson J, Bower GH. Recognition and retrieval processes in free recall. Psychological Review. 1972;79:97–123.
  • Anderson AK, Phelps EA. Lesions of the human amygdala impair enhanced perception of emotionally salient events. Nature. 2001;411:305–309. [PubMed]
  • Anderson A, Christoff K, Stappen I, et al. Dissociated representations of intensity and valence in human olfaction. Nature Neuroscience. 2003;6:196–202. [PubMed]
  • Ashburner J, Friston KJ. Nonlinear spatial normalization using basis functions. Human Brain Mapping. 1999;7:254–66. [PubMed]
  • Bar M. Visual objects in context. Nature Reviews Neuroscience. 2004;5:617–29. [PubMed]
  • Bar M, Aminoff E. Cortical analysis of visual context. Neuron. 2003;38:347–58. [PubMed]
  • Bar M, Kassam KS, Ghuman AS, et al. Top-down facilitation of visual recognition. Proceedings of the National Academy Sciences USA. 2006;103:449–54. [PubMed]
  • Bermpohl F, Pascual-Leone A, Amedi A, et al. Attentional modulation of emotional stimulus processing: an fMRI study using emotional expectancy. Human Brain Mapping. 2006;27:662–77. [PubMed]
  • Brothers L. The social brain: a project for integrating primate behavior and neurophysiology in a new domain. Concepts in Neuroscience. 1990;1:27–51.
  • Calder AJ, Young AW. Understanding the recognition of facial identity and facial expression. Nature Reviews Neuroscience. 2005;6:641–51. [PubMed]
  • Carroll JM, Russell JA. Do facial expressions express specific emotions? Judging emotion from the face in context. Journal of Personality and Social Psychology. 1996;70:205–218. [PubMed]
  • Chabardes S, Kahane P, Minotti L, et al. Anatomy of the temporal pole. Epileptic disorders. 2002;4:9–16. [PubMed]
  • Cox D, Meyers E, Sinha P. Contextually evoked object-specific responses in human visual cortex. Science. 2004;3004:115–17. [PubMed]
  • Damasio AR. Descartes' error: Emotion, reason and the human brain. NY: Grosset/Putnam; 1994.
  • Davenport JL, Potter MC. Scene consistency in object and background perception. Psychological Science. 2004;15:559–64. [PubMed]
  • Deichmann R, Gottfried JA, Hutton C, Turner R. Optimized EPI for fMRI studies of the orbitofrontal cortex. Neuroimage. 2005;19:430–41. [PubMed]
  • den Ouden HE, Frith U, Frith C, Blakemore SJ. Thinking about intentions. Neuroimage. 2005;28:787–96. [PubMed]
  • Dolan RJ, Fletcher P, Morris J, et al. Neural activation during covert processing of positive emotional facial expressions. Neuroimage. 2001a;4:194–200. [PubMed]
  • Dolan RJ, Morris JS, de Gelder B. Crossmodal binding of fear in voice and face. Proceedings of the National Academy Sciences USA. 2001b;98:10006–10. [PubMed]
  • Eifuku S, DeSouza WC, Tamura R, Nishijo H, Ono T. Neuronal correlates of face identification in the monkey anterior temporal cortical areas. Journal of Neurophysiology. 2004;91:358–71. [PubMed]
  • Ekman P, Friesen W, Ellsworth P. What are the relative contributions of facial behavior and contextual information to the judgment of emotion? In: Ekman P, editor. Emotion in the human face. New York: Cambridge University Press; 1982.
  • Elliot R, Dolan RJ, Frith CD. Dissociable Functions in the Medial and Lateral Orbitofrontal Cortex: Evidence from Human Neuroimaging Studies. Cerebral Cortex. 2000;10:308–17. [PubMed]
  • Forgas JP. Mood and judgment: The Affect Infusion Model (AIM) Psychological Bulletin. 1995;117:39–66. [PubMed]
  • Frith CD, Dolan RJ. Brain mechanisms associated with top–down processes in perception. Philosophical Transactions of. Royal. Society. London B. 1997;352:1221–30. [PMC free article] [PubMed]
  • Ganis G, Kutas M. An electrophysiological study of scene effects on object identification. Brain Research: Cognitive Brain Research. 2003;16:123–44. [PubMed]
  • Gilbert DT, Malone PS. The Correspondence Bias. Psychological Bulletin. 1995;117:21–38. [PubMed]
  • Goldberg H. The role of ‘cutting’ in the perception of motion picture. Journal of Applied Psychology. 1951;35:70–71. [PubMed]
  • Gorno-Tempini ML, Rankin KP, Woolley JD, et al. Cognitive and behavioral profile in a case of right anterior temporal lobe neurodegeneration. Cortex. 2004;40:631–44. [PubMed]
  • Gottesman H, Intraub CV. Wide-angled memories of close-up scenes: A demonstration of boundary extension. Behavior Research Methods Instruments and Computers. 1999;31:86–93. [PubMed]
  • Haxby JV, Hoffman EA, Gobbini MI. The distributed human neural system for face perception. Trends in Cognitive Science. 2000;4:223–33. [PubMed]
  • Izquierdo A, Murray EA. Comparison of the effects of bilateral orbital prefrontal cortex lesions and amygdala lesions on emotional responses in rhesus monkeys. Journal of Neuroscience. 2005;37:8534–42. [PubMed]
  • Kawin B. How movies work. Berkeley: University of California Press; 1992.
  • Kim H, Somerville LH, Johnstone T, et al. Contextual modulation of amygdala responsivity to surprised faces. Journal of Cognitive Neuroscience. 2004;16:1730–45. [PubMed]
  • King AJ, Calvert GA. Multisensory integration: Perceptual grouping by eye and ear. Current Biology. 2001;11:322–25. [PubMed]
  • Kitchener EG, Hodges JR, McCarthy R. Acquisition of post-morbid vocabulary and semantic facts in the absence of episodic memory. Brain. 1998;121:1313–27. [PubMed]
  • Kao Y, Davis ES, Gabrieli JDE. Neural correlates of actual and predicted memory formation. Nature Neuroscience. 2005;8:1776–83. [PubMed]
  • Kondo H, Saleem KS, Price JL. Differential connections of the temporal pole with the orbital and medial prefrontal networks in macaque monkeys. Journal of Comparative Neurology. 2003;465:499–523. [PubMed]
  • Kringelbach ML, Rolls ET. The functional neuroanatomy of the human orbitofrontal cortex: evidence from neuroimaging and neuropsychology. Progress in Neurobiology. 2004;72:341–72. [PubMed]
  • Lane RD, Reiman EM, Ahern GL, Schwartz GE, Davidson RJ. Neuroanatomical correlates of happiness, sadness, and disgust. American Journal of Psychiatry. 1997;154:926–33. [PubMed]
  • Lang PJ, Bradley MM, Cuthbert BN. International Affective Picture System (IAPS): Instruction manual and affective ratings; Gainesville, University of Florida. 1999.
  • Levanthal H, Scherer K. The relationship of emotion to cognition: A functional approach to a semantic controversy. Cognition and Emotion. 1987;1:3–28.
  • Lieberman MD, Hariri A, Jarcho JM, Eisenberger NI, Bookheimer SY. An fMRI investigation of race-related amygdala activity in African-American and Caucasian-American individuals. Nature Neuroscience. 2005;8:720–22. [PubMed]
  • Lissek S, Gunturkun O. Out of context: NMDA receptor antagonism in the avian ‘prefrontal cortex’ impairs context processing in a conditional discrimination task. Behavioral Neuroscience. 2005;119:797–805. [PubMed]
  • Loftus EF. Creating false memories. Scientific American. 1997;277:70–75. [PubMed]
  • Lough S, Kipps CM, Treise C, Watson P, Blair JR, Hodges JR. Social reasoning, emotion and empathy in frontotemporal dementia. Neuropsychologia. 2006;44:950–8. [PubMed]
  • Maratos EJ, Dolan RJ, Morris JS, Henson RN, Rugg MD. Neural activity associated with episodic memory for emotional context. Neuropsychologia. 2001;39:910–20. [PubMed]
  • Mazziotta JC, Toga AW, Evans A, Fox P, Lancaster J. A Probabilistic Atlas of the Human Brain: Theory and Rationale for Its Development: The International Consortium for Brain Mapping (ICBM). 21.1995. [PubMed]
  • Mitchell JP, Cloutier J, Banaji MR, Macrae CN. Medial prefrontal dissociations during processing of trait diagnostic and nondiagnostic person information. Social Cognitive and Affective Neuroscience. 2006;1:49–55. [PMC free article] [PubMed]
  • Nakamura K, Matsumoto K, Mikami A, Kubota K. Visual response properties of single neurons in the temporal pole of behaving monkeys. Journal of Neurophysiology. 1994;71:1206–21. [PubMed]
  • Narumoto J, Okada T, Sadato N, Fukui K, Yonekura Y. Attention to emotion modulates fMRI activity in human right superior temporal sulcus. Brain Research: Cognitive Brain Research. 2001;12:225–31. [PubMed]
  • O'Doherty J, Kringelbach ML, Rolls ET, Hornak J, Andrews C. Abstract reward and punishment representations in the human orbitofrontal cortex. Nature Neuroscience. 2001;1:95–102. [PubMed]
  • Oldfield RC. The assessment and analysis of handedness: the Edinburgh inventory. Neuropsychologia. 1971;9:97–113. [PubMed]
  • Pelphrey KA, Viola RJ, McCarthy G. When strangers pass: processing of mutual and averted social gaze in the superior temporal sulcus. Psychological Science. 2004;15:598–603. [PubMed]
  • Penny WH, Friston K. Random effects analysis. In: Frackowiak RSJ, editor. Human Brian Function. London: Academic Press; 2003.
  • Reiman EM, Lane RD, Ahern GL, et al. Neuroanatomical correlates of externally and internally generated human emotion. American Journal of Psychiatry. 1997;154:918–25. [PubMed]
  • Rolls ET, Browning AS, Critchley HD, Inoue K. Face-selective and auditory neurons in the primate orbitofrontal cortex. Experimental Brain Research. 2005;1:1–14. [PubMed]
  • Ross L. The intuitive psychologist and his shortcomings: Distortions in the attribution process. In: Berkowitz L, editor. Advances in experimental social psychology. Vol. 10. New York: Academic Press; 1977. pp. 173–220.
  • Rotshtein P, Henson RN, Treves A, Driver J, Dolan RJ. Morphing Marilyn into Maggie dissociates physical and identity face representations in the brain. Nature Neuroscience. 2005;8:107–13. [PubMed]
  • Russell JA. The circimplex model of affect. Journal of Personality and Social Psychology. 1980;39:1161–78.
  • Sahakyan L, Kelley CM. A contextual change account of the directed forgetting effect. Journal of Experimental Psychology: Learning, Memory and Cognition. 2002;28:1064–72. [PubMed]
  • Small DM, Gregory MD, Mak YE, et al. Dissociation of Neural Representation of Intensity and Affective Valuation in Human Gustation. Neuron. 2003;39:701–11. [PubMed]
  • Smith CA, Griner LA, Kirkby LD, Scott HS. Toward a process model of appraisal in emotion; Paper presented in the ninth meeting of the International Society of Research on Emotions; Toronto, Canada. 1996.
  • Smith AP, Henson RN, Dolan RJ, Rugg MD. fMRI correlates of the episodic retrieval of emotional contexts. Neuroimage. 2004;22:868–78. [PubMed]
  • Tranel D, Damasio H, Damasio AR. A neural basis for the retrieval of conceptual knowledge. Neuropsychologia. 1997;35:1319–27. [PubMed]
  • Trope Y. Identification and Inferential Processes in Dispositional Attribution. Psychological Review. 1986;93:239–57.
  • Tsao DY, Freiwald WA, Tootell RB, Livingstone MS. A Cortical Region Consisting Entirely of Face-Selective Cells. Science. 2006;311:670–74. [PMC free article] [PubMed]
  • Vuilleumier P, Richardson MP, Armony JL, Driver J, Dolan RJ. Distant influences of amygdala lesion on visual cortical activation during emotional face processing. Nature Neuroscience. 2004;7:1271–78. [PubMed]
  • Wallbott H. In and out of context: Influences of facial expression and context information on emotion attributions. British Journal of Social Psychology. 1988;27:327–69.
  • Winston JS, Strange BA, O'Doherty J, Dolan RJ. Automatic and intentional brain responses during evaluation of trustworthiness of faces. Nature Neuroscience. 2002;5:227–83. [PubMed]

Articles from Social Cognitive and Affective Neuroscience are provided here courtesy of Oxford University Press