|Home | About | Journals | Submit | Contact Us | Français|
Understanding the minds of others is one of the great challenges humans face. Accordingly, much work in cognitive neuroscience has explored the brain systems engaged when perceivers share and make inferences about the internal states of social targets. These studies, however, typically use divergent and highly simplified stimuli and methods, and as a consequence have produced largely non-overlapping sets of results that have motivated artificially constrained theories about the processes involved in perceivers' abilities to understand targets. Here we suggest that these difficulties may stem from two main sources: the lack of meaningful behavioral data about the brain bases of perceivers' actual accuracy in inferring target states, and qualitative differences between the social stimuli used in neuroimaging paradigms and the social information perceivers encounter in the real world. We advocate more focus on studies of naturalistic social cognition, which could overcome these limitations and complement current approaches, and discuss work in our lab that has demonstrated the feasibility and utility of such paradigms. Finally, we discuss the relevance of naturalistic social cognition to diagnosing and treating autism spectrum disorder. Overall, using naturalistic paradigms in neuroimaging will be critical to modeling the way the brain actually understands other minds.
One of the great challenges faced by the human mind is the need to comprehend the content of various other minds. Every individual's interaction partners, group members, and competitors provide complex, often-contradictory cues about what they are thinking and feeling (i.e., their internal – mental – states). Moreover, the mental states of such social targets often contain cues that are critical to a perceiver's planning of their own actions. For example, if a target looks in a terrified way at something behind me, I probably should consider attending to, and potentially running away from, that thing. If I wish to gain resources through social means – either by tricking a competitor or cooperating with a partner – understanding the mental states of others becomes central to attaining these resources. This is further underscored by illnesses such as autism spectrum disorder and schizophrenia, in which inabilities to read the mental states of targets cause severe deficits in social function.
Given the importance of understanding others, it is unsurprising that a rapidly increasing body of cognitive neuroscience research has sought to explore the neural bases of social cognitive function. By and large, this work has taken one of two main approaches, which have in turn motivated strikingly different theoretical approaches to the way we understand other minds. In this article we briefly review this work, consider its strengths and weaknesses, and then propose a new direction for research on interpersonal understanding that addresses some of the shortcomings of current work. Finally, we will discuss future directions and implications that research on interpersonal understanding may have on the study of autism.
The first cognitive neuroscience approach to understanding others has concentrated on the neural systems involved in making complex inferences about others, especially when target and perceiver mental states diverge. Consider, for example, that in some situations a perceiver has access to knowledge a target does not. Imagine a case in which a target is looking for an object, such as a coffee mug, and you (the perceiver) know that it is hidden in a non-obvious location (i.e. in the refrigerator instead of the cupboard). In those cases, while trying to infer how the target will behave, you do important cognitive work, such as inhibiting your prepotent tendency to guess that the target will act with the knowledge you have (i.e. by looking in the refrigerator), forming mental representations of the targets' intentions and beliefs based on their observable behavior, and keeping representations of both your beliefs and those of the target in mind simultaneously.
This suite of cognitive processes – which typically are referred to collectively as involving mental state attribution (MSA) – is instantiated in a system of cortical regions outlined in Figure 1. Some of these regions, such as temporoparietal junction (TPJ) and posterior cingulate cortex (PCC) may be involved in allocating attention to salient cues in the environment and assessing their relevance to the self,1-3 while other regions, such as the medial prefrontal cortex (MPFC) may be involved in forming representations of internal mental states.4-7 Interestingly, the MPFC is also engaged in forming representations of perceivers' own internal mental states and qualities,7-9 suggesting that perceivers may use common cognitive processes when forming representations about either themselves or others.
Behavioral data suggests that MSA has – at least in part – piggybacked on more generalized executive processes such as inhibition of prepotent responses. MSA develops in parallel with inhibitory functions during childhood,10, 11 and systematic biases in MSA are introduced when perceivers have to perform a concurrent task while making judgments about targets.12 There is evidence that some information about targets, especially trait attributions, may be processed and retained automatically by perceivers.13-15 Nonetheless, taken together, the data suggest that MSA is not always automatic, but may require controlled processing of cues about target states, especially when a targets' knowledge or mental states diverge from those of a perceiver.16 Data from developmental psychology, as well as cognitive neuroscience, suggest that these controlled processes are critical to accurately understanding a target's mental states, at least in these divergent situations for review, see 17.
In contrast to the first approach to the cognitive neuroscience of interpersonal understanding, the second approach has focused on situations in which perceivers' experiences, sensations, or actions converge with those of targets. For example, imagine watching a friend accidentally burn himself while cooking. While seeing this, you (the perceiver) may vicariously share various aspects of this experience with your friend (the target), such as a general feeling of unpleasantness or anxiety, or even a localized feeling of pain in your own finger. You might also imitate some of your friend's motor actions, such as wincing or pulling your hand back as though it too had been burned.
Behavioral and psychophysiological evidence supports the idea that, in general, people behave as you did in the above example: perceivers tend to align their actions and sensory experiences in synchrony with those they observe in targets. Thus, perceivers become physiologically aroused both when receiving pain directly and when seeing someone else in pain,18 and non-consciously imitate the facial expressions19 and actions of targets.20 It is further possible that imitation could aid a perceivers' cognitive understanding of target states, which has been explored with respect to emotional facial expressions. Posing an expression (i.e. a smile) can lead to an experience of congruent affective states,21 and allow subjects to identify congruent affective states more rapidly,22 even when people are not aware of the expressions they are posing.23
Research beginning in the early 1990s has identified brain systems that could provide a neural substrate for these behavioral effects. The studies exploring these systems have used the logic of shared representations: the idea that perceivers may employ a common cognitive and neural coding to represent both their own states and those of targets. To the extent that common systems are involved in coding responses to one's direct experience and to experiences we observe in others, those systems are said to support shared representations (see Figure 2).
Several types of data support this basic notion. For example, single unit recordings in monkeys and neuroimaging studies in humans have identified subregions of sensory and motor cortex in the parietal lobe, premotor cortex, and inferior frontal gyrus that exhibit “mirror” properties: responding both when perceivers perform actions and when they observe targets performing those actions,24-28 especially if targets' motor intentions are clear.29, 30 More recent studies have demonstrated that areas responsive to the experience of affect – such as the anterior cingulate cortex (ACC) and anterior insula (AI) respond both to a perceiver's experience of pain, disgust, and emotional facial expressions, and to observation of a target experiencing those states.31-34
Although there is no direct behavioral evidence that shared representations instantiated in these systems aid in understanding internal states, theorists have claimed that they are involved in recognizing and responding to the experiences of others.35 More specifically, such theories posit that shared representations are a sensible candidate for supporting interpersonal understanding, especially understanding of basic sensory and affective states such as pain or disgust. Though mirror-like neural and physiological responses to target states are modulated by context,36-39 such systems could work quickly and automatically, for example in underlying non-conscious motor imitation and “contagion” of emotional states.40 The idea is that mechanisms allowing quick and automatic detection of target states through shared representations are most likely helpful in orienting perceivers towards especially salient information in a targets' behavior (i.e. fearful faces indicating an environmental threat).
Given that both MSA and shared representations have been advanced as the source of interpersonal understanding, the independence with which they have been discussed and studied is striking. For example, neuroimaging studies engaging brain regions involved in MSA rarely also engage brain regions underlying shared representations, and visa versa though functional connectivity between these systems has been demonstrated during certain tasks; see 41. This presents a somewhat confusing theoretical picture: these social cognitive processes, while ostensibly supporting the same outcome (allowing a perceiver to understand a target's mind), seem to be operating in relative isolation. As a consequence, a debate has emerged over whether perceivers understand targets through MSA or shared representations.35, 42 More recent viewpoints recognize, however, that it is likely that MSA and shared representations both support interpersonal understanding, and are probably deployed flexibly depending on the type of cognitive resources and social cues available to perceivers.43-45 That being said, there remains a dearth of direct evidence to support this idea.
We believe that two main problems have led to this confusion about the neural bases of interpersonal understanding. First, while studies of MSA and shared representations both claim to explore the basis of a perceivers' ability to share or understand target states, such studies rely almost exclusively on indirect evidence, usually without behaviorally measuring the sharing or understanding of internal states. For example, studies of shared representations usually ask perceivers to observe and imitate target movements, or have them directly experience sensory states and observe those states in targets – all without requiring participants to make any judgments about target states or in any way behaviorally demonstrate a clear understanding of those states. As such, it is impossible to infer whether activations in such studies actually support perceivers' understanding of targets. Similarly, neuroimaging studies of MSA ask perceivers to make judgments about targets presented in simple stimuli (i.e. pictures or cartoons), but these targets are fictional, and the judgments perceivers are asked to make are generally too easy (i.e., there is a ceiling effect) to create any variability in performance that could be taken as evidence that processes supporting understanding have been called into play to varying degrees. As such, these paradigms also fail to afford any direct measure of brain activity supporting interpersonal understanding.
Here it should be noted that studies of MSA and shared representations claiming to demonstrate the brain bases of psychological processes that, in actuality, they do not measure, are committing errors similar to those made by early neuroimaging studies of emotion: they treat a complex cognitive phenomenon as a quality of a stimulus, such as shape, size or color.46, 47 For example, showing people negative or gruesome pictures may cause them to experience negative affect, but without measuring subjective experience or any other behavioral index of emotional responding, it is impossible to know whether brain activity in response to such pictures actually corresponds with the response of interest (i.e, an emotional response), or with some other process engaged by pictures (e.g. subjects distracting themselves, something about processing the perceptual aspects of stimuli, and so on). Similarly, studies of social cognition that manipulate the presence or absence of internal states in stimuli and assume that these stimuli de facto cause social cognitive processing produce results that are inherently ambiguous.
The second problem is that extant studies of shared representations and MSA have most often used simplified stimuli that differ qualitatively from the types of social information perceivers must process in real life social interactions. Using stimuli that vary only along tractable dimensions is critical to achieving tight control over the cognitive processes studied in any experiment, and the use of such stimuli has allowed for crucial progress in mapping distinct social cognitive processes in the brain. However, experimental control can come at a cost, resulting in artificially constrained ideas about the psychological processes involved.
This is especially true for the study of inherently complex phenomena such as social cognition. For example, the neuroimaging studies of MSA and shared representations described above utilize simple stimuli and tasks (i.e. imitation of finger and facial movements or judgments about mental states from pictures), such that they engage single (or limited and circumscribed) sets of processes in relative isolation. This control over stimulus properties may make it unsurprising that such studies show engagement of neural systems responsible either for shared representations or for MSA in isolation from one another.
There is no question that this type of research has been critical in creating a taxonomy of discrete processes involved in social cognition. However, taking the next inferential step – deciding that one, another, or even a combination of processes studied this way account for our social cognitive abilities in the real world – may be less straightforward than current theoretical approaches have assumed. This is because real-life social information differs from such lab stimuli in at least three critical ways. First, cues about target states in the real world are multimodal and involve visual, semantic, and prosodic information. Second, they are dynamic, involving information that is presented serially or simultaneously that has to be integrated by perceivers over time. And third, they are contextually embedded in that perceivers may have access to information (e.g. a targets' beliefs or past behaviors) that can constrain their interpretation of cues about targets' internal states.
To make this concrete, imagine hearing a friend describe their performance on a recent exam. This social target may present semantic cues (i.e. “well, I didn't do that well), and visual cues (smiling) that on the surface are not congruent, but as a perceiver, you may have contextual information (e.g. knowing that your friend is describing her performance within earshot of someone else who failed the same exam) that helps guide your processing of the cues you perceive directly. As the conversation unfolds, your ideas about your friend's internal states will shift as you perceive and account for new information she provides through multiple modalities over time.
These differences between real-life social information and the types of stimuli used in previous studies of MSA and shared representations are salient in that they may produce not only quantitative, but also qualitative differences in associated neural and cognitive processing of social information. For example, integrating information over time (i.e. segmenting event structures, picking out salient environmental cues from a changing visual field) produces unique patterns of neural activity,48, 49 including activity in neural structures involved in motor control and MSA. Furthermore, activity in MSA-related structures during perception of naturalistic events predicts subsequent memory for these events.50 Similarly, access to contextual information can change both the judgments perceivers make about targets' states,51 and the neural activity associated with making such judgments.52
Given these points, we believe that while extant neuroimaging research has done much to advance knowledge about the processes involved in understanding other minds, limitations in the ecological validity of this work suggest a new, complementary direction for the study of social cognition. What is needed are behavioral measures producing variance in performance – and more specifically, variance in the accuracy of judgments about attributes or the perceived intensity of a target's internal states – which can be used to meaningfully connect brain activity to social perceptions or behaviors. Additionally, naturalistic, dynamic stimuli should be employed to probe the neural bases of perceiving social cues that better approximate those encountered by perceivers in the wild.
One approach to expanding the cognitive neuroscience of social cognition in these ways has been developed in our lab over the last few years. Our starting point was empathic accuracy paradigms from social psychology,53, 54 which have two appealing features relevant to naturalistic social cognition. First, they involve complex stimuli that depict actual social targets experiencing internal states dynamically across time: in this case, stimuli are videotapes of individuals discussing emotional autobiographical events. Second, these paradigms critically allow for a continuous, variable measure of social cognitive performance. Just after being taped, targets watch the videos of themselves, and continuously rate how positive or negative they felt while discussing these events. In a subsequent paradigm, perceivers continuously rate how positive or negative they believe targets felt while talking, and time series correlations between perceiver inferences and targets' self ratings are used as a measure of empathic accuracy (for a diagrammatic view of this procedure, see Figure 3).
Research in social psychology has demonstrated that several factors are related to perceivers' empathic accuracy for targets, including but not limited to relationships between perceivers and targets, perceivers' motivation, the modalities of information available to perceivers, and the shared physiological arousal between perceivers and targets.54-59 Furthermore, empathic accuracy predicts social adjustment in adolescents,60 and is impaired in autism spectrum disorder, an illness characterized by deficits in social interactions,61 suggesting that this type of social cognitive performance may meaningfully relate to social functioning.
These aspects of empathic accuracy made the paradigms for studying it appealing for use in neuroimaging. In our first study of this type, we scanned perceivers while they rated several target videos and assessed their accuracy about targets' emotional states on a video-by-video basis. We then searched for brain regions whose activity tracked parametrically with a perceivers' accuracy; or in other words, regions that were selectively engaged during periods of accurate, as opposed to inaccurate inferences (see Figure 4). Results indicated that both regions classically involved in MSA, including the medial PFC and superior temporal sulcus, and parts of the mirror neuron system involved in shared representations, including the inferior parietal lobule and the premotor cortex, tracked with the accuracy of inferences made about these complex social stimuli (Figure 5, see also Zaki et al.62)
This paradigm serves as a demonstration of methodological feasibility, and its results motivate further use of a naturalistic approach to social cognition. By employing a complex, naturalistic task, this study provides evidence that neural systems underlying shared representations and MSA work in concert while perceivers make inferences about targets' internal states. The use of a meaningful performance measure further confirmed that the engagement of both of these systems is related to empathic accuracy – in other words, that this activity tracks with a fully operationalized, empirically defined understanding of a target's mind.
Another advantage of exploring naturalistic social cognition in the brain is that it allows for examination of social cognition as a truly interpersonal phenomenon. While social cognitive theories commonly argue that attributes of both a target and a perceiver contribute actively to that perceiver's inferences,63-65 neuroscience views of empathy and social cognition – in part because of methodological constraints – have focused exclusively on the cognitive processes perceivers engage in when thinking about other minds.
While this approach has advanced knowledge about intrapersonal aspects of social cognition in perceivers, it may ignore critical aspects of social inference processes as they occur outside the decidedly non-social space inside a scanner. For example, behavioral studies have shown that aspects of a targets' personality strongly predict outcomes such as empathic accuracy. Specifically, targets who report being high in dispositional emotional expressivity as indexed using measures developed by Ickes et al.66, 67 are also more affectively “readable,” producing higher levels of empathic accuracy regardless of the perceiver viewing them.64, 68 This may be because high expressivity targets produce social cues (i.e. emotional language and facial expressions) that telegraph corresponding internal states more clearly than cues given off by low expressivity targets.56, 59 Perception of these “high fidelity” cues could in turn influence the social cognitive processes perceivers typically engage to understand targets. In other words, while perceivers may employ common intrapersonal cognitive processes when seeking to understand the mental states present in a cartoon, picture, or real-life target, these processes do not occur in a vacuum, and may importantly be affected by interpersonal factors such as targets' behaviors and dispositions.
Using real social targets as stimuli in cognitive neuroscience studies affords researchers the ability to address this issue by monitoring exactly the ways in which differences between social targets can affect the neural processes of perceivers. To explore this, we used parametric analyses similar to the ones described above to search for perceiver brain regions tracking with the expressivity of a target they were viewing in a given video. Results demonstrated that target expressivity related to perceivers' engagement of several brain regions commonly associated with MSA, including large sections of dorsal MPFC and the premotor cortex, which is part of the mirror neuron system engaged during sharing of sensorimotor representations (see Figure 6). Conjunction analyses revealed that both the MPFC and premotor cortex areas tracking with target expressivity also tracked with perceivers' empathic accuracy. Together, this evidence suggests that expressive targets may produce high-fidelity social cues that in turn cause perceivers to more strongly engage neural and cognitive processes allowing them to be accurate about target states. This interpersonal account of social cognition in the brain provides another example of how naturalistic paradigms can better approximate real-life social cognition and foster exploration of its neural bases.
The use of naturalistic stimuli in neuroimaging is an exciting avenue of research that will allow investigators to probe the roots of cognitive processes that occur in the real world, but heretofore have not been possible to examine within the tight constraints of cognitive neuroscience paradigms. This is especially important to the study of social cognition, because exploring social cognitive processes using simplified stimuli, examining perceivers in isolation, and ignoring behavioral correlates of neural activity can produce research methods and results that may not map on to the way people understand each other in the richly complex social world.
New research employing a naturalistic approach to social cognition will be able to complement the tighter, more controlled work that has dominated the field until now. Two important challenges for this research will be using meaningful behavioral measures of social cognitive success or bias, and employing more realistic stimuli involving social cues that are multimodal, dynamic, and contextually embedded.
Existing work in shared representations has begun addressing the first of these challenges by demonstrating links between the intensity of pain a perceiver rates a target as experiencing and activity in that perceiver's pain matrix.69 Similar paradigms are being developed to monitor the behavioral effects of MSA using variance in reaction time or allocentric biases in spatial perspective taking.70, 71 For example, work in social psychology has demonstrated that perceivers with access to unique knowledge overestimate the extent to which naïve social targets will use that knowledge when making decisions.72, 73 Similarly, perceivers overestimate the extent to which an aspect of the environment that is emotionally salient to them (e.g. the Barry Manilow t-shirt the perceiver has been forced to wear) will be salient to others.74, 75 Presumably, the extent of these biases is not fixed, and perceivers attending most closely to targets' internal states may attenuate perceivers' bias in making such judgments. Adapting such paradigms to a neuroimaging context will allow researchers to understand the brain bases of not only social cognitive inferences, but also cognitive and neural predictors of the efficacy of such inferences.
The second important challenge to imaging naturalistic social cognition also has been addressed in new lines of research. In addition to the work described in this article, the feasibility of using dynamic, naturalistic stimuli has been demonstrated by paradigms employing complex stimuli such as videos successfully in neuroimaging research.48, 50, 76
Of course, even using realistic stimuli and performance measures are not enough to capture much of the richness of social cognition as it occurs in the real world. For example, as we pointed out above, oftentimes when perceivers consider the mental states of targets, they make inferences using not only the cues available to them at the moment, but also pre-existing contextual information they have that may constrain their expectations about targets' mental states. For example, knowing that you are seeing a friend the day after the football team he follows obsessively loses the superbowl, you may have a specific expectation that he will feel upset or dejected, and may see signs of those emotions that you would have otherwise ignored in his behavior. There is some evidence from social psychology and cognitive neuroscience that such contextual information indeed can impact the way perceivers process and judge basic social cues.51, 52, 77 However, the majority of this research has been conducted with static picture stimuli. Examining how contextual information changes judgments about more naturalistic social information will be an important future direction.
Another import factor as of yet unexplored in cognitive neuroscience is the way that interpersonal dynamics contextualize and change how perceivers make inferences outside the lab. In real life, if a perceiver is unsure about what inference to make about a target's internal states, they need not – and probably will not – sit back and passively ponder what a target is experiencing. Instead, they will actively pursue information about targets: asking them how they feel or what they are thinking, or indirectly probing for cues about these states. Further, the information they pursue will often be biased towards confirmation of their previously existing beliefs about social targets.78-80 Similarly, targets will constrain their behavior to match the social roles they wish to fulfill.81 This role will importantly vary based on the perceiver observing that target: imagine a college student's shifting behaviors as they interact with their professors, parents, roommates, and romantic partners. A target's behavioral shifts will in turn alter the way perceivers make inferences about that target, often allowing perceivers to be accurate about targets in some, but not other situations referred to as “circumscribed accuracy” by Swann65 Thus, the process of social cognition involves an interpersonal negotiation that will be quite difficult to capture in controlled experimental laboratory settings, let alone using neuroimaging. Nonetheless, we believe it is important to prioritize attempts at capturing as much of the social cognitive process as possible in experimental settings.
Finally, the use of naturalistic social cognition will be beneficial – and may be critical – to understanding illnesses involving social cognitive deficits. For example, autism spectrum disorder (ASD) is characterized in part by deficits in reciprocal social interactions, which have been long related to difficulties in accurate mental state attribution82, 83 and in spontaneous motor imitation.84, 85 Brain bases of such behavioral abnormalities have been reported more recently: even high functioning individuals with ASD show less activity in the mirror neuron system during imitation tasks,86 and less activity in MSA related regions such as the MPFC while making explicit inferences about target internal states.87, 88
This evidence has motivated several prominent theories concerning the neurocognitive bases of social deficits in ASD. Unfortunately, these theories have often been overly constrained, suggesting that ASD is an illness purely defined either by deficits in shared representations (i.e. the “Broken Mirror Hypothesis” of Ramachandran and Oberman89), or by deficits in explicit MSAs.90 In essence, neuroscientific theories about social cognitive dysfunction in ASD have often reproduced the problems of neuroscientific theories of normative social cognition, by hanging a richly complex problem in social interaction on abnormalities in single cognitive or motor processes. Compounding this problem is the fact that – as described above – the simplified methods used to study social cognitive processes in isolation may not serve as realistic proxies for the social world, social cognition, or its deficits in ASD.
Data support the idea that single-process models of social deficits in ASD are insufficient. For example, not all studies demonstrate problems in MSA when employing simplified tasks.91 Further, the few studies attempting to directly link behavioral deficits in MSA with social symptom severity assessed clinically often have failed to find such relationships.92, 93 Finally, interventions aimed at improving social cognitive performance in simplified tasks (i.e. by training people with ASD to recognize basic emotional facial expressions in pictures) often produce improvements on these tasks, without causing any improvements in clinically assessed social interaction abilities.94-96
These disparities underscore the qualitative differences between tasks used to assess social cognition and the types of social cognition necessary to real-life interactions. They also suggest that the social deficits in ASD may stem from inabilities to carry out inferences about complex, contextually embedded social cues. In fact, the two extant studies examining empathic accuracy in ASD, using paradigms similar to the ones described above, support such assertions. First, while these individuals performed normally on simplified emotion recognition tasks, they showed more severe impairments in naturalistic empathic accuracy tasks.61 Second, these deficits were only exhibited in certain situations: people with ASD were less accurate about targets engaging in an unstructured interaction with each other, but not about targets who were interviewing each other in a structured way asking each other questions, such as “what do you like to do in your spare time?” from a list; see Ponnet et al.97 This suggests that individuals with ASD may be better able to interpret especially clear and transparent cues about target internal states that do not require contextual information to decode. One intriguing possibility is that expressive individuals, who give off more frequent and direct cues about their internal states, could be more “readable” to perceivers with ASD, potentially through the increased engagement of MSA-related brain regions associated with viewing such targets. Were this to be the case, it could motivate a novel form of intervention for ASD, in which caregivers and family members of individuals with ASD could restructure their behavior to provide clear, readable cues about their internal states, thereby improving the ability of people with ASD to understand the minds of others not only in the lab, but in the perpetually complex social world.
The possibility of finding new diagnostic techniques and interventions for ASD highlights the fact that naturalistic paradigms can allow researchers to make headway not possible using current standard techniques for assessing social cognition. By moving towards paradigms that capture the complexity of the real social world, and assessing perceivers' abilities to make accurate inferences about targets, neuroimaging of social cognition can approach more ecologically valid theories about how minds understand each other.
The authors wish to thank L. Fuhrman and J. Spicer for helpful comments on this manuscript, and NIDA Grant 1R01DA022541-01 (to KO) and Autism Speaks Grant 4787 (to JZ) for support of this work.