|Home | About | Journals | Submit | Contact Us | Français|
Deaf signers have extensive experience using their hands to communicate. Using fMRI, we examined the neural systems engaged during the perception of manual communication in 14 deaf signers and 14 hearing non-signers. Participants passively viewed blocked video clips of pantomimes (e.g., peeling an imaginary banana) and action verbs in American Sign Language (ASL) that were rated as meaningless by non-signers (e.g., TO-DANCE). In contrast to visual fixation, pantomimes strongly activated fronto-parietal regions (the mirror neuron system, MNS) in hearing non-signers, but only bilateral middle temporal regions in deaf signers. When contrasted with ASL verbs, pantomimes selectively engaged inferior and superior parietal regions in hearing non-signers, but right superior temporal cortex in deaf signers. The perception of ASL verbs recruited similar regions as pantomimes for deaf signers, with some evidence of greater involvement of left inferior frontal gyrus for ASL verbs. Functional connectivity analyses with left hemisphere seed voxels (ventral premotor, inferior parietal lobule, fusiform gyrus) revealed robust connectivity with the MNS for the hearing non-signers. Deaf signers exhibited functional connectivity with the right hemisphere that was not observed for the hearing group for the fusiform gyrus seed voxel. We suggest that life-long experience with manual communication, and/or auditory deprivation, may alter regional connectivity and brain activation when viewing pantomimes. We conclude that the lack of activation within the MNS for deaf signers does not support an account of human communication that depends upon automatic sensorimotor resonance between perception and action.
The linguistic articulators for sign language are the same as those involved in everyday human actions, such as reaching, grasping, object manipulation, and communicative gesture. Here, we explore the interaction between the neural systems that support human action understanding and those involved in sign language comprehension. Recently, the human mirror neuron system (MNS) has been argued to be the neural mechanism that underlies action understanding through embodied simulation and automatic sensorimotor resonances (e.g., Gallese, 2007; Rizzolatti and Craighero, 2004). The MNS is hypothesized to be a perception-action matching system that is automatically engaged during the observation of both communicative and non-communicative gestures or actions. The neuroanatomical correlates of the human MNS consist of the inferior frontal gyrus (IFG), ventral premotor cortex, and the inferior parietal lobule (IPL) (see Rizzolatti and Sinigaglia, 2008, for review). In addition, other regions outside the MNS play a role in the perception of actions and gestures. Specifically, the superior temporal sulcus (STS) is involved in the perception of biological motion and, more broadly, in processing social communication (e.g., Grossman et al., 2000; Allison, Puce, and McCarthy, 2000). Visual regions, including the fusiform face area (FFA) and the extrastriate body area (EBA), are also recruited during the perception of gestures and actions involving the hands, arms, and face (Montgomery and Haxby, 2008; Astafiev, Stanley, Shulman, and Corbetta, 2004).
We investigated whether knowledge and use of American Sign Language (ASL) has an impact on the neural systems that are recruited during the perception of pantomimes, which are meaningful but non-linguistic (i.e. they are not lexical signs). Specifically, we presented pantomimes, which (unlike signs) can involve the whole body, are not stored in a signer's lexicon, and may violate phonological constraints on form (Klima and Bellugi, 1979). Since signers have different life experiences with manual communication than hearing non-signers, we speculated that observing pantomimes might engage distinct neural regions for deaf signers compared to hearing non-signers. Native deaf signers have been exposed from birth to a manual linguistic system that serves as their primary means of communication. In addition, deaf signers have extensive experience with pantomimic communication through their interactions with hearing non-signers and through storytelling in ASL, which often incorporates pantomimic body and facial gestures (Emmorey, 1999). We hypothesized that these different experiences with manual communication might alter the nature of the neural systems that underlie pantomime recognition for deaf signers.
In support of this hypothesis, Corina et al. (2007) recently reported the surprising result that deaf signers did not engage the fronto-parietal network associated with the MNS when passively viewing manual actions that were self-oriented (e.g., scratch neck, lick lips, rub shoulder) or object-oriented (e.g., bite an apple, read a book, pop a balloon; i.e., the model handled the objects). In contrast, hearing non-signers exhibited robust activation within the MNS when observing these actions. Corina et al. (2007) hypothesized that life-long experience with a visual language shifts neural processing of human actions to extrastriate association areas (including the EBA), regions that were particularly active for the deaf signers. Corina et al. (2007) suggested that this shift arises because signers must actively filter human actions in order to be able to quickly distinguish linguistic from non-linguistic actions for further semantic and syntactic processing. Such pre-processing of human action is not required for non-signers. In the current study, we attempt to replicate and extend this finding by investigating whether differences between signers and non-signers in neural circuitry for action observation extends to processing meaningful pantomimes.
A second question we addressed was whether and how neural regions differ when signers are engaged in processing meaningful hand movements that have linguistic form (ASL signs) vs. meaningful hand movements that are non-linguistic (pantomimes). Comprehension of single lexical signs (even iconic signs) can be impaired in deaf signers with aphasia who nevertheless are able to recognize pantomimes (Corina et al., 1992; Marshall et al., 2004). However, there are no reports of patients with preserved sign language comprehension who are impaired in recognizing pantomimes, suggesting that a double dissociation may not exist between processing sign language and gesture (MacSweeney et al., 2008). There is also some evidence that similar neural circuitry supports processing linguistic signs and non-linguistic gestures. MacSweeney et al. (2004) contrasted perception of signed sentences (British Sign Language) with perception of a set of non-linguistic manual gestures known as Tic Tac, used in racecourse betting (the gestures were not known to the participants in the fMRI study). In general, very similar neural systems were recruited for both types of stimuli, although left perisylvian regions were recruited to a greater extent for the linguistic signs than for the non-linguistic Tic Tac gestures (left IFG, posterior STS, and anterior supramarginal gyrus).
Corina et al. (2007) contrasted perception of meaningful linguistic stimuli (ASL nouns) with perception of manual actions (actions on objects and self-oriented actions), and found that the neural systems recruited during sign perception were different from those recruited during action perception: ASL signs engaged left inferior frontal cortex (BA 46/9), left superior temporal gyrus (BA 41), and the insula, whereas actions engaged bilateral superior frontal cortex (BA 10) and right occipital-temporal cortex, extending into the right temporal pole.
Note that the Tic Tac stimuli used in MacSweeney et al. (2004), although non-linguistic in form, were symbolic and, in this sense, similar to the sign stimuli. The gestures had the potential to communicate and, in fact, participants were instructed to guess which Tic Tac gesture string did not make sense; in other words, participants were looking for meaning in the gestures. In contrast, the actions used in Corina et al. (2007) were neither linguistic nor symbolic. In the current study, we tease apart some of these effects by presenting ASL verbs and pantomimes. Both stimuli are meaningful to deaf signers, but signs participate in a linguistic system of constraints and have stored lexical representations; pantomimes do not. As in Corina et al. (and unlike MacSweeney et al.), participants in our study passively viewed the stimuli, rather than performing a semantic judgment task.
A third question we explored was how neural activation patterns differ when hearing non-signers observe meaningful hand gestures (pantomimes) compared to meaningless hand gestures (ASL verbs). Although the contrast between meaningful and meaningless hand gestures has been of central importance in the apraxia literature (e.g., Buxbaum, 2001; Goldenberg, in press), very few neuroimaging studies have examined whether and how meaningfulness impacts the neural correlates underlying the observation of human movements. Decety et al. (1997) found that viewing pantomimes recruited more left hemisphere structures than ASL signs (which were meaningless to their participants). Greater left hemisphere involvement for meaningful movements can be attributed to the greater role of the left hemisphere in processing semantic information (e.g., Gonzales Rothi, Ochipa, and Hielman, 1991). However, Villareal et al. (2008) recently reported that an extensive, bilateral common neural network was engaged during the recognition of both meaningful hand movements (pantomimes and emblems) and meaningless hand movements (actions involving movements comparable to those involved in meaningful actions but with no goal). Here, we further investigate the extent to which meaningful hand movements (pantomimes) and meaningless hand movements (ASL signs, which were not known to the hearing participants) engage extensive overlapping regions and whether meaningful movements preferentially activate left hemisphere structures.
Finally, we applied functional connectivity analyses to characterize more fully the extent to which sign and gesture processing might build upon the fronto-parietal MNS in deaf signers and hearing non-signers. Functional connectivity analyses are able to identify the extent to which activation levels in two regions are correlated, and this correlation is interpreted as a reflection of the degree to which the two regions are functionally connected (e.g., Friston, 1994). Such analyses have demonstrated differences in functional connectivity within the left perisylvian language network for high- vs. low-capacity readers (Prat, Keller, and Just, 2007) and have revealed altered connectivity patterns within the motor network for patients with multiple sclerosis (Rocca et al., 2007). In this experiment, we selected seed voxels from regions within the MNS (ventral premotor cortex and inferior parietal cortex) and from a region outside the MNS (fusiform gyrus). The fusiform gyrus was chosen because this region is known to be engaged when viewing faces and bodies. In these analyses, correlation coefficients were computed between mean time series in seed voxels and all other voxels in the brain.
Fourteen deaf signers (7 males) and 14 hearing non-signers (6 males) participated in the experiment. All participants were right-handed (Oldfield handedness scores were 85.6 for the deaf group and 88.3 for the hearing group), and all had attended college. The deaf signers (mean age = 22.3 years; range: 19-43 years) were all born into signing families, were exposed to ASL from birth, and reported a hearing loss of ≥ 70dB. The hearing non-signers (mean age = 24.3 years; range: 22-29 years) reported normal hearing and no knowledge of a signed language.
To create the pantomime and ASL stimuli, a deaf actress was filmed performing a large number of pantomimes (N=118) and ASL verbs (N=108), and these stimuli were edited and presented to a separate group of 22 deaf signers and 38 hearing non-signers for meaningfulness ratings. The actress also produced 104 emblematic gestures (e.g., thumbs-up), 81 highly iconic ASL signs (e.g., the verb TO-HAMMER), and 93 nonsense gestures (possible but non-occurring ASL signs). All 504 stimuli were presented in random order, and participants were asked to rate each form for meaning on a scale of 0 – 3, where 0 = no meaning, 1 = weak meaning, 2 = moderate or fairly clear meaning, and 3 = absolute strong/direct meaning. If the form was rated as having some meaning (1-3), participants were asked to provide a brief description or definition of the meaning. Finally, the deaf participants were also asked to decide whether the form was ASL, pantomime, or nonsense.
Based on this norming study, we selected 60 pantomimes that were rated as meaningful by the majority of both the deaf and the hearing participants (given a rating of 2 or 3). In addition, these forms were considered to be pantomimes and not ASL signs by the majority (83%) of the deaf participants. We also selected 60 ASL verbs that were considered to be ASL signs and not pantomime by deaf signers and that were not recognized as meaningful by hearing non-signers; the majority (92%) of the hearing participants rated these ASL signs as having weak or no meaning (a rating of 0 or 1). The pantomimes and ASL verbs are listed in the Appendix, and example stimuli are illustrated in Figure 1.
Pantomimes and ASL verbs along with the three additional classes of stimuli (not reported) were presented while fMRI data were acquired. Each stimulus type consisted of 60 video clips, and the length of each clip was 1.5 seconds. Each stimulus type was presented in four 45 second blocks, 15 clips in each block, with an inter-stimulus-interval of 3 seconds. As a baseline condition, a fixation cross was presented in the same fashion (i.e., sixty 1.5 second clips presented in four 45 second blocks with a 3 second ISI). All stimuli were presented in three fMRI runs with 8 blocks per run. Stimuli were randomized within runs; the same runs were used in each session, but their sequence was randomized across subjects.
All video clips were presented using PsyScope X B45 software (Carnegie Mellon University, Department of Psychology) on a Powerbook G4 Macintosh. Stimuli were displayed at the center of a rear projection screen in the scanner room. Participants saw the stimuli through a mirror attached on the head coil.
The participants were instructed either in ASL or in English to simply watch the video clips attentively without making any response. Specifically, participants were told “you do not need make any response during the experiment, but please pay close attention to all of the videos. We will have a short test about the stimuli after the scan. However, please do not try to memorize the stimuli. We do not want to see brain responses for memorization.” After scanning, participants completed a short recognition test to evaluate their attentiveness during the period of image acquisition. For this task, we presented 25 video clips, and participants were asked to judge whether or not each clip had been presented during the experiment (5 clips were novel and 20 were from the experiment). Accuracy was 74.86% for the deaf group and 76.57% for the hearing group, indicating that participants had attended to the stimuli.
Blood oxygenation level-dependent contrast functional images were acquired on a 3T GE scanner (Milwaukee, WI) using a gradient echo EPI sequence (TR = 2000 ms, TE = 30 ms, FA 90°, FOV = 240 × 240 mm, matrix size = 64 × 64, voxel size = 3.75 × 3.75 × 6 mm3). Twenty-four axial slices were acquired with 6 mm thickness and no gap, covering the whole brain. The MRI data were acquired with a standard head-coil. Four dummy scans (with no data acquisition) were inserted in the beginning for each fMRI run to ensure steady-state magnetization. In total, 555 scans were collected while the participants passively viewed the stimuli or fixation cross.
Data were preprocessed and analyzed using statistical parametric mapping software (SPM2, Wellcome Department of Imaging Neuroscience, http://www.fil.ion.ucl.ac.uk/spm) implemented in MATLAB version 7.1.0 (R14) (Mathworks Inc., Sherbom, MA). Functional scans were realigned using the first volume as reference for motion correction. Subjects in whom head motion in any direction exceeded 1.5 mm were excluded.
Spatial normalization was performed using global linear (affine transform) and local nonlinear (cosine basis functions) transformations to register the EPI images from each subject to the ICBM152 template defined in MNI space (voxel size 3 × 3 × 3 mm) . Images were spatially smoothed with a Gaussian kernel with full-width at half maximum (FWHM) of 9mm.
The study utilized a block design, and task effects were estimated using a general linear model with a delayed boxcar waveform. A high pass filter was implemented using a cut-off period of 128 seconds in order to remove the low frequency drifts from the time series. The task effects were first evaluated at the single subject level. In order to account for inter-subject variance, a random effects model was used to obtain group results with contrast images from each subject. One-sample t-tests assessed the significance of the effects separately for each group. Two sample t tests were performed to estimate differences between the subject groups. Statistical inferences were performed at the voxel level at p < 0.005 uncorrected for multiple comparisons across the entire brain volume. Activation clusters encompassing less than 10 voxels were discarded. To prevent false activations, we used inclusive masking procedures – for example, relative increases in activity in one group are not reported unless they also constitute activations above the fixation baseline in that group.
Functional connectivity was measured by the seed voxel method. The seed voxel regions were selected from conjunctions or contrasts carried out in SPM. The seed voxels that represent the MNS were selected from significant activation clusters from the fMRI contrast of the pantomime vs. fixation in the hearing group. These seeds – in the left IPL (−54 −27 33) and left premotor cortex (−51 0 6) – were confirmed by literature review to be within the human MNS (Rizzolatti and Craighero, 2004). Seed voxels from the peak activation coordinates (see Table 1) were not selected because they were at relatively high Z levels compared to previous neuroimaging studies of action observation (e.g., Decety et al, 2002, Grafton et al., 1996, and Manthey, Schubotz, & von Cramon, 2003). The fusiform seed (−42 −48 −12) was selected from a conjunction analysis of the pantomime and the ASL conditions; this seed was shown to represent a common area of activation outside of the MNS in both the hearing and deaf groups.
Seed voxel linear regression was performed in MEDx 3.44 (Medical Numerics, Inc. USA) in which the time series for each brain voxel was regressed against the time series of the chosen seed voxel. First, all preprocessed images were normalized to the same mean intensity in order to prevent spurious correlations due to between-subject differences in mean intensity. We also ruled out the possibility of spurious correlations due to unequal variances between participant groups by calculating the mean variance across all voxels in each condition. Mean variances were not significantly different between groups. Multiple regressions were applied to the intensity normalized images to generate statistical maps for each subject and condition, for each seed voxel. Beta maps from these regression analyses were subjected to single-group t-tests by condition, which created group z-maps representing correlations of each voxel with the seed voxel for a given condition across all participants within a group. Z values of absolute value greater than 3 were considered significant, and a mask was applied to the z-maps to exclude all values between −3 and 3. Clusters with fewer than 10 voxels were not included in the analysis. The resulting maps depicted significant positive and negative correlations between each voxel and the seed voxel for a given condition. Z maps were projected onto a single subject T1 image to represent results of functional connectivity analysis (only positive correlations are shown).
For both contrast and connectivity analyses, anatomical regions were identified and labeled using the AAL and Brodmann maps provided in MRIcro (http://www.sph.sc.edu/comd/rorden/mricro.html).
As expected, hearing non-signers showed extensive activation within the mirror neuron system when observing meaningful pantomimes compared to fixation baseline, as shown in Figure 2A. Significant bilateral activation was present in the inferior frontal gyrus (IFG), premotor cortex, and the inferior parietal lobule (IPL), extending into the superior parietal lobule (SPL) (Table 1). In contrast, no significant activation was observed in the MNS when deaf signers viewed pantomimes (Figure 2A and Table 1). Both groups showed significant bilateral activation in middle temporal cortex and the fusiform gyrus. There were no neural regions that were more active for deaf signers compared to hearing non-signers, but hearing non-signers exhibited significantly more activation within fronto-parietal cortices compared to deaf signers (Table 2).
For hearing non-signers, perception of ASL verbs (meaningless for these participants) activated regions similar to those for pantomime, although somewhat less extensively (see Figure 2B). For deaf signers, perception of ASL verbs also activated regions similar to those for pantomime; however, activation was also observed in the left inferior frontal gyrus for ASL verbs (see Table 1 and Figure 2B). As with pantomimes, there were no neural regions that were more active when deaf signers processed ASL signs compared to hearing non-signers. Hearing non-signers again showed more activation within the mirror neuron system than deaf signers (Table 2).
As predicted for hearing non-signers, we observed more activation in the inferior and superior parietal lobules for pantomimes compared to (meaningless) ASL signs (see Table 3). This contrast also revealed more activation in the left dorsal precentral gyrus (BA 6), left postcentral gyrus, and left fusiform gyrus for pantomimes than for ASL signs. For deaf signers, we observed greater activation for pantomimes (meaningful but without linguistic form) compared to ASL signs (meaningful with linguistic form) in the right posterior superior temporal gyrus and in the right fusiform gyrus. Lastly, both groups engaged the right inferior occipital gyrus to a greater extent for pantomimes than for ASL signs.
For deaf signers, this contrast did not reveal greater involvement of left perisylvian cortices for ASL verbs (Table 3). For the hearing non-signers, ASL signs engaged the IPL and cuneus bilaterally, left posterior cingulate gyrus, and right superior medial frontal cortex (BA 9) to a greater extent than pantomimes.
Because deaf signers did not exhibit significant activation in left ventral premotor cortex or the left inferior parietal lobule, we present only functional connectivity analyses for seed voxels within these regions for the hearing non-signers.
For hearing non-signers, robust connectivity was observed between the left ventral premotor seed voxel and left parietal cortices for both pantomimes and ASL signs (Table 4). Ventral premotor connectivity was strongly left lateralized within the parietal lobe for both stimulus types. In addition, functional connectivity was observed for pantomimes between the left premotor seed voxel and pre- and postcentral gyri bilaterally, left supplementary motor cortex, left superior temporal cortex, and bilateral middle and inferior temporal cortices. For (meaningless) ASL signs, functional connectivity was less extensive and included left pre- and postcentral gyri and superior and middle temporal cortices.
Left lateralized functional connectivity was observed with premotor cortex for both pantomimes and (meaningless) ASL verbs for the hearing participants (Table 4), and a small cluster of connectivity within right IPL was observed for both stimuli types. For ASL signs, hearing non-signers exhibited connectivity bilaterally with the inferior frontal gyrus and the superior temporal gyrus that was not observed for pantomimes.
For pantomimes, activity within the left fusiform gyrus was correlated with left IPL activity for both the deaf signers and hearing non-signers (Table 5). However, connectivity with left premotor cortex was only observed for the hearing non-signers, and only deaf signers exhibited connectivity with the right hemisphere homologue for both signs and pantomimes (see Figure 3). As with the other seed voxels, fusiform connectivity for hearing non-signers was strongly left lateralized. In addition, deaf signers showed unique functional connectivity between the fusiform gyrus and inferior temporal cortices bilaterally for both pantomimes and ASL verbs. Hearing non-signers exhibited functional connectivity between the left fusiform seed voxel and left STG for both stimulus types, which was not observed for the deaf group.
To examine the degree of right hemisphere connectivity for the deaf and hearing groups, we examined the proportion of suprathreshold voxels in the right hemisphere for the fusiform seed voxel. For deaf signers, 17.6% of the total suprathreshold voxels (133/754) were in the right hemisphere when viewing pantomimes and 29% of suprathreshold voxels (234/830) were in the right hemisphere when viewing ASL verbs. In contrast, for hearing non-signers, only 0.3% (4/1282) and 0.2% (4/1709) were in the right hemisphere when viewing pantomimes and ASL verbs, respectively.
The most striking result of this study was the lack of activation within the mirror neuron system (inferior frontal gyrus, ventral premotor cortex, and inferior parietal lobule) for deaf ASL signers when passively viewing either signs or communicative gestures, compared to a fixation baseline (see Figure 2A). In contrast, hearing non-signers showed robust activation within the MNS for both sets of stimuli despite the fact that, for these participants, pantomimes are meaningful and signs are not. These findings replicate previous studies of human action observation (e.g., Buccino et al., 2001; Grèzes, Armony, Rowe, and Passingham, 2003; Villarreal et al., 2008) and extend Corina et al.'s (2007) study, which found that deaf signers did not engage fronto-parietal cortices during passive viewing of human actions with objects. We interpret these results as indicating that extensive experience with meaningful hand movements (sign language and pantomime) substantially reduces or eliminates the role of the MNS in passively viewing actions that are communicative, either with linguistic form (ASL signs) or without (pantomimes). In addition, such results argue against a universal, automatic resonance of the motor system during action observation (e.g., Rizzolatti and Craighero, 2004). If activation within the MNS occurs automatically in response to observed human body actions, we would expect neural activity in fronto-parietal cortices when passively viewing hand and body movements in signers as well as non-signers.
However, it is not the case that deaf signers never engage the MNS, or elements of the MNS, when comprehending communicative gestures and signs, particularly when an active task is added to the paradigm. Many studies report activation within the left inferior frontal gyrus (BA 44/45) and inferior parietal cortex (BA 40) when signers perform a semantic judgment or memory task with sign language (e.g., Bavelier et al., 2008; Neville et al., 1998; MacSweeney et al., 2002; MacSweeney et al., 2006). As noted earlier, MacSweeney et al. (2004) found fronto-parietal activation when signers were asked to look for meaning in Tic Tac gestures. Thus, when additional linguistic or cognitive demands are superimposed upon passive viewing, activation within the MNS is observed for signers. Interestingly, activation in these areas is observed for non-signers even without such demands.
Mounting evidence from functional neuroimaging reveals that practice or repeated exposure to a particular task can create significant changes in neural representations and functional connectivity (see Clare Kelly and Garavan, 2005, for review). Practice and experience can result in either an increase or a decrease in activation within task-relevant brain areas – or they can cause functional reorganization of neural activity (both increases and decreases across cortical regions). Decreases in the extent or intensity of activation are most commonly attributed to increases in neural efficiency. For example, the overall activation level in cortices that support distributed representations may be reduced because only a minority of neurons fire in response to relevant stimuli, and activity of the majority of other neurons is suppressed (Poldrack, 2000). We hypothesize that deaf signers recognize signs and pantomimes quickly and relatively automatically and that this ease of processing leads to a substantial reduction or a complete lack of neuronal firing within the MNS, in contrast to hearing non-signers who do not have life-long experience with a conventional sign language.
At first glance, our findings and explanation may appear at odds with the results of Calvo-Merino and colleagues. Calvo-Merino et al. (2006) and Calvo-Merino et al. (2005) reported increased activation within the MNS when expert dancers viewed dance movements that they had been trained to perform, compared to movements on which they had not been trained. However, as in the sign language experiments of Bavelier et al. (2008), Neville et al. (1998), and MacSweeney et al. (2002; 2006), the dancers did not passively view dance movements. Rather, they were asked to perform judgment tasks for each video (e.g., “How tiring is each movement?” or “How symmetric is each movement?”). Given the previously mentioned effects of training and experience on neural activity, we suggest that passive viewing of dance movements might reveal reduced activation in the MNS for expert dancers compared to non-dancers. But dancing and manual communication are different phenomena. For deaf signers, manual communication (particularly signing) occurs throughout the day, and sign recognition is immediate and automatic, as evidenced by sign-based Stroop effects (e.g., Vaid and Corina, 1989; Marschark and Shroyer, 1993). For expert dancers, dance movements may be processed by different mechanisms that are asymbolic and more tightly linked to the motor system.
Corina et al. (2007) found evidence for functional reorganization when deaf signers observed non-communicative human actions. These actions were non-communicative because there was no intent to convey information on the part of the actor who produced the self-oriented gestures (e.g., scratching oneself) and who interacted with objects (the fMRI analyses combined both action types). In contrast, we found no evidence for functional reorganization when deaf signers observed communicative human actions, that is, no evidence that deaf signers engaged a qualitatively distinct neural system compared to hearing non-signers when they observed pantomimes, although we did observe unique functional connectivity patterns for the deaf group (see Table 5 and Figure 3). Interestingly, when pantomimes were contrasted directly with ASL verbs, activation within right superior temporal cortex was observed only for deaf signers (see Table 3). Saxe et al. (2004) have argued that a region in the right posterior superior temporal sulcus responds specifically to observed intentional actions. The activation peak for observing pantomimes [63, –45, 15] was just lateral to the local maxima in right STS reported by Saxe et al. (2004) [Exp. 1: 54, –42, 9; Exp. 2: 51, –42, 18]. For deaf signers, a critical difference between observing pantomimes and ASL verbs is that pantomimes depict intentional actions themselves, whereas ASL verbs represent linguistic labels for actions. For this reason, right posterior superior temporal cortex may have been more engaged for pantomimes than for ASL verbs. We hypothesize that hearing non-signers did not show differential activation in right STS for this contrast because they may have been attempting to work out the intentions of the model when she produced ASL verbs.
Replicating previous results, we found that pantomimes engaged more left hemisphere structures than ASL signs for the hearing group, and that pantomimes also engaged parietal regions to a greater extent than ASL signs for the hearing group (see Table 3). Most of the pantomimes involved grasping movements (e.g., pretending to hold and manipulate objects like a hammer, a telephone, or a dart), reaching movements of the arm and hand (e.g., pretending to direct an orchestra, shampoo one's hair, or play the piano), and full body movements (e.g., dancing in place, pretending to jog or swing a golf club). Stronger activation within parietal cortices (BA 40 and BA 7) for pantomimes most likely reflects the recognition and understanding of the model's depiction of picking up, moving, and holding various objects. Several studies have found that the inferior parietal lobule is engaged when observing grasping actions (e.g., Grafton et al., 1996), while the superior parietal lobule is engaged when observing reaching movements of the hand and arm (Filimon et al., 2007), as well as when producing pantomimes (Choi et al., 2001).
For deaf signers, there were no regions that were significantly more engaged for ASL verbs compared to pantomimes. Pantomimes do not have stored lexical representations, but they do convey sentence-level concepts with both an agent (the actress) and a patient (the manipulated object). Thus, pantomimes are likely to involve more extensive semantic processing than single lexical verbs, which might have obscured potential differences in regional activation between signs and pantomimes in a direct contrast. We note that the contrast between ASL verbs and fixation baseline revealed activation in left inferior frontal gyrus (BA 45) that was not present for pantomimes compared to fixation (see Figure 2). Overall, these findings are consistent with those of MacSweeney et al. (2004), who reported largely overlapping patterns of activation for signed sentences and strings of meaningless Tic Tac gestures, but signed sentences exhibited stronger activation in left IFG and left posterior STS extending into supramarginal gyrus. It is likely that we observed even less sign-specific activation than MacSweeney et al. (2004) because pantomimes are meaningful and, in this sense, may be more sign-like than Tic Tac gestures.
For pantomimes, the functional connectivity analyses revealed correlated activity between left premotor cortex and left IPL for the hearing group (Table 4), indicating robust integration within the MNS when viewing either meaningful pantomimes or meaningless gestures. In addition, functional connections of both the left anterior (premotor) and left posterior (IPL) components of the MNS were strongly left lateralized for the hearing group. For the fusiform seed voxel (a region outside the MNS, activated by both the hearing and deaf groups), activity was coupled with the anterior component of the MNS (ventral premotor cortex) for pantomimes, but only for the hearing group. Again functional connectivity was strongly left lateralized for hearing non-signers and bilateral for deaf signers (Figure 3).
Greater right hemisphere connectivity for deaf signers might be an effect of experience with sign language. Some previous studies have found more right hemisphere activation during sign language comprehension, compared to spoken language comprehension (Neville et al., 1998; Capek et al., 2004; but see MacSweeney et al. 2002; Hickok, Bellugi, and Klima, 1998). In addition, a recent morphometry study by Allen et al. (2008) found that both deaf and hearing signers exhibited increased white matter volume in the right insula compared to hearing non-signers. Allen et al. (2008) speculated that the distinct morphology of the right insula for ASL signers might arise from enhanced connectivity due to an increased reliance on cross-modal sensory integration in sign language compared to spoken language. If sign language processing recruits right hemisphere structures to a greater extent than spoken language, signers may develop more extensive functional networks connecting the left and right hemispheres.
Another possibility, which is not mutually exclusive, is that the bilateral connectivity we observed might reflect the effects of congenital and life-long auditory deprivation on brain organization. Kang et al. (2003) investigated the functional connectivity of auditory cortex in deaf children and adults by examining interregional metabolic correlations with 18F-FDG PET. In this study, the mean activity of FDG uptake in the cytoarchitectonically defined A1 region served as a covariate in the interregional and interhemispheric correlation analysis. The authors reported that metabolism of left auditory cortex was strongly correlated with auditory cortices in the right hemisphere for deaf adults and older deaf children (ages 7-15 years), but no such correlation was evident for normally hearing adults. The cross-hemispheric correlation was stronger for deaf adults than for deaf children. The absence of auditory input, perhaps in conjunction with sign language experience, may lead to plastic changes in functional connectivity between the two hemispheres of the brain for deaf individuals.
In summary, deaf signers exhibited distinct patterns of brain activity and functional connectivity when passively viewing pantomimes and ASL signs, compared to hearing non-signers. The fact that no brain activation was found in anterior (ventral premotor) or posterior (inferior parietal) elements of the MNS for deaf signers for either signs or pantomimes argues against an account of human communication that depends upon automatic sensorimotor resonance between perception and action (see also Toni et al., 2008). We hypothesize that life-long experience with manual communication reduces or eliminates involvement of the mirror neuron system during passive viewing of communication via the hands. The neural regions engaged in pantomime and sign language perception were very similar, but non-identical, for signers. Recognizing pantomimes recruited right posterior superior temporal cortex near a region hypothesized to respond to observed intentional actions (Saxe et al., 2004), whereas recognizing ASL verbs preferentially engaged regions within the left inferior frontal gyrus. For hearing non-signers, processing pantomimes engaged parietal cortices to a greater extent than processing meaningless hand movements (ASL signs), reflecting the role of the inferior and superior parietal cortices in grasping and reaching movements. Finally, functional connectivity analyses revealed greater cross-hemisphere correlations for deaf signers and greater left hemisphere integration for hearing non-signers, particularly when viewing pantomimes. We speculate that more right hemisphere integration for deaf signers might arise as a result of long-term experience with sign language processing, plastic changes associated with sensory deprivation, or both.
This research was supported in part by NIH grants R01 HD-13249 and R01 DC-00201. We would like to thank Franco Korpics for help recruiting Deaf participants, Jeannette Vincent for help with stimuli development, Helsa Borinstein for help with the gesture norming study, and Jeffery Solomon for technical support with the connectivity analysis. Finally, we thank all of the Deaf and hearing individuals who participated in the study.
|BAWL-OUT||applying nail polish to fingers|
|CAMPING||biting into a large sandwich|
|CHALLENGE||blowing nose using a tissue|
|CHANGE||blowing soap bubbles|
|CURSE||catching a ball|
|DANCE||combing a mustache and beard|
|DESTROY||covering a cough with the hand|
|DIE||dancing in place|
|DONATE||dialing on the keypad of a phone|
|DREAM||dialing on a rotary phone|
|DROWN||directing an orchestra|
|DRY||drying the face with a towel|
|FIGHT||drying the hands with a towel|
|FIX||eating with one hand from a bowl|
|GRADUATE||holding up a paper and ticking off a list|
|GUESS||injecting arm with a syringe|
|IGNORE||licking an ice cream cone|
|INSULT||lifting a heavy box with both hands|
|INTERPRET||listening on the telephone|
|INTERVIEW||lowering eyeglasses to see a long distance|
|KILL||peeling a banana|
|LIE||pitching a baseball|
|LOCK||playing a guitar|
|MARCH||playing the piano|
|PLAY||pulling a pint of beer|
|PLUG-IN||putting an earring in an ear lobe|
|PUNISH||putting on eyeglasses|
|REPEAT||removing an object from the mouth|
|REQUIRE||removing a splinter from thumb with tweezers|
|ROAM||running in place|
|RUN||sewing with needle and thread|
|RUSH||shaking and throwing dice|
|SELL||shaving the face with a razor|
|SHOUT||shampooing one's own hair|
|SING||shooting a rifle|
|SIT||smelling an object (holding it to the nose)|
|SKIP||sneezing into a tissue|
|SPEAK||swinging baseball bat|
|STAY||taking a neck pulse|
|TAKE-OFF||taking a wrist pulse|
|THROW-AWAY||tearing a sheet of paper in half|
|TRAVEL||testing an invisible glass wall|
|USE||throwing a ball|
|VISIT||throwing a dart|
|WAIT||trimming finger nails|
|WEIGH||turning a door knob and opening a door|
|WISH||typing on a keyboard|
|ZAP (get revenge)||wringing out a towel|
Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
Karen Emmorey, San Diego State University.
Jiang Xu, National Institute on Deafness and other Communication Disorders.
Patrick Gannon, Hofstra University School of Medicine.
Susan Goldin-Meadow, University of Chicago.
Allen Braun, National Institute on Deafness and other Communication Disorders.