|Home | About | Journals | Submit | Contact Us | Français|
Long-term memory integrates the multimodal information acquired through perception into unified concepts, supporting object recognition, thought, and language. While some theories of human cognition have considered concepts to be abstract symbols, recent functional neuroimaging evidence has supported an alternative theory: that concepts are multimodal representations associated with the sensory and motor systems through which they are acquired. However, few studies have examined the effects of cortical lesions on the sensory and motor associations of concepts. We tested the hypothesis that individuals with disease in auditory association cortex would have difficulty processing concepts with strong sound associations (e.g., thunder). Human participants with the logopenic variant of primary progressive aphasia (lvPPA) performed a recognition task on words with strong associations in three modalities: Sound, Sight, and Manipulation. LvPPA participants had selective difficulty on Sound words relative to other modalities. Structural MRI analysis in lvPPA revealed gray matter atrophy in auditory association cortex, as defined functionally in a separate BOLD fMRI study of healthy adults. Moreover, lvPPA showed reduced gray matter density in the region of auditory association cortex that healthy participants activated when processing the same Sound words in a separate BOLD fMRI experiment. Finally, reduced gray matter density in this region in lvPPA directly correlated with impaired performance on Sound words. These findings provide crucial evidence supporting the hypothesis that conceptual memories are represented in the sensory and motor association cortices through which they are acquired.
Our experience of the world is inherently multimodal, integrating visual, auditory, and somatosensory information into unified percepts. Conceptual memory is similarly multimodal, integrating the varied information acquired through perception into unified concepts. These concepts form the basis of long-term memory and support object recognition, thought, and language. Yet the mechanisms through which these concepts are represented in the brain have remained elusive. Some theories of human cognition have considered concepts to be abstract symbols, independent of the sensory and motor systems through which they are acquired (Fodor, 1975; Pylyshyn, 1985; Landauer and Dumais, 1997). In contrast, many current theories suggest that concepts are multimodal representations, recruiting sensory and motor association cortices (Allport, 1985; Martin, 2007; Barsalou, 2008; Kiefer and Pulvermüller, 2011).
Functional neuroimaging evidence is consistent with this hypothesis of multimodal representation, demonstrating that even reading about an object activates sensory-motor regions involved in object perception (Martin, 2007; Kiefer and Pulvermüller, 2011). Hence, reading a word with strong sound associations, like “telephone,” seems to activate acoustic information in auditory association cortex (Kiefer et al., 2008). However, these fMRI studies do not demonstrate that sensory-motor regions are necessary for conceptual processing. An alternative possibility is that these activations are driven by epiphenomenal effects, occurring after abstract conceptual memory has been accessed (Machery, 2007; Mahon and Caramazza, 2008).
Investigations of individuals with brain lesions provide critical evidence for testing the prediction that sensory-motor cortices are necessary for conceptual processing. Visual association cortex atrophy is implicated in difficulty with concrete words (Bonner et al., 2009). Lesions near motor cortex are implicated in difficulty with action relative to object concepts (Arévalo et al., 2007; Grossman et al., 2008; Vigliocco et al., 2011; Kemmerer et al., In press). However, these studies have not examined the convergence of fMRI findings in healthy adults with structural MRI of cortical atrophy, and we are unaware of studies assessing sound-related words in individuals with atrophy of auditory association cortex.
The logopenic variant of primary progressive aphasia (lvPPA) is a focal neurodegenerative condition, usually caused by Alzheimer’s histopathology (Bonner et al., 2010; Grossman, 2010; Gorno-Tempini et al., 2011) and associated with atrophy in superior temporal and inferior parietal regions of the left hemisphere (Grossman, 2010), including canonical auditory association cortex (Maeder et al., 2001; Lewis et al., 2004). Although lvPPA has not previously been associated with a pronounced semantic memory impairment, we predicted relative difficulty in lvPPA for concepts with strong sound associations. We tested this hypothesis with a word recognition study involving words with strong associations in three different modalities: Sound (e.g., thunder), Sight (e.g., pyramid), and Manipulation (e.g., scissors). We expected that, while the overall semantic impairment in lvPPA would be mild, word recognition errors would occur more frequently for words with strong sound associations relative to other modalities. We related behavioral findings to structural MRI analyses of gray matter density in lvPPA, and related this to converging evidence from a separate fMRI experiment using the same task in healthy adults.
We studied 10 participants (5 female) diagnosed with lvPPA according to published criteria (Gorno-Tempini et al., 2011). These patients presented with slowed spontaneous speech, frequent word-finding pauses, difficulty in sentence and phrase repetition, but relatively spared word knowledge, object knowledge, and grammar. All participants had clinically normal hearing. Diagnosis was confirmed in a consensus conference based on a review of a semistructured history, a comprehensive mental status exam, and a complete neurological exam by at least two independent, trained reviewers. A control group of 22 age-matched healthy adult volunteers (14 female) was also studied. Table 1 summarizes the demographic and clinical characteristics of the participants. Participants with lvPPA had mild overall cognitive impairment according to the Mini Mental State Exam (MMSE) (Folstein et al., 1975), difficulty repeating auditory verbal sequences on the Digit Span test (Wechsler, 1987), and relatively preserved performance on a measure of semantic memory, the Pyramids and Palm Trees test (Howard and Patterson, 1992). All participants and their legal representatives participated in an informed consent procedure approved by the University of Pennsylvania institutional review board.
The lexical decision experiment included 120 nouns with strongly associated features in three modalities: Sound (n=40), Manipulation (n=40), and Sight (n=40). The stimuli were selected from a set of 489 nouns probed in a norming study (n=22 young adults) on a scale from 0 to 6 for how strongly they were associated with each of the three modality-specific features: sound, manipulation, and sight. As summarized in Table 2, Sound words had significantly higher sound ratings than Sight words (t(78)=55.2, p<.0001) and Manipulation words (t(78)=34.0, p<.0001). Manipulation words had significantly higher manipulation ratings than Sight words (t(78)=30.5, p<.0001) and Sound words (t(78)=27.2, p<.0001). Sight words had significantly higher sight ratings than Manipulation words (t(78)=8.98, p<.0001) and Sound words (t(78)=8.85, p<.0001). All categories were otherwise matched on letter length, syllable length, and lexical frequency (Francis and Kucera, 1982) (all pairwise comparisons p>.15; Table 2). We also collected familiarity ratings (n=14 young adults). All categories had high familiarity ratings (>5 on a scale of 0 to 6). There were no differences in familiarity across the modality-specific categories (ratings for each category >5.5; all comparisons p>.2). Pronounceable pseudoword foils (n=120) that obeyed the phonotactic rules of English but did not appear in any standard English dictionaries and filler stimuli without strong modality-specific associations (n=40) were included as well. These matched modality-specific words for frequency, letter and syllable length, but familiarity ratings were slightly lower for words without modality-weighted associations (p<.001 compared to all other word categories).
As expected for both controls and lvPPA participants, accuracy was worse for pseudowords (controls: M=90.2, SD=9.4; lvPPA: M=82.0, SD=13.9) than for real words (all p<.05). However, relative performance on these conditions did not differ between lvPPA and controls (p=0.5). Similarly, both lvPPA and controls were worse on items without modality-weighted associations (controls: M=93.8, SD=6.0; lvPPA: M=84.8, SD=16.3) than on the modality-specific word categories (all p<.05), consistent with the pattern of familiarity ratings. Once again, relative performance on these conditions did not differ across lvPPA and controls (p=0.8). These conditions are therefore not further analyzed.
Participants performed an auditory lexical decision task in which they responded by button press to indicate whether or not each stimulus item was a real word. Stimuli were presented over headphones at a comfortable volume (~70dB), and we ensured that each participant could hear the stimuli before starting the experiment. Each trial began with a beep. After stimulus presentation, participants had up to 7 seconds to respond. Stimuli were presented in a random order, and participants were given two rest breaks during the task. Practice sessions were administered to familiarize participants with the task and to ensure that task instructions were understood. Stimulus items from the practice sessions were not re-presented in the experiment. E-Prime 1.0 (Psychology Software Tools, Inc., Pittsburgh, PA) was used to present stimuli and record responses.
Because the accuracy data for the lexical decision task were not normally distributed, we analyzed behavioral performance with nonparametric statistical tests. All p values are corrected for multiple comparisons. Reaction time data were not analyzed for one lvPPA participant who had difficulty performing button press responses with one hand, and for three control participants who did not keep their hands on the response keys during testing.
Nine of the lvPPA participants had a T1-weighted MRI scan within a year of the behavioral task. MRI scans were collected for the lvPPA participants and 38 age-matched controls (a different control group than in the behavioral task; 17 female) with a SIEMENS Trio 3.0T scanner at 1-mm slice thickness and a 195×256 matrix using an MPRAGE protocol (TR=1620 ms, TE=3 ms, flip angle=15°, in-plane resolution=.9766×.9766). Images were preprocessed by deforming into a local template space with a 1-mm3 resolution using PipeDream (https://sourceforge.net/projects/neuropipedream/) and Advanced Normalization Tools (ANTS, http://www.picsl.upenn.edu/ANTS/) (Avants et al., 2008). Images were inhomogeneity-corrected via N4 (Tustison et al., 2010), segmented into tissue probability maps using template-based priors, and registered to MNI-template space. The resulting gray matter probability images are a measure of gray matter density. These were smoothed in SPM8 (Wellcome Trust Centre for Neuroimaging, London, UK) with an 8-mm FWHM Gaussian kernel.
In SPM8, a two-sample t-test contrasted gray matter density between lvPPA participants and healthy controls in order to identify regions of significant cortical atrophy. We used a whole-brain threshold of p<.05 corrected voxelwise for familywise error (FWE). We constrained voxelwise comparisons to gray matter with an explicit gray matter mask, defined by generating a mean gray matter probability image from healthy controls and thresholding at 0.2.
These methods are detailed in an fMRI study (Bonner et al., Submitted) of 20 healthy adults (mean age=23.5 years, SD=4.2). Briefly, for the auditory localizer, participants listened to complex, unnameable sounds (composites of distorted environmental sounds; n=25) and performed a one-back task, pressing a button if the sound repeated. A baseline condition involved the same one-back task using complex, unnameable images (composites of unnameable colored geometric images; n=25). The fMRI version of the lexical decision task involved the same task and lexical stimuli as the behavioral version, with the exception that stimuli were presented on a screen rather than over headphones to minimize stimulus distortion from scanner noise.
BOLD images were acquired with fat saturation, 3-mm3 voxels, flip angle of 90°, TR=3000 ms, TEeff=30 ms, and a 64×64 matrix, acquiring 42 contiguous axial slices through the entire brain every 3 sec. Preprocessing and statistical analyses were performed using SPM5 (Wellcome Trust Centre for Neuroimaging, London, UK). Images were registered to MNI template space and smoothed with an 8-mm FWHM Gaussian kernel. We used a canonical hemodynamic response function and calculated parameter estimates with a general linear model. Estimates were entered into second-level random effects analyses. We specifically defined auditory association cortex by contrasting the auditory localizer task with the visual baseline task, using a cluster-defining threshold of p<.001 uncorrected voxelwise and correcting across the whole brain using cluster extent at p<.05 (FWE), with the smallest cluster being 5257 voxels. We identified a region of interest specific for sound concepts by contrasting Sound words with Sight words within the functionally-defined region of auditory association cortex, using a voxelwise threshold of p<.05 uncorrected and a peak-voxel threshold of p<.005. This resulted in a single cluster showing preferential activity for Sound words.
Figure 1A shows that behavioral performance by lvPPA participants on the word task differed for the Sound word category relative to other word categories and relative to control performance. Performance across word categories within each group was examined using Friedman tests, which revealed a significant effect in lvPPA (χ2(2)=10.17, p=.006) but no effect in controls (χ2(2)=.05, p=.98). We compared performance across categories within lvPPA using Wilcoxon signed ranks tests and found significantly worse performance on Sound words than Sight words (z=−2.04, p=.06) and Manipulation words (z=−2.38, p<.05). Performance on Sight words did not differ from that on Manipulation words (z=−1.29, p>.2). Between-group differences were examined with Mann-Whitney tests. LvPPA were significantly worse than controls on Sound words (U=37.0, p<.005) but did not differ from controls on Sight words (U=71.0, p>.1) or Manipulation words (U=75.5, p>.1). Reaction time analyses demonstrate that the accuracy results in the lvPPA participants do not reflect a speed-accuracy tradeoff: reaction times for Sound words (M=1537 ms, SD=471 ms) were longer than for Sight words (M=1440 ms, SD=515 ms; t(8)=3.46, p<.05) and did not differ from those for Manipulation words (M=1581 ms, SD=570 ms; t(8)=.90, p=.40). Reaction times did not differ across word categories within controls [F(2,36)=2.64, p=.09; Sound words M=1102 ms, SD=166 ms; Sight words M=1064 ms, SD =161 ms; Manipulation words M=1095 ms, SD=159 ms]. Altogether, these findings are consistent with selective difficulty on Sound words relative to other modalities in lvPPA.
We found significant cortical atrophy in lvPPA localized to the posterior temporal and inferior parietal cortices of the left hemisphere, as well as some involvement of inferior frontal cortex, with almost no cortical atrophy in the right hemisphere (Figure 1B). To assess whether this atrophy overlapped with auditory association cortex, we employed a BOLD fMRI functional localizer of auditory association cortex from a separate group of healthy participants. Comparison with the functional localizer demonstrated that cortical atrophy in lvPPA encompasses auditory association cortex in posterior temporal regions (Figure 1B).
We compared the atrophy results with activation from the BOLD fMRI version of the lexical decision task in healthy participants. This revealed that cortical atrophy in lvPPA overlaps with a region in auditory association cortex that is activated by the Sound word condition in the fMRI study (Figure 2, panels A and B), suggesting that impaired performance on Sound words in lvPPA may be related in part to auditory association cortex atrophy. To directly test this hypothesis, we examined the relationship between behavioral performance in lvPPA and cortical atrophy in the peak voxel from the fMRI cluster activated by healthy controls during the Sound word condition. We quantified difficulty specifically with Sound words in lvPPA by subtracting the Sound word accuracy from the average of the other conditions. A Spearman correlation revealed a significant relationship between Sound word performance and gray matter density in this region of superior temporal cortex (r=−.63, p<.05; Figure 2C). There was also a marginally significant correlation of gray matter density with raw Sound word accuracy scores (r=.54, p=.066) but not with performance on the other word categories (p>.1). We verified that the same correlation results were obtained when extracting data from a 10 mm sphere around the peak voxel. A whole-brain regression analysis revealed no other regions that were associated with task performance.
The prevailing theory of semantic memory holds that conceptual memories are multimodal and partly represented in the sensory and motor systems through which they are acquired (Allport, 1985; Martin, 2007; Binder et al., 2009; Kiefer and Pulvermüller, 2011). Many functional activation findings support this theory, yet converging anatomical findings relating functional activation in healthy adults to cortical atrophy in individuals with impaired processing of sensory-motor concepts has been lacking. Here we tested this theory in individuals with lvPPA, a neurodegenerative condition associated with atrophy in superior temporal and inferior parietal cortices (Grossman, 2010), including canonical auditory association cortex (Maeder et al., 2001; Lewis et al., 2004). We demonstrated for the first time that individuals with lvPPA have relative difficulty for words with strong sound associations. Participants with lvPPA were differentially impaired at recognizing Sound words relative to the performance of healthy controls and relative to their own performance on other word categories. Moreover, we provide the first evidence identifying difficulty with auditory-weighed words in a group with focal disease in auditory association cortex. These findings provide crucial evidence supporting the hypothesis that conceptual memories are represented in part in sensory-motor association cortices.
Structural MRI analyses in lvPPA revealed reduced gray matter density centered in posterior temporal and inferior parietal cortices. This included regions of auditory association cortex defined on the basis of an fMRI functional localizer administered to healthy adults. Furthermore, lvPPA participants had reduced gray matter density in a region of posterior temporal cortex that was activated by healthy participants for Sound words during a separate fMRI experiment. We directly examined this region and found that reduced gray matter density in lvPPA correlated with selective difficulty on Sound words. These findings demonstrate a differential impairment in modality-specific conceptual knowledge in lvPPA that is directly related to reduced gray matter density in modality-specific association cortex.
Though several studies have examined the neural representation of words with strong visual or motor associations (Hauk et al., 2004; Bonner et al., 2009; Desai et al., 2010), we know of only one other study examining the neural representation of words with strong sound associations (Kiefer et al., 2008). Using fMRI,Kiefer et al. (2008) demonstrated that auditory association regions in superior and middle temporal gyri are activated when participants perform a word recognition task on words with strong sound associations. This is consistent with our findings relating performance on Sound words to auditory association cortex in the superior temporal lobe.
Previous investigations of semantic memory in individuals with stroke have emphasized dissociations across semantic categories such as animals, plants, tools, and so forth (Gainotti et al., 1995; Gainotti, 2004; Mahon and Caramazza, 2009). The differential weighting of sensory and motor features may be important for some of these categories, but this information was not the basis for defining these categories. Here we examined knowledge of categories that were primarily defined by the weighting of their modality-specific features, matching the approach taken in recent functional neuroimaging investigations (Pulvermüller, 2005; Martin, 2007; Binder et al., 2009) and directly examining the role of modality-specific features in conceptual memory.
Studies of stroke patients pointing to a role for sensory and motor regions in conceptual memory have largely examined controlled semantic retrieval processes, with tasks such as picture naming and category-sorting (Gainotti, 2004) that differ from the simple lexical decision task we used to examine single word meaning. And though previous work has shown that apraxic patients may be impaired at recognizing the sounds of actions (Pazzaglia et al., 2008), this work does not directly address the role of modality-specific representations in word meaning. Furthermore, the neuroimaging technique employed in these stroke studies uses a binary classification approach to label voxels as either lesioned or not lesioned. This differs from our gray matter density analysis, which can relate gradations of cortical tissue loss directly to behavior. Our study is the first to directly examine the sound features of concepts in a patient group with neurodegenerative disease affecting auditory association cortex, and it provides an informative and complementary approach to previous neuropsychological investigations of conceptual memory.
Our correlation analysis related atrophy to a behavioral measure specific for the impairment on Sound words relative to other words. Likewise, our auditory functional localizer was analyzed relative to a visual baseline. This minimized the confounds of associating Sound words with aspects of conceptual memory that may also contribute to Sight and Manipulation words, or that may underlie task performance, such as decision-making or lexical retrieval. This is important because some theories incorporate additional heteromodal components in semantic memory that are not specific to any sensory or motor modality (Koenig and Grossman, 2007; Patterson et al., 2007; Binder et al., 2009; Bonner and Grossman, In Press). From this perspective, severe semantic impairments may accompany damage to heteromodal regions, but damage to a single modality may produce a mild overall semantic impairment that is differentially worse for concepts with strong associations in that modality. This is consistent with the pattern of impairment we observed in the lvPPA participants.
This impairment was observed on a simple word recognition task. Although this task did not explicitly require semantic retrieval, we assume that conceptual representations are automatically activated during word recognition and, thus, likely play a role in the successful performance of this task. This is in line with characterizations of word recognition tasks by many other investigators (Binder et al. 2009). Indeed, it was important to use a simple measure of conceptual knowledge so that we could examine the role of modality-specific representations without requiring participants to engage in mental imagery. This minimized potential confounds related to post-conceptual processing (Machery, 2007; Mahon and Caramazza, 2008) and task-specific effects of controlled semantic retrieval (Peelle et al., 2009).
Although we have contrasted a multimodal account of conceptual representation with an abstract symbolic processing account, it is worth pointing out that even multimodal representations have abstract properties. Abstract symbolic accounts (Fodor, 1975) argue that semantic representations have the same symbolic format regardless of their modality-specific associations and, thus, do not rely on the sensory-motor system. Alternatively, the sensory-motor account (Martin, 2007) argues that concepts are represented as distributed networks of feature representations in sensory and motor association cortices. But these multimodal representations may also have important abstract properties. For one, the regions involved in perceiving a feature may not be identical to those involved in representing that feature in conceptual memory; rather, these regions may be anatomically adjacent (Chatterjee, 2010). Furthermore, multimodal concepts may use information from a range of specific instances to form a prototypical representation. For example, not all apples look the same, but our concept for “apple” can be applied to most of them. Hence, multimodal representations must be compatible with abstract cognitive processing, and we cannot rule out the possibility that regions representing modality-specific semantic features are anatomically adjacent to regions involved in perceiving those features.
A related issue is that the region of superior temporal cortex associated with Sound word processing in both the behavioral and fMRI studies, may not be strictly unimodal. In fact, this area of cortex has also been associated with crossmodal integration of auditory and visual information (Beauchamp et al., 2004). Our findings may be consistent with such an account, since a region that processes crossmodal auditory and visual information would likely be important for the representation of auditory-weighted concepts, which have both auditory and visual features.
Previous investigations of lvPPA have focused on difficulty with repetition of sentences and phrases (Gorno-Tempini et al., 2011), likely stemming from an impaired phonological loop. This problem may contribute to difficulty understanding sentence-length verbal information, but is unlikely to explain selective difficulty for a specific word category on a task involving single word processing. Likewise, difficulty with lexical retrieval is unlikely to explain the pattern of modality-specific performance observed on our word recognition task, given that we would not expect a pure impairment of lexical retrieval to interact with the semantic categories in our study. While previous assessments of conceptual memory in lvPPA have generally shown preserved performance relative to assessments of verbal working memory (Gorno-Tempini et al., 2011), our study is the first to evaluate modality-specific effects in conceptual memory in lvPPA. Our more fine-grained assessment of conceptual memory in lvPPA reveals disproportionate difficulty for concepts with strong sound associations, consistent with the pattern of cortical atrophy in lvPPA affecting canonical auditory association cortex.
In conclusion, we provide evidence from individuals with localized cortical atrophy that conceptual representations rely in part on modality-specific association cortex. We demonstrate that individuals with lvPPA have selective difficulty processing words with strong sound associations, and that this impairment is directly related to reduced gray matter density in a region of auditory association cortex that healthy adults activate when processing the same Sound words in a separate fMRI experiment. These findings suggest that concepts rely on feature representations in sensory association cortices.
This research was supported by the National Institutes of Health AG017586, AG015116, AG032953, NS044266, NS053488, NS054575, and the Wyncote Foundation. We thank Jonathan Peelle for helpful comments on this manuscript, the radiographers at the Hospital of the University of Pennsylvania for their assistance with data collection, and our volunteers for their participation.
Conflict of interest: nothing to declare