PMCCPMCCPMCC

Search tips
Search criteria 

Advanced

 
Logo of nihpaAbout Author manuscriptsSubmit a manuscriptNIH Public Access; Author Manuscript; Accepted for publication in peer reviewed journal;
 
Neuron. Author manuscript; available in PMC Aug 19, 2013.
Published in final edited form as:
PMCID: PMC3746792
NIHMSID: NIHMS501383
Spatiotemporal dynamics of modality-specific and supramodal word processing
Ksenija Marinkovic,* Rupali P. Dhond,* Anders M. Dale,* Maureen Glessner,* Valerie Carr,* and Eric Halgren*§
*Athinoula A. Martinos Center for Biomedical Imaging, Charlestown, MA 02129
Radiology, University of Utah, Salt Lake City, UT 84108
§INSERM E9926, Marseilles, France
The ability of written and spoken words to access the same semantic meaning provides a test case for the multimodal convergence of information from sensory to associative areas. Using anatomically-constrained magnetoencephalography (aMEG), the present study investigated the stages of word comprehension in real time in the auditory and visual modalities, as subjects participated in a semantic judgment task. Activity spread from the primary sensory areas along the respective ventral processing streams and converged in anterior temporal and inferior prefrontal regions, primarily on the left at around 400ms. Comparison of response patterns during repetition priming between the two modalities suggest that they are initiated by modality-specific memory systems, but that they are eventually elaborated mainly in supramodal areas.
Human interaction and thought crucially depend on words, which can take either auditory or visual form. During initial word processing stages, the acoustic signal or letter string are analyzed in their respective sensory modalities, followed by mapping of letter symbols or phonemes onto a word lexicon and, finally, semantic access and integration. The fact that the same semantic knowledge can be accessed by symbols in two different modalities allows exploration of the brain substrate that underlies retrieval of supramodal meaning.
Evidence from functional brain imaging suggests that language comprehension is subserved by modality-specific distributed networks (Cabeza and Nyberg, 2000). Spoken language as well as complex nonspeech stimuli evoke bilateral activation in the superior temporal cortices (Binder et al., 1997; Wise et al., 1991; Zatorre et al., 1992). Leftward speech-related asymmetry has been observed in temporal and left inferior prefrontal cortex (LIPC) (Friederici et al., 2000; Frost et al., 1999; Petersen et al., 1988; Price et al., 1996; Scott et al., 2000). Neuroimaging studies of reading, however, suggest a more clearly left-lateralized activity in ventral temporal and inferior prefrontal regions (Fiez and Petersen, 1998; Gabrieli et al., 1998).
The importance of the left anterior temporal cortex for language processing and semantic memory has been documented with PET (Devlin et al., 2002; Perani et al., 1999), in studies of patients with semantic dementia (Hodges et al., 1992) and with intracranial recordings (Halgren et al., 1994b; Nobre and McCarthy, 1995; Smith et al., 1986). Imaging studies suggest that the LIPC is involved in semantic language tasks and is sensitive to mnemonic manipulations such as priming (Bokde et al., 2001; Gabrieli et al., 1998; Roskies et al., 2001; Wagner et al., 2000). Indeed, the importance of both areas – left temporal and left inferior prefrontal cortices for supramodal semantic processing, has been accentuated by their contributions to repetition priming effects across both auditory and visual modalities (Buckner et al., 2000; Chee et al., 1999). However, the timing and the sequence of their involvement are not clear.
Benefiting from the excellent temporal resolution provided by event-related potentials (ERPs), language studies have described a negativity peaking at ~400 ms (N400) which is evoked by potentially meaningful material including spoken, written or signed words and pictures (Kutas and Federmeier, 2000). The N400 is modulated by priming and is commonly viewed as reflecting attempts to access and integrate a semantic representation into a current context (Brown and Hagoort, 1993; Halgren, 1990; Holcomb, 1993; Rugg and Doyle, 1994). The overlapping scalp distribution of the N400 evoked by auditory and visual stimuli suggests that it may reflect access to a supramodal semantic network, as a final common pathway originating in respective sensory-specific processing areas (Domalski et al., 1991; Gomes et al., 1997; Hagoort and Brown, 2000; Rugg and Nieto-Vegas, 1999). MEG studies of N400 relying on equivalent current dipole modeling method estimed the source of N400m (magnetic equivalent of N400) in the vicinity of the superior temporal sulcus - STS (Helenius et al., 1998; Helenius et al., 2002; Makela et al., 2001; Sekiguchi et al., 2001; Simos et al., 1997). Intracranial recordings suggest a widespread distribution of local N400 generators particularly in the anteroventral temporal lobe (Halgren et al., 1994b; McCarthy et al., 1995; Nobre and McCarthy, 1995; Smith et al., 1986), but also inferolateral prefrontal cortex, occipito-temporal cortex, middle and superior temporal gyri, and supramarginal gyrus (Halgren et al., 1994a; Halgren et al., 1994b).
Taken together, the available neuroimaging evidence suggests that the prefrontal and temporal regions contribute to semantic and mnemonic processing of words presented in both spoken and written form (Buckner et al., 2000; Petersen et al., 1988), and may represent the neural basis of supramodal processing. Conversely, ERPs and MEG provide complementary evidence suggesting that access to the central semantic store is reflected in the N400 component (Kutas and Federmeier, 2000). The present study attempts to integrate the two lines of evidence by comparing brain activity evoked by a semantic judgment task presented in auditory and visual modalities. Our underlying hypothesis is that the left inferior prefrontal and temporal areas are primarily engaged while accessing supramodal semantic knowledge at approximately 400 ms post-stimulus, subsequent to modality-specific and transitional processing stages.
The aim of this study was to examine the spatio-temporal characteristics (“where and when”) of word comprehension in real time in the auditory and visual modalities. High-resolution structural MRI was combined with temporally-precise whole-head high-density MEG and a distributed source modeling approach to estimate the anatomical distribution and hierarchical interdependence of the underlying neural networks (Dale et al., 2000; Dale and Sereno, 1993; Dhond et al., 2001). The task required subjects to estimate the physical size of an animal or object denoted by a word. By using repetition priming, this study also examined the brain areas showing memory-related differential responses to primed items and investigated their time-courses. Finally, the question of the nature of semantic processing and its dependence on the presentation modality was considered by comparing patterns of brain activity obtained in the same subjects across both auditory and visual forms of the same task.
Behavioral performance
Adequate data for both versions of the task were obtained in 9 subjects. Response times and performance accuracy on the task were monitored continuously. Good agreement between the subjects’ judgment and the pre-specified size criteria was suggested by high performance levels. Performance indices (d’, percent correct scores and reaction times) were analyzed with a repeated-measures ANOVA with factors of task (visual, auditory) and repetition (new and repeated stimuli). Based on d’, a bias-free measure of stimulus discriminability, performance was significantly better on repeated than new items, F (1,8) = 14.4, p < 0.005, d’ equals 2.64 and 3.96 for new and repeated items respectively. In contrast, d’ values did not differ between the two task modalities, F (1,8) = 0.1, p > 0.5). Examination of the percent correct scores revealed a slightly better performance on the visual form of the task (92.3 vs. 87.2 correct hits), F(1,8) = 5.38 p < 0.049 (uncorrected for multiple comparisons). However, there were also more false alarms in the visual version of the task (means = 10.4 vs. 3.4), F(1,8) = 6.02 p < 0.04 (uncorrected).
Faster reaction times were observed in the visual, as compared to the auditory task, F(1,8) = 67.8 p < 0.0001, and to the repeated, as compared to new items across both tasks, F(1,8) = 39.9, p < 0.0005. Reaction times and standard errors of the mean were as follows: 960.0 (40.5) and 759.8 (26.9) to novel and repeated written words, 1063.7 (27.2) and 981.8 (32.9) to novel and repeated spoken words respectively.
Estimated activity patterns
Overall pattern
The cortical surface of each individual served as the solution space for current dipoles generating the MEG signal. Estimated dipole strengths across locations were transformed using noise estimates into dynamic statistical parametric maps - dSPMs (Dale et al., 2000). These maps indicate the statistical significance of estimated activity at each latency and cortical location. The maps were averaged across all subjects using cortical surface alignment of corresponding anatomical features (Fischl et al., 1999b). Cortical activity was displayed on “inflated” views of an averaged cortical surface, permitting a view of the activity estimated to lie within the sulci. Inspection of the overall activity patterns averaged across all nine subjects suggested that the earliest activity to novel words was confined to the visual or auditory sensory areas during tasks in the two respective modalities, followed by association brain areas in the anterior temporal and prefrontal areas. Estimated activity to novel words for selected latencies is presented in Figure 1. The overall activity pattern is described here for each modality seriatim.
Figure 1
Figure 1
Group average (N=9) statistical parametric maps of estimated responses at selected latencies to novel words. The earliest significant activity during auditory and visual tasks is estimated to lie in their respective sensory areas. Activity spreads anteriorly (more ...)
In the auditory task, the earliest reliable activity was seen bilaterally at 55 ms in the superior temporal area, as a magnetic analogue of the event-related P50 component measured on the scalp (Figure 1). Activity estimated to lie in the subcentral gyrus most likely reflects blurring of the superior temporal activity; although these areas appear to be distinct on the inflated surface presentation, they are immediately adjoining in the actual brain volume. This initial activity was followed by a second peak at ~100ms (N100m) in the perisylvian/superior temporal plane region. At ~200 ms activity spreads forward and by 250 ms activity was estimated to lie in the anterior regions of the temporal lobe (AT), perisylvian area and posterior inferior prefrontal regions bilaterally. In addition to sustained contribution of the bilateral perisylvian region, the activity was left-dominant after 300 ms, relying primarily on the AT, anterior LIPC and bilateral ventromedial prefrontal (VMPF) areas. Timecourses of the estimated activity were plotted for selected locations. These waveforms (Figure 4) suggest sustained contributions from the left-dominant AT initially peaking at ~410 ms, followed by the aLIPC (at ~490ms) until ~700 ms.
Figure 4
Figure 4
Group average timecourses of the estimated noise-normalized dipole strengths to novel and repeated words in selected cortical locations for auditory and visual task versions. Statistically significant early repeat>novel effect to visual words (more ...)
In the visual task, activity started bilaterally in the occipital area and spread along the ventral visual pathway (Figure 1). The left-dominant activity followed the posterior-anterior axis peaking in the ventral occipitotemporal area at ~170 ms, at ~230 ms in the STS and inferolateral temporal area and at ~350 ms in the AT area, encompassing LIPC and orbitofrontal cortex bilaterally by 400 ms. This left temporo-frontal activity peaking at ~400ms may be the magnetic-equivalent of the N400, or N400m. Activity seen in the posterior occipitotemporal region at ~420 ms appears to be partially due to the visual offset response and it is superimposed over the N400m effects.
The overall activity patterns were analogous across modalities. In both tasks, activity started in modality-specific areas and progressed anteriorly via the respective ventral streams, engaging the left STS and left AT areas after ~250 ms. The anterior LIPC was recruited in both tasks particularly during the N400m period, although its contribution started earlier and was sustained for longer in the auditory version. Both tasks elicited similar patterns of activity during the N400m, including the left AT and LIPC as well as bilateral VMPF areas. Whereas activity was strongly left-lateralized during visual task presentation, the auditory version resulted in bilateral perisylvian activity.
Repetition effects
Waveforms evoked by repeated words were subtracted from the waveforms obtained on novel trials for each individual subject. Dynamic SPMs of these differences were averaged across the nine participants (Dale et al., 2000; Dhond et al., 2001). Selected snapshots illustrate the early (~250 ms) and late repetition effects (after 300 ms) in Figures 2 and and33 respectively. Timecourses of the estimated noise-normalized dipole strengths (Figure 4) offer an alternative insight into the temporal dynamics of the activity estimated for each cortical location. Using within-subject statistical comparisons (Woodward et al., 1990), significance of the novel – repeat differences for the “early” (225–250 ms) and “late” (300–500 ms) estimated activity was additionally tested for several relevant regions of interest (ROI). Because they account for the variation of the activity within the sample, ROI-ANOVAs as performed here allow making inferences that can be generalized to the population and are equivalent to random-effects analysis in fMRI. Additional stringency of this test derives from the requirement that the differences in estimated activity coincide in space AND time across subjects: in space because exactly the same ROIs were used across all subjects not allowing any individual variations in localization estimates; in time since the same time window was tested for all subjects.
Figure 2
Figure 2
Group average early repetition effects in auditory and visual modalities. Novel – repeat subtractions are shown in the first row, whereas rows 2 and 3 indicate estimated overall activity to novel and repeated words respectively. Novel spoken words (more ...)
Figure 3
Figure 3
Group average late repetition effects (novel - repeated subtractions) encompassing the N400m which seems to rely primarily on the left anterior temporal, left STS and anterior inferior prefrontal areas in both tasks (blue arrows) in addition to sustained (more ...)
Early, modality specific repetition effects
Repetition effects in the auditory modality were due to stronger responses to novel words (Figure 2, left panel). Within-subject statistical comparisons (ROI-ANOVA) of the estimated noise-normalized dipole strengths within 225–250 ms latency window suggested significant differences in the superior temporal plane, F(1,8) = 28.2, p < 0.001, STS, F(1,8) = 17.1, p < 0.01, and the temporopolar area, F(1,8) = 7.9, p < 0.05.
In contrast, in the visual modality it was the repeated words that evoked a stronger and earlier activity than novel words, particularly in the left inferior temporal and LIPC (Figure 2, right panel). ROI-ANOVA for the 225–250 ms latency window confirmed the repeat > novel effect only in the left inferolateral temporal, F(1,8) = 6.2, p < 0.05 and aLIPC, F(1,8) = 8.0, p < 0.05) regions. Furthermore, responses evoked by repeated words had shorter peak latencies than novel words in both the anterior temporal, F(1,8) = 10.7, p < 0.01 (means = 229 and 239 ms for repeated and novel words respectively) and LIPC area, F(1,8) = 7.1, p < 0.05 (means = 237 and 248 ms). These effects can be observed in the estimated timecourses (Figure 4). The repeat > novel effects were apparently limited to these regions in the left hemisphere because the differences were not significant in the neighboring regions, nor were they significant in the prefrontal or temporal areas on the right.
Late, supramodal repetition effects
Activity subsequent to these early effects was significantly stronger to novel words in both modalities and the N400m was reflected in the prominent engagement of the left temporal, followed by the left prefrontal region. Dynamic SPMs of the difference between the novel and repeated words presented in Figure 3 suggest a considerable overlap in the estimated substrates of the N400m in the two modalities. In addition, ROI-ANOVAs in the 300–500 ms latency window confirmed significant repetition effects in the left anterior temporal region for both modalities including the temporopolar area, F(1,8) = 20.4, p < 0.01 for the auditory and F(1,8) = 11.3, p < 0.01 for the visual version, and the anterior STS regions, F(1,8) = 11.9, p < 0.01 (auditory) and F(1,8) = 12.8, p < 0.01 (visual). Similarly, significant repetition effects were seen in the anterior LIPC for the auditory, F(1,8) = 21.7, p < 0.01, and visual tasks, F(1,8) = 10.6, p < 0.01, as well as in the left prefrontal area superior to LIPC in both auditory, F(1,8) = 5.6, p < 0.05, and visual task versions, F(1,8) = 14.4, p < 0.01. Significant interactions between the factors of modality and repetition in the right hemisphere in the inferior prefrontal, F(1,8) = 9.0, p < 0.05, temporopolar F(1,8) = 14.9, p < 0.01 and anterior STS area, F(1,8) = 9.6, p < 0.05), suggested a bilateral nature of processing in the auditory and left-lateralized effects in the visual modality at this latency. Indeed, the repetition effects in the right hemisphere were significant for the auditory task in the inferior prefrontal, F(1,8) = 7.0, p < 0.05, temporopolar, F(1,8) = 17.3, p < 0.01, and anterior STS area, F(1,8) = 20.7, p < 0.005. In contrast, no difference was seen for the visual task in the right prefrontal, F(1,8) = 0.5, p < 0.5, right temporopolar, F(1,8) = 0.2, p > 0.5, or anterior STS regions, F(1,8) = 0.6, p < 0.5).
For the auditory task, the average peak latency of the activity estimated to lie in the anterior temporal region was 460 ms, followed by two prominent peaks estimated in the anterior LIPC with latencies of 490 and 600 ms (Figure 4). In contrast, the visual task evoked a bifurcated peak with latencies of 350ms and 410 ms in the AT region, followed by activity in the anterior LIPC peaking at 450 ms.
In sum, whereas repetition effects differed between the auditory and visual tasks in the earlier processing stages (before 300 ms), their substrates overlapped increasingly at later latencies, particularly during the N400m time window. The main overlapping activity was observed in the left temporal region, with a very similar pattern appearing already at ~350 ms, as well as the anterior LIPC and ventromedial prefrontal regions at later latencies, after ~400 ms. The temporo-prefrontal contributions were strongly left-lateralized in the visual task version only.
The view that language relies on distributed but interactive brain areas has been supported by a large number of studies (for reviews see Cabeza and Nyberg, 2000; Fiez and Petersen, 1998; Geschwind, 1965; Ingvar, 1983; Mesulam, 1998; Raichle, 1996; Warburton et al., 1996). This view assumes modality-specific lexical components accessing a central semantic system. In the current study, the auditory task utilized the phonological input route for understanding spoken words, in contrast to the orthographic route subserving reading. Neural substrates of these modality-specific routes were quite distinct during initial processing, but overlapping areas were subsequently activated during stages of semantic and contextual integration. This is particularly evident for the N400m repetition effect. The present evidence favors the claim that modulations in N400 amplitude reflect a supramodal semantic process with primary overlapping contributions from left inferior prefrontal, left temporal and medial prefrontal areas bilaterally. Additional contributions were estimated to originate in the right prefrontal and temporal areas during the auditory task only.
Overall activity in the auditory and visual word processing streams
The spatiotemporal characteristics of the responses to spoken words observed in our study concur with other evidence. The earliest response was estimated to lie in the superior temporal area bilaterally at 55 ms after stimulus onset (Figure 1) and then spread to the auditory “belt” in the perisylvian region as reflected in N100m. Subsequently, activity spread along the auditory “ventral” stream into anterior and lateral areas of the STG, concurring with the analogy to the visual pattern recognition or “what” stream (Binder et al., 1997; Rauschecker and Tian, 2000). Ensuing engagement of the ventral prefrontal areas concurs with other evidence obtained during auditory pattern recognition in human (Rauschecker, 1998) and primate studies (Romanski and Goldman-Rakic, 2002).
In contrast, processing of written words started in the primary visual cortex (Figure 1) and proceeded anteriorly in the ventral visual stream, encompassing middle and superior temporal (Wernicke’s) areas, and finally the LIPC region. The activity pattern observed here corresponds closely to activations reported elsewhere (Dale et al., 2000; Dhond et al., 2001; Fiez and Petersen, 1998; Halgren et al., 2002) and agrees with classical models of language processing (Benson, 1979; Geschwind, 1965).
Whereas processing of written words was strongly left-lateralized in all processing stages subsequent to the visual cortex, hearing spoken words resulted in bilateral perisylvian activity, agreeing with other evidence (Belin et al., 2000; Binder et al., 1997; Cabeza and Nyberg, 2000; Helenius et al., 2002; Hickok and Poeppel, 2000; Wise et al., 1991). After ~300 ms, the response was left-biased in the auditory task version, especially in the anterior LIPC and anteroventral temporal regions, possibly indicating the access of semantic networks predominantly in the left hemisphere (Scott et al., 2000).
Early repetition effects
Faster and more accurate responses to repeated stimuli were observed in both task versions, replicating a well-established behavioral repetition priming effect (Schacter and Buckner, 1998). The earliest repetition priming effects were estimated primarily to the modality-specific areas (Figure 2). During auditory word presentation, novel items evoked stronger responses in the perisylvian areas after ~230 ms, suggesting that the auditory system could clearly distinguish the first phoneme of the highly familiar repeated words (total of ten) from the novel words that were presented only once. This may reflect perceptual priming based on mnemonic functions of the auditory association cortex (Näätänen et al., 2001).
In contrast, repeated written words showed a brief tendency towards increased activity of the ventral visual stream starting at ~230 ms, including left occipito-temporal and left posterolateral temporal regions (Figure 2). A similar effect in a stem-completion task was reported in the comparable area (left ventrotemporal area) and at a similar latency (200–245 ms) using the same aMEG technique as the one reported here (Dhond et al., 2001). In an ERP study of word repetition, Nagy et al. (1989) reported a transient repeated > novel effect with an onset at ~200 ms. Using fMRI, Poldrack et al. (1998) have observed increased activation in lateral inferior temporal regions to well practiced stimuli under the conditions of mirror-reading. This evidence suggests that priming may result in a more rapid and preferential access to certain structures along the ventral stream, presumably subserving a transitional stage between perceptual and conceptual function and potentially embodying direct lexical access (Humphreys and Evett, 1985). The latency and anatomical distribution of this effect is consistent with a suggestion placing the word-form processing stage after the early visual processing and before the phonological and semantic stages (Tulving and Schacter, 1990; Warrington and Shallice, 1980). The apparent activation of the more anterior areas, including AT and LIPC, may represent a fast processing of the highly familiar word-form, providing an “early pass” through the relevant parts of the network, the purpose of which may be to prime them for the impending input. The temporal resolution of the aMEG method made it possible to distinguish an early and brief increase to repeated items from a robust subsequent novel > repeat effect. Thus, processing of written words may benefit from prior exposure through multiple stages, including both perceptual (possibly form-based) and conceptual (semantically mediated) priming (Schacter and Buckner, 1998), resulting in a faster and more accurate stimulus identification.
Late priming effects – N400m
The strongest priming effects were seen during the N400m time period in both auditory and visual tasks (Figure 3). Responses were larger to novel stimuli, and encompassed primarily the left temporal, LIPC and medial prefrontal areas in both modalities. Additional responses in the right temporal and prefrontal areas were observed in the auditory task version.
One of the goals of this study was to compare the N400m priming effect to that observed with hemodynamic methods. Indeed, overall activation patterns evoked by semantic tasks in fMRI studies are similar to the patterns observed in the present study. They encompass the left inferior prefrontal regions, and left temporal (word reading) or bilateral temporal (word hearing) areas, but commonly exclude the anterior temporal region (Binder et al., 1997; Buckner et al., 2000; Fiez and Petersen, 1998). In contrast, in the present study strong contributions to the N400m in both modalities were estimated to originate in the anterior temporal lobes. This observation is in accord with PET studies that reliably detect activation in the anterior temporal area during semantic tasks (Devlin et al., 2002; Perani et al., 1999; Tzourio et al., 1998). Based on direct comparisons of the activation patterns obtained with PET and fMRI with semantic tasks, it has been suggested that the absence of the anterior temporal activation in fMRI is most likely due to susceptibility artifacts in that region (Devlin et al., 2000; Veltman et al., 2000). Additional support for the role of the anterior temporal area in semantic and mnemonic functions comes from intracranial recordings. They indicate large locally-generated responses within the ~300–700 ms latency that are similar to faces and words, and maximal in anteroventral temporal areas (Halgren et al., 1994b; McCarthy et al., 1995). Converging evidence is provided by lesion studies. Atrophy in the left anterolateral temporal cortex seen in patients with semantic dementia correlates with impairments in semantic processing (Hodges et al., 1992).
Engagement of the aLIPC occurred subsequent to the temporal area in both auditory and visual versions of the task, though its contributions seemed stronger and more sustained in the auditory version. It was highly sensitive to item repetition as the activity was significantly reduced to repeated words. Even though the present results are consistent with the view that the aLIPC subserves guided semantic access (Kapur et al., 1994; Poldrack et al., 1999; Tagamets et al., 2000; Wagner et al., 2001), it is not possible to determine whether this activity also represents a phonological analysis (Hagoort et al., 1999), or even a more broadly conceptualized mediation of selection among competing alternatives (Thompson-Schill et al., 1997). Intracranial studies using cognitive tasks have reported locally generated N400 in the ventrolateral prefrontal cortex, (Halgren et al., 1994a), which is consistent with our estimated localizations.
The initial semantic access takes place in overlapping regions and at comparable latencies in both modalities. The observed lag in peak latency of ~50 ms and a protracted activity profile to spoken words estimated to the temporo-frontal areas suggest that the semantic integration of a spoken word may commence based on incomplete auditory input and proceeds by sustained integration of the unfolding acoustic stream. This is consistent with previous models of spoken word recognition (Hagoort and Brown, 2000; Marslen-Wilson, 1987) whose main idea is that the comprehension of a spoken word may emerge from a sustained interplay between the continual acoustic input and the “top-down” facilitation by the higher association areas (Van Petten et al., 1999). In this study, reaction times to spoken novel words were delayed for ~100 ms on average. In addition to a delay in the onset of semantic integration, other processes such as applying the task criterion to the results of semantic access/contextual integration, mapping and releasing motor response, may have contributed to that difference.
Results of other MEG studies of N400 (Helenius et al., 1998; Helenius et al., 2002; Makela et al., 2001; Sekiguchi et al., 2001; Simos et al., 1997) generally estimate the N400m source to the superior temporal sulcus, overlapping partially with our estimates. The differences could be explained by the adopted source modeling technique. A recent study by Halgren et al., (2002) analyzed the same sentence-terminal word data using two methods: fitting of an equivalent current dipole (ECD) that best matches the observed field pattern, as well as a distributed source modeling technique employed in the present study. The first method estimated the N400m ECD in the vicinity of the left superior temporal sulcus, replicating results of other similar studies (Helenius et al., 1998; Simos et al., 1997). However, the distributed solution additionally included anteroventral temporal, orbitofrontal and posteroventral prefrontal cortices on the left, consistent with the present findings and evidence from other neuroimaging techniques.
In the present study the N400m, as an index of supramodal contextual integration (Kutas and Federmeier. 2000), was estimated to originate primarily in the left inferior prefrontal and anterior temporal areas in both modalities. This apparently supramodal network that is engaged during contextual integration corresponds to neuroimaging studies of auditory vs. visual priming (Buckner et al., 2000; Chee et al., 1999). Its sensitivity to repetition priming implicates the N400m in the close mutually-reinforcing interaction between semantic and mnemonic processing. Our results are consistent with the “unitary semantic hypothesis” (Caramazza et al., 1990) stipulating that supramodal semantic stores can be accessed from any modality, after appropriate lexical access through the phonological or orthographic lexicon. Thus, behavioral priming is a result of “savings” in the multiple stages of processing starting relatively early during a modality-specific phase and continuing through the stages of accessing supramodal semantic stores. Even though our results suggest that the observed activity is largely overlapping between the two modalities, they are consistent with the view that the sensory input continues to contribute to the comprehension process.
Subjects
Nine healthy males participated in MEG/EEG recordings during auditory and visual versions of the task on two separate occasions, in addition to the structural MRI scan. Subjects were all right-handed native English speakers between 22 and 30 years of age (mean = 24.7, SE = 0.77), without hearing or other neurological impairments. No structural brain abnormalities were apparent on their MRI scans. Signed statements of consent were obtained from all participants. They were monetarily reimbursed for their participation.
Task
During a “size” judgment task, participants were presented with words denoting objects, animals, or body parts and were instructed to respond to those larger than one foot (e.g. tiger, shirt), and to refrain from responding to those smaller (e.g. cricket, medal). During practice, ten words were presented repeatedly and subsequently became “repeats”, presented on half of the trials and randomly mixed among “novel”, nonrepeated words that were presented only once. Most likely, this task was carried out by accessing propositional information rather than visual imagery. It presumably imposed rather low-level imagery demands because only the stimuli that were clearly larger or smaller than the reference standard were included in the lists and because all other visual characteristics were irrelevant to the performance.
Two parallel versions of the task with no word overlap were administered to the same participants in auditory and visual modality on two separate occasions, 4 months apart on average. Word lists used in the auditory and visual tasks were balanced for word frequency with means of 13.28 and 12.51 per million respectively, (Francis and Kucera, 1982). The words used in the auditory task were slightly shorter (1.4 syllables and 5.2 letters) than the words used in the visual task (2.1 syllables and 6.6 letters on average), due to presentation timing constraints. The repeated words were chosen to be representative of their respective category with respect to word frequency and length.
Auditory format
Word stimuli were recorded by a native male speaker and were equated for sound onset and duration (500 ms), as well as amplitude level by digitally editing the recorded waveforms. A total of 780 nouns were presented binaurally through plastic tubes at a comfortable level every 2.2 sec. During practice, ten words were presented 6 times and they subsequently became “repeats”, presented on half of the trials and randomly mixed among the 390 “novel”, nonrepeated words that appeared only once during the experiment. The response hand was switched midway through the experiment, in a balanced manner across subjects.
Visual format
Words were presented on a computer-driven projection screen in front of a subject and subtended a <5% visual angle. The words were presented for 300 ms in helvetica font as white letters on a black background every 2 sec. During practice, ten words were presented four times each and subsequently became “repeats”, randomly presented on half of the trials. Six out of nine subjects were presented with a total of 320 words, one subject performed the task with a total of 480 words and two subjects had a total of 640 words. The subjects responded with their left hands throughout the experiment.
MEG Recording
MEG was recorded from 204 channels (102 pairs of planar gradiometers) from the entire head with a planar dc-SQUID Neuromag Vectorview system in a magnetically and electrically shielded chamber. The signals were recorded continuously with 600 Hz sampling rate and minimal filtering (0.1 to 200 Hz). Category-based averages were constructed on-line from trials free of eyeblinks or other artifacts and were low-pass filtered at 30 Hz. Averaged waveforms from a representative subject are shown in Figure 5. The position of magnetic coils attached to the skull and main fiduciary points such as the nose, nasion and preauricular points were digitized with 3Space Isotrak II system for subsequent precise co-registration with MRI images.
Figure 5
Figure 5
MEG waveforms from one subject in auditory and visual task versions. Responses to novel (black lines) and repeated (gray lines) words are superimposed. Note the similar waveforms recorded over anterior temporal areas (arrows).
Spatio-temporal analysis: anatomically-constrained MEG
MEG signals directly reflect the magnetic fields associated with synaptic currents with a millisecond precision. However, the spatial configuration of the intracranial generators cannot be uniquely determined based on extracranial measurements without prior assumptions (Hämäläinen et al., 1993). Anatomically-constrained MEG (aMEG) relies on the anatomical localization of each individual’s cortical surface from their MRI to constrain the inverse solution (Dale and Halgren, 2001). This approach assumes that the synaptic potentials giving rise to the summated MEG lie in the cortical gray matter of each subject, obtained from a high-resolution anatomical MRI (Dale et al., 2000; Dale and Sereno, 1993).
The cortical surface reconstructed for each individual from high-resolution 3-D T1-weighted MRI structural images (1.5T Picker Eclipse) was subsampled to ~2500 dipole locations per hemisphere (Dale et al., 1999; Fischl et al., 1999a). This cortical surface served as the solution space for the estimated current generators (dipoles), constraining the MEG solution to the gray matter. Dipole orientation was unconstrained. The forward solution was calculated using a boundary element model (Oostendorp and van Oosterom, 1991). Using a linear estimation minimum norm approach (Dale and Sereno, 1993; Hämäläinen and Ilmoniemi, 1994), dipole strength power is estimated at each cortical location every 5 ms and is divided by the predicted noise power (Dale et al., 2000). This has the effect of transforming power maps into dynamic statistical parametric maps (dSPMs), as well as making the point-spread function relatively uniform across the cortical surface (Liu et al., 2002). These noise-normalized estimates of the local current dipole power for each location fit the F-distribution and can be viewed as “brain movies”, or a series of SPM frames unfolding across time. Intersubject averaging comprises morphing each subject’s reconstructed surface into an average representation after aligning their cortical sulcal-gyral patterns (Fischl et al., 1999b) and averaging individual inverse solutions (Dhond et al., 2001). In addition to the overall activity patterns presented in Figure 1, repetition priming effects are presented as group average dSPMs of differences in the waveforms evoked by the novel and repeated words conditions. Significance of the average estimated activations is displayed on the images.
An alternative insight into the temporal dynamics of the activity estimated for each cortical location is offered by plotting the estimated noise-normalized dipole strength across all time points at selected locations (Figure 4). In order to further explore the statistical significance of particular comparisons, regions of interest (ROIs) were chosen for the relevant areas on the cortical surface (Figure 4). The same set of ROIs was used for all subjects by an automatic spherical morphing procedure (Fischl et al., 1999b). Since the same noise estimation is used across conditions, the differences in significance at a given location indicate differences in activity, permitting a direct statistical comparison of the activity to novel vs. repeated words. For the same reason, however, estimated dipole strengths cannot be directly compared across different locations. Average estimated noise-normalized dipole strength values were calculated from the estimated timecourses of the cortical points comprised in each ROI for each subject across both auditory and visual tasks and for both novel and repeated conditions within the selected latency windows. These values were submitted to within-subject ANOVAs for the “early” (225–250 ms) and “late” (300–500 ms) repetition effects.
Supplementary Material
Acknowledgments
We are grateful to Thomas Witzel, Sharelle Baldwin, Bruce Fischl, Jeremy Jordin, John Klopp, Jeffrey Lewine, Arthur Liu, Dave Post, Kim Paulson and Bruce Rosen. Supported by NIH grants NS18741 (EH), AA13402 (KM), NS39581 (AMD), EB00790 (AMD), EB00307 (AMD) and the MIND Institute (DOE grant DE-FG03-99ER62764).
  • Belin P, Zatorre RJ, Lafaille P, Ahad P, Pike B. Voice-selective areas in human auditory cortex. Nature. 2000;403:309–12. [PubMed]
  • Benson DF. Aphasia, alexia, and agraphia. New York: Churchill Livingstone; 1979.
  • Binder JR, Frost JA, Hammeke TA, Cox RW, Rao SM, Prieto T. Human brain language areas identified by functional magnetic resonance imaging. J Neurosci. 1997;17:353–62. [PubMed]
  • Bokde AL, Tagamets MA, Friedman RB, Horwitz B. Functional interactions of the inferior frontal cortex during the processing of words and word-like stimuli. Neuron. 2001;30:609–17. [PubMed]
  • Brown C, Hagoort P. The processing nature of the N400: Evidence from masked priming. Journal of Cognitive Neuroscience. 1993;5:34–44. [PubMed]
  • Buckner RL, Koutstaal W, Schacter DL, Rosen BR. Functional MRI evidence for a role of frontal and inferior temporal cortex in amodal components of priming. Brain. 2000;123(Pt 3):620–40. [PubMed]
  • Cabeza R, Nyberg L. Imaging cognition II: An empirical review of 275 PET and fMRI studies. J Cogn Neurosci. 2000;12:1–47. [PubMed]
  • Caramazza A, Hillis AE, Rapp BC, Romani C. The multiple semantics hypothesis: Multiple confusions? Cognitive Neuropsychology. 1990;7:161–189.
  • Chee MW, O’Craven KM, Bergida R, Rosen BR, Savoy RL. Auditory and visual word processing studied with fMRI. Hum Brain Mapp. 1999;7:15–28. [PubMed]
  • Dale AM, Fischl B, Sereno MI. Cortical surface-based analysis. I. Segmentation and surface reconstruction. Neuroimage. 1999;9:179–94. [PubMed]
  • Dale AM, Halgren E. Spatiotemporal mapping of brain activity by integration of multiple imaging modalities. Curr Opin Neurobiol. 2001;11:202–8. [PubMed]
  • Dale AM, Liu AK, Fischl BR, Buckner RL, Belliveau JW, Lewine JD, Halgren E. Dynamic statistical parametric mapping: combining fMRI and MEG for high-resolution imaging of cortical activity. Neuron. 2000;26:55–67. [PubMed]
  • Dale AM, Sereno MI. Improved localization of cortical activity by combining EEG and MEG with MRI cortical surface reconstruction: A linear approach. Journal of Cognitive Neuroscience. 1993;5:162–176. [PubMed]
  • Devlin JT, Russell RP, Davis MH, Price CJ, Wilson J, Moss HE, Matthews PM, Tyler LK. Susceptibility-induced loss of signal: comparing PET and fMRI on a semantic task. Neuroimage. 2000;11:589–600. [PubMed]
  • Devlin JT, Russell RP, Davis MH, Price CJ, Moss HE, Fadili MJ, Tyler LK. Is there an anatomical basis for category-specificity? Semantic memory studies in PET and fMRI, Neuropsychologia. 2002;40:54–75. [PubMed]
  • Dhond RP, Buckner RL, Dale AM, Marinkovic K, Halgren E. Sequence of brain activity underlying word-stem completion. Journal of Neuroscience. 2001;21:3564–3571. [PMC free article] [PubMed]
  • Domalski P, Smith ME, Halgren E. Cross-modal repetition effects on the N4. Psychological Science. 1991;2:173–178.
  • Fiez JA, Petersen SE. Neuroimaging studies of word reading. Proc Natl Acad Sci U S A. 1998;95:914–21. [PubMed]
  • Fischl B, Sereno MI, Dale AM. Cortical surface-based analysis. II: Inflation, flattening, and a surface-based coordinate system, Neuroimage. 1999a;9:195–207. [PubMed]
  • Fischl B, Sereno MI, Tootell RB, Dale AM. High-resolution intersubject averaging and a coordinate system for the cortical surface. Hum Brain Mapp. 1999b;8:272–84. [PubMed]
  • Francis WN, Kucera H. Frequency analysis of English usage: Lexicon and grammar. Boston: Houghton Mifflin; 1982.
  • Friederici AD, Meyer M, von Cramon DY. Auditory language comprehension: an event-related fMRI study on the processing of syntactic and lexical information. Brain Lang. 2000;74:289–300. [PubMed]
  • Frost JA, Binder JR, Springer JA, Hammeke TA, Bellgowan PS, Rao SM, Cox RW. Language processing is strongly left lateralized in both sexes. Evidence from functional MRI, Brain. 1999;122:199–208. [PubMed]
  • Gabrieli JD, Poldrack RA, Desmond JE. The role of left prefrontal cortex in language and memory. Proc Natl Acad Sci U S A. 1998;95:906–13. [PubMed]
  • Geschwind N. Disconnexion syndromes in animals and man. I, Brain. 1965;88:237–94. [PubMed]
  • Gomes H, Ritter W, Tartter VC, Vaughan HG, Jr, Rosen JJ. Lexical processing of visually and auditorily presented nouns and verbs: evidence from reaction time and N400 priming data. Brain Res Cogn Brain Res. 1997;6:121–34. [PubMed]
  • Hagoort P, Brown CM. ERP effects of listening to speech: semantic ERP effects. Neuropsychologia. 2000;38:1518–30. [PubMed]
  • Hagoort P, Indefrey P, Brown C, Herzog H, Steinmetz H, Seitz RJ. The neural circuitry involved in the reading of German words and pseudowords: A PET study. J Cogn Neurosci. 1999;11:383–98. [PubMed]
  • Halgren E. Insights from evoked potentials into the neuropsychological mechanisms of reading. In: Scheibel AB, Wechsler AF, editors. Neurobiology of higher cognitive function. New York: Guilford; 1990. pp. 103–150.
  • Halgren E, Baudena P, Heit G, Clarke JM, Marinkovic K, Chauvel P, Clarke M. Spatio-temporal stages in face and word processing. 2. Depth-recorded potentials in the human frontal and Rolandic cortices [published erratum appears in J Physiol Paris 1994;88(2):following 151] J Physiol Paris. 1994a;88:51–80. [PubMed]
  • Halgren E, Baudena P, Heit G, Clarke JM, Marinkovic K, Clarke M. Spatio-temporal stages in face and word processing. I. Depth-recorded potentials in the human occipital, temporal and parietal lobes [corrected] [published erratum appears in J Physiol Paris 1994;88(2):following 151] J Physiol Paris. 1994b;88:1–50. [PubMed]
  • Halgren E, Dhond RP, Christensen N, Van Petten C, Marinkovic K, Lewine JD, Dale AM. N400-like magnetoencephalography responses modulated by semantic context, word frequency, and lexical class in sentences. NeuroImage. 2002;17:1101–1116. [PubMed]
  • Hämäläinen M, Hari R, Ilmoniemi RJ, Knuutila J, Lounasmaa OV. Magnetoencephalography - theory, instrumentation, and applications to noninvasive studies of the working human brain. Reviews of Modern Physics. 1993;65:413–497.
  • Hämäläinen MS, Ilmoniemi RJ. Interpreting magnetic fields of the brain: minimum norm estimates. Med Biol Eng Comput. 1994;32:35–42. [PubMed]
  • Helenius P, Salmelin R, Service E, Connolly JF. Distinct time courses of word and context comprehension in the left temporal cortex. Brain. 1998;121:1133–1142. [PubMed]
  • Helenius P, Salmelin R, Service E, Connolly JF, Leinonen S, Lyytinen H. Cortical activation during spoken-word segmentation in nonreading-impaired and dyslexic adults. J Neurosci. 2002;22:2936–2944. [PubMed]
  • Hickok G, Poeppel D. Towards a functional neuroanatomy of speech perception. Trends Cogn Sci. 2000;4:131–138. [PubMed]
  • Hodges JR, Patterson K, Oxbury S, Funnell E. Semantic dementia. Progressive fluent aphasia with temporal lobe atrophy, Brain. 1992;115:1783–806. [PubMed]
  • Holcomb PJ. Semantic priming and stimulus degradation: implications for the role of the N400 in language processing. Psychophysiology. 1993;30:47–61. [PubMed]
  • Humphreys GW, Evett LJ. Are there independent lexical and nonlexical routes in word processing? An evaluation of the dual-route theory of reading. Behav Brain Sci. 1985;8:689–740.
  • Ingvar DH. Serial aspects of language and speech related to prefrontal cortical activity. A selective review, Hum Neurobiol. 1983;2:177–89. [PubMed]
  • Kapur S, Rose R, Liddle PF, Zipursky RB, Brown GM, Stuss D, Houle S, Tulving E. The role of the left prefrontal cortex in verbal processing: semantic processing or willed action? Neuroreport. 1994;5:2193–6. [PubMed]
  • Kutas M, Federmeier KD. Electrophysiology reveals semantic memory use in language comprehension. Trends Cogn Sci. 2000;4:463–470. [PubMed]
  • Liu AK, Dale AM, Belliveau JW. Monte Carlo simulation studies of EEG and MEG localization accuracy. Hum Brain Mapp. 2002;16:47–62. [PubMed]
  • Makela AM, Makinen V, Nikkila M, Ilmoniemi RJ, Tiitinen H. Magnetoencephalographic (MEG) localization of the auditory N400m: effects of stimulus duration. Neuroreport. 2001;12:249–53. [PubMed]
  • Marslen-Wilson WD. Functional parallelism in spoken word-recognition. Cognition. 1987;25:71–102. [PubMed]
  • McCarthy G, Nobre AC, Bentin S, Spencer DD. Language-related field potentials in the anterior-medial temporal lobe: I. Intracranial distribution and neural generators, J Neurosci. 1995;15:1080–9. [PubMed]
  • Mesulam MM. From sensation to cognition. Brain. 1998;121:1013–52. [PubMed]
  • Näätänen R, Tervaniemi M, Sussman E, Paavilainen P, Winkler I. “Primitive intelligence” in the auditory cortex. Trends Neurosci. 2001;24:283–8. [PubMed]
  • Nagy ME, Rugg MD. Modulation of event-related potentials by word repetition: the effects of inter-item lag. Psychophysiology. 1989;26:431–6. [PubMed]
  • Nobre AC, McCarthy G. Language-related field potentials in the anterior-medial temporal lobe: II. Effects of word type and semantic priming, J Neurosci. 1995;15:1090–8. [PubMed]
  • Oostendorp T, van Oosterom A. The potential distribution generated by surface electrodes in inhomogeneous volume conductors of arbitrary shape. IEEE Trans Biomed Eng. 1991;38:409–17. [PubMed]
  • Perani D, Cappa SF, Schnur T, Tettamanti M, Collina S, Rosa MM, Fazio F. The neural correlates of verb and noun processing. A PET study, Brain. 1999;122:2337–44. [PubMed]
  • Petersen SE, Fox PT, Posner MI, Mintun M, Raichle ME. Positron emission tomographic studies of the cortical anatomy of single- word processing. Nature. 1988;331:585–9. [PubMed]
  • Poldrack RA, Desmond JE, Glover GH, Gabrieli JD. The neural basis of visual skill learning: an fMRI study of mirror reading. Cereb Cortex. 1998;8:1–10. [PubMed]
  • Poldrack RA, Wagner AD, Prull MW, Desmond JE, Glover GH, Gabrieli JD. Functional specialization for semantic and phonological processing in the left inferior prefrontal cortex. Neuroimage. 1999;10:15–35. [PubMed]
  • Price CJ, Wise RJ, Warburton EA, Moore CJ, Howard D, Patterson K, Frackowiak RS, Friston KJ. Hearing and saying. The functional neuro-anatomy of auditory word processing, Brain. 1996;119:919–31. [PubMed]
  • Raichle ME. What words are telling us about the brain. Cold Spring Harb Symp Quant Biol. 1996;61:9–14. [PubMed]
  • Rauschecker JP. Cortical processing of complex sounds. Curr Opin Neurobiol. 1998;8:516–21. [PubMed]
  • Rauschecker JP, Tian B. Mechanisms and streams for processing of “what” and “where” in auditory cortex. Proc Natl Acad Sci U S A. 2000;97:11800–6. [PubMed]
  • Romanski LM, Goldman-Rakic PS. An auditory domain in primate prefrontal cortex. Nat Neurosci. 2002;5:15–6. [PMC free article] [PubMed]
  • Roskies AL, Fiez JA, Balota DA, Raichle ME, Petersen SE. Task-dependent modulation of regions in the left inferior frontal cortex during semantic processing. J Cogn Neurosci. 2001;13:829–43. [PubMed]
  • Rugg MD, Doyle MC. Event-related potentials and stimulus repetition in direct and indirect tests of memory. In: Heinze H, Munte T, Mangun GR, editors. Cognitive Electrophysiology. Boston: Birkhauser; 1994.
  • Rugg MD, Nieto-Vegas M. Modality-specific effects of immediate word repetition: electrophysiological evidence. Neuroreport. 1999;10:2661–4. [PubMed]
  • Schacter DL, Buckner RL. Priming and the brain. Neuron. 1998;20:185–95. [PubMed]
  • Scott SK, Blank CC, Rosen S, Wise RJ. Identification of a pathway for intelligible speech in the left temporal lobe. Brain. 2000;123(Pt 12):2400–6. [PubMed]
  • Sekiguchi T, Koyama S, Kakigi R. The effect of stimulus repetition on cortical magnetic responses evoked by words and nonwords. Neuroimage. 2001;14:118–28. [PubMed]
  • Simos PG, Basile LF, Papanicolaou AC. Source localization of the N400 response in a sentence-reading paradigm using evoked magnetic fields and magnetic resonance imaging. Brain Res. 1997;762:29–39. [PubMed]
  • Smith ME, Stapleton JM, Halgren E. Human medial temporal lobe potentials evoked in memory and language tasks. Electroencephalogr Clin Neurophysiol. 1986;63:145–59. [PubMed]
  • Tagamets MA, Novick JM, Chalmers ML, Friedman RB. A parametric approach to orthographic processing in the brain: an fMRI study. J Cogn Neurosci. 2000;12:281–97. [PubMed]
  • Thompson-Schill SL, D’Esposito M, Aguirre GK, Farah MJ. Role of left inferior prefrontal cortex in retrieval of semantic knowledge: a reevaluation. Proc Natl Acad Sci U S A. 1997;94:14792–7. [PubMed]
  • Tulving E, Schacter DL. Priming and human memory systems. Science. 1990;247:301–6. [PubMed]
  • Tzourio N, Crivello F, Mellet E, Nkanga-Ngila B, Mazoyer B. Functional anatomy of dominance for speech comprehension in left handers vs right handers. Neuroimage. 1998;8:1–16. [PubMed]
  • Van Petten C, Coulson S, Rubin S, Plante E, Parks M. Time course of word identification and semantic integration in spoken language. J Exp Psychol Learn Mem Cognit. 1999;25:394–417. [PubMed]
  • Veltman DJ, Friston KJ, Sanders G, Price CJ. Regionally specific sensitivity differences in fMRI and PET: where do they come from? Neuroimage. 2000;11:575–88. [PubMed]
  • Wagner AD, Koutstaal W, Maril A, Schacter DL, Buckner RL. Task-specific repetition priming in left inferior prefrontal cortex. Cereb Cortex. 2000;10:1176–84. [PubMed]
  • Wagner AD, Pare-Blagoev EJ, Clark J, Poldrack RA. Recovering meaning: left prefrontal cortex guides controlled semantic retrieval. Neuron. 2001;31:329–38. [PubMed]
  • Warburton E, Wise RJ, Price CJ, Weiller C, Hadar U, Ramsay S, Frackowiak RS. Noun and verb retrieval by normal subjects. Studies with PET, Brain. 1996;119:159–79. [PubMed]
  • Warrington EK, Shallice T. Word-form dyslexia. Brain. 1980;103:99–112. [PubMed]
  • Wise R, Chollet F, Hadar U, Friston K, Hoffner E, Frackowiak R. Distribution of cortical neural networks involved in word comprehension and word retrieval. Brain. 1991;114:1803–17. [PubMed]
  • Woodward JA, Bonett DG, Brecht ML. Introduction to linear models and experimental design. San Diego: Harcourt Brace Jovanovich; 1990.
  • Zatorre RJ, Evans AC, Meyer E, Gjedde A. Lateralization of phonetic and pitch discrimination in speech processing. Science. 1992;256:846–9. [PubMed]