We employed EEG–MEG to investigate the timecourse of syntactic comprehension, measuring effects of semantic reversibility and syntactic complexity. The combination of these two factors is known to induce comprehension failure in agrammatic aphasics (Caramazza and Zurif, 1976
), and in healthy adults under adverse conditions (Dick et al., 2001
), but opinions vary on whether the difficulty relates to a failure of automatic syntactic parsing or instead to enhanced working memory demands in reversible complex sentences. Caplan and Waters (1999
) distinguish between interpretive
processing, with the latter referring to task-dependent processes extending far beyond the time when a sentence is heard or read. Thus far, few neurocognitive studies have attempted to draw a temporal dissociation between these stages.
In order to distinguish between these temporal stages, we examined the effects of syntactic complexity and semantic reversibility on the brain's electromagnetic responses in an auditory sentence-picture-matching task occurring in several time windows, including a relative clause, post-relative clause sentence material, and crucially, a 3-s delay period prior to picture onset. Effects of sentence content occurring during the subsequent delay period can be unambiguously attributed to post-interpretive reanalysis and working memory.
The behavioral data in this experiment indicated that the sentence-picture-matching task is more difficult for reversible sentences, and also that object-embedded clauses within reversible sentences pose a special challenge to comprehension beyond reversibility alone. Through analysis of oscillatory activity, we found that reversibility modulates the brain's response to language input both during and after the sentence, suggesting that the need to explicitly consider syntactic information induces stronger neural activity during online comprehension and also during working memory for sentence content. In contrast, syntactic complexity in the form of object-embedded clauses modulated oscillatory activity only during the post-sentence delay period, suggesting that enhanced brain responses to these sentences are attributable to processes of post hoc reanalysis rather than online comprehension.
Reversible sentences induced a decrease in spectral power compared to irreversible sentences, beginning during the sentence and persisting throughout the delay period, in a broad frequency range including alpha and beta bands. Previous comparisons of MEG and fMRI have suggested that power decreases in alpha and beta bands correspond closely to fMRI activation and are reliably associated with increased
neural activity (Brookes et al., 2005
; Kim and Chung, 2008
). Although the mechanism behind power decrease is not completely understood, it is generally assumed that task-related neural activity interferes with ongoing “idling” rhythms in cortex, resulting in desynchronization of oscillatory activity as detected at the scalp. At present, the mechanisms of oscillatory desynchronization are not well understood; for discussion, see (Neuper and Pfurtscheller, 2001
; Jones et al., 2009
). Here, we use SAM to map 8–30
Hz desynchronization as an indicator of enhanced neuronal activity in specific time periods. Our results demonstrate the utility of desynchronization in the alpha and beta bands as a brain mapping modality for language experiments. In our experiment, we did not detect a robust increase in gamma activity (>35
Hz) accompanying the alpha and beta decreases as have sometimes been reported (e.g. Brookes et al., 2005
). Although gamma oscillations were evoked by linguistic stimulation (Figures A,B), they did not significantly distinguish between conditions. Comparisons of intracranial and non-invasive recordings have suggested that gamma oscillations are coherent over a smaller spatial extent, providing superior localization of activity, but are present at a much smaller signal-to-noise ratio at the surface, limiting their utility in non-invasive experiments (Crone et al., 2006
). Furthermore, gamma activity is frequently observed most strongly in occipital regions under conditions of visual stimulation (e.g. Brookes et al., 2005
; Meltzer et al., 2008
), but may not be as general an indicator of neural activity in other paradigms as is low-frequency desynchronization.
Synthetic aperture magnetometry analyses suggested that online comprehension of reversible sentences recruits enhanced activity in core regions of receptive language function, including the anterior temporal lobe, posterior superior temporal lobe, and the temporoparietal junction. This effect is consistent with models of language comprehension that consider full syntactic parsing to be an optional process employed as needed (Ferreira et al., 2002
; Sanford and Sturt, 2002
), thus leading to increased activation in language areas when semantic reversibility demands explicit consideration of word order information to derive the correct meaning of the sentence. Enhanced power decrease in the 8–30
Hz band during the delay period following reversible sentences occurred in overlapping regions with those seen during the sentence, but with a more frontal distribution, recruiting regions of left inferior frontal gyrus and extending up into the dorsal premotor cortex. In fMRI, we observed enhanced activation to reversible sentences in all of these regions (Meltzer et al., 2010
). The present results suggest that the enhanced activity in frontal cortex occurs later than the activity in posterior regions, despite the fact that the temporal difference is not large enough to be dissociable in fMRI. Such a conclusion is also bolstered by the analyses of “non-specific” activity combined across all conditions, which showed that speech comprehension produced activation in bilateral auditory cortex and in widespread portions of the left temporal lobe (Figure D), but working memory delay period produced a much wider left lateralized pattern of activation in posterior temporal lobe, and large portions of frontal and parietal cortex (Figure E).
While the manipulation of syntactic complexity (RSO–RSS) featured sentences that were exactly matched for the semantic and lexical content of the post-RC sentence segment that followed the critical relative clause, a limitation of the present design is that the reversible and irreversible sentences involved different main clause verbs and different action scenarios. Therefore, it is possible that part of the difference between these conditions may be attributable to semantic factors beyond syntactic parsing. Reversible sentences involve social or physical interactions between two human beings, and therefore may engage cognitive mechanisms related to such factors rather than purely linguistic processing. However, we believe that the contribution of such factors cannot fully explain the present findings, on the basis of our previous fMRI results with the same materials (Meltzer et al., 2010
). In that study, we also compared the brain's responses to the pictures
following reversible vs. irreversible sentences, finding increased activity for reversible pictures only in bilateral posterior middle temporal regions. In contrast, increased activity for reversible sentences
occurred throughout the left perisylvian areas, as seen again in the present MEG study. We would expect that activity related to the general semantic content of the reversible scenarios would be induced by the pictures depicting the actions as well, and therefore we conclude that such factors cannot fully explain the perisylvian activations induced by reversible sentences in the context of a task that demands a thematic role judgment.
Another concern related to the task design is that subjects had to be prepared to make the more difficult syntactic discriminations for the reversible sentences, whereas they may have caught on that no such decision was required for the irreversible sentences, which were tested using only lexical foils. This may have resulted in increased attention and arousal for the reversible sentences, contributing to greater brain activity during and following these sentences. However, our results are unlikely to be driven by general arousal, given that the increased activation to reversible sentences was almost completely left lateralized and localized to perisylvian language areas. This argues that the increased attention induced by reversible sentences is rather specific to the linguistic processing demands of the task, rather than a more general modulation of neural excitability.
The pattern of selective activations observed to the most difficult sentences, the non-canonical RSO condition (contrasted with RSS), was clearly dissociable from activation related to reversibility alone. We observed selective activity to RSO sentences only during the post-sentence delay period. This activity occurred in anterior temporal and frontal regions, and was remarkably symmetric bilaterally. The finding of enhanced activity in bilateral frontal regions is consistent with patterns of activation in these regions observed in fMRI studies of verbal working memory load (Rypma et al., 1999
; Narayanan et al., 2005
), supporting the hypothesis that enhanced processing of non-canonical sentences occurs mainly in a late reanalysis stage in working memory. This finding is also consistent with the fMRI results showing increased activity specific to the RSO condition timelocked to the onset of the picture probes in bilateral frontal cortex (Meltzer et al., 2010
). These selective activations were interpreted as reflecting effortful reanalysis of both the sentence and the pictures. However, the fMRI technique, being dependent on hemodynamic deconvolution with a jittered trial design, was unable to fully resolve activity that peaks in the interval between the sentence presentation and the pictures. Activity occurring during the memory delay but before the picture probes may have appeared to be associated with the initial sentence processing, due to the hemodynamic delay. In the present MEG study, we observed that selective activation of bilateral anterior temporal and frontal regions occurred in response to RSO sentences, but only in the delay period, not during the sentence itself. This distinction would not have been apparent in the BOLD data, which integrates neuronal activity over a timescale of several seconds. With beamforming analysis of MEG data, we have been able to demonstrate for the first time that comprehension of syntactically complex sentences, so often impaired in aphasic patients, does not lead to an appreciable increase in neural activity during the perception of the sentence (at least for the measures employed here), but rather during the period of task-related memory use following the sentence. These findings suggest that comprehension deficits in impaired populations may stem from an impaired capacity for working memory use, rather than a loss of core syntactic competence.
Our results do not necessarily imply that there is no specialized neuronal activity involved in processing syntactic complexity in real time, but given the effects seen in this study, any such real-time activity seems to be dwarfed in magnitude by post hoc
processes. Previous ERP studies have associated syntactic reanalysis with such “late” components as the P600 potential (Friederici et al., 1996
; Frisch et al., 2002
). However, we observed that syntactically challenging sentences induced power decreases that spanned the entire 3-s length of the delay period (as seen in Figure ). This finding has important implications for processing models. In some models, “reanalysis” in complex syntax implies that subjects take extra time to derive the meaning of a sentence, but do arrive at a stable meaning eventually. Our results suggest instead, that rather than arriving at a stable meaning, subjects continue to engage in mental reanalysis of complex sentences (including but not limited to subvocal rehearsal processes) until the task demands are fulfilled. Furthermore, this extra analysis extends to the picture selection period, as evidenced by the increased reaction time following RSO sentences, as well as the increased hemodynamic responses to picture probes following RSO sentences observed in our previous fMRI study (Meltzer et al., 2010
). This notion of temporally extended reanalysis is more in line with that described in Caplan and Waters (1999
) as a post-interpretive process, rather than a more immediate process described as reanalysis in the P600 literature.
One potential objection to these findings is that they may be related to the task demands, which involved an extended memory delay and a challenging picture-matching probe. Subjects may not engage in such extended reanalysis in the course of ordinary language comprehension. However, we would argue that this “optional” reanalysis of language is an important component of comprehension during the understanding of more difficult sentences, and that its frequent absence in ordinary listening accounts for misinterpretations of complex sentences as shown by Ferreira (2003
). Furthermore, the sentence-picture-matching tasks used in offline comprehension tests in clinical populations allow for untimed responses, during which even healthy participants may engage in reanalysis over several seconds. Our observations of increased brain activity following complex sentences both during the memory delay (shown with MEG) and following the actual picture presentation (shown previously with fMRI) suggest that participants must continue to process the most difficult sentences until the task demands are completed. One might expect that participants would quickly determine who did what to whom after the sentence is completed, forming a conceptual representation of the action unrelated to the actual syntactic structure. Our results argue that this does not occur; subjects are more likely to continue to rehearse complex sentences “verbatim” instead of transforming them to a simpler representation.
We are not the first to propose that brain activation in response to syntactically complex sentences is largely due to working memory demands. In an ERP study, King and Kutas (1995
) observed Left Anterior Negativities to object-embedded vs. subject-embedded relative clauses, beginning after the main verb of the sentence and extending until after its completion. This activity was much stronger in subjects with higher working memory capacities. It is uncertain why we did not observe increased activity for object-relative sentences during the sentences themselves, either in oscillatory responses or time-domain averages of evoked responses (data not shown), but our interpretation is consistent with theirs: rehearsal and reanalysis of complex sentences is most likely responsible for increased brain activation, rather than automatic parsing processes triggered by syntactic complexity. It is possible that our task design encouraged subjects to delay the reprocessing of syntactically complex sentences to a greater degree than the subjects in the King and Kutas study. In any case, both our results and theirs argue against an automatic parsing process being responsible for the observed neural activity.
The present results suggest that comprehension of syntactically complex sentences may rely upon a highly distributed network of (predominantly frontal) brain regions that are also involved in more general cognitive processes, which are especially activated by the need for continued rehearsal of these sentences in working memory. Such an interpretation can reconcile divergent findings in the literature on syntactic comprehension in aphasia. Although studies have shown an association between damage to Broca's area and impaired comprehension of non-canonical sentences (Drai and Grodzinsky, 2006
), other studies have shown that this association is imperfect, with the same comprehension patterns sometimes occurring after damage to different areas (Berndt et al., 1996
; Caplan et al., 2007b
). Our findings point to an extended reanalysis involving large portions of the frontal lobe, suggesting that damage to a wide variety of areas could be sufficient to disrupt it. An inability to engage in extended reanalysis of complex sentences may underlie comprehension deficits for complex material in aphasic patients whose comprehension of less syntactically complex material is otherwise undisturbed.