|Home | About | Journals | Submit | Contact Us | Français|
To make sense of a sentence, we must compute morphosyntactic and semantic-thematic relationships between its verbs and arguments and evaluate the resulting propositional meaning against any preceding context and our real-world knowledge. Recent electrophysiological studies suggest that, in comparison with non-violated verbs (e.g. “…at breakfast the boys would eat…”), animacy semantic-thematically violated verbs (e.g. “..at breakfast the eggs would eat…”) and morphosyntactically violated verbs (e.g. “…at breakfast the boys would eats…”) evoke a similar neural response. This response is distinct from that evoked by verbs that only violate real-world knowledge (e.g. “…at breakfast the boys would plant…”). Here we used fMRI to examine the neuroanatomical regions engaged in response to these three violations. Real-world violations, relative to other sentence types, led to increased activity within the left anterior inferior frontal cortex, reflecting participants’ increased and prolonged efforts to retrieve semantic knowledge about the likelihood of events occurring in the real world. In contrast, animacy semantic-thematic violations of the actions depicted by the central verbs engaged a frontal/inferior parietal/basal ganglia network known to mediate the execution and comprehension of goal-directed action. We suggest that the recruitment of this network reflected a semantic-thematic combinatorial process that involved an attempt to determine whether the actions described by the verbs could be executed by their NP Agents. Intriguingly, this network was also activated to morphosyntactic violations between the verbs and their subject NP arguments. Our findings support the pattern of electrophysiological findings in suggesting (a) that a clear division within the semantic system plays out during sentence comprehension, and (b) that semantic-thematic and syntactic violations of verbs within simple active sentences are treated similarly by the brain.
In order to make sense of a sentence, we must integrate multiple different types of relationships. These include the morphosyntactic and semantic-thematic relationships between its verbs and arguments, and the relationship between the resulting propositional representation of meaning, any preceding context, and the associations between words and events that have been encountered before and that are stored within long-term semantic memory (our real-world knowledge). This study aimed to examine the neural networks that are engaged in processing and evaluating these three different types of relationships – syntactic, semantic-thematic and real-world knowledge – by examining how the brain responds when each are violated. A large behavioral and electrophysiological literature suggests that the introduction of such violations may tax the underlying neurocognitive processes that are engaged in computing and evaluating these types of relationships (Marslen-Wilson, Brown et al. 1988; Osterhout and Holcomb 1995). This approach has led to claims that aspects of semantic and syntactic processing are instantiated by distinct neurocognitive networks: violations of real-world semantic knowledge and syntactic rules each lead to distinct temporal and spatial patterns of neural activity, as indexed using event-related potentials (ERPs) and functional magnetic resonance imaging (fMRI) techniques (Osterhout and Holcomb 1995; Kuperberg 2007; Osterhout, Kim et al. 2007).
In ERP studies, words that are incongruous with their preceding context, and/or with real-world knowledge stored within semantic memory, lead to the production of a distinct negative-going waveform, peaking at approximately 400 msec – the N400 (Kutas and Federmeier 2000). An increased N400 is evoked by violations of semantic associative (Bentin, McCarthy et al. 1985; Rugg 1985) and categorical (Deacon, Grose-Fifer et al. 2004; Grose-Fifer and Deacon 2004) relationships in word-pair semantic priming paradigms, as well as by semantic anomalies in sentence paradigms (Kutas and Hillyard 1980; Kutas and Hillyard 1984). An increased N400 is also evoked by words that are either unlikely, e.g. by “plant” in sentences describing an unlikely event such as “Every morning at breakfast the boys would plant …” (Kuperberg, Sitnikova et al. 2003), or untrue, e.g. by “white” in sentences such as “The Dutch trains are white….”, presented to Dutch comprehenders who know that Dutch trains are, in fact, yellow (Hagoort, Hald et al. 2004).
Using fMRI, several studies have demonstrated that a left inferior frontal and temporal network is modulated in numerous paradigms examining the structure and function of semantic memory. The left inferior frontal gyrus (Brodmann Areas, BA 45 and 47) is thought to play a role in the explicit, controlled retrieval (Wagner, Pare-Blagoev et al. 2001) and/or the controlled selection (Thompson-Schill, D’Esposito et al. 1997; Thompson-Schill, D’Esposito et al. 1999) of the meaning of individual words. It is also activated when semantic violations are introduced within sentences (Ni, Constable et al. 2000; Newman, Pancheva et al. 2001; Kiehl, Laurens et al. 2002; Kuperberg, Holcomb et al. 2003; Hagoort, Hald et al. 2004), including violations of what is likely to occur in the world (Kuperberg, Sitnikova et al. 2003) and violations of what we know to be true (Hagoort, Hald et al. 2004), suggesting that it may also mediate the retrieval of real-world semantic knowledge.
In many of these paradigms, although not all (e.g. Hagoort et al., 2004), the left inferior frontal cortex is co-activated with temporal cortices (Van Petten and Luka 2006). The left middle temporal cortex may function together with the left inferior frontal cortex in explicitly retrieving semantic information (Wagner, Pare-Blagoev et al. 2001), while activity within more inferior temporal cortices may reflect a more implicit and automatic activation of stored associative semantic knowledge (Alexander, Hiltbrunner et al. 1989; Chao, Haxby et al. 1999; Martin and Chao 2001; Bar 2004; Wheatley, Weisberg et al. 2005; Gold, Balota et al. 2006, Kuperberg, Lakshmanan et al. 2007).
In ERP studies, syntactic violations or ambiguities within sentences produce several different electrophysiological responses that are distinct from the N400 (reviewed by A. Friederici & Weissenbor, 2007). One such waveform is a positive-going deflection that peaks at approximately 600 msec after the onset of a syntactic violation (or ambiguity), e.g. “The spoiled child throw the toys on the floor.” (Hagoort, Brown et al. 1993) and that has been termed the P600 (Osterhout and Holcomb 1992). The P600 effect evoked by syntactic anomalies or ambiguities is thought to reflect a late cost of syntactic processing, although it is debated whether this constitutes a syntactic or thematic reanalysis after a first stage of structural syntactic assignment (Friederici and Weissenbor 2007) or a prolonged attempt at integrating a word into the preceding thematic structure of the context (Kaan, Harris et al. 2000). Given that their polarity and time courses are distinct, what is undisputed is that the P600 reflects neurocognitive processes that are distinct from those reflected by the N400.
There have been several fMRI studies examining the effects of introducing different types of syntactic violations within sentences (Kuperberg, McGuire et al. 2000; Ni, Constable et al. 2000; Moro, Tettamanti et al. 2001; Newman, Pancheva et al. 2001; Friederici, Ruschemeyer et al. 2003; Kuperberg, Holcomb et al. 2003; Ruschemeyer, Fiebach et al. 2005). Of most relevance to the current study are two studies that introduced violations of morphosyntactic infection between a verb and its arguments. Ni et al. (2000, Experiment 2) reported that, in comparison to non-violated sentences, agreement violations such as “Trees can grew” elicited increased activity within bilateral inferior frontal gyri, pericentral cortices, right middle (BA46) and superior frontal gyri (BA 8&9), as well as within the right inferior parietal lobule. Kuperberg et al. (2003a) found that sentences with violations of inflectional morphosyntax such as “Every morning at breakfast the boys would eats toast and jam”, relative to both non-violated sentences as well as violations of what is likely to occur in the real world, led to robust increases in activity within bilateral parietal cortices (BA 40 and 7). At a less stringent significance threshold, these sentences were also associated with increased activity within posterior inferior frontal (BA 44) and adjacent premotor regions (BA 6), as well as within the left posterior occipito-temporal cortex. Intriguingly, much of this network has been implicated in executing and comprehending the meaning of goal-directed action (Buccino, Binkofski et al. 2001; Rizzolatti, Fogassi et al. 2001). Thus, one possibility is that encountering these morphosyntactic violations on verbs lead to a semantic-thematic reanalysis or integration process (Kaan, Harris et al. 2000; Kuperberg, Holcomb et al. 2003) that involves a computation and evaluation of the meaning of the action depicted by the verb in specific relation to the meaning of its arguments.
Semantic-thematic relationships between nouns and their arguments lie at the interface between semantic memory and syntactic structure. Despite their semantic nature, they have direct implications for how the combination of words within a sentence plays out in terms of syntactic structure (Chomsky 1981; Jackendoff 1987; Levin 1993; Levin and Rappaport Hovav 2005). They describe those aspects of meaning of a verb and its arguments (a subset of all their meanings) that have a direct bearing on the number and types of morphosyntactic arguments assigned by a verb (Pinker 1989; Levin and Rappaport Hovav 2005). They encompass those aspects of meaning of a NP argument that are particularly relevant to the action described by the verb (Jackendoff 1978; Jackendoff 1987). So, for example, consider the semantic-thematic violation in the sentence, “…at breakfast the eggs would eat…”. This sentence is violated because “eggs” is inanimate in nature and, as such, does not have the semantic properties to be able to perform the action of eating. The sentence would also be anomalous if its subject was “computers” but not if it was “boys” or “dogs”. The same would be true of other verbs such as “kiss” or “kick” that can only assign the role of Agent (performer of the action) to an animate subject NP for a sentence to make sense. Of note, such semantic-thematic relationships between an animate Agent and a verb can be used to describe the event structure not only of verbs describing bodily action (e.g. “eat”), visual motion (e.g. “run”), or verbs of contact (e.g. “hit”), but also of other generalizable semantic classes of verbs, such as verbs of transfer (e.g. “donate”, “invest”).
Interestingly, in ERP studies, animacy violations of semantic-thematic relationships between verbs and arguments (e.g. “…at breakfast the eggs would eat…”) sometimes fail to evoke an N400 effect but rather lead to a Late Positivity effect (Kuperberg, Sitnikova et al. 2003; Hoeks, Stowe et al. 2004; Kuperberg, Kreher et al. 2007; Kuperberg 2007). This Late Positivity is similar in morphology and scalp distribution to the P600 evoked by inflectional morphosyntactic violations between a subject NP and verb (Kuperberg, Caplan et al. 2006). Although different hypotheses have been proposed to explain why these types of semantic violations evoke a P600 effect rather than an N400 effect (Kuperberg, Sitnikova et al. 2003; Hoeks, Stowe et al. 2004; Kim and Osterhout 2005; Kolk and Chwilla 2007; Kuperberg, Kreher et al. 2007; Kuperberg 2007), they tend to agree that the P600 is often triggered by a temporary conflict between the output of more than one processing stream and that it reflects a continued analysis or reanalysis of the critical word with respect to its context. What is debated is the nature of such processing streams and their output representations, as well as the specificity of such reanalysis to a particular level of linguistic processing. We have suggested that the P600 evoked by both semantic-thematic and syntactic anomalies reflects, at least in part, a continued semantic-thematic analysis of the meaning of arguments against more minimal semantic requirements of the action described by the central verb (Kuperberg, Kreher et al. 2007). Consistent with this hypothesis, the P600 evoked by animacy semantic-thematic violations is also similar to the Late Positivity elicited by violations of visual real-world actions depicted within short, silent movie clips (Sitnikova, Kuperberg et al. 2003; Kuperberg 2007; Sitnikova, Holcomb et al. 2007; Sitnikova, Holcomb et al. 2007), suggesting that such a semantic-thematic integration or reanalysis might engage similar neural processes to those engaged in relating people, objects and action during real-world event comprehension (Sitnikova, Kuperberg et al. 2003; Kuperberg 2007; Sitnikova, Holcomb et al. 2007; Sitnikova, Holcomb et al. 2007). It is also possible that such a semantic-thematic analysis is one component of a more general reanalysis of all aspects of the original linguistic input including orthographic, phonological, lexico-semantic and syntactic information (Vissers, Chwilla et al. 2006; Kolk and Chwilla 2007).
Taken together, ERP findings suggest that the brain may differentiate between two broad types of semantic violations: 1) violations of semantic associative, categorical, and real-world relationships that are violated in sentences such as “…at breakfast the boys would plant..”, and that are based mainly on the frequency of co-occurrence of words or events, as stored within semantic memory (Kutas and Federmeier 2000), and 2) semantic-thematic verb-argument relationships, violated in sentences such as “…at breakfast the eggs would eat…”, that may rely on prolonged attempts to semantically-thematically integrate or reanalyze the meaning of arguments against more minimal semantic requirements of the central verb (Kuperberg, Sitnikova et al. 2003; Kuperberg, Kreher et al. 2007; Kuperberg 2007). Moreover, there is some evidence that such semantic-thematic integration or reanalysis may also be engaged upon encountering morphosyntactic violations (Kaan, Harris et al. 2000; Kuperberg, Caplan et al. 2006) and may share commonalities with processes engaged upon encountering violations of action in the visual world (Kuperberg 2007; Sitnikova, Holcomb et al. 2007).
The present study aimed to determine whether similar broad neural distinctions between these sentence types were observable using fMRI. We examined the neuroanatomical regions engaged in response to introducing three types of violation within sentences. First, we introduced verbs that violated our knowledge about what events are likely to occur in the world, given a particular context. For example, in a sentence such as, “Every morning at breakfast the boys would plant…”, the violation lies at the level of integrating the likelihood of boys planting with what we know about events that usually occur at breakfast. We use the term, ‘real-world pragmatic1 violations’ to describe these types of sentences. Second, we introduced verbs whose thematic structure was such that it violated the animacy of its preceding Agent NP. For example, in a sentence such as, “Every morning at breakfast the eggs would eat…”, the violation lies in the relationship between the inanimate subject NP, “eggs”, and the verb of the sentence, “eat” (that has a thematic-semantic structure such that it assigns the role of Agent to an animate rather than inanimate subject), without necessarily considering the preceding context, “Every morning at breakfast”, or even real-world knowledge. We use the term ‘animacy semantic-thematic violations’ to describe these types of sentences. Third, we introduced inflectional violations between subject NPs, an auxiliary when present, and the main verbs. For example, in a sentence such as, “Every morning at breakfast the boys would eats…”, the verb does not agree in number with the subject NP. We use the term ‘morphosyntactic violations’ to describe these types of sentences.
We contrasted each of these types of sentences to non-violated sentences, and to one another. The design was based on our previous fMRI study described above in which we examined the effects of two of these violations – real-world pragmatic and morphosyntactic (Kuperberg, Holcomb et al. 2003). On the basis of this previous study as well as the findings of Hagoort et al. (2003), we predicted that the real-world pragmatically violated sentences (e.g. “…at breakfast the boys would plant…”), in comparison with non-violated sentences and morphosyntactically violated sentences, would lead to increased activity within the left inferior frontal cortex (BA 45/47). On the basis of this previous study as well as the findings of Ni et al. (2000, Experiment 2), we also predicted that the morphosyntactically violated sentences (e.g. “…at breakfast the boys would eats…”), in comparison with both non-violated and real-world pragmatically violated sentences, would recruit inferior parietal cortices and possibly also posterior inferior frontal cortices and the left occipito-temporal cortex.
Our main aim was to examine the pattern of activity evoked by the animacy semantic-thematic violated sentences (e.g. “…at breakfast, the eggs would eat…”), in relation to each of the other sentence types. On the basis of the distinct ERP effects evoked by real-world pragmatic and animacy semantic-thematic violations, we predicted that the animacy semantic-thematic violations would lead to the recruitment of a neuroanatomical network that was distinct from the left inferior frontal cortex recruited by the real-world pragmatic violations. More specifically, given that what was violated was the relationship between the action denoted by the critical verb and the inanimate properties of the Agent carrying out this action, we predicted that these violations would evoke activity within a widespread network engaged in evaluating such action-based, semantic-thematic relationships. This might include premotor and parietal cortices, known to be engaged in processing both verbal and non-verbal representations of goal-directed physical action (Buccino, Binkofski et al. 2001; Rizzolatti, Fogassi et al. 2001). It might also include dorsolateral and superior prefrontal cortices that may be engaged in processing representations of goal and motivation (Ruby, Sirigu et al. 2002).
Based on the similarity of the Late Positivity/P600 evoked by these types of animacy semantic-thematic violations and the P600 evoked by morphosyntactic violations (Kuperberg, Caplan et al. 2006), as well as on the theory that encountering morphosyntactic violations between a verb and its argument might also lead to a prolonged semantic-thematic analysis (Kaan, Harris et al. 2000), we hypothesized that the neural network engaged by animacy semantic-thematic violations would, at least partially, overlap with the network engaged by these types of morphosyntactic violations.
The stimuli have been described in detail elsewhere (Kuperberg, Sitnikova et al. 2003; Kuperberg, Caplan et al. 2006). Two-hundred-and-forty Agent-Theme or Experiencer-Theme verbs were chosen as critical words and 6–10-word sentences providing a constraining context, with an animate subject NP, were constructed for each of them (see Table 1). Most verbs were dynamic (nine were stative) (Dowty 1989). The critical verb fell on the sixth to the ninth word of the sentence and none of the critical verbs were sentence-final words.
The animacy semantic-thematic violated sentences were constructed by replacing the subject animate NP with an inanimate NP, which was in keeping with the overall preceding context, such that the sentence became anomalous on the critical verb. The syntactically violated sentences were constructed by introducing an inflectional morphosyntactic violation between the subject, an auxiliary when one was present, and the main verb (either by violating subject-verb agreement or by using a finite in place of an infinitival verb). The real-world pragmatically violated sentences were constructed by replacing the critical verbs and its subsequent words with verbs and subsequent words that were chosen pseudorandomly from sentences of another list (see below), such that the sentences were rendered relatively less likely or less plausible, with respect to their preceding contexts and real-world knowledge, than the non-violated sentences.
So that no participant would encounter the same word more than once (leading to repetition priming effects) but so that, across all participants, all critical verbs (and words subsequent to the critical verb) would be seen in all four conditions, the sentences were divided into four lists that were counterbalanced across subjects. Thus, in each list, there were 240 test sentences: 60 normal sentences, 60 real-world pragmatically violated sentences, 60 animacy semantic-thematically violated sentences and 60 morphosyntactically violated sentences.
Sixteen (11 male and 5 female; mean age: 42, SD: 9) participants were recruited by advertisement. All participants were right-handed as assessed using the modified Edinburgh Handedness Inventory (Oldfield 1971; White and Ashton 1976). Selection criteria required all participants to have normal or corrected-to-normal vision, to be native speakers of English and not to have learned another language before age 5 years. In addition, participants were not taking any medication and were screened to exclude the presence of psychiatric and neurological disorders and to exclude the contraindications for MRI. Written consent was obtained from all subjects before participation according to the established guidelines of the Massachusetts General Hospital Institutional Review Board.
During scanning, each trial began with the presentation of a yellow fixation point at the center of the screen for 400 msec, followed by a 100 msec blank screen, followed by the first word of the sentence. Each word appeared on the screen for 400 msec with an interstimulus interval (ISI) of 100 msec separating words. Thus, the critical verb always started after 3 sec from the beginning of the sentence trial. The final word of each sentence was followed by a question mark cue that remained on the screen until the subject made his/her response at which point the next trial started. The subject’s task was to decide whether or not each whole sentence was acceptable by pressing one of two buttons on a response box (using their left hand, with their index and middle fingers, counterbalanced across subjects). Subjects were told that sentences may not be acceptable in different ways and that if a sentence seemed at all “odd” or unlikely or if it had an obvious grammatical error, they should indicate that it was unacceptable. Subjects were instructed to wait until the question mark cue before responding. Accuracy and judgment reaction times (RTs) were recorded.
Sentence trials were pseudo-randomly presented amongst fixation trials in which subjects were asked to fixate on a white “+” symbol for variable durations ranging from 2 through 20 seconds until the next sentence trial. The fixation trials constituted 32% of all trials. The random interleaving of these ‘fixation’ or ‘null-events’ amongst the sentences was critical for the efficient estimation and deconvolution of the entire hemodynamic response (Burock, Buckner et al. 1998).
Participants’ accuracy (judging a non-violated sentence as acceptable and a violated sentence as unacceptable) and RTs to each sentence type (collapsed across individual sentences) were entered into a repeated-measures ANOVA with subjects as a random effect and sentence type as a within-subject factor. One participant’s average RTs to the animacy semantic-thematic violated sentences was more than three standard deviations above the mean RT to the animacy semantic-thematic violations (across all participants) and was excluded from analyses. Significant effects of sentence type were followed up using planned t-tests comparing the different sentence types. Alpha was set to 0.05. All RT analyses were repeated using correctly-answered responses only and revealed the same pattern of findings.
Imaging took place on a 3 Tesla MR scanner (Siemens Trio) with echoplanar (EP) imaging capability. Head motion was minimized using pillows and cushions around the head. Subjects underwent two conventional high-resolution 3D structural scans, constituting a spoiled GRASS (SPGR) sequence (128 sagittal slices, 1.33 mm thickness, TR = 2530 msec, TE = 3.77 msec, flip angle = 7 degrees, bandwidth = 200 Hz, in-plane resolution = 1 mm × 1.33 mm). These scans were followed by the acquisition of T1-weighted anatomic images (30 slices, 3 mm thickness, skip 1 mm between slices) and then a T2-weighted image acquired in plane with the functional images to assist in the manual registration of the functional data to the high resolution anatomical scans.
Each subjects then viewed one of the four counterbalanced sentence lists with the sentence trials and fixation trials, divided over eight functional runs. Each functional run lasted between 420 seconds during which T2*-weighted echoplanar (EP) images were acquired (30 slices covering the whole brain, 210 images per slice, 3 mm thickness, in-plane resolution of 3.125 mm, slices oriented approximately 30\grad axially, 1 mm between slices), using a gradient echo (GR) sequence (TR = 2 sec; TE = 25 msec; flip angle = 100\grad).
Following motion correction, the two high-resolution structural scans for each participant were averaged to increase the signal:noise, and this high signal:noise resulting volume was used to reconstruct a model of each individual’s cortical surface using FreeSurfer, developed at the Martinos Center for Biomedical Imaging, Charlestown, MA, http://surfer.nmr.mgh.harvard.edu/ (Dale and Sereno 1993; Dale, Fischl et al. 1999; Fischl, Liu et al. 2001). The surface representing the gray/white border was inflated (Dale and Sereno 1993; Fischl, Sereno et al. 1999), differences between individuals in the depth of gyri/sulci were normalized, and, for the purposes of averaging functional data across subjects (see below), each subject’s reconstructed cortical surface was morphed/registered to an average spherical surface representation that optimally aligned sulcal and gyral features across subjects while minimizing metric distortion (Fischl, Sereno et al. 1999; Fischl, Sereno et al. 1999).
For each participant, the acquired native functional volumes were first corrected for potential motion using the AFNI algorithm (Cox 1996). Next, the functional volumes were spatially smoothed using a 3-D Gaussian filter with a full-width half-max (FWHM) of 6 mm. Global intensity variations across runs and participants were removed by rescaling all voxels and time points of each run such that the mean in-brain intensity was fixed at an arbitrary value of 1000.
The functional images were then analyzed with a General Linear Model (GLM) using the FreeSurfer Functional Analysis Stream (FS-FAST). The hemodynamic response for each trial was modeled using three components, each constituting a canonical hemodynamic response function (HRF) (Friston, Fletcher et al. 1998), convolved with a box car of an appropriate length. The first component was modeled as a single regressor and lasted between 3500 msec and 4500 msec and corresponded to the onset of each sentence trial until the offset of the word before the critical word. This did not differ between experimental conditions. The second component lasted from the onset of the critical verb until the onset of the question mark and lasted from 620 msec to 1950 msec and was modeled separately for each sentence type. The third component lasted from the onset of the question mark until each participant’s response that moved him or her on to the next trial. In addition, mean offset and linear trend regressors were included to remove low-frequency drift.
These GLM parameter estimates and residual error variances of each participant’s functional data were resampled onto his or her inflated cortical surface and then onto the average cortical spherical representation (see anatomical reconstruction above). Each participant’s data were then smoothed on the surface tessellation using an iterative nearest-neighbor averaging procedure (equivalent to applying a two-dimensional Gaussian smoothing kernel with a FWHM of approximately 8.5 mm). Each participant’s functional data were also resampled into Talairach space (Talairach and Tournoux 1988) for averaging of subcortical data. This was carried out by first aligning the functional to the structural data with a six degrees-of-freedom transform using SPM software, and then aligning the structural data to the MNI305 template using a twelve degrees-of-freedom alignment procedure (Collins, Neelin et al. 1994).
A contrast was generated that compared all sentences across the whole trial (all three regressor components) with the baseline fixation condition. In addition, pair-wise contrasts comparing activity to each sentence type relative to every other sentence type were constructed using the regression weights of the second canonical HRF component (that differed across conditions). These contrasts were tested using a t statistic at each voxel on the spherical surface (to identify cortical clusters) and in Talairach space (to identify subcortical clusters) using a random effects model. To correct for multiple comparisons, significant clusters of activated voxels were identified on the basis of a Monte Carlo simulation (Doherty, West et al. 2004) using a cluster size threshold of 300 mm2 (for cortical clusters) and 300 mm3 (for subcortical clusters) using a threshold for rejection of the null hypothesis of p < 0.05. Cortical maps were then overlaid on one another in order to identify regions that were commonly activated across contrasts of interest. Statistical maps were also generated that included only activity to correctly-answered trials. Any difference in findings revealed by these maps and the maps generated from all trials are noted in the Results tables.
In addition to modeling the hemodynamic response for each trial in each individual using an HRF (Friston, Fletcher et al. 1998), functional images were also analyzed using a finite impulse response (FIR) model, that gave estimates of the hemodynamic response every 2 sec (at each TR), allowing us to examine the hemodynamic response at successive intervals without assumptions about the shape of the hemodynamic response (Burock, Buckner et al. 1998; Dale 1999; Burock and Dale 2000). By qualitatively comparing the results yielded by the HRF-modeled and the FIR-modeled datasets, we were able to determine whether any apparent differences in the amplitude of hemodynamic activity between the sentence types yielded by the HRF-modeled dataset actually reflected differences across sentence types in the time course of hemodynamic activity within the same networks.
There were significant differences in accuracy across the four sentence types, F(3, 45) = 5.59, p = 0.002. Follow up paired t-tests showed that participants responded more accurately to the animacy semantic-thematic violated sentences than to the non-violated sentences, t(15) = 3.99, p < 0.001, the real-world pragmatically violated sentences, t(15) = 4.14, p < 0.001, and the morphosyntactically violated sentences, t(15) = 2.981, p < 0.009. There were no differences in accuracy between any of the other sentence types (all ps > 0.1).
There were also significant differences in RTs across the four sentence types, F(3, 42) = 3.55, p < 0.037. Follow-up paired t tests showed that RTs to the real-world pragmatically violated sentences were marginally longer than to the non-violated sentences, t(15) = 1.9, p < 0.071, and significantly longer than to the animacy semantic-thematic violated sentences, t(14) = 2.88, p < 0.02, and to the morphosyntactically violated sentences, t(15) = 2.76, p < 0.015. This pattern of RT findings was the same when only the correctly-answered trials were analyzed.
As expected, a large network distributed across frontal, temporal and occipital cortices showed more activity to the sentences (all four types) than to the low-level baseline fixation condition (Table 3A, Figure 1). In addition, consistent with previous studies, a large network showed less activity to the sentence trials than to the fixation trials; these so-called ‘deactivated’ regions included the middle frontal, medial frontal, inferior parietal, medial parietal and medial temporal cortices (Table 3B, Figure 1).
There was significantly more activity in response to the real-world pragmatic violations than to each of the three other sentence types within the left anterior inferior frontal gyrus (Table 4, top row, and Figure 2). An overlap map confirmed that this region was activated in all three contrasts (Figure 2). The contrast between the real-world pragmatically violated and the non-violated sentences also showed increased activity within the bilateral anterior ventromedial temporal cortices (fusiform and parahippocampal gyri) and within extrastriate cortices.
There was relatively more activity in response to the morphosyntactically violated sentences than to both the non-violated sentences and the real-world pragmatically violated sentences within widespread regions (Table 5, Figure 3). Many of the same regions were activated in both these contrasts, and included bilateral posterior inferior frontal and precentral cortices, bilateral middle frontal gyri, the left inferior parietal lobule, bilateral insula, superior temporal gyri, anterior medial prefrontal cortices, the left lateral occipitotemporal cortex and the basal ganglia. An overlap map confirmed that many of the same clusters were activated in both contrasts (Figure 3, bottom row).
With the exception of the left inferior parietal lobule (BA 40) and bilateral anterior medial prefrontal cortices (BAs 32/23/10), most of these regions did not show significant modulation in a direct contrast between the morphosyntactically violated and the animacy semantic-thematic violated sentences.
The animacy semantic-thematic violated sentences showed more activity than both the non-violated sentences and the real-world pragmatically violated sentences within a widespread network (Table 6, Figure 4). Once again, many of the same regions were activated in both these contrasts, including the bilateral posterior inferior frontal/precentral cortices, bilateral middle frontal cortices, insula, and the left inferior parietal lobule and the basal ganglia. An overlap map confirmed that many of these clusters were activated in both contrasts (Figure 4, bottom row).
Once again, a direct contrast between the animacy semantic-thematic violated and the morphosyntactically violated sentences failed to reveal significant modulation in most of these regions: only the left posterior inferior frontal gyrus showed more activity in response to the animacy than to the morphosyntactic violations.
As noted above, many of the same regions were activated in response to the morphosyntactically violated sentences (relative to both the real-world pragmatically violated and the non-violated sentences) as well as to the animacy semantic-thematic violated sentences (relative to both the real-world pragmatically violated and the non-violated sentences). Cortical regions of overlap are shown in Figure 5 and constituted clusters within bilateral posterior inferior frontal gyri, precentral gyri, middle frontal gyri, the left inferior parietal lobule, the left insula and the left lateral occipitotemporal cortex. Consistent with our previous study, some of these regions fell within the network showing less activation to sentences as a whole than to fixation (Figure 1). In addition, subcortically, the basal ganglia (putamen and caudate) were commonly activated to morphosyntactically and animacy semantic-thematic violated sentences relative to the real-world pragmatically violated sentences.
The left and right anterior medial fusiform cortices showed increased activity when each of the violated sentences was compared with the non-violated sentences (cluster number 10 in Tables 4, ,55 and and6,6, Figure 6, top row). An overlap map of each of these three contrasts confirmed that this was the only region commonly activated (Figure 6, bottom).
A qualitative comparison between the statistical maps yielded by the HRF-modeled datasets (as described above) and an FIR-modeled datasets showed very similar patterns of findings. The pattern of activity to each sentence type at each TR did not indicate the same increases or decreases of activity coming into play at different time points (data not shown). When FIR activity across multiple TRs was summed and the same contrasts as described above were examined, the distinctions in the neuroanatomical areas modulated to each sentence type relative to one another were very similar to those reported in the Results. This suggests that the differences in hemodynamic activity described above did, in fact, reflect true integrated differences in the amplitude of hemodynamic activation over the time-range of interest, rather than reflecting differences across sentence types in the time course of hemodynamic activity within the same network.
Real-world pragmatic violations uniquely activated the left anterior inferior frontal gyrus (BA 47) in comparison with all other sentence types. Both the morphosyntactic and the animacy semantic-thematic violations (relative to the non-violated sentences and the real-world pragmatic violations) activated a common network that included bilateral posterior inferior frontal gyri (BA 44), bilateral precentral gyri (BA 6), bilateral middle frontal gyri (BA 46/9), the left inferior parietal lobule (BA 40), the left insula, the left lateral occipito-temporal cortex and the basal ganglia. Of these, the left posterior inferior frontal gyrus showed relatively more activity in response to the animacy semantic-thematic violations than to the morphosyntactic violations, and the left inferior parietal lobule (BA 40) showed relatively more activity in response to the morphosyntactic violations than to the animacy semantic-thematic violations. Bilateral medial anterior prefrontal cortices (BAs 32/23/10) also showed more activity in response to the morphosyntactic violations than to the animacy semantic-thematic violations. Finally, relative to the non-violated sentences, all three types of violated sentences showed increased activity within bilateral anterior medial fusiform gyri (BA 36/37).
We used fMRI to determine which neuroanatomical regions were engaged upon encountering violations of three different types of relationships within simple, active sentences: (a) relationships between the propositional representation of the sentence and knowledge about the likelihood of the event described occurring in the real world, (b) semantic-thematic relationships, based on animacy constraints, between the Agent subject NP and the action described by the main verb, and (c) morphosyntactic inflectional relationships between the subject NP and the main verb. We compared each type of violated sentence with non-violated sentences, and with one another. Real-world pragmatic violations led to increased activity within the left anterior inferior frontal cortex. In contrast, sentences containing inanimate NP Agents that violated the functional requirements of the action denoted by the verb, engaged a widespread frontal-parietal-occipitotemporal network that overlapped with the regions engaged by morphosyntactic violations between verbs and their subject NPs. Finally, bilateral anterior ventromedial fusiform cortices were activated in contrasting all three types of violated sentences with the non-violated sentences.
Below, we begin by considering some of the potential confounds in interpreting these pattern of findings. We then discuss the patterns of activity to the real-world pragmatic and morphosyntactic violations in relation to previous studies. The main focus of the discussion will be on the pattern of neural activity to the animacy semantic-thematic violations and its overlap with activity to the morphosyntactic violations. We then consider activity to all types of violations relative to non-violated sentences. We conclude by highlighting some of the implications of our findings.
During self-paced word-by-word reading, previous studies have shown that it takes longer to process morphosyntactic violations (relative to non-violated verbs) at the point of the anomaly, while reading times at sentence-final words are similar in morphosyntactically-violated and non-violated sentences. In contrast, real-world pragmatic violations do not generally lead to increases in processing load at the point of the anomaly, but rather are associated with longer reading times to words following the critical word including the sentence-final word (De Vincenzi, Job et al. 2003; Ditman, Holcomb et al. 2007). Also, in speeded end-of-sentence acceptability judgment or anomaly detection tasks, participants are slower to detect real-world pragmatic violations than syntactic violations (Fodor, Ni et al. 1996; Kuperberg, Holcomb et al. 2003; Kuperberg, Caplan et al. 2006) or animacy semantic-thematic violations (Kuperberg, Caplan et al. 2006). This is consistent with the pattern of end-of-sentence RT judgments observed in the present study: RTs to the real-world pragmatically violated sentences were longer than to both the morphosyntactically violated sentences and the animacy semantic-thematic violated sentences. Taken together, these findings suggest a temporal distinction in processing syntactic, thematic and real-world pragmatic information; the output of a semantic memory-based analysis is delayed relative to that of a syntactic or semantic-thematic analysis, probably because it requires a longer search through semantic memory to determine that a sentence is implausible with respect to real-world knowledge than to decide whether or not a sentence is anomalous based on a more finite set of semantic-thematic or syntactic constraints between the verb and the subject NP.
In interpreting differences in the hemodynamic response between sentence types, our assumption is that the BOLD response captured and integrated differences in both the nature as well as the timing of neurocognitive processes engaged upon both initially detecting the different types of violations, as well as integrating or reanalyzing them with respect to their preceding contexts and the words that followed until the end of the sentence. Before making this assumption, however, we consider two potential confounds.
A first possibility is that the differences in BOLD activity observed across sentence types did not primarily reflect differences in neural activity during the sentences themselves, but rather reflected differences in decision times that occurred after the sentences had ended. We attempted to address this by modeling the hemodynamic activity to each of the four sentence types from the onset of the critical verb until the end of the sentence; the decision itself was modeled separately for each trial. Thus, we factored out any differences between conditions in judgment RTs on a trial-by-trial basis. A second possibility is that, rather than reflecting differences in the amplitude of the BOLD response between sentence types (multiple hemodynamic response functions integrated over the time span specified), the observed differences in the hemodynamic responses across conditions actually reflected differences in the timing of the same neurocognitive processes; in other words, they reflected differences in the latency of the same overall hemodynamic response. We addressed this possibility by conducting additional analyses in which we examined the hemodynamic response to each sentence type at successive TR intervals, without any assumptions about the time course of the hemodynamic response function. When we examined the patterns of activity to each sentence type at each of these TRs, we did not see the same increases or decreases of activity coming into play at different time points. Thus, it seems unlikely that any differences in activity between conditions simply reflected the same cognitive processes being picked up hemodynamically at different temporal stages.
The only region that showed more activity to the real-world pragmatically violated sentences, relative to all other sentence types, was the anterior left inferior frontal cortex (BA 47). This is consistent with previous findings (Kuperberg, Sitnikova et al. 2003; Hagoort, Hald et al. 2004), and with the known role of the left inferior frontal cortex, particularly its anterior portion, in retrieving information from within semantic memory (Wagner, Pare-Blagoev et al. 2001). As discussed above, we suggest that its increased recruitment to the real-world pragmatically violated sentences, relative to the other sentence types, reflected participants’ increased and more prolonged efforts to search and retrieve information from semantic memory in order to determine whether the event described in the sentence matched previously stored information, for example about what events are likely to occur at breakfast (see Kuperberg, Holcomb et al. 2003 for further discussion).
Additional activity observed within bilateral anterior inferior temporal cortices (fusiform/parahippocampal cortices) is also consistent with some previous fMRI (Newman, Pancheva et al. 2001; Kiehl, Laurens et al. 2002) and MEG (Halgren, Dhond et al. 2002) studies that have introduced semantic violations within sentences. It may reflect an implicit automatic activation of prestored semantic associative information (Alexander, Hiltbrunner et al. 1989; Chao, Haxby et al. 1999; Martin and Chao 2001; Wheatley, Weisberg et al. 2005; Gold, Balota et al. 2006; Kuperberg, Lakshmanan et al. 2007). As discussed below, however, activation of the anterior inferior temporal cortex was not specific to this contrast.
The animacy semantic-thematic violated sentences engaged a frontal/inferior parietal/basal ganglia network that was distinct from the anterior inferior frontal cortex engaged to the real-world pragmatic violations. This hemodynamic dissociation is broadly consistent with our previous ERP data that demonstrated clear distinctions in the electrophysiological signatures to these two types of conceptual violations: whereas real-world pragmatic violations evoke an N400 effect, thought to reflect, in part, a semantic memory-based process (Federmeier and Kutas 1999; Kutas and Federmeier 2000), these animacy semantic-thematic violations, similar to syntactic violations, fail to evoke an N400 effect, but rather evoke a P600 effect (Kuperberg, Sitnikova et al. 2003; Kuperberg, Caplan et al. 2006; Kuperberg, Kreher et al. 2007). The essential difference between these two sentence types is as follows: In the real-world pragmatically violated sentences, there is no problem in assigning the thematic role of Agent to the animate subject NP at the point of the verb; the violations arise only at the level of relating the event described to its preceding context and real-world knowledge. The animacy semantic-thematic violations, on the other hand, although also violating real-world knowledge, can be detected purely through computing the relationship between the Agent NPs and the actions described by the verbs.
As discussed below, there are several possible interpretations of this pattern of findings. In this section, we discuss the possibility that the engagement of the frontal/inferior parietal/basal ganglia circuitry to these violations reflected a continued analysis (or reanalysis) based, in part, on semantic-thematic constraints related to the action described the verb and the semantic properties of its NP Agent argument2. This interpretation is consistent with the activation of some of these regions (left inferior frontal, left ventral premotor, and parietal cortices) in association with German sentences in which thematic relationships between arguments were relatively difficult to assign because of a mismatch between the thematic argument hierarchy and syntactic word order (Bornkessel, Zysset et al. 2005).
We suggest that such a semantic-thematic analysis proceeds without the reader necessarily referring to semantic real-world knowledge about the likelihood of the event to occur in the world, explaining why, unlike the real-world pragmatic violations, the animacy thematic violations were not associated with an increased recruitment of left anterior inferior prefrontal cortices (see above). Indeed, based on our ERP studies, we have suggested that the increased demand for semantic-thematic analysis to these animacy thematic role violations may even ‘switch off’ a semantic memory-based analysis involving a referral to real-world knowledge (Kuperberg, Sitnikova et al. 2003; Kuperberg, Kreher et al. 2007).
Intriguingly, many of the regions constituting this frontal/inferior parietal/basal ganglia network to the animacy semantic-thematic violated sentences (relative to both non-violated and real-world pragmatically violated sentences) have been previously implicated in both carrying out and visually comprehending action (Rizzolatti, Fogassi et al. 2001), possibly through activity of ‘mirror neurons’ (di Pellegrino, Fadiga et al. 1992; Gallese, Fadiga et al. 1996). This raises the interesting possibility that the semantic-thematic analysis described above may be mediated by similar action-based neural systems that compute relationships between actors, actions and objects, around an event, in the visual world. This explanation would be consistent with cumulating evidence suggesting that visual events continuously influence the assignment of thematic roles during sentence comprehension (Tanenhaus, Spivey-Knowlton et al. 1995; Kamide, Altmann et al. 2003), and with our recent findings suggesting that the electrophysiological signatures of processing these types of animacy semantic-thematic violations and violations of real-world action are similar (Sitnikova, Kuperberg et al. 2003; Kuperberg 2007; Sitnikova, Holcomb et al. 2007; Sitnikova, Holcomb et al. 2007). More generally, this interpretation is also consistent with evidence for links between real-world action and language systems (Glenberg and Kaschak 2002; Arbib 2005; Kemmerer 2006; Fischer and Zwaan 2007).
We speculate that different parts of this action-based network played different roles in evaluating these Agent-Action semantic-thematic relationships. For example, activity within the left posterior inferior frontal gyrus (Broca’s area), together with adjacent motor and premotor cortices, may have reflected access to the meaning of the physical bodily actions denoted by many of the verbs used (Bak, O’Donovan et al. 2001; Hauk, Johnsrude et al. 2004)3. Activity within the posterior lateral occipito-temporal cortex which lies near area MT (specialized for perceiving biological motion) may have mediated the retrieval of representations of motion features of the actions described by some of the verbs in these sentences (Martin, Haxby et al. 1995; Damasio, Grabowski et al. 2001; Kable, Kan et al. 2005). The left inferior parietal activity may have reflected participants’ attempts to relate the action described by the verb with the functional properties or visuomotor affordances (Gibson 1979; Glenberg and Kaschak 2002) of the Agents executing these actions (Chao and Martin 2000; Buccino, Binkofski et al. 2001; Damasio, Grabowski et al. 2001; Fogassi, Ferrari et al. 2005). Activity within bilateral mid-dorsolateral prefrontal cortex may have reflected the computation of functional relationships between Agents and Action that were more complex, e.g. evaluating whether inanimate agents possessed the goals and motivation to carry out the actions described by the verb (see Ruby et al. 2002 for a similar interpretation). Finally, activity within the bilateral basal ganglia that are also known play a role in selecting and sequencing action (Vakil, Kahan et al. 2000) and that may be a source of the P600 effect produced by semantic-thematic violations (Kotz, Frisch et al. 2003), may have reflected participants’ attempts to repair these sentences, see Stowe et al. (2004) for a similar interpretation of basal ganglia function with respect to repairing syntactically ambiguous sentences.
The network activated in association with the morphosyntactically violated sentences (relative to the non-violated and real-world pragmatically violated sentences) overlapped largely with that activated by the semantically-thematically violated sentences. We offer several possible functional explanations for the common activation of this network in the next section. At this stage, we note that the increased activity to morphosyntactic violations (relative to non-violated sentences) within posterior inferior frontal and premotor cortices, as well as within the left middle frontal gyrus, is consistent with the pattern of activity to agreement violations reported by Ni et al. (2000). It also partially replicates the findings of our previous study using similar stimuli in which we reported the most robust activity to morphosyntactically violated sentences (relative to non-violated and real-world pragmatically violated sentences) within parietal cortices (Kuperberg, Holcomb et al. 2003)4. Activity within the basal ganglia is consistent with data from patients with subcortical lesions and with Parkinson’s disease that have implicated this region as a source of the P600 evoked by other types of syntactic violations (Friederici, von Cramon et al. 1999; Friederici, Kotz et al. 2003). It also accords with previous neuroimaging and patient studies that have associated the basal ganglia with aspects of syntactic processing (Ullman 2001; Friederici and Kotz 2003).
The overlapping pattern of hemodynamic activity to the animacy thematic-semantically violated and morphosyntactically violated sentences, and its neuroanatomical distinction from activity to the real-world pragmatically violated sentences, is particularly interesting as it parallels the pattern of previous ERP findings to the same types of anomalies: both produce P600 effects, as opposed to real-world pragmatic violations that produce N400 effects (Kuperberg, Caplan et al. 2006). In the ERP literature, explanations for why a P600 effect is evoked by both these types of anomalies vary in the degree to which the underlying neurocognitive processes are considered domain-general or specific to a particular level of linguistic analysis. They also vary in the degree to which they emphasize neurocognitive processes engaged upon anomaly detection versus continued analysis or reanalysis occurring as a consequence of anomaly detection. Below we consider some of these neurocognitive processes in relation to the neuroanatomical regions activated to these types of anomalies. However, it is important to note that these accounts are not mutually exclusive of one another. In addition, although it is tempting to infer that these hemodynamic responses reflect the underlying neural sources of the P600, just as in our previous parallel ERP/fMRI investigation (Kuperberg, Holcomb et al. 2003), we are very cautious about such an interpretation. This is because ERPs and fMRI index neural activity at very different time scales. As discussed above, BOLD activity reflected neural activity not only to the critical words but also additional neurocognitive processes occurring after the critical words that are not reflected by ERPs.
One explanation for why the semantic-thematic and morphosyntactic violations engaged overlapping regions (relative to the other sentence types) is that, as discussed above, they were both more easily detectable at the point of the critical word. Although the measured hemodynamic response integrated the neural responses across multiple words within the sentence, the immediate detection of the semantic-thematic and syntactic anomalies at the point of the critical word may have captured attention to a greater degree than to the non-violated or real-world pragmatically violated sentences and this attentional shift may have driven some of the hemodynamic response to these sentence types. Specifically, the increased recruitment of the parietal cortex to both types of anomalies may have reflected the known role of this region in allocating attention to popout or novel events (Ipata, Gee et al. 2006) and perhaps updating such events with their surrounding context. This is similar to explanations of neurocognitive processes proposed to underlie the well-known P300 event-related potential (Donchin and Coles 1988) that may be functionally linked to the P600 (Coulson, King et al. 1998; but see Osterhout, McKinnon et al. 1996 and Osterhout and Hagoort 1999) and that is likely to reflect activity within a widespread network, including some of the parietal and prefrontal regions activated here (Daffner, Scinto et al. 2003; Bledowski, Prvulovic et al. 2004)
Another explanation is that the network engaged to the animacy semantic-thematic and morphosyntactic anomalies reflected the detection of conflict. As outlined in the Introduction, the P600 effect may be triggered by a temporary conflict between the output of more than one parallel processing stream (Vissers, Chwilla et al. 2006; Kuperberg 2007). Kolk, Chwilla and colleagues have hypothesized that this detection of conflict is mediated by a non-domain-specific general monitoring network that, during language comprehension, serves the function of preventing comprehension errors (Kolk, Chwilla et al. 2003; Vissers, Chwilla et al. 2006; Kolk and Chwilla 2007). They have proposed that such monitoring is similar or analogous to the types of conflict monitoring that are thought to occur in the action domain and that lead to the production of an error-related negativity ERP component and increased activity within the anterior cingulate cortex (van Veen and Carter 2006; Taylor, Stern et al. 2007). In addition, they have also suggested that activity within the left inferior frontal gyrus may mediate such a general monitoring function, again linking this to its possible role in the resolution of conflict and selection amongst competing alternatives (Novick, Trueswell et al. 2005). Our findings are only partially consistent with this theory: the syntactically violated sentences (relative to non-violated sentences) did engage the anterior cingulate cortex, but the animacy thematically violated sentences (relative to the non-violated and real-world violated sentences) did not recruit this region. Both morphosyntactically and animacy thematic violations recruited the posterior (but not the anterior) portion of the left inferior frontal gyrus.
Other explanations for why overlapping neural systems were engaged to the animacy semantic-thematically and morphosyntactically violated sentences (relative to the other sentence types) focus on the types of continued analyses or reanalyses processes engaged after anomaly detection. Kolk et al.’s monitoring hypothesis holds that such reanalysis is very general in nature, involving all aspects of language processing: phonological, orthographic, semantic, and syntactic (Vissers, Chwilla et al. 2006; Kolk and Chwilla 2007). It is possible that some of the activity observed in association with these violations reflected such a general reanalysis, although this account might predict even more widespread activity to that observed here. In order to test this hypothesis more directly, one would need to determine whether the same network is activated by other types of violations that lead to conflicts between alternative representations, such as orthographic violations within highly predictable sentences (Vissers, Chwilla et al. 2006).
It may be that the regions activated to the animacy semantic-thematic and morphosyntactic violations reflected more specific reanalysis processes that focused on more specific levels of language processing. Above we considered the possibility that encountering the animacy semantic-thematic violations led to an increased or prolonged analysis that was based partially on semantic-thematic constraints related to the action described by the verb and the semantic properties of its NP Agent argument. It is possible that, like animacy semantic-thematic violations, the inflectional morphosyntactic violations also triggered such a semantic-thematic analysis. This explanation is similar to a proposal by Kaan and colleagues (Kaan, Harris et al. 2000) who suggested that participants may attempt to semantically-thematically integrate a morphosyntactically violated verb with the subject of a sentence, but that such integration is rendered more difficult, if not impossible, because of the grammatical number mismatch; see also discussion by Kuperberg et al. (2006). From a theoretical perspective, a link between processing aspects of syntax and processing semantic-thematic relationships would be interesting as the latter lies at the semantic/syntactic interface: although semantic in nature, thematic relationships, based on properties such as animacy, represent the part of the semantic system that has implications for combination of words around a verb (Pinker 1989; Goldberg 1995; Levin and Rappaport Hovav 2005).
Interestingly, as in our previous study, some of the regions modulated by both animacy semantic-thematic and morphosyntactic violations (relative to other sentence types) overlapped with those that showed ‘deactivation’ when the sentences were compared with the low-level fixation condition (see Figure 1)5. The precise significance of this observation remains unclear but it suggests that the types of neurocognitive processes mediating either a more specific semantic-thematic analysis/reanalysis or a more general detection, monitoring and reanalysis, may be suppressed during the first half of the sentence and only come into play upon encountering these anomalies.
All three types of violated sentences engaged anterior ventro-medial fusiform cortices, relative to the non-violated sentences. There has been much speculation about what role this region plays during language comprehension. It has been touted as playing a particular role in storing and activating semantic associative, categorical and contextual relationships within semantic memory (Mummery, Shallice et al. 1999; Rossell, Price et al. 2003; Bar 2004; Wheatley, Weisberg et al. 2005; Gold, Balota et al. 2006; Kuperberg, Lakshmanan et al. 2007). It may be one source of the N400 effect as indicated by some intracranial electrode studies (McCarthy, Nobre et al. 1995; Nobre and McCarthy 1995). Other studies, however, have emphasized the involvement of anterior temporal cortices more specifically in sentence comprehension beyond the single word (Mazoyer, Tzourio et al. 1993). and some have suggested that parts of this overall region may even be specialized for aspects of syntactic processing (Dronkers, Wilkins et al. 1994; Humphries, Love et al. 2005; Stowe, Haverkort et al. 2005). The ventral anterior temporal cortex may therefore be sensitive to multiple types of linguistic information rather than only one type, and be involved in integrating these multiple sources of information together to construct higher order propositional meaning. Indeed, its common engagement to real-world pragmatic, morphosyntactic and semantic-thematic violations in the present study is consistent with our findings in an early blocked-design study in which we demonstrate that the ventral temporal fusiform cortex was the only region commonly activated to real-world pragmatic, selection-restriction and subcategorization violations between verbs and object NPs, each in comparison with non-violated sentences (Kuperberg, McGuire et al. 2000).
We have demonstrated similarities in the pattern of hemodynamic response to animacy semantic-thematic violations and morphosyntactic violations between verbs and subject NP arguments, and have shown that this response is distinct from that engaged to real-world pragmatic violations of these verbs. This parallels the overall pattern of ERP findings seen to the same types of violation – a P600 effect to animacy semantic-thematic and syntactic violations, but an N400 effect to real-world pragmatic violations (Kuperberg 2007). We have suggested that the left anterior inferior frontal cortex to real-world pragmatic violations acts to search, retrieve and integrate real-world knowledge into an incoming sentence representation. In contrast, we have discussed several possible non-mutually exclusive neurocognitive processes that may be reflected by activity within a widespread frontal/inferior parietal/basal ganglia network to both semantic-thematic and morphosyntactic violations. Most specifically, we have suggested that such activity may, in part, reflect a continued thematic-semantic analysis or reanalysis of such sentences that may involve attempts to relate the action described by the verb to the affordances, goals and other relevant semantic properties of its Agent NP arguments. In keeping with the ERP literature, our assumption is that both the neurocognitive processes of semantic memory-based integration and semantic-thematic prolonged analysis or reanalysis are particularly taxed when their respective relationships are violated. This, however, is an empirical question. Future studies will determine whether the same regions are also engaged at points of processing difficulty during the processing of non-violated sentences (of note, both N400s and P600s are indeed evoked at points of semantic unexpectancy and syntactic ambiguity respectively during normal sentence processing).
Our findings indicate that, at a neuroanatomical level, the brain respects a distinction between two different aspects of meaning: the semantic relationship between the propositional content of a sentence and our knowledge about what is likely in the real world, and the semantic-thematic relationship between a verb and its arguments within a sentence. Thus, at a neural level, a division within the semantic system into components that are relevant and irrelevant for action (Martin, Haxby et al. 1995) appears to play out during sentence processing. Most intriguingly, these data are consistent with ERP findings in suggesting that violations of semantic-thematic relationships, based on animacy constraints, between a verb and its preceding Agent argument are treated by the brain as similar to morphosyntactic inflectional violations between a verb and its subject NP argument (Kuperberg, Caplan et al. 2006; Kuperberg 2007).
This research was supported by NIMH (R01 MH071635) and the Institute for Mental Illness and Neuroscience Discovery (MIND). Gina R Kuperberg was also supported by a NARSAD (with the Sidney Baer Trust) and a Claflin Distinguished Scholars Award from Mass. General Hospital. We thank Marianna Eddy and Fujiro Ozawa for their help in data acquisition, and David Caplan, Mante Nieuwland and Tali Ditman for their comments on the manuscript.
1The term ‘pragmatic’ is used to describe these violations following Marslen-Wilson, Brown et al. (1988) and Tyler (1992) who originally discussed violations of real-world knowledge in these terms, and in order to be consistent in terminology with our previous studies using the same types of sentences (Kuperberg, Holcomb et al. 2003; Kuperberg, Sitnikova et al. 2003; Kuperberg, Caplan et al. 2006; Kuperberg, Kreher et al. 2006; Kuperberg, Sitnikova et al. 2006; Kuperberg, Kreher et al. 2007; Kuperberg 2007). However, this is an umbrella use of the term, ‘pragmatic’, referring to the fact that such real-world knowledge is situationally based and context dependent, rather than traditionally ‘linguistic’. Note that this descriptive and general use of the term is distinct from ‘pragmatics’ when used to refer to the relationship between sentence meaning and speaker’s meaning and/or to refer to a variety of phenomena at the level of discourse.
2Based on a series of ERP studies, we have suggested that such prolonged semantic-thematic processing is triggered by a conflict between the actual syntactic assignment of the subject NP (“eggs”) to the role of Agent at the point of the verb, and two types of semantic processes: (a) a semantic-thematic process (either heuristic or combinatorial) that initially assigns the role of Theme to the subject NP “eggs” based on its inanimate properties (Kuperberg, Sitnikova et al. 2003; Kuperberg, Caplan et al. 2006; Kuperberg, Kreher et al. 2007; Kuperberg 2007), and (b) a semantic memory-based process that recognizes the close semantic associative relationship between “eggs” and “eat” (Kim and Osterhout 2005; Kuperberg, Caplan et al. 2006; van Herten, Chwilla et al. 2006; Kuperberg 2007).
3Although, we cannot exclude the possibility that some of this motor activity reflected participants’ button-press responses, it seems unlikely that such motor responses can explain all differential activity within these regions. First, participants also made motor responses to the non-violated and pragmatically violated sentences, but activity within motor cortices was significantly less to both these sentence types than to the animacy and syntactically violated sentences. Second, we modeled hemodynamic activity to each of the four sentence types from the onset of the critical verb until the end of the sentence; the decision itself was modeled separately for each trial. It will, however, be important to determine whether these findings can be replicated as participants simply read the sentences without making any judgments.
4In this previous investigation, the posterior inferior frontal gyrus, left premotor cortex and the left posterior occipito-temporal cortex were also activated to morphosyntactically violated sentences relative to non-violated sentences at subthreshold levels of significance, as shown in Figure 6 in G.R. Kuperberg et al. (2003a). One reason for the discrepancy in findings may be the method of analysis used: in our previous study, we focused on the modulation of the hemodynamic response at and straight after its peak. By focusing in on activity at particular time points, we may have diminished power to capture activity within regions that were not modulated precisely at these time points. In the present study, we modeled the data from the onset of the critical word until end of the sentence using a hemodynamic response function. We also averaged data across participants using a random effects model, rather than a fixed effects model (as in our previous study), perhaps making our current findings more generalizable.
5In our previous study in which the trials followed on from one another at a fixed rate, we considered the possibility that differential degrees of deactivation may have reflected the relatively longer periods of rest in between trials, arising because participants were generally fastest to make acceptability judgments to the morphosyntactically violated sentences (G. R. Kuperberg et al., 2003a). This relatively trivial explanation for differential degrees of deactivation seems less likely to account for the pattern of findings in the current study as participants’ acceptability decisions moved them on to subsequent trials, eliminating any rest periods in between trials.