|Home | About | Journals | Submit | Contact Us | Français|
Active reading requires coordination between frequent eye-movements (saccades) and short fixations in text. Yet, the impact of saccades on word processing remains unknown, as neuroimaging studies typically employ constant eye fixation. Here we investigate eye-movement effects on word recognition processes in healthy human subjects using anatomically-constrained magnetoencephalography, psychophysical measurements, and saccade detection in real-time. Word recognition was slower and brain responses were reduced to words presented early vs. late after saccades, suggesting an overall transient impairment of word processing after eye-movements. Response reductions occurred early in visual cortices and later in language regions, where they co-localized with repetition priming effects. Qualitatively similar effects occurred when words appeared early vs. late after background-movement that mimicked saccades, suggesting that retinal motion contributes to postsaccadic inhibition. Further, differences in postsaccadic and background-movement effects suggest that central mechanisms also contribute to postsaccadic modulation. Together, these results suggest a complex interplay between visual and central saccadic mechanisms during reading.
In reading, rapid eye-movements (saccades) bring words onto the high acuity fovea for detailed analysis. Current psycholinguistic models of reading are based on eye-movement measurements of the position, duration, and sequence of eye fixations in text (Reichle et al., 2003; Rayner and Reichle, 2010), but fail to consider the impact of the eye-movement itself on word processing. Moreover, neuroimaging studies typically employ words presented during constant fixation (Sereno and Rayner, 2003), thus eliminating saccadic influences. Studies of active vision in animal models and humans indicate, however, that perceptual and cognitive processes are modified before, during and after an eye-movement via both central and visual mechanisms (Wurtz 2008). Similar neural pathways and mechanisms may be crucial for successful, fluent reading.
Saccades challenge the visual system by producing abrupt changes in the retinal stimulus as the visual field image moves over the retina. Our brain ignores the retinal motion and compensates for the repositioning of gaze, generating perceptual constancy. Psychophysical studies in humans and electrophysiological data in primates indicate that, although not perceived, visual stimulation during saccades continues to be processed in the visual system, influencing processes at re-fixation (Ibbotson and Cloherty, 2009). The extent to which retinal motion modulates word processing in reading remains unknown.
In addition to such visual effects, central mechanisms mediated by brain regions that control eye-movements and attention alter visual processing after saccades. In primates, thalamic recordings typically reveal transsaccadic suppression followed by enhancement (Reppas et al., 2002; Royal et al., 2006). This pattern has been identified in a number of cortical visual areas (Ibbotson and Krekelberg, 2011), although results remain variable at the single-neuron level and controversial (Wurtz, 1969; DiCarlo and Maunsell, 2000; Gawne and Martin, 2002; MacEvoy et al., 2008; Ibbotson et al., 2008). Central suppression, reported from ~100 ms before onset to ~50 ms after the end of saccades, is thought to decrease the sensation of image motion in active vision (Burr et al., 1994; Ross et al., 2001). Postsaccadic facilitation lasting ~200–400 ms presumably amplifies visual sensitivity at fixation (Ibbotson and Cloherty, 2009). In the absence of behavioral measures it is not known, however, if and how these opposite neural effects, individually or together, alter perception. Further, there is yet no evidence that central postsaccadic mechanisms modulate word processing.
Here, we use anatomically-constrained magnetoencephalography (MEG) to investigate the impact of saccades on the spatiotemporal dynamics of word processing during a one-back word recognition task. We performed two sets of parallel experiments in the same participants. Firstly, we compared behavioral and brain responses to words presented early vs. late after natural saccades detected in real-time. Secondly, we simulated the retinal motion associated with saccades by moving the background image, and compared responses to words presented early vs. late after such movement. We examined whether repetition priming effects, previously implicated in lexical and semantic processing, co-localize with post-saccadic effects. Our results provide the first evidence for transient modulation of word processing after saccades, with contributions from both visual effects associated with image motion and central mechanisms.
Participants’ approval was obtained and informed consents were signed before each measurement. Seven healthy right-handed adults (5 males/2 females) underwent two MEG sessions for Experiments 1–2, and also a structural MRI scan.
During a 1-back word recognition task, subjects waited for an auditory go-cue at the beginning of each trial to make a saccade between two fixed strings of five-crosshairs, 10 degrees apart (Fig. 1A). Saccades were detected in real time using the horizontal EOG signal and triggered the subsequent foveal word appearance at the new fixation either 76 ms (‘early’ postsaccadic latency condition) or 643 ms (‘late’ condition) later. These latencies ensured in individual subjects that words appeared at re-fixation only after the end of saccades (see offline computations below), allowing control of stimulus timing (onset and duration) across conditions. Latencies include a fixed delay of 33 ms between the stimulus trigger pulses sent by the Presentation program and the stimulus appearance on the projection screen. The stimuli were five-letter novel words (50%) and one repeated word presented for 250 ms. Brain activity evoked by saccades alone was assessed from ‘no-stimulus’ trials wherein a string of five Xs presented 1243 ms after the saccade detection marked the end of trial. Early, late and no-stimulus trials appeared in pseudo-randomized order, with 1300–1500 ms interstimulus interval. Subjects were instructed to read the stimulus silently and respond as accurately and quickly as possible by pressing a button with their right index finger if the stimulus was the same as that in the previous trial (10%, match trials), and another with their left finger otherwise (90%, non-match trials). We collected 110 trials/condition in 20 blocks, with short 1–3 min. breaks between blocks and a total recoding time of 90 min. Two additional blocks were used to familiarize the subject with the task before recordings. During recordings, subjects rested their upper jaw on a custom-made bite-bar while comfortably leaning their head against the back of the dewar’s helmet; this approach maintained a steady position of the head relative to the MEG sensors within as well as across recording sessions.
Words were balanced across conditions with respect to lexical frequency (Kucera and Francis 1967), concreteness index, and stress index. Stimuli were presented on a computer-driven projection and subtend a < 5 degrees visual angle; the whole projection screen subtends 47 degrees. In our approach, we rationalize that the ‘late’ postsaccadic latency condition is equivalent to a static fixation condition because visual and central influences related to saccades are expected to fade > 600 ms following the eye movement.
Occasionally, the electronic circuit did not detect a saccade and as a result failed to trigger the word appearance. For these trials (< 10% of all trials), feedback was provided immediately by the appearance of the word ‘error’ at the missed saccade target location, which cued the subject to correct gaze by fixating the missed location and await a new trial.
To examine the contributions of visual effects attributed to the retinal image motion during saccades vs. central postsaccadic influences, in parallel experiments we examined the brain activity to words presented after background motion that mimicked an eye movement (Fig. 1B). Following the auditory cue, subjects were instructed to maintain their gaze stationary in the center of the screen while the two strings of five-crosshairs 10 degrees apart were moved to mimic the retinal motion during an eye movement. Words were presented foveally at fixation either early (59 ms) or late (626 ms) after the background motion offset. Based on data in Experiment 1, for each subject we computed the mean and variance of the saccade onset latencies with respect to the auditory cue. Random numbers following this distribution were generated and used here to set the onset time of the image motion relative to the auditory cue. Motion velocity and duration matched the average values obtained for saccades in preliminary experiments. Experiment 2 paralleled Experiment 1 in every other aspect regarding word stimuli, inclusion of no-stimulus trials, number of trials per condition and number of blocks, collection of behavioral data as well as task instructions.
Whole-head MEG (306 sensors arranged in triplets of two orthogonal planar gradiometer and a magnetometer, dc-SQUID Neuromag Vectorview system, Elekta-Neuromag Ltd., Helsinki, Finland) was recorded in a magnetically and electrically shielded room, band-pass filtered (0.01–200 Hz), and digitized at 600 samples/s. The horizontal and vertical components of eye-movements were recorded concurrently with MEG using two pairs of bipolar EOG electrodes. For subsequent coregistration with the structural MRI and to record the position of the head relative to the sensor array, the locations of four head-position indicator (HPI) coils attached to the scalp, three fiducial landmarks (nasion and auricular points), and additional scalp surface points were digitized using a 3 Space Fastrak system (Polhemus, Colchester, VT) integrated with the Vectorview system.
Saccades were detected in real time using the EOG signal for horizontal eye movements, which was sent online to a saccade-detection circuit employing filters and a threshold comparator. Saccade detection triggered the stimulus presentation with different delays adjusted in preliminary experiments so that the display change occurred only after the end of the saccade.
The times of word presentation relative to saccades were confirmed offline. The beginning and endpoint of saccades were computed for each condition based on the low-passed filtered and averaged EOG signal, using in-house software in MATLAB (Mathworks, Natick, MA). An automated algorithm started at peak velocity and searched the first derivative of the eye-position trace backwards and forward to fixation. Saccade onset and offset were defined as the first point in time preceding peak velocity and the last point following peak velocity, respectively, for which the slope of the eye-position trace was larger than 3.3 SDs from the mean baseline value (p < 0.001). These computations confirmed that in the early condition words appeared between 103–145 ms (mean ± s.e., 119.9 ± 2.9 ms) after the onset and 2 – 72 ms (31.5 ± 3.3 ms) after the offset of saccades; in the late condition words appeared between 669–718 ms (686.7 ± 3.0 ms) after the onset and 568 – 640 ms (597.4 ± 3.6 ms) after offset of saccades.
MRI recordings (1.5 T Sonata scanner, Siemens Medical Solutions, Erlangen, Germany) consisted of two structural 3D magnetization prepared rapid gradient echo (MPRAGE) scans (TR/TE/TI = 2.73 s/3.31 ms/1 s, flip angle = 7°, 128 × 1.3 mm-thick sagittal slices at an in-plane resolution of 1 mm2) and two multi-echo multi flip angle (5° and 30°) fast low-angle shot (FLASH) scans (TR/TE = 20 ms/(1.8 + 1.82 × n) ms, n = 0–7). The standard MPRAGEs were used for individual cortical surface reconstructions with FreeSurfer (http://surfer.nmr.mgh.harvard.edu) and for registering MEG data to the individual subject’s anatomy (Dale et al., 1999; Fischl et al., 1999a,b, 2001). The FLASH sequences were used to compute the inner skull surface for the boundary element model (BEM). This information was then employed in computing the MEG forward solution. Cortical surfaces were inflated to visualize both gyri and sulci and to morph the hemispheres into a sphere for inter-subject registration based on the sulcal-gyral pattern (Fischl et al., 1999a,b).
Trials free of eye blinks or other artifacts and with correct subject responses were low-passed filtered at 40 Hz and averaged for each stimulus condition (N ~ 100 trials). Average waveforms generated by saccades or background movement alone were obtained from no-stimulus trials, and subsequently subtracted from each stimulus waveform (Fig. 2B–C), resulting in a differential signal that allowed comparison across conditions of responses generated by words alone. For the saccade experiment this approach effectively eliminated the artifacts caused by horizontal saccades.
This differential MEG signal was further analyzed here to estimate the corresponding patterns of brain activity (current sources) across cortical locations and time. MEG signals measure the magnetic fields generated by synaptic currents in the brain. These current sources (dipoles) were estimated using the linear minimum-norm estimate (MNE) approach (Hamalainen and Ilmoniemi, 1994; Dale and Sereno, 1993) and information of the head anatomy obtained from anatomical MRI data using the MNE software (http://www.nmr.mgh.harvard.edu/martinos/userInfo/data/sofMNE.php). The solution space for the estimated currents was constrained to the gray/white matter boundary reconstructed for each individual from the structural MRI, which was subsampled to 4098 dipole elements per hemisphere with ~5-mm spacing (Dale et al., 1999; Fischl et al., 1999a). A forward solution for the source space was computed using a one-layer BEM model (Hämäläinen and Sarvas, 1989). The noise covariance matrix was calculated from 200-ms baseline periods prior to saccades (Experiment 1) or background motion (Experiment 2) in individual trials. The noise covariance matrix and the forward solution were used to create a linear inverse operator (Dale et al., 2000) that was applied to the data at each time point, producing time-courses of activity at each cortical location. Current orientations were approximately constrained to be perpendicular to the cortical surface by setting source variances for the transverse components to be 0.6 times the variance of the normal components (Lin et al., 2006). For analysis across participants, the inverse solutions were registered to the average cortical surface computed across all subjects (Fischl et al., 1999b). The current estimate at each cortical location was divided by the estimated baseline variance, resulting in an F-like statistic (Dale et al., 2000). The square root of the F statistic, which is a signal-to-noise ratio estimate, is analogous to a z-score and allows the visualization of results as dynamic statistical parametric maps (dSPM). The dSPM identifies locations where the current strength estimates are most reliable based on their signal-to-noise ratio.
Effects of saccades, background movement and word repetition were quantified in regions of interest (Tables 1–2) selected a priori based on their implication in previous studies of visual word recognition. ROIs were manually drawn on the omnibus dSPM solution combining all subjects and conditions from both experiments (Marinkovic et al., 2003; Wehner et al., 2007) and represented on the average brain of all subjects. Each ROI included at least 30 contiguous sources (range 30–58) that had the maximum dSPM value within the 70–500 ms time window significant at p uncorrected < 0.0001; this corresponds to a corrected p < 0.014 (0.0001 × 4098 sources per hemisphere/30 sources). The same ROIs were used for all subjects by automatic spherical morphing of original labels to individual subjects (Fischl et al., 1999b). Anatomical labels generated in Freesurfer for occipital pole, occipitotemporal gyrus and lateral occipitotemporal sulcus showed large activity levels that met our criteria and were used here instead of manually-drawn labels.
The absolute current values within an ROI were averaged across voxels at each time point, producing regional time-courses for individual subjects and conditions. For an individual ROI and time interval of interest fixed across subjects, we compared activity across conditions using a single repeated-measure three-way ANOVA with within-subject factors of word presentation time (early or late after saccades or background movement), stimulus type (novel or repeated words) and movement direction (right or left).
Time windows were selected based on previous MEG (Tarkiainen et al., 1999; Dhond et al., 2001; Marinkovic et al., 2003; Solomyak and Marantz, 2009; McDonald et al., 2010; Lehtonen et al., 2011) and intracranial studies (Halgren et al., 1994; Halgren et al., 2006) that reveal approximate time courses associated with visual feature processing (80–120 ms), early orthographic processing (120–160 ms) and word-form identification (165–215 ms), lexical access (~200–240 ms) and semantic processing (~300–500 ms); and repetition priming effects (~200–240 ms; ~240–300 ms; ~300–500 ms).
Seven healthy volunteers performed a one-back word recognition task while reading words presented foveally after horizontal eye movements 10 degrees apart (Fig. 1A). Both novel words (50% of trials) and one repeated word were presented for 250 ms either early (76 ms) or late (643 ms) after saccade detection in real time using the electrooculogram (see Methods). The selection of these time windows allowed us to control the duration of word viewing at the fixation following a saccade, ensuring that, in each individual, words appeared either immediately after the end of the saccade (range 2 – 72 ms, 31.5 ± 3.3 ms), when we expected postsaccadic influences, or ~600 ms later, when we expected these influences to fade. Subjects’ task was to determine whether the word at fixation was the same as that in the previous trial (10% match-trials) or different (90% non-match trials). High overall accuracy scores were observed in both match (average percent correct = 80.06%) and non-match trials (99.79%). All further analyses were based on the large number of non-match correct trials that were free of blinks and other artifacts not related to eye movements.
Percent correct scores and reaction times (RTs) were analyzed with repeated-measure three-way ANOVAs with within-subject factors of word presentation time (early or late after saccades), word type (novel or repeats), and saccade direction (right or left). There were no main effects or interactions on accuracy (all F(1, 6)s < 4.1, p’s > 0.05). Fig. 1Ac illustrates average RTs for individual conditions. RTs were slower when words appeared early vs. late after a saccade, F(1, 6) = 22.01, p < 0.004, suggesting that word recognition is overall temporarily impaired after an eye movement. Slower RTs were also observed for right vs. left saccades, F(1, 6) = 19.41, p < 0.005, with an additional interaction between saccade direction and word presentation time, F(1, 6) = 8.9, p < 0.007. This stronger modulation for right saccades is likely to have arisen because words appeared on average 21 ms earlier after the end of right than left saccades (range 14 – 30 ms, see Methods). In addition, slower RTs were found for repeated vs. novel words, F(1, 6) = 50.32, p < 0.0004.
In parallel experiments, the same subjects read words presented foveally during constant fixation, either early (59 ms) or late (626 ms) after the end of background movement that mimicked saccades (Fig. 1B). As above, we found high overall accuracy scores for both one-back match (80.16%) and non-match trials (99.61%). Further analyses were again based on non-match correct trials (90% of all trials) that were free of artifacts. Accuracy scores and RTs were analyzed with repeated-measure three-way ANOVAs with within-subject factors of word presentation time (early or late after background movement), word type (novel or repeats), and movement direction (right or left). We found that both accuracy and RTs varied across conditions. Accuracy was worse to words presented early vs. late after movement, F(1, 6) = 28.01, p < 0.002 (trials with incorrect responses < 2.5% for individual conditions). Similar to the saccade task, RTs were slower when words appeared early vs. late after movement (Fig. 1Bc), F(1, 6) = 50.92, p < 0.0004, indicating that saccade-like retinal motion disrupts subsequent word processing. Here too, slower RTs were found for repeated vs. novel words, F(1, 6) = 24.01, p = 0.0027. An additional interaction was found between word type and word presentation time, F(1, 6) = 8.39, p < 0.008, reflecting larger background movement effects on repeated vs. novel words. There was no main effect of movement direction (p > 0.05).
To directly compare the behavioral effects of natural saccades and background movement (measured as changes in RTs for early minus late word presentation conditions), we performed a three-way repeated-measure ANOVA with factors of experiment, word type and image movement direction. There was no main effect of experiment (p = 0.49), indicating similar increases in RTs to words presented early vs. late after saccades (74.8 ms) and background movement (65.3 ms). Consistent with the results reported above, we found a significant interaction between experiment and movement direction (F(1, 6) = 10.34, p < 0.004) reflecting the slower RTs after right vs. left natural saccades, but no such effects of direction in Experiment 2. This analysis also confirmed the slower RTs on repeated vs. novel words, F(1, 6) = 6.73, p < 0.041.
Previous studies report changes in spontaneous activity and visual responses related to central effects around the time of saccades in early visual areas (Sylvester et al, 2005; Royal et al., 2006; Rajkai et al., 2008; Cloherty et al., 2010), which could modulate the variability in RT to words following saccades, rendering it different from that following background movement (Burr et al., 1994; Diamond et al., 2000; Watson and Krekelberg, 2011). We compared the RT standard deviations using a repeated-measure four-way ANOVA with within-subject factors of experiment, latency, word type and image movement direction. There was no main effect of experiment (F(1, 6) = 2.19, p = 0.19) or interactions (p’s > 0.05), indicating similar variability in RTs after saccades and background movement.
Average MEG waveforms were computed for each condition in each subject. Figure 2B illustrates a representative example of responses to novel words presented early and late after the end of right saccades. Averages from no-stimulus trials illustrated activity generated by saccades alone as well as the eye-movement related artifacts (see Methods). This latter waveform was subtracted from the waveform in each condition (Fig. 2B), resulting in a differential signal that reflects responses to words alone. Due to the stereotypical nature of eye movements across trials, this approach effectively eliminated the eye-movement related artifacts.
This differential MEG signal was further analyzed to estimate the patterns of cortical activity across locations and time using a distributed source modeling approach that constrained current sources to the cortical surface of each participant reconstructed from structural MRI (Dale et al. 1993). Noise-normalized dynamic statistical parametric maps (dSPMs) (Dale et al., 2000) were computed for individual conditions to evaluate the statistical significance of estimated responses relative to pre-stimulus baseline activity. Figure 3 illustrates snapshots of average dSPMs across subjects, at selected latencies after word presentation. The evolution of the activity patterns was consistent with that reported in previous MEG studies of visual word processing (Dhond et al., 2001; Marinkovic et al., 2003). Briefly, activity began in occipital pole (peak at ~95 ms), and subsequently spread anteriorly within the ventral visual stream, recruiting the occipitotemporal and posterior superior temporal regions, followed by ventral and anterior temporal cortices, and prefrontal cortex (early peaks between ~140–155 ms). Subsequent peaks of left lateralized activity were found in occipitotemporal cortex at ~170 ms, in anterior temporal cortex at ~210 ms, and within the 300–500 ms (N400) time window, distributed across anterior occipitotemporal, anterior temporal and prefrontal cortices. For words presented early vs. late after saccades, we observed a prominent reduction in the response within multiple regions, at multiple stages of cortical processing, beginning with the earliest stage in occipital pole. These reduced responses, as well as repetition effects, were further quantified in regions of interest (ROIs), as described below.
ROIs were selected on the basis of previous studies of visual word processing and they all exhibited large activity values here (see Methods). Regional time-courses of estimated currents were computed in each ROI and individual subject, by averaging values across all voxels. Figure 4 illustrates regional time-courses averaged across subjects for early vs. late word presentation time. For each ROI, effects were evaluated in response windows selected a priori based on previous MEG and intracranial studies of the time course of visual word recognition (see Methods). Relevant time windows included 80–120 ms for early visual processing in occipital pole, 120–160 ms for the earliest response phase in downstream ROIs, 165–215 ms for orthographic processing and word-form access in occipitotemporal cortex, 190–240 ms for lexical processing and 300–500 ms (N400) for semantic processing in distributed language networks including inferior and superior temporal and prefrontal cortices. For an individual ROI and time range of interest, fixed across subjects, a single repeated-measure three-way ANOVA with within-subject factors of word presentation time (early or late), word type (novel or repeated) and saccade direction (right or left) was performed to compare currents across conditions. The detailed comparisons of all individual tests for the main effects of word presentation time and word repetition are presented in Tables 1. Effects of word presentation time, i.e. post-saccadic effects, are summarized in the text below.
Postsaccadic effects were similar for right and left saccades, and also for novel and repeated words (e.g., Fig. 6). For the repeated word condition, we repeatedly presented a single word either early or late after right and left saccades in order to reliably assess postsaccadic effects in early visual areas that are sensitive to the visual attributes of the stimulus.
In occipital pole, saccades diminished both the earliest (80–120 ms) as well as later phases of the response (e.g., 120–160), suggesting an overall suppression of activity to words entering the fovea via saccades (Fig. 4, Table 1). Attenuated activity from 120–160 ms was found in bilateral ventral occipitotemporal (vOT) junction, bilateral occipitotemporal (OT), right anterior inferior temporal (IT), bilateral anterior superior temporal sulcus (STS), right posterior STS, and bilateral inferior Sylvian fissure (SF); a similar trend was observed in left anterior IT, left posterior STS, right planum temporale, and left inferior prefrontal cortex (IPFC).
Unlike the strong effects at earlier processing stages, activity in OT cortex (both fusiform area and lateral occipitotemporal sulcus) from 165–215 ms was reduced only at trend level (p < 0.1). In downstream areas, reduced activity from 190–240 ms occurred in the left hemisphere in anterior STS, inferior SF, and at trend level in IPFC; and in the right hemisphere in posterior STS and planum temporale. Reduced activity from 250–350 ms was found at trend level in left planum temporale, during a pronounced, left-lateralized response component; and also in right posterior STS. Activity from 300–500 ms (N400) was attenuated within a distributed temporofrontal network including bilateral OT and anterior IT, right anterior STS, right posterior STS, right inferior SF and right IPFC. Together, these results demonstrate postsaccadic effects of variable strength in both visual and higher cortical areas during early and late phases of the word-evoked response.
To evaluate effects of repetition, we focused on response windows that have been previously associated with repetition priming, from 190–240 ms, 240–300 ms and 300–500 ms (see Methods); we also report results at earlier spatiotemporal stages of the response described above (Table 1). Average responses across subjects to novel vs. repeated words in Figure 5 illustrate robust repetition effects left-lateralized in occipitotemporal, anterior temporal, superior temporal and prefrontal regions, within the same ROIs that revealed significant postsaccadic modulation. Below we evaluate early (< 300 ms) and late (300–500 ms) repetition effects.
We found early repetition enhancement (repeats > novel) from ~120–160 ms in left OT cortex, with a similar tendency in occipital pole, left vOT junction and anterior temporal regions. This was followed by repetition suppression (novel > repeats) from 190–240 ms in bilateral planum temporale and left inferior SF. Robust left-lateralized repetition suppression from ~240–300 ms occurred in left OT, left anterior IT, left anterior STS, and left inferior SF, as well as bilaterally in planum temporale.
Repetition suppression extended to 300–500 ms response range in anterior temporal and inferior prefrontal regions. These late effects were of smaller magnitude and occurred relatively earlier (e.g., peaks at ~350 ms) compared with previous reports. Significance was evaluated within four 50-ms windows from 300–500 ms, and was found in left anterior STS (300–400 ms), left inferior SF (300–400), and left IPFC (300–400 ms). In addition, from 400–450 ms repetition enhancement (repeat > novel) was found in right anterior IT. No significant interactions were found between postsaccadic and repetition effects. Overall, these results indicate co-localization of postsaccadic and repetition effects previously associated with lexical and semantic processing.
Figure 2C illustrates average MEG waveforms to novel words presented early and late after background movement for the same subject as in 2B. Similar to the analysis in Experiment 1, waveforms from no-stimulus trials that reflect activity generated by movement alone were subtracted from the waveform in each condition; this resulted in differential signals that reflect responses to words alone (Fig. 2C). Estimated activity patterns were qualitatively similar to those in the saccade task (Fig. 3). Again, we found reduced activity for words presented early vs. late after background movement, consistent with a contribution of retinal image motion to the overall suppressive postsaccadic effects.
Similar to the saccade task, diminished responses, although varying in magnitude, were observed for words presented early vs. late after background movement (Fig. 4B, Table 2). In occipital pole, background movement attenuated the earliest phase of the response from 80–120 ms although, unlike saccades, not later activity. From 120–160 ms we found reduced activity in bilateral vOT junction, left OT, left anterior IT, bilateral posterior STS, and right inferior SF, with a similar trend in left anterior STS.
Significant attenuation from ~165–215 ms was found in left OT cortex. In downstream areas, reduced activity from 190–240 ms was seen in left anterior IT, bilateral anterior STS, right inferior SF, and bilateral IPFC; reduced activity in this time range also occurred in bilateral posterior STS. Effects were found also from ~250–350 ms in the left planum temporale and posterior STS, as well as in the right posterior STS. Similar to the saccade task, attenuated responses from 300–500 ms occurred within a distributed temporofrontal network including bilateral OT and anterior IT, bilateral anterior STS, right posterior STS, bilateral planum temporale, right inferior SF, right IPFC. Overall, these results indicate that the visual effects of image motion are carried over from early visual to language areas, and likely contribute to postsaccadic suppression.
The spatiotemporal pattern of repetition effects in the background movement task was similar to that in the saccade task. Figure 5 illustrates left lateralized repetition effects that co-localized with background movement effects in occipitotemporal, anterior inferior and superior temporal, posterior temporal and planum temporale, and prefrontal regions. The detailed comparisons of all individual tests are presented in Table 2.
While diminished responses were found after both saccades and background movement, the degrees of response change varied significantly across experiments and cortical regions. We compared effects of saccades and background movement in three selected ROIs, including occipital pole, ventral occipitotemporal junction and occipitotemporal cortex (Fig. 6). Response attenuation was significantly larger in the saccade than background movement experiment in occipital pole from 80–120 ms (early/late responses, 65.9% vs. 83.4%, F(1, 6) = 16.7, p < 0.007), and in vOT junction from 120–160 ms (66.6% vs. 83.4%, F(1, 6) = 19.83, p < 0.0043). Unlike these earlier effects, the degrees of response modulation in left OT cortex were similar in the two experiments.
Here we provide the first report of the impact of saccades on the spatiotemporal dynamics of word processing. Our approach employed anatomically-constrained MEG and eye-movement detection in real-time in two sets of parallel experiments in the same participants. In the natural saccade experiments, we found robust postsaccadic effects consisting of slower reaction times and reduced cortical responses to words presented early vs. late after saccades, suggesting an overall transient impairment of word processing after an eye movement. Response reductions occurred to various degrees in early visual as well as higher cortical areas where they co-localized with repetition priming effects, suggesting that saccades have consequences at both early and late stages of word recognition. In the background movement experiments, qualitatively similar effects occurred when words were presented early vs. late after the movement mimicking the visual motion across the retina during saccades. This suggests that image motion caused by saccades suppresses responses to words at re-fixation and contributes to postsaccadic inhibition. Importantly, however, the response modulation was more pronounced following saccades than after background movement in selected cortical regions, consistent with central postsaccadic mechanisms modifying word processing in addition to the influences of image motion.
The suppression after saccade-like image motion observed here is consistent with visual ‘masking’ effects whereby a visual stimulus (here the image motion) reduces visibility and responses to a second stimulus (here a word) presented close in time and space. Visual masking is known to contribute to perceptual constancy: the retinal motion during saccades is ‘masked’ by the activation produced by stimuli at fixation (Judge et al., 1980; Wurtz 2008). Conversely, recent psychophysical and physiological studies suggest that, although not perceived, visual stimulation during saccades continues to be processed in the visual system and modulates visual responses at fixation via mechanisms that probably include adaptation (Ibbotson and Cloherty, 2009; Judge et al., 1980). In the present experiment, we minimized residual visual activity from a previous fixation by introducing ~1.5 sec of stationary fixation between trials. Thus, our results probably reflect suppressive interactions between cortical activity to image motion itself and the word at re-fixation. Our findings further suggest that these visual effects are carried over through the ventral stream into language regions, disrupting word processing.
To examine the contribution of central saccadic influences on word processing, we contrasted the effects of natural saccades with the effects of background movement. We found similar behavioral suppression (longer RTs) in both experiments. However, the neural response reductions in the occipital pole and in downstream ventral occipitotemporal junction were significantly larger after saccades than after background movement. This suggests that central mechanisms modulate early stages of word processing after saccades, even though here they did not impact the behavioral performance. This is consistent with electrophysiological evidence in primates of central saccadic mechanisms that produce a biphasic modulation of visual sensitivity in several visual areas (Reppas et al., 2002; Royal et al., 2006; Rajkai et al., 2008; Ibbotson et al., 2008; MacEvoy et al., 2008). Central suppression, reported from ~100 ms before saccades until ~50 ms after the end of saccades, likely contributes to perceptual constancy by decreasing our sensation of image motion (Ross et al., 2001). Following suppression, central postsaccadic enhancement for ~200–400 ms presumably promotes stimulus processing at fixation (Ibbotson and Krekelberg, 2011), although behavioral data linking neuronal effects to enhanced performance in primates or humans are currently missing.
Although we observed a further reduction in the word-evoked response after saccades in selected regions, it is important to note that this may still reflect a central postsaccadic facilitatory signal predicted by previous studies. Indeed, to generate the MEG waveforms related to words alone, we subtracted out the waveform generated by saccades, including the eye-movement artifact and also brain activity associated with the saccade itself. Therefore, we likely subtracted any postsaccadic changes in activity attributed to central enhancement. Thus, while our results provide evidence for a central postsaccadic mechanism that modulates word processing, they cannot distinguish between its overall facilitatory or suppressive nature.
We found effects of saccades and background movement in both visual and higher cortical areas. Although the present experiment was not designed to examine specific representations activated during word recognition, effects were assessed within specific regions and time windows implicated in previous studies of visual word recognition. We found various degrees of response modulation at the occipital pole (80–120 ms), reflecting early visual feature processing, and in occipitotemporal areas (120–160 ms; 165–215 ms corresponding to the M170) implicated in orthographic and word-form access (Tarkiainen et al., 1999; Dehaene et al., 2002; Solomyak and Marantz, 2009). Reduced responses also occurred within the planum temporale (190–240 ms and 250–350 ms) that has been implicated in grapheme-to-phoneme coding, as well as in anterior temporal and inferior prefrontal regions previously associated with lexico-semantic processing (Halgren et al., 1994; Marinkovic et al., 2003; Lehtonen et al., 2011). Finally, reduced responses were found within a distributed temporofrontal network between 300–500 ms, corresponding to the N400 time-window, which has been implicated previously in lexico-semantic processing (Marinkovic et al., 2003; Matsumoto et al., 2005).
Differential activity to novel versus repeated words has been used previously to study the neural representations activated during word processing. We found modest early repetition enhancement from ~120–160 ms within the left occipitotemporal and anterior temporal regions. This partially replicates previous findings by Dhond et al., 2001 and Marinkovic et al., 2003 who showed similar early enhancement effects (although at a later latency of 190–240 ms), possibly reflecting a temporal advantage for repeats. This was followed by a robust, left-lateralized suppression effect between 200–300 ms and 300–500 ms (N400) in occipitotemporal, anterior temporal, superior temporal and prefrontal cortex. This is in general agreement with previous MEG studies of word repetition priming (Marinkovic et al., 2003; Matsumoto et al., 2005; McDonald et. al, 2010) and probably reflects facilitated lexical access for repeated words that share the same representations with their primes. At still later latencies (> 400 ms), we observed repetition enhancement in anterior inferior temporal cortex, and repeated words also led to longer reaction times than unrepeated words. There remains debate about the precise mechanisms driving these latter effects. Multiple repetitions of a single word here likely generates conscious recollection known to enhance responses in recognition memory regions that were not the focus of our analysis (Dale et al., 2000; McDonald et. al, 2010); interactions between these memory processes and word processing may have led to the observed effects. Of most relevance here, all regions that showed word repetition effects also showed postsaccadic modulation (Figs 4–5). This provides additional evidence that saccades influenced cortical areas that subserve language processing.
As noted above, comparison of saccadic and background movement effects suggests central postsaccadic influences in the occipital pole and ventral occipitotemporal junction. In contrast, effects were similar across the two experiments in downstream occipitotemporal cortex. This is consistent with evidence from vision research that centrally-mediated saccadic suppression and facilitation vary across regions and are probably related to the functional specialization of different areas (Ibbotson and Krekelberg, 2011). Future studies designed to isolate specific levels of language processing are necessary to determine a functional role of central saccadic effects, and in particular postsaccadic enhancement, in different language regions during reading.
Our experimental paradigm differed from normal reading in several ways. While reading text on a page, words enter the fovea via an eye-movement. Here, they were presented at fixation within 50 ms after the end of saccades. This approach avoided any uncertainty about exactly when word viewing began at the end of saccades, as measured with the electrooculogram, allowing us to control stimulus timing (onset and duration) across conditions. It is possible, however, that this gap before the word appeared at re-fixation influenced the neural response. In addition, the eye-movements employed here were larger than those in reading (10 deg vs. 1–2 deg), possibly producing effects of different strength and duration.
These caveats aside, our findings have important implications for understanding the neural mechanisms of natural reading. Converging evidence from studies of perceptual stability and attention have led to the view that natural vision is an attentional process (Berman and Colby, 2009) which relies on constant interactions between regions that control eye-movements and attention and those serving perception and cognition. These interactions are probably critical in fluent reading, where visual constancy is a prerequisite. Interestingly, deficits in eye-movement control and perceptual-motor integration are found in many disorders in which reading is impaired, including dyslexia, attention-deficit hyperactivity disorder and schizophrenia (Leigh and Kennard, 2004; Shaywitz and Shaywitz, 2008). The present study is the first to show robust behavioral and neuronal effects of eye-movements on the processing of single written words. Our results suggest that visual interactions across saccades and central saccadic modulation influence not only visual cortices but activity throughout the language system. Understanding the interactions between eye-movements and language processes, and their uncoupling, will be critical for the development of complete neurocognitive models of reading and its disorders.
This work was supported by NIH grants HD050627 (S.T.), EB006385 (E.N.B. and M.S.H.), EB009048 (M.S.H.), MH071635 (G.K.), National Center for Research Resources Grant P41 RR14075 (M.S.H.), and a Claflin Distinguished Scholars Award from Massachusetts General Hospital (S.T.).