|Home | About | Journals | Submit | Contact Us | Français|
Visual attention and saccades are typically studied in artificial situations, with stimuli presented to the steadily fixating eye, or saccades made along specified paths. By contrast, in the real world saccadic patterns are constrained only by the demands of the motivating task. We studied attention during pauses between saccades made to perform 3 free-viewing tasks: counting dots, pointing to the same dots with a visible cursor, or simply looking at the dots using a freely-chosen path. Attention was assessed by the ability to identify the orientation of a briefly-presented Gabor probe. All primary tasks produced losses in identification performance, with counting producing the largest losses, followed by pointing and then looking-only. Looking-only resulted in a 37% increase in contrast thresholds in the orientation task. Counting produced more severe losses that were not overcome by increasing Gabor contrast. Detection or localization of the Gabor, unlike identification, were largely unaffected by any of the primary tasks. Taken together, these results show that attention is required to control saccades, even with freely-chosen paths, but the attentional demands of saccades are less than those attached to tasks such as counting, which have a significant cognitive load. Counting proved to be a highly demanding task that either exhausted momentary processing capacity (e.g., working memory or executive functions), or, alternatively, encouraged a strategy of filtering out all signals irrelevant to counting itself. The fact that the attentional demands of saccades (as well as those of detection/localization) are relatively modest makes it possible to continually adjust both the spatial and temporal pattern of saccades so as to re-allocate attentional resources as needed to handle the complex and multifaceted demands of real-world environments.
People cannot attend to multiple objects or events without some loss of perceptibility. These so-called attentional bottlenecks, key to the understanding of immediate visual experience and awareness, have attracted the interest of researchers for decades (James, 1890; Broadbent, 1958; Treisman, 1969; Sperling & Dosher, 1986; Sperling & Melchner, 1978; Shaw, 1982; Neisser & Becklin, 1975; Mack & Rock,1998; Bonneh, Cooperman & Sagi, 2001; Lavie, Hirst, de Fockert & Viding, 2004; Kahnemann, Beatty & Pollack, 1967; Carrasco, Ling & Reed, 2004; Kastner, DeWeerd, Desimone & Ungerleider, 1998).
Much of what we know about the characteristics of attentional bottlenecks in vision has come from studies in which sequences of brief stimuli are presented to the steadily fixating eye. For example, several studies have shown that attending to a central visual task (visual search, typically) can impair the ability to discriminate the contrast, orientation or spatial frequency of eccentric gratings (Morrone, Denti & Spinelli, 2004; Huang & Dobkins, 2005; Lee, Itti, Koch & Braun, 1999; Dosher & Lu, 2005; Joseph, Chun, & Nakayama, 1997; Schwartz, Vuilleumier, Hutton, Maravita, Dolan & Driver, 2005). In general, the more difficult the central task, the greater the impairment of perceptual discrimination of the eccentric stimuli.
Studying attention during maintained fixation, as the prior work has done, has real virtues. It allows precise control over the spatial and temporal properties of stimuli, making it possible to study the fine grain properties of attention during brief and manageable intervals of time. But this admittedly artificial situation, in which attention is held at a single central region for prolonged periods, does not represent how attention normally functions. Under normal circumstances attention is not rigidly held in place by a fixation target, but is in continual motion, as saccadic eye movements take the line of sight from one region to another.
Studies of attention during steady fixation have another drawback, namely, they cannot consider the role played by attention in the control of the saccades themselves. Perceptual attention must be allocated to the target of a saccadic eye movement during the interval preceding each saccade to avoid saccadic error or delays (Hoffman & Subramaniam, 1995; Kowler, Anderson, Dosher & Blaser, 1995; Deubel & Schneider, 1996; Godijn & Theeuwes, 2003; McPeek, Maljkovic & Nakayama, 1999; Cohen, Schnitzer, Gersch, Singh & Kowler, 2007; Moore & Armstrong, 2003). Under some conditions the resulting enhancement of visual performance at the saccadic goal can spread to different locations along the planned saccadic path (Gersch, Kowler & Dosher, 2004; Gersch, Kowler, Schnitzer & Dosher, 2008; Baldauf & Deubel, 2008; Godijn & Theeuwes, 2003). These prior results came from studies of saccades made to designated targets, another convenient, but artificial, situation. Dictating the choice of saccadic targets is a useful experimental tactic for removing ambiguity about where subjects intend to look, thus facilitating the comparison of visual performance on and off a planned saccadic path. Nevertheless, such restrictions do not allow us to address basic questions about the links between attention and saccades when the targets are chosen freely, rather than being specified by the experimenters.
The goal of the present paper was to reduce the artificial restrictions on attention and eye movements that have characterized the past work – both the psychophysical and oculomotor research – and study attention during intersaccadic pauses while saccades were made to freely-chosen targets. The way we approached this problem was influenced by prior work. In particular:
During the free-scanning tasks we tested, the planned pathways of sequences of saccades could not be known in advance. As a result, we did not set out to compare perceptual attention at locations on and off the planned saccadic path, as prior work on saccades and attention has done (e.g., Gersch et al., 2008). Instead we investigated the extent to which the performance of the different primary tasks (counting, pointing, and looking-only) impaired performance of the secondary, orientation identification task.
The main primary task in Experiment 1 was to count the number of elements (10 to 19 randomly-positioned dots) shown in the display and report the result at the end of each trial. Counting was chosen because it readily encourages, and benefits from, the use of saccades (Landolt, 1891; Kowler & Steinman, 1977, 1979), and because it can be done with simple, unstructured visual displays. Counting tasks can be performed during steady fixation, although the accuracy of the reports suffers (Kowler & Steinman, 1977, 1979). Note that while we asked subjects to count the dots, we did not attempt to control or to monitor the various estimation or grouping strategies that subjects might have adopted (Liss & Reeves, 1983; Dehaene, Piazza, Pinel & Cohen, 2003; Gelman & Gallistel, 1978; Hurewitz, Gelman & Schnitzer, 2006). Our goal was to keep the subjects engaged in a task that generated saccades, and not to test a particular model of numerical judgments.
Counting was deemed to be the primary task (task instructions will be discussed in more detail below). The secondary visual task – identifying the orientation of a briefly-presented tilted Gabor patch – was used to characterize attention during randomly-chosen intersaccadic pauses. Gabor orientations were separated widely (+/− 22.5 deg), well above discrimination thresholds under conditions of full attention with high contrast targets (Regan & Beverly, 1985). These procedures allowed us to determine how attention to the primary task affected contrast thresholds for orientation identification (Dosher & Lu, 2000a, b; Gersch et al., 2004), and how effects due to attention depended on stimulus contrast.
Orientation identification during counting was compared to that obtained during a steady-fixation baseline (no counting and no saccades). We also tested two other versions of the primary task, namely, (a) counting during steady fixation, with shifts of attention substituting for the saccades, and (b) making comparable patterns of saccades without counting. This allowed us to evaluate the demands on attention made by saccades with respect to those made by the cognitive or perceptual requirements of counting itself. Finding strong suppression of secondary task performance when saccades are made (either during counting, or simply when looking around the display) relative to performance during steady fixation (with or without counting) would indicate that saccades contributed substantially to the overall attentional demand. On the other hand, finding strong suppression of secondary task performance during counting (with or without saccades), and less suppression while making saccades without counting, would support a weaker contribution of saccades to the overall demand on attention relative to the demands imposed by other perceptual or cognitive aspects of the motivating task.
Stimuli were displayed on a Dell P793 CRT monitor (13 deg × 12 deg; viewing distance 115 cm, 1.46 pixels/min arcs; refresh rate 75 Hz, non-interlaced).
The stimulus for the counting task was a set of 10–19 dark grey dots (8′ radius, 15 cd/m2) presented on a medium gray (54 cd/m2) background. Dots were randomly distributed about the screen (minimum edge separation 3′) within the central 8 deg of the display. The Gabor probe appeared in one of four locations (north, south, east, or west), centered on a location 3 deg from the center of the display. The Gabor probe was generated according to the following:
where f is the spatial frequency (2.24 cycles/deg), l0 is the mean luminance (54 cd/m2), θ is the orientation (+/− 22.5 deg from vertical), σ the standard deviation of the Gaussian window (0.89 deg), (x, y) the spatial coordinates in the display, and a the amplitude. Maximum luminance was 108 cd/m2. Amplitude was determined from the contrast (the difference between maximum and minimum luminance divided by twice the mean luminance), and contrast was selected at random from 6 or 7 values ranging from 4 to 96%. Three frames of the Gabor were interleaved with single frames of background luminance. The background luminance frames were replaced with external noise frames in a control condition, to be described below. All frames of the Gabor appeared within a 66 ms time period.
Before each trial the display contained a white fixation cross located either in the center of the display (for conditions requiring saccades) or at a randomly-chosen location that would later be occupied by one of the dots (for conditions requiring steady fixation). The subject started each trial when ready by pressing a button. Two hundred ms later the array of dots appeared and remained on throughout the 3 sec trial.
For conditions requiring saccades, the Gabor probe appeared during a randomly-chosen fixation pause. The fixation pause was selected by detecting on-line the first saccade to occur after a randomly-selected delay (600 ms – 2 s) following the start of the trial. The Gabor then appeared after an additional random delay of 25–100 ms so that the Gabor would appear at various times after saccadic offset but before the next saccade to be made (this was verified by off-line data analyses). For conditions requiring steady fixation, the timing of the Gabor was the same except that the Gabor appeared immediately after the first randomly-selected delay (600 ms–2s). Note that due to the random choice of fixation position at the start of the steady fixation trials, the retinal eccentricities of the Gabors (which takes into account both their position on the display and the position of fixation) were the same across conditions (see Table 2 for verification). Trials in which the Gabor appeared during any part of a saccade, regardless of the experimental condition, were eliminated. Figure 1 shows the display (dots and Gabor probe) along with representative eye movement traces from trials requiring saccades and from trials requiring steady fixation.
Immediately after the trial was over, an arrow appeared that disclosed the location of the Gabor. This ‘post-cue’ was included in all conditions to remove uncertainty about the location on which to base the response (Dosher & Lu, 2000a, b). The subject then reported by means of button presses: (1) the number of dots (in conditions requiring counting), and (2) the tilt (left or right) of the Gabor. Feedback was given after each trial by showing the display frame containing the Gabor for 500 ms and by announcing the number of dots displayed.
There were 4 experimental conditions tested in separate experimental sessions (75–100 trials each), and each condition was typically tested once (75–100 trials each) per day. The 4 conditions used the same displays but different instructions: (1) Steady fixation (baseline): subjects maintained a steady line of sight on the initial fixation cross, and continued fixating the same location when the trial started and the fixation cross was replaced by the random dots. (2) Counting with saccades: Subjects used any pattern of saccades they chose while counting the dots. At the end of the trial, the number of dots was reported followed by the report of the orientation of the Gabor. (3) Counting during steady fixation: Steady fixation was maintained, as in the baseline condition, and subjects counted as best they could by shifting attention instead of shifting the line of sight (Kowler & Steinman, 1977, 1979). (4) Saccades only: Subjects inspected the dots with saccades, attempting to reproduce the approximate pattern of saccades they used during counting, without actually counting or reporting the count at the end of the trial. (Note: As will be shown below, subjects had no difficulty producing saccadic patterns that resembled the patterns used during counting.) A total of 700 – 1100 trials/subject were run for each of the 4 experimental conditions listed above.
For conditions in which counting was required (either with or without saccades), counting was defined as the primary task. Subjects were told to try to continue counting without interruption either in anticipation of, or in reaction to, the appearance of the Gabor.
A few variations of the basic experiment were included: (1) An easy perceptual discrimination (500 Hz vs. 1000 Hz tone, 200 ms) was tested in place of the report of the Gabor. This control was included to verify that any errors in reports about the Gabor were not due to the need to deliver two responses at the end of the trial (the dot number and the Gabor report); (2) Effective Gabor contrast was increased by 67% by presenting 5 consecutive frames of the Gabor (rather than 3 frames interleaved with background luminance); (3) Gabors were oriented vertically with a spatial frequency either 1.7 or 2.7 c/d. The task was to report which frequency (the higher or lower of the pair) was presented; (4) Single frames containing high-contrast visual noise (diam 80′) were interleaved with the frames of the Gabor, and shown at all 4 of the possible Gabor locations; (5) The post-trial arrow disclosing Gabor location was eliminated and instead observers reported Gabor location (N, S, E, W) rather than orientation. Subjects ran in 1500–2000 trials for each of these 5 additional experimental variations.
Four volunteers were tested (JT, SDK, GT, ES), all with uncorrected normal vision, and all naïve as to the purpose of the experiment. Two additional naïve observers (AW and LM) were tested in the version of the experiment with interleaved noise frames.
Horizontal and vertical movements of the right eye were recorded using a Generation IV Double Purkinje Image Tracker (Crane & Steele, 1978). The left eye was covered and the head was stabilized with a dental biteboard. The tracker’s voltage output was fed on-line through a low pass 100 Hz filter to a 12-bit analog to digital converter (ADC). The ADC, controlled by a PC, sampled the eye’s position every 2 ms. The digitized voltages were stored for analysis. The time of appearance of the Gabor was monitored by a photocell, which received a signal from a small white square on the display, out of the subject’s view, whenever the Gabor appeared. The output of the photocell was fed to a channel of the ADC and recorded along with the eye position to ensure accurate temporal synchronization between the stimulus display and the eye movement recording.
Tracker noise level was measured with an artificial eye after the tracker had been adjusted so as to have the same first and fourth image reflections as the average subject’s eye. Filtering and sampling rate were the same as those used in the experiment. Noise level, expressed as a standard deviation of position samples, was 0.4′ for horizontal and 0.7′ for vertical positions. Recordings were made with the tracker’s automatically movable optical stage (auto-stage) and focus servo disabled.
The beginning and ending positions of saccades were detected off-line by means of a computer algorithm employing an acceleration criterion (Gersch et al., 2004). The value of the criterion was determined empirically for individual observers by examining a large sample of analog recordings of eye positions. Saccades as small as the microsaccades that may be observed during maintained fixation (Steinman, Haddad, Skavenski, & Wyman, 1973) could reliably be detected by the algorithm.
Trials were eliminated if tracker lock was lost during the trial (1.3%) or if the Gabor appeared during a saccade (10.2%). The eliminated trials were distributed approximately equally across all experimental conditions.
Performance on the orientation identification task was analyzed by fitting Weibull functions to the psychometric data using the ‘psignifit’ algorithm (Wichman & Hill, 2001). Contrast thresholds were taken at the 75% correct level. Given that performance in many cases never reached this level even at maximum contrast, we will also report the peak performance at the 100% contrast level.
Figure 2 contains the main result of this experiment: the proportion of correct identifications of Gabor orientation as a function of contrast under the 4 conditions tested.
Counting resulted in substantially poorer identification performance than found for the steady fixation baseline, even at the highest Gabor contrast tested. Orientation identification during counting with saccades did not exceed about 75% correct, which means (given that chance performance was 50% correct) subjects failed to identify the orientation of the highest contrast Gabor on about half of the trials. The poor performance at the highest Gabor contrast, and the fact that there was little improvement with increases in contrast above 32%, suggests that the suppression of the Gabor during counting was not due exclusively to a reduction in the effective Gabor contrast due to inattention (sometimes referred to as a reduction in contrast gain; e.g., Morrone et al., 2004; Huang & Dobkins, 2005). The pattern of results suggest that failures to identify the high contrast Gabors was due in part to an overall response suppression that cannot be overcome completely by increases in stimulus contrast.
Losses in orientation identification while counting during maintained fixation followed the same pattern, but were smaller than losses while counting with saccades. Performance reached about 85% correct at the highest contrast tested, better than obtained for counting with saccades, but still considerably poorer than the baseline, steady fixation condition.
Making saccades without counting had modest effects on orientation identification and revealed a more complex pattern of losses. Two subjects (ES, GT) showed a loss only at the lower Gabor contrasts, one (SDK) only at high contrasts, and one (JT) at both.
The pattern of results in Figure 2 shows that, using orientation identification as a benchmark of attention, counting was more demanding of attention than the act of making saccades across the display. Counting could, on about half the presentations, render the orientation of even high contrast Gabors nearly impossible to perceive.
Why was orientation identification so poor at high stimulus contrasts during counting? We made additional observations that allowed us to reject a few possibilities unrelated to interference from attention to the primary counting or saccadic tasks:
The poor performance in the orientation identification task during counting could have been due to a failure to detect the Gabor, or, alternatively, to difficulty creating or maintaining an accurate representation of its features. The distinction between detection and identification comes up often in the attention literature, with some studies showing that a primary task interferes with the ability to notice (i.e., detect) unexpected stimuli (Mack & Rock, 1998; Bonneh et al., 2001), and others showing that detection (and localization) are far less demanding of attention than identification (Sagi & Julesz, 1985; Huang et al., 2007). In agreement with these latter observations, one of our subjects (JT) confidently asserted after the experimental data were collected that he usually knew where the Gabor was even though he could not always determine its properties.
To distinguish detection from identification, the secondary task was changed to one of localizing the Gabor. All other aspects of the experiment remained the same except for the post-trial display, where the arrow disclosing Gabor location while waiting for the subject’s report was removed.
Figure 4 confirms JT’s impressions and shows that reports of the location of the Gabor (N,S,E, or W) were much more accurate than reports of its orientation. Two subjects (JT and SDK) showed only a small loss relative to baseline in any of the experimental conditions. ES and GT showed losses during counting relative to the baseline conditions, but not as severe as found for the orientation identification task, particularly at the high Gabor contrasts where localization performance in the 4AFC task reached 85% correct.
The high level of performance in the localization task shows that poor orientation identification during counting cannot be explained solely by failure to notice the Gabor stimulus, or by confusions about its location during the trial. The main effect of counting is an interference with the representation of the features of the Gabor.
Another indication that the presence of the Gabor was registered by the visual system came from an analysis of the duration of pauses between saccades. Figure 5a, b, based on data from the identification task, shows that the duration of the intersaccadic pause containing the Gabor increased with Gabor contrast. This was true even for trials in which the Gabor orientation was reported incorrectly (Fig. 5a). The slopes of the functions in Fig. 5a, b were significantly greater than 0 (average slope=.67, t=6.51, p=.000043) with pause durations increasing by 25–100 ms over the full range of contrasts tested.
If the longer intersaccadic pause durations were due solely to the transient flash of the Gabor, then the pause durations also should have increased during the localization task. Figure 5c, d shows that in the localization condition the slopes of the functions relating Gabor contrast and intersaccadic pause duration were shallower (mean=.06) and not significantly different from 0 (t=.42, p=.68). This leads to an interesting speculation, namely, that the increased pause durations during the orientation task was not an automatic reaction to the appearance of the Gabor, but rather a strategic response tailored to the orientation task, an attempt to briefly interrupt counting, or looking, in order to devote more attention to the time-consuming task of identifying the orientation of the Gabor.
We also examined the reports of the number of dots in the display. Performance was represented by the correlation between the actual number of dots displayed and the reports of dot number. The correlation provides a reasonable measure of how much useful information was obtained from the dot displays across the different conditions.
Figure 6a shows that the correlations were substantially higher when saccades were used (triangles) than during steady fixation (circles), as was expected. Correlations did not vary systematically with Gabor contrast, suggesting that detection of the Gabor (which was more likely at higher contrasts; Fig. 4) did not interfere with obtaining information from the dot displays. The same pattern of results was obtained regardless of whether the Gabor was reported correctly or incorrectly. Figure 6a also shows that for 3 subjects, correlations for the counting task did not differ as a function of the type of Gabor report (orientation or location) that was required.
These results show that the higher levels of performance obtained in the Gabor localization task did not come as a result of a tradeoff with counting. There was, however, evidence for other tradeoffs. The arrows along the righthand ordinates of Figure 6a show counting performance for experimental sessions in which counting alone was tested, without any judgments about the Gabor. In cases where counting was performed using saccades (black arrows), correlations for counting alone were either the same as (subjects JT and GT) or slightly higher than (ES and SDK) those obtained when both the counting and Gabor tasks were done in the same trials. When counting was carried out during steady fixation (blue arrows), correlations were reduced relative to the counting-along baseline by the addition of the Gabor task for all subjects except GT.
The losses in both Gabor judgments and counting performance when both tasks were done together, relative to performance when either task was done alone, are shown in the form of Attentional Operating Characteristics (Sperling & Dosher, 1986) in Fig. 6b. These results confirm the mutual interference of the counting and the Gabor judgments. The fact that small losses were observed even in the primary counting task suggests that there were few resources to spare, and that any attempts to improve Gabor judgments would have come at the expense of counting.
The perceptual losses found for orientation identification during counting were substantial. Would such losses be limited to a task such as counting, with a demonstrable cognitive load (Gelman & Gallistel, 1978; Hurewitz, Gelman & Schnitzer, 2006; Dehaene et al., 2003), or would they obtain more generally with any purposeful task involving saccades? To address this question we performed a second experiment that used a different and less cognitively-demanding primary task to motivate the saccades. The task we used was pointing to sets of dots using a visible cursor controlled by moving an unseen pen across a digitizing tablet. Pointing, like counting, benefits from the use of saccades (Biguer, Prablanc, & Jeannerod, 1985).
The stimulus for the pointing task was the same as for the counting task except that fewer dots (7–16) were displayed per trial.
Steady fixation (baseline) was the same as in Experiment 1. Pointing-with-saccades: Subjects used the pen of a digitizing tablet to move a visible cursor from one dot to the next. The tablet and pen were out of view, located on a shelf mounted below a tabletop in front of the subject. No constraints were imposed on the cursor movements, other than to make a reasonable attempt to hit each dot. For technical reasons (incompatibilities in software) we did not record the output of the digitizing pen, but did verify by inspection of the movements of the visible cursor on the display screen that subjects were performing the task. Saccades only: Subjects were asked to look from one dot to the next, as if they were pointing to the dots with the cursor. A total of 1000–2700 trials were run/subject, of which 9.1% were eliminated due either to loss of tracker lock or to a Gabor occurring during a saccade.
Six subjects were tested, including 4 who participated in the counting experiment (SDK, GT, AW, LM) and 2 new subjects (AS, JW). Other than author JW, all the subjects were naïve as to the purpose of the experiments.
Table 1 summarizes characteristics of the saccades. During the pointing task, saccades were smaller and intersaccadic pause durations longer than during counting. As was the case for counting, saccades were larger than when solely looking around the display. Five of the 6 subjects showed longer intersaccadic pauses during pointing than when solely looking. Intersaccadic pause durations once again increased with Gabor contrast when looking only (Fig. 7b). Increases in pause duration as a function of contrast were not found during pointing (Fig. 7a), perhaps because the intersaccadic pause durations were already prolonged due to the requirements of the pointing task.
The effect of pointing on orientation identification is shown in Fig. 8. Four subjects (JW, SDK, GT and LM) showed losses during pointing, and while looking-only without pointing. The magnitude of the losses during pointing was less than during counting (Figs. 2 and and3),3), particularly for LM. Subjects AW and AS showed little or no effect of either pointing, or looking-only, on orientation identification.
Note that two of the subjects with good orientation identification during the pointing task (LM and AW) had shown substantial losses in orientation identification during counting in Experiment 1 (Fig. 3c). Both also showed losses in the looking-only condition, with LM showing losses in Experiment 2 (Fig. 8) and AW in Experiment 1 (Fig. 3c). Thus, their good performance during the pointing task can be attributed to differences in the distribution of attention in the visuomotor tasks tested (pointing or looking-only), rather than to a global immunity from interference from any of the primary tasks.
The results presented so far showed substantial losses in orientation identification during counting, smaller losses during pointing, and smaller losses still while making saccades by themselves.
Figure 9 provides a summary of the effects of counting, pointing and saccades on orientation performance. Fig. 9a shows mean contrast thresholds in two conditions, baseline steady-fixation, and saccades-only. Thresholds were averaged over all 14 instances where orientation identification was measured under these two conditions (Figs. 2, 3a, 3c and and8).8). Analyses of thresholds are limited to these two conditions because orientation performance during either counting or pointing usually did not reach levels high enough in a sufficient number of cases to allow meaningful threshold comparisons.
Fig. 9a shows that average contrast threshold increased from 16% in the steady fixation baseline to 22% while making saccades. The magnitude of this increase in average threshold (37%) is in the range of the effects of saccades on attention found during tasks where saccades were made along specified paths (Gersch et al., 2004), as well as the effects of attentional cues measured during steady fixation (Dosher & Lu, 2000a, b; Carrasco, Penpeci-Talgar, & Eckstein, 2000).
Figure 9b summarizes the effects of the different tasks on peak performance (performance at 100% Gabor contrast). Of the different primary tasks, counting led to the poorest orientation identification, and looking-only to the best. Performance when counting during steady fixation was better than when counting with saccades. Performance during pointing was better than either counting task, as well as more variable due to the differences among subjects (see Fig. 8). Note that both measures, threshold and peak performance, show that the attentional demands of purposeless saccades were significant, but smaller in magnitude than when the saccades were motivated by a purposeful task.
Can any of the variation in perceptual performance during the primary tasks be traced to overt differences in characteristics of the saccades? Figure 10 contains scatterplots showing the relationship between the perceptual measures in the orientation task (thresholds and peak performance) and two measures of saccades: average saccade size and average intersaccadic pause durations. Each datum point in the scatterplot represents the results of one subject tested with one of the primary tasks studied (counting, pointing, or saccades-only).
Figure 10 shows weak relationships between the saccadic measures and perceptual performance. There was a small tendency for lower levels of perceptual performance to be associated with either longer intersaccadic pauses (top) or shorter saccades (bottom). Neither of these saccadic characteristics should have hurt identification of the Gabor (if anything, longer pauses should have helped). We suspect that the longer pauses, and perhaps the shorter saccades, did not impair Gabor performance, but rather are indicators of increased attention to the primary task. Thus, the correlations are consistent with the view that the more difficult primary tasks were associated with less attention to the Gabor targets.
Much of what we know about attention, and the relationship between attention and saccades, is based on studies in which the patterns of eye movements are restricted in the interests of achieving precise experimental control. In the case of psychophysical investigations of attention, one or more stimuli are typically presented briefly to the steadily fixating eye. In the case of investigations of the link between attention and saccades, a single saccade, or a sequence of saccades, are made to specified targets. As useful as such experimental approaches have been, and will continue to be, for testing hypotheses about attention, if we are to accept the assumption that the processes discovered are relevant to attention and saccades in natural viewing, then it becomes necessary to attempt comparable psychophysical investigations when the constraints on eye movements are removed.
The present work removed some of the conventional constraints by studying attention during the pauses between saccades while subjects were engaged in different active visual tasks, namely, counting, pointing, or simply looking around the display. These active tasks were performed with patterns of saccades that were not dictated by the experimenter, but were chosen subject only to the needs of the motivating task. Attention was evaluated during intersaccadic pauses by means of a psychophysical task (identifying the orientation of a briefly-presented Gabor probe).
We found that all of the active tasks we studied, including looking-only, led to losses in performance of the secondary, orientation identification task. Finding such performance losses shows that the attentional requirements attached to the control of saccades is not limited to tasks where saccadic targets were specified by the experimenters, and can apply as well to saccades made along freely-chosen paths. (There were differences across tasks and individuals in the losses associated with saccades; this issue will be taken up later in the discussion.)
Saccades made less of a demand on resources than other perceptual or cognitive aspects of the tasks. Orientation identification was poorer during counting, either with or without saccades, than during either looking or pointing. The pattern of losses during counting was particularly striking: Orientation identification reached a level of only 75–80% correct at a moderate contrast (32%), and did not improve with further contrast increases. These losses at high stimulus contrast are similar to those found in experiments where discrimination thresholds were measured during performance of a competing task (e.g., Morrone et al., 2004; Huang & Dobkins, 2005). In cases like these, where the loss of performance appears to persist despite increasing contrast, losses can be due to any number of processes, including failure to accurately encode the features of the stimulus, or failure to maintain a representation of the stimulus long enough for the perceptual decision to be made.
In contrast to the poor identification of the orientation of the Gabor during counting, the ability to detect and locate the Gabor was largely spared. Detection and localization require less attention, and less processing time, than identification of stimulus features, a result described by Sagi & Julesz (1985), and confirmed recently under different conditions by Huang, Treisman & Pashler (2007).
The relatively good detection/localization performance was consistent with the increased intersaccadic pause duration found for the high contrast Gabors (see also Henderson & Hollingworth, 2001). The transient appearance of the Gabor could have contributed to increasing pause duration, but was not the only factor responsible because pause duration was not inflated by the appearance of the Gabors during the perceptual localization task. Thus, the increase in intersaccadic pause duration during the identification task was associated with task difficulty, and was perhaps a sign of an attempt to shift some processing resources to the Gabor in order to better identify its features. These increases in pause duration did not improve orientation performance, presumably because the Gabor was gone, along with any persisting memory of its features, before orientation could be determined.
The pointing and the looking-only tasks produced clear losses in orientation performance, although less than those produced by counting. These effects are consistent with the view that attention is engaged by the need to plan and program freely-chosen patterns of saccades. 1
The magnitude of the effects of looking-only, or pointing, differed across experiments and observers, and there were instances where either pointing or looking did not impair orientation performance (Fig. 8, AW and AS; Fig. 3, ES and LM). Individual differences, even large ones, are certainly not without precedent in studies of attention (Shaw & Shaw, 1977), even studies of attention in monkey (Boudreau, Williford & Maunsell, 2006). The individual differences we observed were not correlated with overt characteristics of saccades, such as the average sizes or intersaccadic pause durations. Thus, there was no evidence that good identification performance while looking or pointing was achieved by sacrificing the control of the saccades. And, given that all observers showed substantial losses in orientation performance during counting, and comparable identification performance in the steady fixation baseline, we do not believe that the individual differences stemmed from phenomena related to the orientation identification task itself. This leaves us with the issue of attentional strategies, and how attention was distributed while making saccades.
Both the pointing and the looking-only tasks may have allowed flexibility in the choice of attentional strategies. In a prior study of attention preceding single saccades, Kowler et al. (1995) found that it was possible to voluntarily allocate some attention to non-goal locations without much cost either to saccadic accuracy or saccadic latency (see Attentional-Operating Characteristics in their Fig. 11). These findings (and comparable ones applying to smooth pursuit; Khurana & Kowler, 1987) show that a given pattern of eye movements can be produced either by allocating full attention to the target, or by sparing some attention in order to analyze non-target stimuli.
Flexibility is interesting because of what it implies about the underlying relationship between perceptual attention and saccades. All that eye movements may ever require in order to maintain an acceptable level of spatial and temporal accuracy is a modest alteration in the distribution of attention, large enough so that the peak level coincides with the saccadic target during either the entire latency interval of the saccade, or some critical portion of the latency interval. With such a distribution of attention, the spatial goal of the saccade can be determined by a “winner take all” network that finds the locus of peak attentional strength (Koch & Ullman, 1985; Tsotsos, 1990). Such networks would make it possible to allocate considerable attention to locations other than the saccadic goal, with little or no cost to saccades. Recently, Gersch et al. (2008) found evidence supporting this view in a study of attention during saccadic sequences. They found that when saccades were made along color cued paths, perceptual attention could be distributed along the path to locations both ahead and behind the immediate saccadic goal without disrupting the saccadic pattern. (Peak attentional strength remained at the saccadic goal.) It is possible that the subjects in our study who showed good performance on the orientation task during pointing or looking may have found a way to distribute attention efficiently, so that the peak strength remained at the immediate saccadic goal, but the distribution of attention across the display was sufficient to support the orientation task.
Counting was another matter. Performance on the orientation task in all observers suffered during counting, even at high stimulus contrasts. The flexible strategies of distributing attention, which might have been adopted by some subjects during pointing or looking, were evidently no longer available. This was a surprising outcome in that counting a display of dots makes only modest demands on pattern recognition systems, and thus would appear to present little basis for competition with identification of the features of the Gabor.
What aspects of the counting task were responsible for the grab of resources? Selection of a dot or a group of dots as a saccadic target, or shifts of spatial attention to the targets, could have contributed to the losses. But given that target selection was also required during looking or during pointing, it is not likely that target selection accounted for all of the losses during counting. We can also rule out processes related to generating and remembering a report of the number of dots since we did not find comparable losses in other tasks (detection and localization; or auditory discrimination) that also required a second response.
This leaves us with the operations performed at stages in between perceptual selection of a local group of dots and the generation of the counting report. These operations include estimating the number of dots in a group, keeping track of the running sum, and retrieving and generating the count-words (see, for example, Dehaene et al., 2003; Gelman & Gallistel, 1978; Hurewitz et al., 2006; Liss & Reeves, 1983). In addition, demands are made on both memory and executive processes to keep track of which dots have already been counted, and ensure the process continues in an effective way over time.
These operations may have been demanding enough to use up all the perceptual or cognitive resources available during a given fixation pause (Lavie et al., 2004; Epelboim & Suppes, 2001), and thus prevent the maintenance of a representation of the Gabor in a relatively durable form (see the following section for a related point). But an overload of fixed-capacity processors is not the only explanation for the difficulties in identifying the Gabor. An alternative explanation involves attentional strategies. Specifically, as part of a strategic adjustment to the difficulty of counting, attentional filters may have been invoked to shield visual or cognitive analyzers from inputs unrelated to the primary counting task. Difficult or demanding primary tasks, such as counting, may encourage such filters to spring into action even when there is capacity to spare. Other circumstances might as well. For example, studies have found that even a relatively low-demand primary task could encourage a conservative strategy of ignoring the secondary task under circumstances where the probability of a secondary target appearing is low (Droll et al., 2005; Boudreau et al., 2006). The relative roles of attentional strategies and processing capacity in accounting for dual task performance has been a fundamental issue in evaluating dual-task performance (e.g., Navon & Gopher, 1979; Sperling & Dosher, 1986), and needs to be re-examined in the context of attempts to understand the role and uses of attention in active and complex tasks. Although we cannot definitively distinguish the roles of capacity and strategies in determining our results, the fact that there were some losses in the primary counting task (Fig. 6), accompanying the losses in the Gabor tasks, is an indication that performance limits had been approached, and that any inducements to devote more time or attention to the Gabor would have led to even more errors in the counting task.
Our results can be related to a finding from a very different domain. Supèr, Spekreijse & Lamme (2001a, b) found a correlation between the late component of activity in V1 in monkey and the performance of perceptual tasks. In these studies monkeys had to make a saccade to a motion-defined or a texture-defined target that appeared in a randomly-chosen location in the display. The probability of making a saccade to the correct location (i.e., the probability of correctly finding the target) was positively correlated with the strength of the late V1 response to the target figures. The late V1 response began about 100 ms following stimulus onset, and could last as long as 1–2 seconds, even when the target was no longer present on the screen. The late activity in V1 was attributed to feedback signals from higher-level cortical areas.
The relevance of late and persisting neural signals to perception indicates the importance of preserving representations of selected stimuli for periods of time at least as long as the duration of a typical fixation pause. One consequence, however, of relying on extended processing time is that it enhances the opportunity for competition among concurrent tasks. Persisting neural signals may remain in a fragile state, subject to disruption by other targets or tasks present at the same time (Supèr et al., 2001b; also, Sigman & Dehaene, 2005). Such disruption of the representation of the Gabor may have occurred in our experiments due to the attention to the dot stimuli and to the operations connected to performing the different primary tasks. This disruption would account for failures to identify the Gabor, even at high stimulus contrasts. It would be of considerable interest to know whether, in a dual-task behavioral paradigm comparable to ours, maintained attention to a primary task is able to abolish the late-V1 response to newly-presented secondary targets, and with it, the ability of the animal to identify critical features of these targets.
Studying attentional bottlenecks during free-viewing, where the spatial and temporal patterns of saccades are not constrained by instruction, is interesting for obtaining some insight about strategies of compensation for the bottlenecks. Free-viewing permits a re-allocation of resources in at least two ways, namely, by allowing intersaccadic pause duration to increase long enough to accommodate multiple task demands, or by commanding shifts of attention or shifts of gaze to a new place.
We found that tasks connected to such reallocation of resources made modest demands. These included the saccadic shifts of gaze to look around the display, and the perceptual detection or perceptual localization of the Gabor probe. These tasks could be done in combination without substantial loss. By comparison, the perceptual identification of features (orientation) was costly, as was a cognitive task – counting – that involved perception, memory and thinking. These tasks could not be done well in combination.
These results show that the operations needed to detect or locate targets, and to control saccades, can continually run in the background, enabling rapid re-distribution of the limited analytical and memory resources as needed. This is useful. Given that demanding tasks are required so often in real-world settings, the ability to rapidly re-orient and re-allocate processing resources by appropriate modifications of saccadic patterns is both a valuable skill and a necessary option in natural viewing. Our results show that the limits on the ability to perform multiple operations during single fixation pause are so severe that vision reduces to a serial process in which the continual management of resources by saccades becomes a crucial component of successful task performance.
Supported by NIH EY15522. We thank the reviewers for valuable comments and suggestions.
1While it is possible that losses in orientation identification during intersaccadic pauses could be due to factors connected to the retinal changes produced by saccades, rather than to attention or saccadic planning, we previously (Gersch et al., 2004) showed that any such “intersaccadic suppression” could be at most responsible for a small portion of the elevation in visual thresholds for orientation identification during pauses between saccades.
Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.