|Home | About | Journals | Submit | Contact Us | Français|
Salient stimuli are characterized by their capability to perturb and seize available cognitive resources. Although the striatum and its dopaminergic inputs respond to a variety of stimuli categorically defined as salient, including rewards, the relationship between striatal activity and saliency is not well understood. Specifically, it is unclear if the striatum responds in an all-or-none fashion to salient events or instead responds in a graded fashion to the degree of saliency associated with an event. Using functional magnetic resonance imaging, we measured activity in the brains of 20 participants performing a visual classification task in which they identified single digits as odd or even numbers. An auditory tone preceded each number, which was occasionally, and unexpectedly, substituted by a novel sound. The novel sounds varied in their ability to interrupt and reallocate cognitive resources (i.e., their saliency) as measured by a delay in reaction time to immediately subsequent numerical task-stimuli. The present findings demonstrate that striatal activity increases proportionally to the degree to which an unexpected novel sound interferes with the current cognitive focus, even in the absence of reward. These results suggest that activity in the human striatum reflects the level of saliency associated with a stimulus, perhaps providing a signal to reallocate limited resources to important events.
Due to the limitations of our cognitive resources, it is impossible to process every stimulus in the environment at a given moment. Although much of our surroundings are predictable, we constantly encounter unexpected changes of potential behavioral significance. Consequently, our available cognitive resources preferentially shift toward salient events at the expense of less salient stimuli. Saliency, in this context, refers to the capability of important or arousing stimuli to interrupt the current cognitive focus and cause the reallocation of available resources, i.e., elicit an attentional or behavioral switch. The exact mechanisms underlying this prioritized processing of unexpected, potentially important stimuli remain unclear, but it has been proposed that the striatum and its major dopaminergic inputs may play a role by providing a signal that facilitates the reallocation of resources toward unexpected, salient events (Redgrave et al., 1999a,b).
Activity in the striatum has long been associated with reward exclusively, but there is considerable evidence supporting the hypothesis that activity in the striatum signals saliency in general. Midbrain dopamine neurons that project to the striatum respond to a large category of arousing events, including novel stimuli (Schultz and Romo, 1990; Legault and Wise, 2001), aversive stimuli (Young, 2004), and high intensity visual and auditory events (Steinfels et al., 1983; Strecker and Jacobs, 1985; Horvitz, 2000; Horvitz et al., 1997), in addition to unexpected rewards (Mirenowicz and Schultz, 1994; Schultz, 1998). Similarly, human neuroimaging studies have demonstrated that striatal activations follow unexpected rewards and reward-related stimuli (Delgado et al., 2000; Knutson et al., 2000; Berns et al., 2001), but also follow aversive stimuli (Becerra et al., 2001; Jensen et al., 2003), and neutral arousing visual stimuli (Zink et al., 2003).
Although the striatum responds to a variety of stimuli that can be categorically defined as salient, saliency is not a discrete quality (not ‘‘either/or’’). It lies instead on a continuum, so that the higher the saliency, the greater the likelihood of perturbing the current focus of attention or behavior. It follows that brain regions that drive the reallocation of resources should signal the degree of saliency associated with a stimulus, such that the strength of the signal determines the likelihood of an attentional or behavioral switch. A precise relationship between striatal activity and the degree to which events perturb limited cognitive resources has not yet been established, which is essential to determine a striatal role in prompting the reallocation of resources to unexpected potentially important stimuli in our environment.
We used functional magnetic resonance imaging (fMRI) to investigate human neuronal responses to unexpected stimuli that were associated with varying levels of saliency, in the absence of reward. We hypothesized that if the striatum provides a signal to switch resources to important events, then its activity following a stimulus should increase with the level of saliency associated with the stimulus.
Twenty right-handed, healthy adults (ten males; ten females), ages 18–30, participated in the fMRI experiment. Participants had no history of neurological or psychiatric disorder and gave written, informed consent for a protocol approved by the Emory University Institutional Review Board.
The software package, Cogent 2000 (FIL, University College London), was used for stimulus presentation and response acquisition.
While in the scanner, participants performed four runs of a modified version of a classification task (Fig. 1) implemented in previous studies (Escera et al., 1998, 2003). In each run, a single digit number (2, 3, 4, 5, 6, 7, 8, or 9) appeared for 200 ms in the center of the screen, with a 3.3 s interstimulus interval. Upon the appearance of the number, participants were instructed to press one of two buttons with either their right index finger or middle finger to indicate if the number was ‘‘even’’ (2, 4, 6, or 8) or ‘‘odd’’ (3, 5, 7, or 9). The numbers were presented in a pseudorandom order with each number appearing 18 or 19 times per run. The buttons assigned to ‘‘even’’ numbers and ‘‘odd’’ numbers were counterbalanced across participants. 600 ms preceding the appearance of each number, an outline of a circle appeared in the middle of the screen and remained on the screen until the end of the trial, so the numbers always appeared in the center of the circle outline. A 500 ms task-irrelevant sound was presented to the participants via headphones when the outline of the circle appeared 600 ms prior to the appearance of a number. The visual circle outline at the time of the sound presentation, together with the instructions to focus on the visual stimuli and the constant timing of stimuli appearance, minimized the possibility that the auditory stimuli would come to represent timing cues for upcoming numerical task-stimuli. The sound was either a 600 Hz ‘‘standard’’ tone (85% of trials), a 700 Hz ‘‘deviant’’ tone (5% of trials), or a ‘‘novel’’ sound (10% of trials). The standard tones set the background experimental context. The novel category contained a variety of sounds, some identifiable and some unidentifiable. Although the saliency of the novel sounds lay on a continuum, about half of the novel sounds (5% of trials) were designed to be particularly salient, based on their identifiable or alerting nature (e.g., a siren), whereas the other novel sounds (5% of trials) were considered minimally salient (e.g., various types of noise), as were the deviant tones and standard tones. The deviant tones served as a control for changes in the environment not associated with a range of saliency. The standard and deviant tones were created with the software Matlab 6.5 (The MathWorks Inc., Natick, MA). The novel sounds were selected from a variety of sources, and all the sounds and tones were modified with the software GoldWave v5.06 (GoldWave Inc.; Newfoundland, Canada) to be 500 ms in length and normalized to the same loudness. Prior to starting the task, participants were told that they would hear a sound, usually a tone, when the circle appeared but that the sound was completely task-irrelevant, and they should simply ignore the sound and respond as accurately and quickly as possible to the numbers. Each of the four runs lasted 8.8 min, consisting of 225 scans and 150 auditory–visual stimuli pairs. 126 standard tones, 8 deviant tones, and 16 novel sounds were presented per run. On average, 17 s separated consecutive nonstandard sounds (novel and deviant). During acquisition of the structural image (see fMRI imaging section below), the participants performed a short practice consisting of 10 auditory–visual pairs to familiarize the participant with the task and ensure that the participants could hear the auditory stimuli during scanning. Only standard tones were used in the practice session.
Immediately following the scanning session, each participant completed a rating questionnaire to assess the identifiable, alerting, and pleasurable nature of the novel sounds. Specifically, participants listened to each novel sound and indicated whether or not they could identify the sound and how certain they were regarding their judgment. They rated their confidence on a scale from 1 (not at all confident) to 5 (extremely confident). We considered a sound identifiable to a given participant only if it was correctly identified and confidence was rated as 4 or 5. Using a one-way repeated measures ANOVA, we compared reaction times to the numerical task-stimuli preceded by deviant tones, identifiable novel sounds, and unidentifiable novel sounds. Post hoc comparisons were computed using the Tukey test. Participants also indicated whether they considered the sound alerting (yes or no) and rated the pleasurable nature of each sound on a scale from 1 (extremely unpleasant) to 5 (extremely pleasant), 3 being neutral. A one-way repeated measures ANOVA was performed to compare reaction times to the numerical task-stimuli following novel sounds associated with different pleasure rating levels. In order to be consistent with the data entered in the parametric modulation in the fMRI analysis (see below), the reaction times were expressed as percent change from the median reaction time following the standard tones in a given run.
Scanning was performed on a 3.0 T Siemens Magnetom Trio whole-body scanner. For each participant, a T1-weighted structural image was acquired for anatomical reference, followed by four whole-brain functional runs of 225 scans each to measure the T2*-weighted blood oxygenation level-dependent (BOLD) effect (gradient-recall echo-planar imaging; repetition time: 2350 ms; echo time: 40 ms; flip angle: 90°; 64 × 64 matrix; field of view: 192 mm; 35 3.5 mm axial slices acquired parallel to the anteroposterior commissural line, with an interleaved order of slice acquisition). Head movement during scanning was minimized with padding.
The data were analyzed using Statistical Parametric Mapping (SPM2) (Friston et al., 1995b). Slice timing correction was used to adjust for time differences due to multislice imaging acquisition. Motion correction to the first functional scan was performed within participants using a six-parameter rigid-body transformation. For each individual, the mean of their functional images was spatially normalized to the Montreal Neurological Institute (MNI) template conforming to the Talairach orientation system (Talairach and Tournoux, 1988) by applying a 12-parameter affine transformation followed by nonlinear warping using basis functions (Ashburner and Friston, 1999). The computed transformation parameters were applied to all the functional images, interpolating to a final voxel size of 3 × 3 × 3 mm3. Images were subsequently spatially smoothed with an 8 mm isotropic Gaussian kernel.
A random-effects, event-related statistical analysis was performed with SPM2 (Friston et al., 1995a, 1999), in a two-level procedure. At the first level, a separate general linear model (GLM) was specified for each participant. The BOLD responses to standard tones, deviant tones, novel sounds, and error trials were modeled separately, time-locked to the sound onset, by convolving the onset vectors with a synthetic hemodynamic response function as implemented by SPM2. For the deviant tones and novel sounds, additional regressors were also included in the model to estimate the linear modulation of the hemodynamic response by reaction times to the immediately subsequent numeric task-stimuli (RT). RTs were entered as percent change from the median reaction time following the standard sounds in a given run (on average across participants, the novel RTs ranged from −29.0% to 111.7%; the deviant RTs ranged from −26.3% to 81.1%). The RT regressors were scaled by Euclidean normalization and mean-corrected by the SPM2 GLM routine. The Euclidean normalization transforms the set of parametric values entered into a set of new values such that their sum of squares will equal one. Because error trials were modeled separately, on average, 63.25 novel trials (out of 64 total) and 31.2 deviant trials (out of 32 total) were included in the model for each participant. At the model estimation stage, the data were high-pass filtered (threshold = 128 s), and serial correlations were accounted for by an autoregressive model of the first order. For each participant, contrast images were calculated for: NOVEL sounds modulated by RT; DEVIANT tones modulated by RT; and ‘‘(NOVEL modulated by RT) > (DEVIANT modulated by RT).’’ The individual contrast images were then entered into a second level analysis, using a one-sample t test. Due to the a priori hypothesis concerning the striatum, the resulting summary statistical map for the contrast ‘‘(NOVEL modulated by RT) > (DEVIANT modulated by RT)’’ was thresholded at P < 0.001 (uncorrected for multiple comparisons) with a voxel extent of 10 contiguous voxels (Friston, 1997). We separately considered the contrast, ‘‘DEVIANT modulated by RT’’ at a lenient threshold of P < 0.50 uncorrected for multiple comparisons, to confirm negative results.
Both the deviant tones and novel sounds occurred infrequently relative to the standard tones, but the current experimental design contained twice as many novel sounds as deviant tones so the novel category could have a wide range of saliency associated with the sounds. To assure that the different number of events in the two categories did not affect the results, we performed a separate analysis in which we randomly divided the novel sounds into two groups, each with the same number of events as the deviant tones. We created a new statistical model with separate regressors for each of the novel groups and performed the original fMRI analysis separately for each of the two groups of novel sounds.
During the scanning session, skin conductance responses (SCRs) were acquired while participants performed the experimental task. The signal was acquired with the MP150 system (Biopac Systems, Goleta, CA) equipped with AcqKnowledge 3.7 software (Biopac System). The SCR data were sampled at 125 Hz and a 1 Hz low-pass filter and a 0.05 Hz high-pass filter were applied to the data during acquisition. Interference from the scanner produced an artifact in the SCR signal in the form of spikes of high amplitude and short duration. These artifacts were subsequently filtered out with the software Matlab 6.5 (The MathWorks Inc., Natick, MA) using an ad-hoc moving average process. The filter calculated the absolute value of the difference between a given data point (i) and the following point (i + 1), and if the resulting value was greater than 0.01 μS, then i was replaced by ((i − 1) + (i + 5))/2. To analyze the SCR to the deviant tones and novel sounds (separated by an average of 17 s), we computed the integral of the SCR signal over a 5 s interval, beginning 2 s following stimulus onset and ending 7 s following stimulus onset, which includes most of the SCR rise, peak, and return to baseline. To compare the SCR to the deviant tones, identifiable novel sounds, and unidentifiable novel sounds, we performed a one-way repeated measures ANOVA on the resulting data. Post hoc comparisons were computed using the Tukey test.
On average, participants made less than 3 errors per run. The SCR and reaction time data were used to confirm that the novel sound category did contain sounds associated with different levels of saliency, as intended. The novel sounds were designed to vary in their saliency based on their identifiability, and therefore, to confirm the saliency manipulation, we divided the novel sounds into two categories for the behavioral and physiological data analysis: identifiable (‘‘high salience’’) and unidentifiable (‘‘low salience’’). The deviant tones served as a control for a minimally salient change in the environment. A one-way repeated measures ANOVA on the reaction times to the number stimuli in the classification task yielded a significant effect of preceding sound type [ F(2,38) = 7.527; P = 0.002]. Post hoc comparisons (Tukey test) revealed that the median reaction times to the numerical task-stimuli following ‘‘high salience’’ novel sounds (mean ± SEM = 593.7 ± 24.6 ms) were significantly longer than median reaction times to the numerical task-stimuli following ‘‘low salience’’ novel sounds (mean ± SEM = 551.2 ± 19.8 ms; P = 0.002) and deviant tones (mean ± SEM = 561.1 ± 20.0 ms; P = 0.020). The median reaction times following ‘‘low salience’’ novel sounds and deviant tones were not significantly different from each other ( P = 0.662). Furthermore, a one-way repeated measures ANOVA on the integrated skin conductance response (SCR) following the sounds yielded a significant effect of sound type [ F(2,38) = 7.482; P = 0.002]. Post hoc comparisons (Tukey test) revealed that the integrated SCR following ‘‘high salience’’ novel sounds (mean ± SEM = 1.50 ± 0.07 μS s) was greater than the integrated SCR following ‘‘low-salience’’ novel sounds (mean ± SEM = 1.40 ± 0.06 μS s), with a strong trend toward significance ( P = 0.059), and significantly greater than the integrated SCR following deviant tones (mean ± SEM = 1.34 ± 0.04 μS s; P = 0.001). The integrated SCR following ‘‘low salience’’ novel sounds and deviant tones was not significantly different from each other ( P = 0.319).
To directly test the hypothesis that striatal activity increases with the degree of saliency associated with each novel sound, we modeled the expected brain response to novel stimuli as a synthetic hemodynamic response function whose amplitude was parametrically modulated by the corresponding reaction time to the immediately subsequent task-related numerical stimulus (‘‘NOVEL modulated by RT’’). As a control for variations in reaction times that were not due to the saliency of the preceding sounds—for example, consequences of task-unrelated thoughts or fatigue—we adopted a similar parametric modulation of the hemodynamic response function to the deviant tones (‘‘DEVIANT modulated by RT’’). The contrast of interest for determining the relationship between striatal activity and the degree of stimulus saliency was, therefore, ‘‘(NOVEL modulated by RT) > (DEVIANT modulated by RT).’’ We performed a whole-brain analysis, but because of a specific a priori hypothesis concerning the striatum, maps were thresholded at P < 0.001, uncorrected for multiple comparisons, voxel extent = 10 voxels (Friston, 1997). At this threshold, significant activations were only observed in the striatum (Fig. 2), specifically in the right caudate (peak MNI coordinates: 15, 3, 18; cluster size = 24 voxels; peak z score = 3.99) and the left caudate (peak MNI coordinates: −9, 6, 3; cluster size = 12; peak z score = 3.66). The ventral striatum did not contain significant activations at the original threshold, but a significant activation in the left nucleus accumbens (peak MNI coordinates: −15, 3, −12; peak z score = 3.10) resulted at a slightly less stringent threshold ( P < 0.005; peak P = 0.001; cluster size = 6 voxels), which may be reasonable given the small size of this structure; however, such a threshold may not adequately protect against type I errors.
The novel sound category contained twice as many sounds as the deviant category to ensure a wide range in saliency. Even when we randomly divided the novel sounds into two groups (to have an equal number of events in the novel and deviant categories) and reanalyzed the data for the two novel samples separately, the striatal activations were similar to the original analysis for both novel groups, but less extensive (group 1: P < 0.001; peak MNI coordinates: 12, 6, 6; cluster size = 18 voxels; peak z score = 3.69; and P < 0.005; peak MNI coordinates: −12, 0, 12; cluster size = 5 voxels; peak z score = 3.09; group 2: P < 0.005; peak MNI coordinates: 15, 6, 9; cluster size = 3 voxels; peak z score = 2.80; and P < 0.005, peak MNI coordinates: −12, 12, 3; cluster size = 13 voxels; peak z score = 3.15).
Because the effect ‘‘DEVIANT modulated by RT’’ served as a control for variations in reaction times that were unrelated to the sound stimuli, it was important to confirm a lack of striatal activations in this effect. We therefore considered ‘‘DEVIANT modulated by RT’’ separately at a highly lenient threshold of P < 0.50, uncorrected for multiple comparisons. Even at such a threshold, ‘‘DEVIANT modulated by RT’’ did not reveal striatal activation (Fig. 3B).
When judging the novel sounds in terms of their pleasantness using a rating scale ranging from 1 (extremely unpleasant) to 5 (extremely pleasant), with 3 being neutral, participants rated the sounds 2.6 on average (SEM = 0.06). We used the delay in reaction time following novel sounds in each pleasure rating level to confirm that the correlation between striatal activity and magnitude of reaction time delays following novel sounds was unrelated to pleasure (reward). The reaction times to numbers preceded by novel sounds (expressed as percent change from the median reaction time following the standard tones in a given run) were not significantly different across the five levels of pleasantness ratings associated with the novel sounds [ F(4,57) = 0.294; P = 0.881; one-way repeated measures ANOVA].
While previous studies have reported that the dopamine/striatal system is engaged when one unexpectedly encounters salient stimuli in the environment, the present findings explicitly link the level of activity in the striatum with the level of saliency associated with a particular event. Specifically, striatal activations in the bilateral caudate, and to a lesser extent in the left nucleus accumbens, are related to the degree that a stimulus interrupts the current cognitive focus. The novel sounds in the present experiment served as stimuli of interest because they were unexpected events, designed to be associated with varying levels of saliency based on their identifiability and alerting nature—ranging from minimally salient to highly salient. As such, the novel sounds elicited varying delays in reaction times to the subsequent numerical task-stimuli. The deviant tones did not vary in their saliency, and therefore variations in reaction times following deviant tones were attributed to task-unrelated thoughts or fatigue, rather than attributed to an effect of the preceding deviant tone itself.
The saliency manipulation within the novel sound category was confirmed by the SCR and behavioral data. SCRs provide a physiological measure indicating orienting responses or saliency of a particular event (Boucsein, 1992). The SCRs following novel sounds were significantly greater than SCRs following deviant tones and close to significantly greater than SCRs following unidentifiable novel sounds, confirming that the novel sound category did indeed contain sounds associated with greater saliency compared to the deviant tones. The variance in reaction times following novel sounds and deviant tones was equal, but the reaction times to numerical task-stimuli preceded by identifiable novel sounds were significantly longer than the reaction times to numerical task-stimuli preceded by unidentifiable novel sounds and deviant tones, indicating that the identifiable nature of the novel sounds affected their saliency.
The novel sounds were separated into two groups (identifiable and unidentifiable) for behavioral confirmation of our saliency manipulation, but the saliency of the novel sounds, as reflected in the reaction times to subsequent numerical task-stimuli, lays on a continuum rather than being divided into two discrete groups. Therefore, we opted for a parametric modulation approach to assess the relationship between brain activity and the level of saliency associated with a particular event. The resulting striatal activation in our contrast of interest, ‘‘(NOVEL modulated by RT) > (DEVIANT modulated by RT),’’ was specifically attributed to varying degrees of saliency associated with the novel sounds because ‘‘DEVIANT modulated by RT’’ controlled for variations in reaction times that were not due to the nature of the sound stimuli. Even at an extremely lenient threshold of P < 0.50, striatal activations were not observed for ‘‘DEVIANT modulated by RT,’’ providing strong evidence that the observed striatal response in the contrast, ‘‘(NOVEL modulated by RT) > (DEVIANT modulated by RT),’’ reflected a correlation between striatal activity and the degree of saliency of the novel sounds, rather than task-unrelated thoughts or fatigue.
The sound stimuli in the task design were completely task-irrelevant, which was also important for the interpretation of the results. If a contingency between sound type and number type existed or was perceived by the participants (i.e., the occurrence of a novel sound indicated an upcoming odd or even number), then a prediction error, rather than the saliency of the novel sounds, could have elicited delayed reaction times and the resulting increase in striatal activation. To avoid such a confound, the task was carefully designed so that within each run, the novel sounds were followed by all the numbers, 2–9, twice (16 novel trials total), and the deviant tones were followed by all the numbers once (8 deviant trials total), both in random order. Although no contingency between sounds type and number type existed, to ensure that a contingency was not perceived by the participants, they were explicitly instructed that the sounds were completely task-irrelevant and could be ignored. Furthermore, in post-session interviews, no participants indicated that he/she perceived such a contingency, even when asked.
Although striatal activations follow categorically defined salient events, different interpretations of such data have led to the view that the striatum and its dopaminergic inputs process unexpected rewards or reward prediction error exclusively, rather than saliency in general (Schultz, 1998; Ungless, 2004). Furthermore, data that implicate activity in the striatum with reward-related stimuli are more numerous than data that link the dopamine/striatal system with salient, nonrewarding events, which have been studied less frequently in humans. Rewards have been defined as events that elicit approach and consumatory behavior, serve as positive reinforcers of behavior, and induce subjective feelings of pleasure (Schultz, 1998). In order to be able to attribute the present results to saliency in general, rather than reward, we made sure that the novel sounds were not rewarding to our participants and, perhaps more importantly, that the saliency level of the sounds was not confounded by whatever pleasure they might have conferred. In terms of the pleasantness associated with the novel sounds, participants rated the novel sounds slightly below neutral. Thus, the novel sounds were not considered rewarding and did not elicit hedonic feelings. Furthermore, the reaction times to numbers preceded by novel sounds did not differ across the five levels of pleasantness ratings associated with the novel sounds, indicating that the saliency associated with a particular novel sound was unrelated to its pleasure rating.
The contention that striatal activation reflects the potential importance of a stimulus, and therefore plays a role in the reallocation of resources, is still consistent with decades of research linking the dopamine/striatal system with coding unexpected rewards and reward-related events. Unexpected rewards are highly salient stimuli that are prone to interrupt and redirect the current focus of attention and behavior. In fact, as it has been recently pointed out (Maunsell, 2004), reward manipulation and attentional effects have not been clearly separated in many previous reward studies. Thus, the so-called ‘‘reward response’’ of the striatum may be more appropriately, and more generally, categorized as a ‘‘saliency response.’’ Recent studies have suggested a link between reward magnitude and level of activity in the striatum of monkeys (Cromwell and Schultz, 2003) and humans (Delgado et al., 2003), which, in accordance with the present findings, may be related to higher magnitude rewards receiving higher saliency assignments.
In conclusion, the present findings demonstrate that striatal activity corresponds to the degree to which unexpected stimuli perturb cognitive resources. These results provide strong evidence for the claim postulated by Redgrave et al. (1999b) that striatal activation drives the reallocation of available resources to process salient events with priority, rather than just providing a reward signal, especially when no obvious reward is present. Beyond their contribution to the interpretation of striatal activation, the present findings have implications regarding a variety of neuropsychiatric disorders that affect the dopamine/striatal system, including schizophrenia, attention deficit hyperactivity disorder, and drug addiction. Current theories that relate these disease states with a breakdown of proper saliency assignments (Kapur, 2003; Volkow et al., 2004a,b) are well supported by our results, and their future clinical development may eventually lead to better preventions and treatments.
We thank the Biomedical Imaging Technology Center at Emory University, especially Keith Heberlein and Robert Smith for their technical assistance during scanning sessions. We also thank Rosa Aurora Chavez-Eakle for helpful comments and Carles Escera for providing sound stimuli. Novel sounds were also selected from various websites: a1freesoundeffects.com, soundamerica.com, simplythebest.com, and cepl.nyspi.org. This work was gratefully supported by grants from the National Institutes of Health, DA00367 and DA016434 to G.S.B. and F31 MH067348 to C.F.Z.