PMCCPMCCPMCC

Search tips
Search criteria 

Advanced

 
Logo of nihpaAbout Author manuscriptsSubmit a manuscriptHHS Public Access; Author Manuscript; Accepted for publication in peer reviewed journal;
 
J Neurosci. Author manuscript; available in PMC 2010 June 2.
Published in final edited form as:
PMCID: PMC2830793
NIHMSID: NIHMS175688

Spatially global representations in human primary visual cortex during working memory maintenance

Abstract

Recent studies suggest that visual features are stored in working memory (WM) via sensory recruitment, or sustained stimulus-specific patterns of activity in cortical regions that encode memoranda. One important question concerns the spatial extent of sensory recruitment. One possibility is that sensory recruitment is restricted to neurons that are retinotopically mapped to the positions occupied by the remembered items. Alternatively, specific feature values could be represented via a spatially global recruitment of neurons that encode the remembered feature, regardless of the retinotopic position of the remembered stimulus. Here, we evaluated these alternatives by requiring subjects to remember the orientation of a grating presented in the left or right visual field. fMRI and multivoxel pattern analysis were then used to examine feature-specific activations in early visual regions during memory maintenance. Activation patterns that discriminated the remembered feature were found in regions of contralateral visual cortex that corresponded to the retinotopic position of the remembered item, as well as in ipsilateral regions that were not retinotopically mapped to the position of the stored stimulus. These results suggest that visual details are held in WM through a spatially global recruitment of early sensory cortex. This spatially global recruitment may enhance memory precision by facilitating robust population coding of the stored information.

Keywords: Working Memory, Perception, Sensory Recruitment, fMRI, Pattern Classification, Positional Specificity

Working Memory (WM) supports the maintenance of information in an online state. Human neuroimaging (Serences et al., 2009a; Harrison & Tong, 2009) and single-unit studies in non-human primates (Miller et al., 1996; Super et al., 2001) suggest that WM storage is mediated by sensory recruitment, or sustained activity in early cortical regions that encode information. For example, Serences et al. (2009a) required subjects to remember the orientation or color of a foveally presented grating. Using fMRI and multivoxel pattern analysis (MVPA), they demonstrated that sustained patterns of activation in primary visual cortex (V1) during WM storage discriminated the specific value of the remembered attribute, even in the absence of changes in overall response amplitude.

One important question concerns the spatial extent of sensory recruitment. On one hand, recruitment may be confined to cortical regions that were activated during the encoding of a stimulus. Dill and Fahle (1998) showed subjects two dot patterns separated by a short interval and asked them to determine whether these stimuli were identical. Relative to a condition in which the stimuli appeared in the same spatial location, performance was reliably reduced when they appeared in different locations (see also Zaksas et al, 2001; Hollingworth, 2006; 2007). These findings suggest that object representations held in WM are spatially-specific, thus motivating the hypothesis that sensory recruitment is confined to cortical regions that are retinotopically mapped to the spatial location occupied by the memoranda.

Alternatively, sensory recruitment may extend to cortical regions that were not engaged during stimulus encoding. For example, a specific orientation could be represented via the broad recruitment of cortical neurons that encode orientation, regardless of the retinotopic position of the remembered stimulus. This possibility is suggested by studies demonstrating a spatially global spread of feature-based attention during perception (Treue & Martinez-Trujillo, 1999; Serences & Boynton, 2007a). For example, Serences and Boynton (2007a) instructed subjects to monitor one of two directions of motion carried by stimuli located in one hemifield of a display. Using fMRI and MVPA, these authors demonstrated that patterns of activity in visual areas ipsilateral to the stimuli discriminated the attended direction of motion, despite the fact that these regions were not being driven by a bottom-up signal. During WM maintenance, this kind of spatially global recruitment might enhance the precision of mnemonic representations by recruiting additional neurons to facilitate robust population coding of the stored information (Pouget et al., 2003).

Here, we attempted to determine whether sensory recruitment during WM maintenance is spatially local or global. Subjects were required to remember the orientation of a grating presented in the left or right visual field. Using fMRI and MVPA, we found that patterns of activity in early cortical regions (V1) both contralateral and ipsilateral to a stimulus discriminated the remembered orientation. Under identical sensory conditions where WM maintenance was not required, these effects were abolished. Furthermore, patterns of activity in contralateral and ipsilateral ROIs were qualitatively similar during perception and WM maintenance, consistent with sensory recruitment accounts of WM storage.

Methods

Subjects

Twenty neurologically intact subjects participated in a single two-hour scanning session. All subjects gave informed written consent in accordance with the Office for the Protection of Human Subjects at the University of Oregon. Data from three subjects were discarded due to technical difficulties (n = 2) or voluntary withdrawal from the study (n = 1). Prior to scanning, each subject participated in a 1.5h training session to identify performance thresholds in each behavioral task (see Staircase Procedure, below). Subjects were compensated at a rate of $8 per hour for behavioral testing and $25 per hour for scanning.

Working Memory Task

Stimuli were generated in Matlab (Version 7.1; Mathworks, Natick, MA) using Psychophysics Toolbox software (Brainard, 1997; Pelli, 1997) and rendered in black on a medium-grey background via a rear projection system. Subjects were instructed to maintain fixation on a central dot (subtending approximately 0.2° from a viewing distance of 58 cm) throughout each scan.

The sequence of events on each trial is presented in Figure 1. At the start of each trial, a “sample” grating stimulus (radius 4°, 1 cycle/°) was displayed in the upper left or right portion of the display (horizontal and vertical eccentricity of ±8.5° and +5°, respectively). The sample was rendered with an orientation of 45° or 135° (jittered on a trial-by-trial basis by a randomly selected value in the range of ±10° to discourage the use of categorical labels), and cycled on and off at a rate of 2 Hz (250 ms on, 250 ms off) for a total of 6 seconds. To attenuate the potency of retinal afterimages, the spatial phase of the sample was randomized on each cycle. After the sample epoch, a 2 second change in the color of the fixation point (to red or green) instructed subjects to remember the orientation of the sample for comparison with a subsequent test grating (a “memory” trial) or wait for the start of the next trial (a “no-memory” trial). For nine subjects, a green cue was used to denote memory trials and a red cue denoted no-memory trials. For the remaining eight subjects, this mapping was reversed. The cue was followed by a 14 second delay period. On memory trials, the delay period was followed by the presentation of a static, 1 second test grating and a 2 second inter-trial interval. Subjects made a button press response during this 3-second interval to indicate whether the sample and test stimuli shared the same orientation (50% of trials). On no-memory trials, the delay period was instead followed by an additional 3 seconds of fixation. Stimulus orientation (45° or 135°), position (left or right visual field), and trial type (memory or no-memory) were balanced within each scan (where “scan” refers to a continuous, 404s-long block of 16 trials). Subjects completed a total of 6 (n = 1), 7 (n = 7), 8 (n = 7), or 9 (n = 2) scans as time permitted.

Figure 1
WM Task. On each trial of the task, observers were shown a sample grating in the upper left or right visual field for 6s. A subsequent change in the color of the fixation point (to red or green; shown here as white) informed observers whether they should ...

Perceptual Monitoring (PM) Task

During the same scanning session, subjects also performed a perceptual monitoring task. The stimuli and display parameters were closely modeled after those used in the WM task. On each trial, a sample grating oriented to 45° or 135° (randomly jittered on a trial-by-trial basis by a value in the range of ±10°) was presented in the upper left or right visual field. This stimulus cycled on and off at a rate of 2 Hz (250ms on, 250 ms off) for the duration of each 15 second trial. Subjects were instructed to monitor this stimulus and make a manual button press response whenever they detected a brief (one cycle) change in its orientation (a target event; these occurred at unpredictable intervals 2–3 times per trial; see Staircase Procedure for information about the size of these angular deviations). Each 260s-long scan contained a total of 12 real and 3 “null” (i.e., 15 seconds of fixation) trials, and each observer completed a total of 3 (n = 2) or 4 (n = 15) scans as time permitted.

Staircase Procedure

To ensure that both behavioral tasks were sufficiently challenging, we adjusted their difficulty for each subject during a separate behavioral testing session (completed 1 to 3 days prior to the scanning session). Sample-test (for the WM task) and sample-target (for the PM task) disparities were independently adjusted for each orientation category (45° or 135°) until a criterion level of performance (75% accuracy) was reached. The resulting orientation disparities were used to set sample-test and sample-target disparities during the scan session.

Eye Tracking

To assess compliance with fixation instructions during PM and WM scans, eye position was visually monitored with the aid of an ASL (Bedford, MA) 5000 infrared tracking system in all subjects. For four subjects, eye position data during WM scans were explicitly recorded. This enabled more fine-grained assessments of whether there were any reliable differences in eye position as a function of the orientation stored in working memory. Recording was performed at 60 Hz, and data were filtered for blinks and corrected for linear drift offline.

fMRI Data Acquisition and Analysis

fMRI data were collected using a 3T Siemens Allegra system at the Robert and Beverly Lewis Center for Neuroimaging at the University of Oregon. Anatomical images were acquired using a spoiled-gradient-recalled T1-weighted sequence that yielded images with a 1-mm3 resolution. Whole-brain echo-planar images (EPIs) were acquired in 33 transverse slices (3mm in-plane resolution, 2000-ms TR, 30-ms TE, 90° flip angle, 64 × 64 matrix, 192-mm FoV, 3.5-mm slice thickness, no gap). EPIs were slice-time corrected, motion corrected (both within and between scans), and high-pass filtered (three cycles per run). Image preprocessing and data analysis were performed using BrainVoyagerQX (Version 1.9) and custom time-series and pattern-classification routines written in Matlab.

Retinotopic Mapping and Voxel Selection

Retinotopic mapping data were acquired using a rotating checkerboard stimulus flickering at 8 Hz and subtending 45° of polar angle (following Engel et al., 1994; Sereno et al., 1995). Each observer completed one scan lasting 480s. This procedure was used to identify visual areas V1, V2v, V3v, V4v, V2d, V3d, and V3a in each cortical hemisphere. To aid in the visualization of these regions, data were projected onto a computationally inflated representation of each observer’s grey/white matter boundary.

To identify spatially selective voxels in these visual areas, we constructed a general linear model (GLM) with a single boxcar regressor (denoting stimulus location, i.e., left vs. right visual field) using data from all four PM scans. This regressor was then convolved with a gamma function to account for the assumed shape of the hemodynamic response (see Boynton, Engel, Glover, & Heeger, 1996). Voxels that showed a stronger response during epochs of contralateral (relative to ipsilateral) stimulation (p < .05, Bonferroni corrected) were used to define ROIs in V1, V2v, V3v, hV4v, and V3a (as stimuli were presented in the upper visual field, ROIs in V2d and V3d typically contained too few voxels to enable a meaningful analysis; see Supplementary Table 1 for information on ROI sizes). The mean response amplitude across the 60 most spatially selective voxels in each visual region (collapsed across corresponding ROIs in each cortical hemisphere) identified using this method is shown in Supplementary Figure 1.

Multivoxel Pattern Analysis

MVPA provides a powerful complement to conventional univariate analyses of fMRI data. Posterior cortical regions such as V1 contain submillimeter columns of neurons that are selective for different stimulus features such as orientation. MVPA assumes that if a particular fMRI voxel contains slight preponderance of columns tuned to a specific feature value, it should give rise to a weak but detectable response bias. By considering patterns across multiple weakly selective voxels, it is possible to infer the specific feature values that subjects are attending (e.g., Haxby et al., 2001; Haynes & Rees, 2005; Kamitani & Tong, 2005; Norman et al., 2006; Peelen & Downing, 2007; Serences & Boynton, 2007a; 2007b; Serences et al. 2009b), imagining (Stokes et al., 2009) or remembering (Serences et al., 2009a; Harrison & Tong, 2009), even in the absence of sustained changes in overall response amplitude (Serences et al., 2009a).

To perform MVPA, we first extracted the raw time series from each voxel in a given ROI during a time period extending from 6 to 16 seconds following the offset of the sample stimulus (i.e., 12–22 seconds after the start of each trial). Each time series was normalized on a scan-by-scan basis using a z transform and sorted into one of eight bins according to three factors: ROI (contralateral or ipsilateral to the stimulus location), orientation (45° or 135°), and memory condition (memory or no-memory). We then defined a “training” data set using the data from all but one scan (here, “scan” refers to a single, continuous block of trials; thus, the training and test sets were always independent). Within the training data set, we computed activation patterns comprising the mean response of each voxel during 45° and 135° trials. This was done independently for each location and memory condition pair, yielding a total of 8 activation patterns. Data from the remaining scan were used to define a “test” data set using an analogous approach. Before classification, we removed the overall mean from each activation pattern in the training and test sets. Finally, we trained a Support Vector Machine (SVM; specifically, the OSU-SVM implementation, http://sourceforge.net/projects/svm) to discriminate sample orientation using patterns from the training set, then used it to infer the orientation of the sample stimulus on each trial in the test set. Classification was performed separately for each stimulus location and memory condition pairing, so chance performance was always 50%. This analysis was iterated using a hold-one-out cross-validation procedure (see, e.g., Kamitani & Tong, 2005) until data from every scan had been used as the test set. Overall classification accuracy for each observer was then defined as the average classification accuracy for each observer across all 6, 7, 8, or 9 permutations (depending on how many WM scans the observer completed) of the hold-one-out procedure.

Results

The primary goal of this study was to use fMRI and MVPA to examine the spatial extent of sensory recruitment during WM maintenance. We examined response profiles within several regions of occipital cortex (V1, V2v, V3v, hV4v, V3a); however, we focus on V1 as previous demonstrations show that this region displays robust patterns of orientation-selective activity during both perception (Haynes & Rees, 2005; Kamitani & Tong, 2005; 2006; Serences & Boynton, 2007a; Serences et al. 2009b) and WM maintenance (e.g., Serences et al., 2009a; Harrison & Tong, 2009). For all analyses, data are presented as collapsed across corresponding ROIs in each visual area (e.g., left and right V1) as no differences in classification accuracy were observed across cortical hemispheres. All analyses reported here are based on the 60 most spatially selective voxels in each ROI. Specific V1 ROIs in three subjects contained fewer than 60 voxels. For these subjects, analyses were performed using all available voxels within each ROI. Critically, we observed no systematic relationship between ROI sizes and overall classification accuracy, indicating that the findings reported here were not unfairly influenced by subjects with larger ROIs. In addition, all findings are robust across a broad range of pattern sizes (i.e., 50 – 75 voxels).

Behavioral performance in the PM and WM tasks is summarized in Table 1. Overall, orientation discrimination thresholds were higher during perceptual monitoring (p < .001). However, no differences in discrimination thresholds were observed between orientation categories (45° and 135°) in either task, and overall accuracy remained at a level near that dictated by the staircase procedure.

Table 1
Mean (±S.E.M.) discrimination thresholds and accuracy for 45° and 135° trials during PM and WM maintenance.

Multivoxel Pattern Analysis

If sensory recruitment operates in a spatially global fashion, then sustained patterns of activation in ROIs both contralateral and ipsilateral to a remembered stimulus should discriminate its orientation. Figure 2 depicts the results of an MVPA analysis based on the 60 most spatially selective voxels (as identified using data from the perceptual monitoring task, see Retinotopic Mapping and Voxel Selection, Methods) from each V1 ROI during a time period extending from 6 to 16 seconds following the offset of the sample stimulus. An analysis of variance revealed a main effect of memory condition, F(1, 16) = 11.07, p < .001, no effect of stimulus location (contralateral or ipsilateral; F(1, 16) = 1.49, p > .24), and no interaction between these two factors, F(1, 16) < 1. Planned comparisons revealed that classification accuracy was higher on memory trials relative to no memory trials. Critically, this was true in both contralateral (t(16) = 3.48, p < .01) and ipsilateral (t(16) = 2.84, p < .05) ROIs. Moreover, we failed to observe consistent above-chance classification in either ROI on no memory trials (t(16) = .19, p > .8 and t(16) = −1.99, p > .06 for contralateral and ipsilateral ROIs, respectively; apparent below chance classification in ipsilateral ROIs during no-memory trials was not robust across variations in pattern sizes – see Supplementary Figure 2). As sensory conditions were equivalent during memory and no memory trials, these findings suggest that the above-chance classification observed in both ROIs during memory trials reflects active WM maintenance rather than a lingering sensory response to the sample stimulus. Qualitatively similar findings were observed across a range of pattern sizes (50, 75 voxels; see Supplementary Figure 2) and classification algorithms (linear discriminant function), indicating that above-chance classification is not idiosyncratically dependent on the use of a SVM.

Figure 2
Multivoxel Pattern Analysis of WM Data. The results of a multivoxel pattern analysis using the 60 most spatially selective voxels in each V1 ROI during a time period extending from 6 to 16 s following the offset of the sample stimulus. The horizontal ...

MVPA was also applied to data from extrastriate visual areas V2v, V3v, hV4v, and V3a. However, contralateral and ipsilateral ROIs in each of these areas failed to support above chance classification accuracy during memory or no-memory trials (Supplementary Figure 3; see Serences et al., 2009a, for similar findings using foveal stimuli). Qualitatively similar results were observed over a wide range of pattern sizes (i.e., 50, 60, and 75 voxels) and different classification algorithms (e.g., linear discriminant function).

These results are consistent with the hypothesis that sensory recruitment during WM maintenance is spatially global. However, there are a number of possible issues that might also account for these effects. One such issue is head motion. Subtle differences in head position during 45° and 135° trials could differentially alter the intensity of fMRI voxels and bias the results of MVPA. To quantify any differences in head position as a function of the remembered orientation, we computed the mean change (relative to the start of each trial) in six motion parameters (translation and rotation in the X, Y, and Z planes) during a window extending from the 6–20s following the start of each memory trial (an earlier time window than that used for MVPA was selected to account for hemodynamic lag). These data were then submitted to a 2 (sample orientation, 45° or 135°) × 2 (stimulus location, left or right visual field) × 6 (motion parameter) ANOVA. This analysis revealed a main effect of visual field, F(1,16) = 17.49, p < .001, and a marginal effect of motion parameter, F(1.92, 31.87) = 2.24, p = .06 (a Greenhouse-Geisser correction was applied to all model terms containing motion parameter data due to nonsphericity of this factor). Critically, this analysis failed to reveal any effects of stimulus angle, F(1,16) < 1, nor did this factor interact with any other term (all ps > .15). As shown in Supplementary Figure 4, subjects exhibited a tendency to move or rotate in a direction consistent with the spatial location of a remembered stimulus. However, we failed to observe any biases in head position as a function of stimulus orientation, suggesting that above-chance orientation classification observed on memory trials was not solely due to subtle differences in head motion.

A second potential issue is eye position. Although subjects were instructed to maintain fixation for the duration of each WM scan (as assessed by visually monitoring ongoing eye position in all subjects with the aid of an ASL tracking system) it is possible that they made subtle eye movements consistent with the orientation of a remembered grating. These eye movements could produce systematic modulations in activation patterns within retinoptically organized regions of visual cortex that could bias the results of a MVPA. To further assess compliance with fixation instructions and examine whether there were any subtle differences in eye position as a function of the remembered orientation, eye position data were recorded and tagged with trial-by-trial information regarding the specific angle stored for four subjects during scanning. To quantify differences in eye movements during WM scans, we identified all stable fixations (defined as a 200ms epoch during which eye position did not deviate more than 0.25°) that occurred outside of a 0.5° window encompassing the center of the screen during a period extending from 6–20s following the start of each memory trial. The X and Y positions of fixations made by each of the four observers are shown in Supplementary Figure 5. Critically, we observed no differences in the mean or variance of these fixations as a function of remembered orientation (45° or 135°) across several parameters (X position, Y position, polar distance [relative to the center of the screen], and polar angle; paired-samples t-tests, all ps > .3), suggesting that above chance classification accuracy during memory trials cannot be fully explained by subtle biases in eye movements.

Recent human neuroimaging (Serences & Boynton, 2007a) and single-unit (Martinez-Trujillo & Treue, 2004) studies in nonhuman primates have reported spatially global feature-based attention effects during perceptual processing. To examine whether similar effects could be observed in the present study, we performed a classification analysis on patterns observed in each ROI during a period 4–10s following the onset of the sample stimulus. Because the sample epoch of memory and no-memory trials were identical, both types of trial were included in this analysis. In contrast to the spatially global effects we observe during WM maintenance, neither contralateral nor ipsilateral V1 ROIs reliably discriminated the orientation of the sample stimulus (two-tailed t-tests against chance, all ps > .1). We suspect that this lack of generality is due to the relatively low salience of our stimuli. For example, MVPA was also applied to data from the PM task using a longer analysis window (6–16s following stimulus onset). Following Serences and Boynton (2007a), patterns of activity in contralateral and ipsilateral V1 ROIs reliably discriminated the orientation of a monitored grating. However, when the same analysis was repeated using a shorter analysis window (4–10s following stimulus onset), these effects were abolished. We speculate that with a longer sample epoch, patterns of activity in both contralateral and ipsilateral ROIs would support robust classification of sample orientation.

Because classification was carried out independently for each ROI, these findings do not establish whether contralateral and ipsilateral patterns within a given region are similar. To evaluate this possibility, we repeated our classification analysis using data from contralateral memory trials as the training set and examined whether this would allow accurate classification of the stored orientation when the same cortical ROI was ipsilateral to the sample stimulus (using the same hold-one-out cross validation procedure described above; see Multivoxel Pattern Analysis, Methods). Across a range of pattern sizes (50, 60, 75 voxels), this analysis failed to reveal above chance classification accuracy (one-sample t-tests against chance, all ps > .1). Thus, although patterns of activation in both contralateral and ipsilateral ROIs discriminate the orientation of a remembered stimulus (see Figure 2), the pattern of activity in a given ROI is not necessarily identical across trials in which that region is contralateral and ipsilateral to the stored stimulus (see Serences & Boynton, 2007a, for a qualitatively similar finding in the context of a perceptual discrimination task). We speculate that this lack of generalization may be due to differences in lingering bottom-up activity in when a given ROI is contralateral and ipsilateral to the stored stimulus. However, it is also possible that patterns of activation in ipsilateral ROIs may be qualitatively different from those observed in contralateral ROIs.

Event-related Analysis of WM Data

Data from the WM task were also submitted to a conventional univariate analysis. Figure 3 depicts the mean amplitude of the BOLD response in each V1 ROI during both memory and no-memory trials (no significant differences in mean response amplitude as a function of angle were observed; thus the data are shown collapsed across this factor). To quantify differences between memory conditions and ROIs during WM maintenance, we computed the mean BOLD response within each condition during a period extending from 6–16s following the offset of the sample stimulus (12–22s in Figure 2). A 2 (memory condition) × 2 (ROI) ANOVA of these data revealed a main effect of ROI, F(1, 16) = 11.69, p < .001, a main effect of memory condition, F(1, 16) = 8.45, p < .01, and a significant interaction between these factors, F(1, 16) = 10.72, p < .01. Curiously, mean response amplitudes on no-memory trials were higher than those on memory trials (M = .16 vs −.02, respectively; standardized units), particularly towards the end of the memory period. Mean response amplitudes were also greater in ipsilateral ROIs (M = −.08 vs. −.13, for contralateral and ipsilateral ROIs, respectively), though this effect was not robust across variations in voxel number. Qualitatively similar effects were also observed in extrastriate visual areas (see Supplementary Figure 7).

Figure 3
Event-related Analysis of WM Data. The mean response of the 60 most spatially selective voxels in V1 (collapsed across hemispheres) is shown as a function of memory condition and ROI. Data are collapsed across stimulus orientation (i.e., 45° vs. ...

Similarities between spatially global signals during perception and WM maintenance

Recent studies (Serences et al., 2009a; Harrison & Tong, 2009) have reported a high degree of similarity between stimulus-specific patterns of activation in early sensory regions during perceptual processing and WM maintenance, consistent with the hypothesis that WM involves the ingemination of a specific perceptual event. For example, Serences et al. (2009a) trained a classification algorithm to recognize stimulus-specific patterns in V1 using data from a perceptually-demanding orientation discrimination task, then used this classifier to successfully discriminate the identity of a remembered orientation during a separate WM task (see Harrison & Tong, 2009, for a similar demonstration). However, this analysis was restricted to cortical ROIs that corresponded to the location of the monitored or remembered stimulus. Here, we asked whether a qualitatively similar result would be observed in ROIs that were not retinotopically mapped to the position of the stimulus that was stored in WM (i.e., ROIs ipsilateral to the stimulus’ location). To examine this possibility, we trained a classifier to recognize stimulus orientation using data from PM scans. Recall that the stimuli used in PM scans were identical to those used in the WM task. However, during PM scans, stimuli cycled on and off at 2 Hz (250 ms on, 250 ms off) for the duration of each 15 second trial, and thus, WM maintenance was not explicitly required. The classifier trained using the PM task was then used to infer the specific orientation that subjects were remembering during each trial of the WM task.

Figure 4 depicts the results of this analysis using the 75 most spatially-selective voxels from each cortical ROI. Data from one subject were excluded from this analysis due to 0% classification accuracy in each ROI across all tested pattern sizes. This was likely due to a large motion artifact between PM and WM scans that interfered with alignment of the functional images. The findings reported here represent the remaining 16 subjects. Collapsed across ROIs, this analysis revealed a successful generalization between patterns of activity observed during PM and WM maintenance, but only when active memory maintenance was required (two-tailed t-tests against chance, p < .01 and p = .44 for memory and no-memory conditions, respectively; in addition, a one-tailed paired-samples t-test revealed significantly higher classification accuracy during memory trials, t(15) = 1.93, p = .036). Qualitatively similar results were also observed when 65 voxels were used to perform this analysis. Thus, the pattern of activity observed within a specific ROI – whether it was contralateral or ipsilateral to the sample stimulus – was similar in the PM and WM tasks. These findings lend further support to the hypothesis that WM involves the reiteration of a specific perceptual event. The same analysis failed to reveal consistent above chance classification in extrastriate regions V2v, V3v, hV4v, and V3a.

Figure 4
Global Orientation-selective Patterns are Similar during PM and WM Maintenance. A classification algorithm was trained to recognize orientation-selective patterns using the 75 most spatially selective voxels from each V1 ROI during PM (both LVF and RVF ...

These findings reflect conditions in which the sample stimulus appeared in the same spatial location during the PM and WM tasks. We next asked whether training on a contralateral stimulus in the PM task would allow successful classification of ipsilateral stimuli during the WM task. These data are shown in Supplementary Figure 7. This analysis failed to reveal consistent above-chance classification accuracy in any of the visual areas that we examined (two-tailed t-tests against chance, all ps > .15), suggesting that similarities between PM and WM maintenance may be restricted to situations where a monitored and remembered stimulus occupy the same spatial location.

Discussion

An emerging perspective – informed by human neuroimaging and single-unit recordings in non-human primates – is that WM storage is mediated by sensory recruitment, or sustained activity in posterior cortical regions that encode memoranda (Miller et al., 1996; Awh & Jonides, 2001; Super et al., 2001; Jonides et al., 2005; Pasternak & Greenlee, 2005; Postle, 2006; D’Esposito, 2007). Here, we attempted to determine the spatial extent of this recruitment. Positional specificity effects in WM (e.g., Dill & Fahle, 1998; Zaksas, et al., 2001; Hollingworth 2006; 2007) suggest that sensory recruitment is spatially local. However, several studies have reported spatially global feature-based attention effects during perceptual processing (e.g., Treue & Martinez-Trujillo, 1999; Saenz & Boynton, 2002; Martinez-Trujillo & Treue, 2004; Bichot et al., 2005; Serences & Boynton, 2007a). Given known similarities between mechanisms of visual attention and working memory (e.g., Desimone, 1996; Rainer, Assad, & Miller, 1998; Awh & Jonides, 2001), these findings suggest that sensory recruitment may be spatially global. Here, subjects were required to remember the orientation of a grating presented in one visual hemifield. Using fMRI and MVPA, we found that patterns of activity in early visual areas (e.g., V1) both contralateral and ipsilateral to this stimulus discriminated its orientation (see Figure 2). These findings cannot be explained by subtle differences in head or eye movements. Moreover, under identical sensory conditions where WM maintenance was not explicitly required, these effects were abolished. Thus, above-chance classification observed on memory trials was not caused by a sustained sensory response to the sample stimulus. Additionally, we found that patterns of activity in contralateral and ipsilateral ROIs were qualitatively similar during perceptual processing and WM maintenance (specifically, activity patterns in contralateral ROIs during PM were qualitatively similar to patterns observed in contralateral ROIs during WM maintenance, while activity patterns observed in ipsilateral ROIs during PM were qualitatively similar to patterns observed in ipsilateral ROIs during WM maintenance; see Figure 4), suggesting that similar mechanisms may support both processes. Together, these results suggest that sensory recruitment during WM maintenance is spatially global.

Several recent studies have reported successful classification of stimulus orientation in extrastriate visual areas during sensory encoding (Kamitani & Tong, 2005) and WM maintenance (Harrison & Tong, 2009). Likewise, studies of feature-based attention have documented spatially global effects in extrastriate visual areas (e.g., Treue & Martinez-Trujillo, 1999; Serences & Boynton, 2007a). However, the spatially global WM effects reported here were confined to primary visual cortex. We suspect that this difference is due to the relatively low salience of our stimuli. For example, the large foveal stimuli used by Harrison and Tong (2009) activated several hundred voxels within extrastriate visual areas V2-hV4v. In contrast, our small, peripheral stimuli produced a relatively weak response in only a few dozen voxels within each extrastriate visual area we examined (see Supplementary Table 1). It is possible that with a larger response, patterns of activity in each of these visual areas would support robust classification of the remembered orientation.

Our findings complement recent studies that have reported feature-specific patterns of activation in retinotopically organized visual areas during WM maintenance in the absence of sustained changes in overall response amplitude (e.g., Serences et al., 2009a; Harrison & Tong, 2009). In the current study, we observed a relative decrease in response amplitude during WM maintenance (relative to a condition where WM maintenance was not required). One possibility is that WM maintenance involves the recruitment of only those neurons tuned to the remembered feature value, while neurons tuned to other values are suppressed (relatively speaking). This might lead to an overall decrease in response amplitude during WM storage. However, we emphasize that this account is purely speculative as changes in response amplitude have been inconsistent across different studies that have examined activity in primary visual cortex. For example, Serences et al. (2009a) failed to observe sustained increases in response amplitude during WM maintenance (see also Offen et al., 2009). In contrast, Harrison and Tong (2009) observed sustained increases in response amplitude during WM maintenance in a subset of their subjects, and no sustained changes in others. These studies, in concert with the findings reported here, raise questions about the extent to which sustained increases in response amplitude are diagnostic of an area’s involvement in WM maintenance.

Our findings are consistent with a growing number of studies that have reported spatially global signals in human and monkey retinotopic cortex during visual perception (e.g., Zaksas & Pasternak, 2005; Serences & Boynton, 2007a; Williams et al, 2008). In one example, Williams et al. (2008) required subjects to compare the category membership of two stimuli presented in the periphery of a visual display. Using fMRI and a multivariate analysis, these authors identified category-specific information in foveal retinotopic cortex, despite the fact that all stimuli were presented in spatial locations corresponding to cortical regions outside of this area. One difference between the task devised by Williams et al. and the WM task used in this study is that in the former, all stimuli were present for the duration of each trial. However, we speculate that similar feedback mechanisms support the spatially global mnemonic representations observed in this study. One possibility is that feedback directly facilitates stimulus-specific patterns of activity in both cortical hemispheres. Alternatively, feedback signals might be directed to a contralateral ROI and spread to ipsilateral ROIs via hard-wired cross-hemispheric connections between similarly tuned neurons in each hemisphere (for example, recent work has described a network of callosal axons linking cortical regions that represent the same orientation and spatial location of visual stimuli in cat visual areas 17 and 18; see Rochefort et al., 2009). By this account, similarly tuned neurons in each cortical hemisphere are connected in a mutually excitatory manner; the efficacy of these connections might be modified by WM maintenance, giving rise to stimulus-specific modulations in the ipsilateral ROI. However, we believe this possibility unlikely given that patterns of activity in a given ROI during contralateral and ipsilateral trials are not necessarily identical (see Multivoxel Pattern Analysis, Results). Finally, inhibitory connections between corresponding visual areas in each hemisphere may play a role in producing stimulus-specific activation patterns. This possibility is consistent with the lack of generalization across patterns of activation evoked by contralateral and ipsilateral memoranda in a given ROI.

An important question concerns the boundary conditions that govern the spatially global mnemonic representations observed in this experiment. Several studies have demonstrated that performance on a memory-limited change detection task is degraded when the sample and test stimuli can occupy different spatial locations (Dill & Fahle, 1998; Zaksas, et al., 2001). For example, Zaksas et al. (2001; Experiment 1) trained monkeys to perform a change detection task while varying the spatial separation of the sample and test stimuli. These conditions were blocked, so it was always possible to infer the location of the test stimulus given the location of the sample. On some trials, the experimenters presented a random dot mask during the delay period; this stimulus could appear at the location of the sample stimulus, the impending test stimulus, or elsewhere in the display. Task performance was significantly impaired by the presentation of this mask, but only when it appeared at the location of the impending test stimulus. The selective interference produced by this stimulus suggests that memory for the random dot aperture was localized in the cortical regions that were retinotopically mapped to the position of the impending test stimulus. However, as Zaksas et al note, the masking effects observed in this study may simply mean that the discrimination judgment required by the task was biased towards information stored in the cortical regions that processed the eventual test location, even if the relevant information was also stored in other cortical regions. It is therefore still possible that the relevant directional information was stored in a spatially global manner during this task.

Do the spatially global activation patterns observed in this study have functional consequences for WM maintenance? Posterior visual areas V1-hV4v are thought to represent information about feature properties such as orientation via population response profiles (e.g., Pouget et al., 2001; Pouget et al., 2003; Ma et al., 2006). One possibility is that the spatially global representations observed in this study act to enhance the precision of mnemonic representations (e.g., Awh, Barton, & Vogel, 2007; Zhang & Luck, 2008; Barton, Ester, & Awh, in press). For example, spatially global sensory recruitment would increase the number of neurons dedicated to representing a remembered feature attribute, which may in turn improve the efficiency or signal-to-noise ratio of population responses and thus enhance the precision of mnemonic representations. Future research is needed to examine this possibility.

In summary, our findings demonstrate that sensory recruitment during active WM maintenance is spatially global. We propose that spatially global gain modulations such as those observed in this study may serve to enhance the precision of mnemonic representations by recruiting additional sensory neurons that are not directly driven by the stimulus.

Supplementary Material

Supp1

Acknowledgements

Supported by NIH-R01MH077105-01A2 to E. A. E.A., J.T.S., & E.F.E. conceived and designed the experiment, E.F.E. collected and analyzed data, E.F.E., J.T.S., & E.A. wrote the manuscript.

References

  • Awh E, Barton B, Vogel EK. Visual working memory represents a fixed number of items, regardless of complexity. Psychol Sci. 2007;18:622–628. [PubMed]
  • Awh E, Jonides J. Overlapping mechanisms of attention and spatial working memory. Trends Cogn Sci. 2001;5:119–126. [PubMed]
  • Barton B, Ester EF, Awh E. Discrete resource allocation in visual working memory. J Exp Psychol Hum Percept Perform. (in press) [PMC free article] [PubMed]
  • Bichot NP, Rossi AF, Desimone R. Parallel and serial neural mechanisms for visual search in macaque area V4. Science. 2005;308:529–534. [PubMed]
  • Boynton GM, Engel SA, Glover GH, Heeger DJ. Linear systems analysis of functional magnetic resonance imaging in human V1. J Neurosci. 1996;16:4207–4221. [PubMed]
  • Brainard DH. The Psychophysics Toolbox. Spat Vis. 1997;10:433–436. [PubMed]
  • D’Esposito M. From cognitive to neural models of working memory. Philos Trans R Soc Lond B Biol Sci. 2007;362:761–772. [PMC free article] [PubMed]
  • Desimone R. Neural mechanisms for visual memory and their role in attention. Proc Natl Acad Sci USA. 1996;93:13494–13499. [PubMed]
  • Dill M, Fahle M. Limited translation invariance of human visual pattern recognition. Percept Psychophys. 1998;60:65–81. [PubMed]
  • Engel SA, Rumelhart DE, Wandell BA, Lee AT, Glover GH, Chichilnisky EJ, Shadlen MN. fMRI of human visual cortex. Nature. 1994;369:525. [PubMed]
  • Harrison SA, Tong F. Decoding reveals the contents of visual working memory in early visual areas. Nature. 2009;458:632–635. [PMC free article] [PubMed]
  • Haxby JV, Gobbini MI, Furey MI, Ishai A, Schouten JL, Pietrini P. Distributed and overlapping representations of faces and objects in ventral temporal cortex. Science. 2001;293:2325–2430. [PubMed]
  • Haynes JD, Rees G. Predicting the orientation of invisible stimuli from activity in human primary visual cortex. Nat Neurosci. 2005;8:686–691. [PubMed]
  • Hollingworth A. Scene and position specificity in visual memory for objects. J Exp Psychol Hum Learn. 2006;32:58–69. [PubMed]
  • Hollingworth A. Object-position binding in visual memory for natural scenes and object arrays. J Exp Psychol Hum Percept Perform. 2007;33:31–47. [PubMed]
  • Jonides J, Lacey SC, Nee DE. Processes of working memory in mind and brain. Curr Dir Psychol Sci. 2005;14:2–5.
  • Kamitani Y, Tong F. Decoding the visual and subjective contents of the human brain. Nat Neurosci. 2005;8:1248–1254. [PMC free article] [PubMed]
  • Kamitani Y, Tong F. Decoding seen and attended motion directions from activity in the human visual cortex. Curr Biol. 2006;16:1096–1102. [PMC free article] [PubMed]
  • Ma WJ, Beck JM, Latham PE, Pouget A. Bayesian inference with probabilistic population codes. Nat Neurosci. 2006;9:1432–1438. [PubMed]
  • Martinez-Trujillo JC, Treue S. Feature-based attention increase the selectivity of population responses in primate visual cortex. Curr Biol. 2004;13:655–662. [PubMed]
  • Miller EK, Erickson CA, Desimone R. Neural mechanisms of visual working memory in prefrontal cortex of the macaque. J Neurosci. 1996;16:5154–5167. [PubMed]
  • Norman KA, Polyn SM, Detre GJ, Haxby JV. Beyond mind-reading: multi-voxel pattern analysis of fMRI data. Trends Cogn Sci. 2006;10:424–430. [PubMed]
  • Offen S, Schluppeck D, Heeger DJ. The role of early visual cortex in visual short-term memory and visual attention. Vision Res. 2009;49:1352–1362. [PMC free article] [PubMed]
  • Pasternak T, Greenlee M. Working memory in primate sensory systems. Nat Rev Neurosci. 2005;6:97–107. [PubMed]
  • Peelen MV, Downing PE. Using multi-voxel pattern analysis of fMRI data to interpret overlapping functional activations. Trends Cogn Sci. 2007;11:4–5. [PubMed]
  • Pelli DG. The VideoToolbox software for visual psychophysics: Transforming numbers into movies. Spat Vis. 1997;10:437–442. [PubMed]
  • Postle BR. Working memory as an emergent property of the mind and brain. Neuroscience. 2006;139:23–38. [PMC free article] [PubMed]
  • Pouget A, Dayan P, Zemel R. Information processing with population codes. Nat Rev Neurosci. 2000;1:125–132. [PubMed]
  • Pouget A, Dayan P, Zemel R. Inference and computation with population codes. Annu Rev Neurosci. 2003;26:381–410. [PubMed]
  • Rainer G, Asaad WF, Miller EK. Selective representation of relevant information by neurons in the primate prefrontal cortex. Nature. 1998;393:577–579. [PubMed]
  • Rochefort NL, Buzas P, Quenech’du N, Koza A, Eysel UT, Milleret C, Kisvarday ZF. Functional selectivity of interhemispheric connections in cat visual cortex. Cereb Cortex. 2009;19:2451–2465. [PubMed]
  • Serences JT, Boynton GM. Feature-based attentional modulations in the absence of direct visual stimulation. Neuron. 2007a;55:301–312. [PubMed]
  • Serences JT, Boynton GM. The representation of behavioral choice for motion in human visual cortex. J Neurosci. 2007b;27:12893–12899. [PubMed]
  • Serences JT, Ester EF, Vogel EK, Awh E. Stimulus-specific delay activity in human primary visual cortex. Psychol Sci. 2009a;20:207–214. [PMC free article] [PubMed]
  • Serences JT, Saproo S, Scolari M, Ho T, Muftuler LT. Estimating the influence of attention on population codes in human visual cortex using voxel-based tuning functions. Neuroimage. 2009b;44:223–231. [PubMed]
  • Sereno MI, Dale AM, Reppas JB, Kwong KK, Belliveau JW, Brady TJ, et al. Borders of multiple visual areas in humans revealed by functional magnetic resonance imaging. Science. 1995;268:889–893. [PubMed]
  • Stokes M, Thompson R, Cusak R, Duncan J. Top-down activation of shape-specific population codes in visual cortex during mental imagery. J Neurosci. 2009;29:1565–1572. [PubMed]
  • Super H, Spekreijse H, Lamme VAF. A neural correlate of working memory in the monkey primary visual cortex. Science. 2001;293:120–124. [PubMed]
  • Treue S, Martinez -Trujillo JC. Feature-based attention influences motion processing gain in macaque visual cortex. Nature. 1999;382:539–541. [PubMed]
  • Zaksas D, Bisley JW, Pasternak T. Motion information is spatially localized in a visual working-memory task. J Neurophysiol. 2001;86:912–921. [PubMed]
  • Zaksas D, Pasternak T. Area MT neurons respond to visual motion distant from their receptive fields. J Neurophysiol. 2005;94:4156–4167. [PubMed]
  • Zhang WW, Luck SJ. Discrete fixed-resolution representations in visual working memory. Nature. 2008;453:233–235. [PMC free article] [PubMed]