PMCCPMCCPMCC

Search tips
Search criteria 

Advanced

 
Logo of nihpaAbout Author manuscriptsSubmit a manuscriptHHS Public Access; Author Manuscript; Accepted for publication in peer reviewed journal;
 
J Neurosci. Author manuscript; available in PMC 2013 October 10.
Published in final edited form as:
PMCID: PMC3664518
NIHMSID: NIHMS466290

Distributed Patterns of Activity in Sensory Cortex Reflect the Precision of Multiple Items Maintained in Visual Short-Term Memory

Abstract

Traditionally, load sensitivity of sustained, elevated activity has been taken as an index of storage for a limited number of items in visual short-term memory (VSTM). Recently, studies have demonstrated that the contents of a single item held in VSTM can be decoded from early visual cortex, despite the fact that these areas do not exhibit elevated, sustained activity. It is unknown, however, whether the patterns of neural activity decoded from sensory cortex change as a function of load, as one would expect from a region storing multiple representations. Here, we use multi-voxel pattern analysis (MVPA) to examine the neural representations of VSTM in humans across multiple memory loads. In an important extension of previous findings, our results demonstrate that the contents of VSTM can be decoded from areas that exhibit a transient response to visual stimuli, but not from regions that exhibit elevated, sustained load-sensitive delay-period activity. Moreover, the neural information present in these transiently activated areas decreases significantly with increasing load, indicating load sensitivity of the patterns of activity that support VSTM maintenance. Importantly, the decrease in classification performance as a function of load correlates with within-subject changes in mnemonic resolution. These findings indicate that distributed patterns of neural activity in putatively sensory visual cortex support the representation and precision of information in VSTM.

Although performance on many everyday cognitive tasks depends on the ability to maintain and manipulate multiple items in visual short-term memory (VSTM), it remains unclear precisely how multiple visual items are represented simultaneously in the brain. A widely accepted hallmark of VSTM storage is the presence of sustained, elevated delay-period activity (e.g., Fuster and Alexander, 1971; Funahashi et al., 1989), which is thought to underlie the “active” mechanisms of short-term maintenance. In human neuroimaging, the sensitivity of this activity to load is considered even stronger evidence for VSTM storage. For example, functional magnetic resonance imaging (fMRI) signal in intraparietal sulcus (IPS) increases with increasing memory load, and asymptotes at apparent capacity limitations (Linden et al., 2003; Todd and Marois, 2004; Xu and Chun, 2006).

More recently, studies using multi-voxel pattern analysis (MVPA) have demonstrated that the identity of a single item held in VSTM can be decoded during the delay period from early visual regions (e.g., V1-V4, MT), even in the absence of sustained delay-period activity (Harrison and Tong, 2009; Serences et al., 2009; Linden et al., 2012; Riggall and Postle, 2012). Although these more recent findings suggest that VSTM representations are coded in distributed patterns of activation in early visual cortex, it remains unclear whether such presumably sensory-based representations respond to changes in VSTM load, and critically, how these changes may reflect load-dependent changes in VSTM performance.

The present study investigated the sensitivity of sensory cortex-based short-term memory representations to memory load by acquiring fMRI while subjects performed delayed recall of one, two, or three presented directions of motion (Figure 1A). Pattern classifiers were trained to classify the remembered direction of motion, and we examined how direction-specific classifier evidence changed as a function of VSTM load. Importantly, the manipulation of load also enabled us to investigate the neural basis of precision in VSTM. Recent evidence suggests that there is a measureable decline in the level of detail or “precision” of VSTM representations as information load increases (Bays and Husain, 2008; Zhang and Luck, 2008; van den Berg et al., 2012). According to these models, this decrease in mnemonic resolution reflects a decrease in the proportion of neural resources dedicated to each representation. Accordingly, we predicted that this loss of mnemonic resolution as a function of load should be reflected in classification performance, reasoning that declines in both classification performance and behavioral precision may result from noisier neural representations. Relating classification performance to behavioral precision would be strongly consistent with the idea that patterns of activity in sensory cortex reflect the neural representation of information held in VSTM, even in the absence of sustained, elevated delay-period activity in these regions.

Figure 1
Behavioral paradigm and performance. (A) Schematic of the delayed-recall task performed by participants while undergoing fMRI. On every trial participants were presented with three patches of dots displayed within a single 2 s TR and instructed to remember ...

Methods

Participants

Ten right-handed volunteers (3 females) 23 – 31 (M = 25.5) years of age from the University of Wisconsin – Madison community participated in the study for a small remuneration ($15/hour). All subjects provided informed consent according to the procedures approved by the Health Sciences Institutional Review Board at the University of Wisconsin – Madison. Subjects had normal or corrected-to-normal vision, no contraindications for MRI, and no reported history of neurological disease.

Stimuli and Procedure

Participants underwent functional magnetic resonance imaging (fMRI) while performing six runs of a delayed recall task for visual motion (Figure 1A). A grey fixation cross (~0.22°) was presented at the beginning of each trial on a black background, and changed to white 2 s before the onset of the memory sample. The memory sample lasted a total of 2 s (1 TR), during which three patterns of dots were presented. Dot patterns were circular patches (~11° diameter) of dots (~0.13°) with a density of approximately 0.7 dots-per-square degree. Dot patterns were centered around the fixation, and uniquely and randomly presented in red, green, or blue, without repeating colors within a trial. Each dot pattern was presented for 500 ms, and stimuli were separated by an ISI of 250 ms. On any given trial, one, two, or three of the dot patterns could be moving with 100% coherence at 2.75°/s, while the remaining dots (loads 1 and 2) remained static. On 90% of trials, one of the presented directions was sampled from one of three directions (7°, 127°, 247°). These were the directions used as labels for the patterns used to train the classifier, and were also selected as the probed (target) directions on those trials. Because we wanted behavioral estimates of precision to correspond directly to the neural data, the probed item was always the stimulus with one of these three directions of motion. The remaining moving patterns on these trials were sampled randomly from the 360° space. No restrictions were placed on the selected directions of these remaining items. On the remaining 10% of trials, all moving patterns were randomly selected, including the probed direction, in order to prevent participants from learning the target directions. The sample display was followed by an 8 s delay period, over which participants were instructed to remember the direction of motion of all moving patterns. Following the delay, a probe display appeared for 5s. The probe consisted of a colored circle (~11° in diameter) with a line (~4.5°) extended between the centre of the screen and random location on the circumference of the circle. Participants were instructed to rotate the line so that it matched as closely as possible the direction of motion of the color-matching dot pattern. Participants rotated the line by pressing buttons that rotated the line 10° clockwise, 10° counter-clockwise, or flipped it 180°. The starting position of the line segment was randomly selected, with the constraint that it was presented within a multiple of 10° of the probed sample direction. On those trials in which one of the critical items was presented, the probe color always matched the color of the critical stimulus. On the remaining 10% of trials, the probed stimulus direction was randomly selected. The temporal position of the target stimulus was also counter-balanced within each block. A 7s inter-trial interval (ITI) followed the probe, during which the fixation cross changed to grey.

Participants performed 180 trials over the 6 runs in the scanner. Each block contained 27 critical trials, 3 of each load x direction x temporal position pairing, as well as 3 non-critical trials, one of each load. The color of the sample was always randomly selected. Experimental stimuli were controlled by the Psychophysics Toolbox (Brainard, 1997) (http://psychtoolbox.org) running in MATLAB (The Mathworks, Natick, MA, USA) and presented using a 60-Hz projector (Silent Vision 6011, Avotec INC.), and viewed through a coil-mounted mirror.

Behavioral Analysis

Behavioral performance was assessed using the mixture-model developed by Zhang and Luck (Zhang and Luck, 2008) and extended by Bays and colleagues (Bays et al., 2009; Zokaei et al., 2011). This model breaks down the distribution of error (the angular deviation between the sample direction and the indicated response direction) into estimates of target accuracy (proportion of responses towards the target direction), error (non-target responses and guesses), as well as a measure of precision (κ, or concentration; the variability of recall of the target and non-target responses). That is, responses are modeled to be a mixture of a Von Mises distribution (a circular analog of the Gaussian distribution) with a concentration of κ for all target and non-target responses, plus a uniform distribution of random responses (guesses). The concentration parameter, κ, corresponds to the variability of the target (and non-target) responses, with larger values indicating less variability. The model is defined as

equation M1
(1)

where θ is the target motion direction, and [theta w/ hat] is the response direction on a given trial. α denotes the probability of reporting the target direction and β is the probability of reporting a non-target direction (of m non-target directions). γ is defined as 1 - α - β, and represents the probability of responding at random (see Bays et al., 2009 or Zokaei et al., 2011 for figures depicting the relationship between the response distribution and the model parameters).

Parameter estimates for α, β, γ and κ were obtained using maximum-likelihood estimation (Expectation Maximization) using MATLAB routines (available at http://www.bayslab.com). Responses were entered in for each trial, and separate estimates were obtained for each participant and condition.

Imaging Analysis

Functional data were preprocessed using the Analysis of Functional NeuroImages (AFNI) software package (Cox, 1996) (http://afni.nimh.nih.gov). All volumes were spatially aligned to the final volume of the final run using a rigid-body realignment and corrected for slice-time acquisition. Linear, quadratic and cubic trends were removed from each run to reduce the influence of scanner drift. For univariate analyses, data were spatially smoothed with a 6 mm FWHM Gaussian. For classification analyses, data were z-scored separately within run for each voxel. Data were not smoothed and were left in their native space.

Univariate Region-of-Interest Analysis

The goal of these analyses was to identify regions-of-interest (ROIs) whose activity, either in response to stimulus presentation or during the delay period, varied monotonically with load. We did this by solving a modified general linear model (GLM; implemented in AFNI) in which regressors modeling the sample presentation and delay period were generated as 2 and 8 second boxcars, respectively, and convolved with the canonical hemodynamic response function (HRF) supplied with AFNI. This approach is commonly used to discriminate transient, “stimulus-evoked” activity that returns to baseline upon the offset of the sample from activity that persists at an elevated level across the delay period despite the absence of stimulus input (e.g., Postle, et al., 2000). The model also included three levels of stimulus load (1, 2, and 3).

Figure 2 illustrates regions whose “sample” and/or “delay” activity varied significantly with load. To construct ROIs for hypothesis-testing MVPA analyses, the parameter estimates used to produce the map in Figure 2 were used to identify regions for which load sensitivity of the sample epoch was statistically greater than load sensitivity of the delay epoch (“Sample ROI); and to identify regions for which the converse was true (“Delay ROI”).

Figure 2
Regions of interest used for classification analysis. Initial analysis was restricted to those areas that showed parametrically-increasing BOLD activity as a function of load that were either transiently evoked by the initial sample or sustained throughout ...

Anatomically Defined Regions-of-Interest

In addition to the univariate ROIs, MVPA was also performed on anatomical ROIs created with the automatic parcellation routines (Fischl et al., 2008) implemented by FreeSurfer (http://surfer.nmr.mgh.harvard.edu/). These ROIs included: V1 and V2, MT, intraoccipital sulcus (IOS, including the transverse occipital and superior occipital sulci) and intraparietal sulcus (including the transverse parietal sulcus). All voxels within the anatomical ROIs were included in the MVPA analyses.

Note that we did not include frontal areas among anatomical ROIs for two reasons. First, many previous studies using fMRI, fMRI-guided repetitive transcranial magnetic stimulation (rTMS), and patients with frontal lesions have either a) failed to find load sensitive delay-period activity in frontal cortex (Feredoes and Postle, 2007; Postle, et al., 1999) or b) failed to find that disruption of frontal cortex affects the retention of multiple items (D’Esposito and Postle, 1999; Feredoes et al., 2007; Koenigs et al., 2011; Postle et al., 2006). Second, two recent studies using MVPA methods similar to those used here failed to find evidence for delay-period retention of stimulus-related information in frontal cortex (Christophel et al., 2012; Riggall and Postle, 2012).

Pattern Classification Analyses

To examine the neural representations associated with VSTM maintenance, we trained pattern classifiers to classify each of the three critical directions of motion, and examined classifier sensitivity for each direction using a leave-one-trial-out approach. Classification was performed using the Princeton Multi-Voxel Pattern Analysis (www.pni.princeton.edu/mvpa/) toolbox and custom routines in MATLAB. Preprocessed fMRI data from individual trial time points were used to train separate classifiers to classify the direction of motion (3 possible directions) or the memory load (3 possible loads). Classification was accomplished using L2-regularized logistic regression (LR) with a lambda penalty term of λ = 25. Training for direction was performed collapsed across all three loads; similarly, training for load was performed collapsed across all three directions. Note that all MVPA analyses were performed on data that were neither smoothed nor time-shifted in any other way. Classification was performed on the top 2000 voxels from the sample- and delay load-sensitive GLM-derived ROIs, and on all voxels within the anatomically-defined ROIs.

For each 2 s TR of fMRI data, the trained classifier produced a probability estimate (from 0 to 1) of the extent to which the observed pattern on the tested trial matched the trained pattern for each of the trained items (e.g., the 3 directions of motion). Classification performance was determined using leave-one-trial-out cross validation, in which the classifier was trained on data from all but one trial, and then tested on the remaining trial, rotating through all possible permutations. Training was performed using all possible trials (collapsing across the untrained dimension of load or direction). For each of the three motion directions, we computed a receiver-operating characteristic (ROC) based on the values obtained across all tested trials, then averaged across the dimensions of interest. That is, this method evaluated whether, across trials, evidence for one direction of motion was higher when that direction was in fact presented on that trial relative to when it was not presented. The area under the ROC curve, averaged across the three directions of motion, provides a measure of overall classifier sensitivity, with chance performance at 0.5. Analogous methods of classification and ROC-based sensitivity analysis were used to classify memory load, collapsing across direction of motion.

To examine the dynamics of the memory representation, each classifier was trained on data from only a single time point in the trial (e.g., TR 5) and then tested on all points in the left-out trial. This resulted in a time-course of decoding sensitivity for the entire trial that was determined by the weights of the single training point, and provides a measure of how stable the representations were over time.

The significance of classifier performance was determined using one-tailed, one-sample t-tests, testing against chance performance of 0.5. Tests performed across individual TRs were corrected for multiple comparisons using the Larzelere and Mulaik (1977) method. Within-subject correlations were computed using the method of Bland and Altman (1995) by calculating an analysis of covariance (ANCOVA) with the factors of classifier sensitivity, and subjects as the covariates, and behavioral performance as the dependent measure, and fitting the model to parallel lines.

Within-subject correlation

Although classification performance was significantly above chance at the group level, there was significant variance between subjects in the ability to classify direction of motion. Consequently, we attempted to examine the relationship between classification performance and behavioral precision while accounting for these individual differences in classification performance. A number of studies of VSTM have typically normalized between-subject differences in neural signal by calculating pair-wise differences between loads (Vogel and Machizawa, 2004; Vogel et al., 2005; Emrich et al., 2009). Here, we instead attempted to examine whether changes in the relative amount of neural information across all three tested memory loads predicted the change in the quality of representations maintained in VSTM. That is, we took advantage of the variation in dependent measures across all three loads by using a within-subject correlation approach (Bland and Altman, 1995) to examine the change in both classifier performance and behavioral precision across loads. Specifically, we used analysis of covariance (ANCOVA) to remove between subject differences in classification performance and measured the remaining variation in classification performance explained by variation in behavioral precision.

Results

Behavioral Results

Behavior was analyzed according to the method of Bays et al. (2009; Zokaei et al., 2011). This method uses a mixture-model of the behavioral response error (the angular distance between the reported direction and the actual direction of the probed memory stimulus) to obtain maximum-likelihood estimates for the proportion of correctly-recalled targets, the precision of those responses (the distribution of error around the target), as well as the proportion of errors (both non-target responses and guesses; see Methods).

Examining the concentration parameter (K, the estimate of the variability of memory precision) of this model using repeated-measures ANOVA revealed a significant effect of load, F(2,18) = 8.55, MSE = 500.4, p = 0.0025 (Figure 1B). A similar effect was observed for the proportion of target responses (PT), F(2,18) = 7.97, MSE = .011, p = .0033. That is, consistent with previous studies examining memory for direction of motion (Zokaei et al., 2011), both the number of correctly recalled targets and the precision of those responses decreased as the amount of to-be-remembered information increased. We also observed a significant number of non-target (i.e., “binding”) errors that increased at larger set sizes, F(2,18) = 7.15, MSE = .01, p = .0052.

Because we wanted behavioral responses to match as closely as possible the decoded patterns of brain activity, participants were probed on the three critical directions on a majority of trials. This procedural limitation could have allowed participants to learn which directions were most likely to be probed, thereby affecting their behavioral responses. To determine whether or not such learning occurred, we obtained parameter estimates for the first and second half of the experiments, collapsed across load. This analysis revealed no significant changes in either the concentration of target responses, t(9) = −1.31, SD = 12.33, p = .22, or the likelihood of reporting the target, t(9) = −1.12, SD = .043, p = .29, consistent with previous studies that failed to show learning for individual items in tasks of VSTM (Olson and Jiang, 2004; Beck et al., 2008). Moreover, the estimates obtained here are very similar to those obtained in a task that used a similar design, but that selected probes at random (Zokaei et al., 2011), suggesting that participants’ responses were unaffected by non-randomzied target probes.

Classification Results

Region-of-interest analysis

Before examining whether classification performance changes as a function of VSTM load, we first tested whether it was possible to successfully decode the direction of motion of one of the memory stimuli while multiple directions were held in VSTM. A previous study by our group (Riggall and Postle, 2012) reported that decoding the direction of a single item is possible within those areas that support motion perception (i.e., calcarine and extra-calcarine occipital cortex, including MT+) but not in areas that showed sustained activity during the delay (IPS, frontal areas). Accordingly, we focused our initial analysis on non-overlapping regions-of-interest that demonstrated load-sensitive BOLD signal intensity either in a phasic response to the memory sample (“Sample ROI”) or in a response that was sustained throughout the delay period (“Delay ROI”). In this way, we sought to determine whether information about the remembered direction of motion was coded in cortical areas sensitive to visual stimulation independently from those that exhibited load-sensitive, sustained, delay-period BOLD signal.

Examining the patterns of activity in the Sample ROI, classification of the direction of motion resulted in significantly greater-than-chance sensitivity, averaged across all three loads (Figure 3C). This was true both when the classifier was trained on the time point that captured the peak of the sample-evoked response and when it was trained on the final time point of the delay period, a time point that would be dominated by sustained delay-period activity. In contrast, those areas that exhibited sustained, elevated delay-period BOLD signal (i.e., the Delay ROI) showed no evidence of direction sensitivity during any point in the trial (Figure 3D), replicating the results of Riggall and Postle (2012). As a control analysis, training the classifier on inter-trial fixation activity resulted in chance-level classification performance for both ROIs.

Figure 3
BOLD signal intensity and classification performance in functional and anatomical ROIs. (A) illustrates load-sensitive stimulus-evoked activity, but no sustained delay activity, in the functionally-defined, Sample ROI. (B) illustrates elevated, sustained ...

It is possible that by focusing on areas whose load-sensitive response was specific to either the sample or delay periods, this analysis overlooked important voxels that are critical to the maintenance of information in VSTM (e.g., those that show elevated levels of BOLD signal intensity to both encoding and maintenance – those that appear yellow in Figure 2). However, further analysis restricted to several anatomically-defined areas confirmed that the majority of information about direction of motion during the delay period was exhibited in calcarine and extra-calcarine occipital cortex (Figure 3E). Notably, classification performance was not significantly better than chance in the IPS. Thus, although elevated delay-period activity was absent from the observed BOLD signal intensity in areas associated with visual perception (Figure 3A), the ability to maintain the representation of multiple items in VSTM appears to be supported by distributed patterns of activity in these areas, rather than in those areas that exhibit elevated delay-period activity.

Load Sensitivity

A critical aim of the present study was to examine whether classification performance would decrease as a function of memory load in a manner that reflected the decline in behavioural precision. To test this question, we trained a classifier on the final time point of the delay period for both of the Sample and Delay ROIs. Next, for each ROI, we decoded classification performance at each time point from each of the three load conditions, resulting in a classification time course that reflected the extent to which stimulus-specific neural representations were present across the entirety of the trial.

Examining the entire delay period (4 – 12 seconds) in the Sample ROI, mean classification sensitivity was significantly greater than chance for all three loads, ps < .01, adjusted (Figure 4A). Thus, even in the presence of competing within-category representations, memory representations can be identified in the patterns of activity in sensory cortex. Importantly, direction sensitivity significantly decreased during the delay period at higher loads, F(2,18) = 5.63, MSE = .009, p = 0013, consistent with the decline in behavioural performance at higher set sizes. One concern with this result might be that presenting multiple stimuli in a single location might lead to a loss in classifier sensitivity at higher loads, independent of VSTM. That is, by presenting multiple items at one location, the same neurons (within the same voxels) in visual cortex would be required to process the perceptual information of each of the directions of motion, resulting in a loss in sensitivity for any given direction. If this alternative account were correct, one would expect to see the same load-related decline in classification during the stimulus presentation epoch of the trial. However, no such decrease as a function of load was observed when this procedure was repeated with classifiers trained on the time point that captured the peak of the sample-evoked response (i.e., 4–6 sec) and testing classification sensitivity during the sample-evoked period (i.e., 4 – 8 sec; F(2,18) = 2.5, MSE = .006, p = .11). Thus, our results are consistent with the interpretation that the load-dependent decrease in classification sensitivity during the delay period is due to a reduction in the proportion of neural resources dedicated to representing that information in VSTM.

Figure 4
Load sensitivity of patterns of neural activity. Classifiers were trained to a single TR (denoted by the colored dots) and tested across the entire trial. (A–B) Classification sensitivity for direction of motion as a function of load in the functionally-defined ...

Additionally, even though delay-period BOLD signal intensity in the Sample ROI was not significantly affected by load (Figure 2A), F(2,18) = 2.64, MSE = .0052 p = .0991, VSTM load could be successfully decoded from sensory regions (Figure 4C). This finding provides further evidence that the patterns of delay-period activity in areas involved in visual perception reflects the representation of multiple items in VSTM, independent of changes in BOLD signal intensity identified in univariate signal intensity. In contrast, despite the presence of load-sensitive BOLD signal in the Delay ROI (Figure 2B), F(2,18) = 5.11, MSE = .01386, p = .0174, classification performance of the contents of VSTM (i.e., direction of motion) was not significantly above chance at any load, ps > .05 (Figure 4B). Thus, although BOLD activity in these regions (including IPS) is sensitive to VSTM load, we failed to find evidence that these areas are involved in the representation of the contents of VSTM per se. In the Discussion section we consider other aspects of task performance to which this activity may correspond.

Relating classification performance to response precision

Does the decrease in classifier performance as a function of memory load reflect the associated decline in mnemonic precision? To examine this question, we specifically focused on within-subject changes across load (see Methods), because there may be sources of error that affect between-subject classification performance that are unrelated to differences in behavior (e.g., differences in encoding strategies, Linke et al., 2011b; Vicente-Grabovetsky et al., 2012, or individual differences in signal-to-noise ratio of BOLD signal, Tong et al., 2012). This analysis revealed that, across loads, the observed decrease in peak classifier sensitivity is significantly correlated with an individual’s change in mnemonic precision, r = .58, p = .006. (Figure 5A). This was also true when the analysis was restricted to the individual TRs corresponding most closely to the encoding (TR 4; r = .45, p = .04) or delay activity (TR 6; r = .53, p = .01).

Figure 5
The relationship between Sample ROI classifier sensitivity and behavioral performance. Within subjects, changes in classifier sensitivity across load are significantly correlated with changes in behavioral precision (A) but not with changes in the proportion ...

It is possible, however, that the decrease in classifier performance reflects the decreased probability that the probed (target) direction was stored in memory at higher loads. That is, given that both the proportion of correctly-recalled targets, and the precision of those responses, decreased as a function of load, the decline in classifier sensitivity could reflect the proportionally fewer trials on which the target direction was successfully encoded and maintained. Contrary to this alternative account, however, the probability of recalling the target (PT) was uncorrelated with peak classifier sensitivity, r = .21, p = .35 (Figure 5B). In other words, the change in classifier sensitivity across memory loads cannot be predicted by a decrease in the percentage of trials on which the critical direction of motion was stored in VSTM. Thus, the decrease in classification performance appears to reflect the decline in mnemonic resolution, rather than reflect decreases in other aspects of behavior, or random decreases that are independent of the internal representations that guide behavior (i.e., noise).

Discussion

In the present study, we sought to investigate how the neural patterns associated with the contents of VSTM change as a function of load. Consistent with several recent studies, we observed that stimulus-specific properties of information maintained in VSTM can be decoded from the patterns of activity in sensory cortex (Ester et al., 2009; Harrison and Tong, 2009; Serences et al., 2009; Riggall and Postle, 2012). We extend these findings by demonstrating that the successful classification of the contents of VSTM is not an all-or-none phenomenon; instead, classification sensitivity varies as a function of VSTM load. Thus, these findings reveal the presence of load sensitivity in the patterns of activity in sensory cortex, providing strong evidence that these patterns may represent a neural correlate of VSTM maintenance.

In addition, we also observed a significant within-subject correlation between classifier sensitivity and mnemonic precision. This finding provides some of the first evidence that variation in mnemonic precision can be extracted from the neural patterns associated with the storage of information in VSTM. A number of recent psychophysical models have established that the quality of VSTM representations decreases as a function of memory load (Wilken and Ma, 2004; Bays and Husain, 2008; Zhang and Luck, 2008; Bays et al., 2009; van den Berg et al., 2012). Importantly, while the models that explain these data differ in their stance on whether VSTM performance is limited primarily by a finite pool of resources, or whether it also contains a fixed capacity of a limited number of slots, these models all emphasize that the precision of maintained representations varies with load, particularly at low set sizes (i.e., when the number of items is below any putative capacity limits). Thus, “slots + resource” (Alvarez & Cavanagh, 2004; Buschman et al., 2011) and “slots + averaging” (Zhang and Luck, 2008, 2011; Anderson et al., 2011;) hybrid models, as well as pure resource models (Wilken and Ma, 2004; Bays and Husain, 2008; van den Berg et al., 2012) can accommodate the findings reported here, in that all hold that the precision of maintained representations will vary with load at small set sizes. Specifically, our results are consistent with the prediction that, as the amount of to-be-remembered motion information increases, fewer resources will be dedicated towards representing a given item, resulting in less signal present in the patterns of neural activity that reflect this representation. Critically, this relationship between VSTM precision and classification performance provides evidence consistent with the idea that sensory visual cortex may play a critical role in supporting the representation of information in VSTM (Postle, 2006). That is, these findings take the important step of linking the neural patterns associated with content-specific VSTM encoding and maintenance with the internal representations associated with behavior.

One possible concern with these results is that a decrease in decoding accuracy with increasing load, such as we present here, could also result from decoding noise that is unrelated to the mnemonic representation. For example, the neural response to multiple objects has been shown to be similar to the weighted sum (MacEvoy and Epstein, 2009) or average (Reddy et al., 2009) of individual responses. This alternative seems unlikely, however, for a number of reasons. First, each condition contained the same number of stimuli, but varied only in how many of the dot patterns contained motion. Thus, this control reduces the stimulus differences between conditions that might affect the weighted averaging or summation of neural signal. Second, classification training was performed over all three tested loads. Accordingly, classification performance was based on patterns of activity that included the increased noise present on load 2 and 3 trials. Third, classification performance was unaffected by load when training and testing to the patterns of activity transiently evoked by the sample stimuli, indicating that classification performance did not decrease obligatorily due to a decrease in the signal-to-noise ratio at higher set sizes. Finally, a recent study examining the response patterns of single neurons in monkeys has shown that the neural information present about an item encoded and stored in VSTM decreases when multiple items are presented within the same hemifield (Buschman et al., 2011), providing evidence that this effect is not idiosyncratic to the classification technique used here. Together, these facts support our conclusion that classification performance can be accounted for by a decrease in the relative amount of information about the target stimulus present in the patterns activity in sensory cortex.

Although our findings are consistent with studies that have failed to observe evidence for the contents of VSTM in the IPS (Linden et al., 2012; Riggall & Postle, 2012), one previous study did observe above-chance classification in posterior parietal cortex (Christophel et al., 2012). A possible explanation for these discrepant findings may be that the study by Christophel et al. (2012) used artificial stimuli that required significant training to differentiate. Thus, decoding accuracy in posterior parietal cortex in this task may reflect processes that are not specific to VSTM maintenance per se. In addition, although the use of the data-driven MVPA approach in the current study has its advantages, there are limitations to this methodology. For example, evidence indicates there is a coarse-scale topographic map of orientation preference in human V1 that is strongly correlated with the retinotopic map (Freeman et al., 2011). Thus, it is possible that multivariate classification methods could exploit these biases, thereby driving up classification sensitivity only in those areas that have strong topographic organization (e.g., V1 and V2). However, other studies have demonstrated that classification performance of orientation-selective activity is predicted by the average BOLD amplitude in a given region (Tong et al., 2012). Thus, given that we observed greater BOLD amplitude during the delay period in the IPS than in sensory visual cortex, it is unlikely that our findings can be explained by differences in classification sensitivity across different regions. Nevertheless, future studies should examine load-dependent VSTM classification sensitivity on a range of stimuli (e.g., colors, faces) in order to determine the reliability of this effect across different features and cortical regions.

If the maintenance and precision of VSTM representations is mediated by patterns of activity in sensory cortex, what is the role of sustained, elevated, delay-period activity? Previous studies have provided evidence that delay-period activity may reflect general cognitive or attentional demands of the task (Mitchell and Cusack, 2008; Magen et al., 2009), as well as goal- or action-related information (Curtis and Lee, 2010; Riggall and Postle, 2012), rather than reflecting VSTM maintenance. One possibility is that whereas VSTM representations may be supported by patterns of activity in sensory cortex, sustained, elevated, delay-period activity (particularly in IPS) may reflect the directed attention towards a limited number of representations (Anderson et al., 2011). This mechanism could potentially account for both apparent capacity limits (i.e., those within the focus of attention), as well as a larger capacity for lower-resolution representations (Sligte et al., 2008, 2009, 2010).

The finding that the precision of VSTM representations is predicted by areas that mediate perceptual encoding is largely consistent with the evidence that VSTM performance is likely limited by encoding processes (Buschman et al., 2011; Linke et al., 2011b; Emrich and Ferber, 2012; Mazyar et al., 2012; van den Berg et al., 2012). That is, the continued recruitment of the same populations of neurons that mediate encoding and perception for VSTM will limit the precision of memory representations to that of encoding. Additionally, when attention is allocated across two different memory categories and then retroactively cued to a single item, classifier evidence for the un-cued item dropped to chance, suggesting that the allocation of resources in VSTM may be dynamic (Lewis-Peacock and Postle, 2012; Larocque et al., 2013). Thus, while it is possible that the fidelity of VSTM may be further affected by subsequent processes, such as by fluctuations in attention (Van den Berg et al., 2012) or stochastic degradation (Fougnie et al., 2012), the correlation between classifier performance and VSTM precision during early stages of encoding rule out the possibility that all variation in precision reflects limitations in later stages of processing (e.g., during recall or comparison; Awh et al., 2007; Barton et al., 2009).

At the neural level, the successful classification of multiple items in VSTM from sensory cortex, as well as its relationship with mnemonic precision, further calls into question the widely held assumption that VWM maintenance of multiple items occurs in a network of fronto-parietal regions (Courtney et al., 1997; Linden et al., 2003; Todd and Marois, 2004; Vogel and Machizawa, 2004; Xu and Chun, 2006; Reinhart et al., 2012). In particular, it calls into question the role of elevated, sustained, load-sensitive delay-period activity in IPS as a hallmark of VSTM maintenance (but see Christophel et al., 2012). Instead, our findings support the view that VSTM maintenance is accomplished by actively maintaining representations in “lower level” sensory cortex (Postle, 2006; Ester et al., 2009; Harrison and Tong, 2009; Serences et al., 2009; Linke et al., 2011a; Riggall and Postle, 2012).

Acknowledgments

This work was supported by the National Institutes of Health Grant R01-MH064498 (B.R.P.). Stephen M. Emrich is now located at the Department of Psychology, Brock University, St Catharines, ON, L2S 3A1, Canada.semrich/at/brocku.ca.

References

  • Alvarez GA, Cavanagh P. The capacity of visual short-term memory is set both by visual information load and by number of objects. Psychological Science. 2004;15:106–111. [PubMed]
  • Anderson DE, Vogel EK, Awh E. Precision in visual working memory reaches a stable plateau when individual item limits are exceeded. Journal of Neuroscience. 2011;31:1128–1138. [PubMed]
  • Awh E, Barton B, Vogel EK. Visual working memory represents a fixed number of items regardless of complexity. Psychological Science. 2007;18:622–628. [PubMed]
  • Barton B, Ester EF, Awh E. Discrete resource allocation in visual working memory. Journal of Experimental Psychology: Human Perception and Performance. 2009;35:1359–1367. [PMC free article] [PubMed]
  • Bays PM, Catalao RFG, Husain M. The precision of visual working memory is set by allocation of a shared resource. Journal of Vision. 2009;9:7.1–11. [PMC free article] [PubMed]
  • Bays PM, Husain M. Dynamic shifts of limited working memory resources in human vision. Science. 2008;321:851–854. [PMC free article] [PubMed]
  • Beck MR, Angelone BL, Levin DT, Peterson MS, Varakin DA. Implicit learning for probable changes in a visual change detection task. Consciousness and Cognition. 2008;17:1192–1208. [PubMed]
  • Bland JM, Altman DG. Calculating correlation coefficients with repeated observations: Part 1--Correlation within subjects. BMJ (Clinical research ed) 1995;310:446. [PMC free article] [PubMed]
  • Brainard DH. The Psychophysics Toolbox. Spatial Vision. 1997;10:433–436. [PubMed]
  • Buschman TJ, Siegel M, Roy JE, Miller EK. Neural substrates of cognitive capacity limitations. Proceedings of the National Academy of Sciences of the United States of America. 2011;108:11252–11255. [PubMed]
  • Christophel TB, Hebart MN, Haynes J-D. Decoding the Contents of Visual Short-Term Memory from Human Visual and Parietal Cortex. Journal of Neuroscience. 2012;32:12983–12989. [PubMed]
  • Courtney SM, Ungerleider LG, Keil K, Haxby JV. Transient and sustained activity in a distributed neural system for human working memory. Nature. 1997;386:608–611. [PubMed]
  • Cox RW. AFNI: software for analysis and visualization of functional magnetic resonance neuroimages. Computers and Biomedical Research. 1996;29:162–173. [PubMed]
  • Curtis CE, Lee D. Beyond working memory: the role of persistent activity in decision making. Trends in Cognitive Sciences. 2010;14:216–222. [PMC free article] [PubMed]
  • D’Esposito M, Postle BR. The dependence of span and delayed-response performance on prefrontal cortex. Neuropsychologia. 1999;37:1303–1315. [PubMed]
  • Emrich SM, Al-Aidroos N, Pratt J, Ferber S. Visual search elicits the electrophysiological marker of visual working memory. PloS one. 2009;4:e8042. [PMC free article] [PubMed]
  • Emrich SM, Ferber S. Competition increases binding errors in visual working memory. Journal of Vision. 2012;12:1–16. [PubMed]
  • Ester EF, Serences JT, Awh E. Spatially global representations in human primary visual cortex during working memory maintenance. Journal of Neuroscience. 2009;29:15258–15265. [PMC free article] [PubMed]
  • Feredoes E, Postle BR. Localization of load sensitivity of working memory storage: quantitatively and qualitatively discrepant results yielded by single-subject and group-averaged approaches to fMRI group analysis. Neuroimage. 2007;35:881–903. [PMC free article] [PubMed]
  • Feredoes E, Tononi G, Postle BR. The neural bases of the short-term storage of verbal information are anatomically variable across individuals. Journal of Neuroscience. 2007;27:11003–11008. [PubMed]
  • Fischl B, Rajendran N, Busa E, Augustinack J, Hinds O, Yeo BTT, Mohlberg H, Amunts K, Zilles K. Cortical folding patterns and predicting cytoarchitecture. Cerebral Cortex. 2008;18:1973–1980. [PubMed]
  • Fougnie D, Suchow JW, Alvarez GA. Variability in the quality of visual working memory. Nature Communications. 2012;3:1229. [PMC free article] [PubMed]
  • Freeman J, Brouwer GJ, Heeger DJ, Merriam EP. Orientation decoding depends on maps, not columns. Journal of Neuroscience. 2011;31:4792–4804. [PMC free article] [PubMed]
  • Funahashi S, Bruce CJ, Goldman-Rakic PS. Mnemonic coding of visual space in the monkey’s dorsolateral prefrontal cortex. Journal of Neurophysiology. 1989;61:331–349. [PubMed]
  • Fuster JM, Alexander GE. Neuron activity related to short-term memory. Science. 1971;173:652–654. [PubMed]
  • Harrison SA, Tong F. Decoding reveals the contents of visual working memory in early visual areas. Nature. 2009;458:632–635. [PMC free article] [PubMed]
  • Koenigs M, Barbey AK, Postle BR, Grafman J. Superior parietal cortex is critical for the manipulation of information in working memory. Journal of Neuroscience. 2009;29:14980–14986. [PMC free article] [PubMed]
  • Larocque JJ, Lewis-Peacock JA, Drysdale AT, Oberauer K, Postle BR. Decoding Attended Information in Short-term Memory: An EEG Study. Journal of Cognitive Neuroscience. 2013;25:127–142. [PMC free article] [PubMed]
  • Larzelere RE, Mulaik SA. Single-sample tests for many correlations. Psychological Bulletin. 1977;84:557.
  • Lewis-Peacock JA, Postle BR. Decoding the internal focus of attention. Neuropsychologia. 2012;50:470–478. [PMC free article] [PubMed]
  • Linden DEJ, Bittner RA, Muckli L, Waltz JA, Kriegeskorte N, Goebel R, Singer W, Munk MHJ. Cortical capacity constraints for visual working memory: dissociation of fMRI load effects in a fronto-parietal network. NeuroImage. 2003;20:1518–1530. [PubMed]
  • Linden DEJ, Oosterhof NN, Klein C, Downing PE. Mapping brain activation and information during category-specific visual working memory. Journal of Neurophysiology. 2012;107:628–639. [PubMed]
  • Linke AC, Vicente-Grabovetsky A, Cusack R. Stimulus-specific suppression preserves information in auditory short-term memory. Proceedings of the National Academy of Sciences of the United States of America. 2011a;108:12961–12966. [PubMed]
  • Linke AC, Vicente-Grabovetsky A, Mitchell DJ, Cusack R. Encoding strategy accounts for individual differences in change detection measures of VSTM. Neuropsychologia. 2011b;49:1476–1486. [PubMed]
  • MacEvoy SP, Epstein RA. Decoding the representation of multiple simultaneous objects in human occipitotemporal cortex. Current Biology. 2009;19:943–947. [PMC free article] [PubMed]
  • Magen H, Emmanouil T-A, McMains SA, Kastner S, Treisman A. Attentional demands predict short-term memory load response in posterior parietal cortex. Neuropsychologia. 2009;47:1790–1798. [PMC free article] [PubMed]
  • Mazyar H, Van den Berg R, Ma WJ. Does precision decrease with set size? Journal of Vision. 2012;12 [PMC free article] [PubMed]
  • Mitchell DJ, Cusack R. Flexible, capacity-limited activity of posterior parietal cortex in perceptual as well as visual short-term memory tasks. Cerebral Cortex. 2008;18:1788–1798. [PubMed]
  • Olson IR, Jiang Y. Visual short-term memory is not improved by training. Memory & Cognition. 2004;32:1326–1332. [PubMed]
  • Palmer J. Attentional limits on the perception and memory of visual information. Journal of Experimental Psychology: Human Perception and Performance. 1990;16:332–350. [PubMed]
  • Postle BR. Working memory as an emergent property of the mind and brain. Journal of Neuroscience. 2006;139:23–38. [PMC free article] [PubMed]
  • Postle BR, Berger JS, D’Esposito M. Functional neuroanatomical double dissociation of mnemonic and executive control processes contributing to working memory performance. Proceedings of the National Academy of Sciences. 1999;96:12959–12964. [PubMed]
  • Postle BR, Ferrarelli F, Hamidi M, Feredoes E, Massimini M, Peterson M, Alexander A, Tononi G. Repetitive transcranial magnetic stimulation dissociates working memory manipulation from retention functions in the prefrontal, but not posterior parietal, cortex. Journal of Cognitive Neuroscience. 2006;18:1712–1722. [PubMed]
  • Postle BR, Zarahn E, D’Esposito M. Using event-related fMRI to assess delay-period activity during performance of spatial and nonspatial working memory tasks. Brain Research Protocols. 2000;5(1):57–66. [PubMed]
  • Reddy L, Kanwisher NG, VanRullen R. Attention and biased competition in multi-voxel object representations. Proceedings of the National Academy of Sciences of the United States of America. 2009;106:21447–21452. [PubMed]
  • Reinhart RMG, Heitz RP, Purcell BA, Weigand PK, Schall JD, Woodman GF. Homologous mechanisms of visuospatial working memory maintenance in macaque and human: properties and sources. Journal of Neuroscience. 2012;32:7711–7722. [PMC free article] [PubMed]
  • Riggall AC, Postle BR. The relation between working memory storage and elevated activity, as measured with functional magnetic resonance imaging. Journal of Neuroscience. 2012;32:12990–12998. [PMC free article] [PubMed]
  • Serences JT, Ester EF, Vogel EK, Awh E. Stimulus-specific delay activity in human primary visual cortex. Psychological Science. 2009;20:207–214. [PMC free article] [PubMed]
  • Sligte IG, Scholte HS, Lamme VAF. Are there multiple visual short-term memory stores? PloS one. 2008;3:e1699. [PMC free article] [PubMed]
  • Sligte IG, Scholte HS, Lamme VAF. V4 activity predicts the strength of visual short-term memory representations. Journal of Neuroscience. 2009;29:7432–7438. [PubMed]
  • Sligte IG, Vandenbroucke ARE, Scholte HS, Lamme VAF. Detailed sensory memory, sloppy working memory. Frontiers in Psychology. 2010;1:175. [PMC free article] [PubMed]
  • Todd JJ, Marois R. Capacity limit of visual short-term memory in human posterior parietal cortex. Nature. 2004;428:751–754. [PubMed]
  • Tong F, Harrison SA, Dewey JA, Kamitani Y. Relationship between BOLD amplitude and pattern classification of orientation-selective activity in the human visual cortex. NeuroImage. 2012;63:1212–1222. [PMC free article] [PubMed]
  • Van den Berg R, Shin H, Chou W-C, George R, Ma WJ. Variability in encoding precision accounts for visual short-term memory limitations. Proceedings of the National Academy of Sciences of the United States of America. 2012;109:8780–8785. [PubMed]
  • Vicente-Grabovetsky A, Carlin JD, Cusack R. Strength of Retinotopic Representation of Visual Memories is Modulated by Strategy. Cerebral Cortex 2012 [PubMed]
  • Vogel EK, Machizawa MG. Neural activity predicts individual differences in visual working memory capacity. Nature. 2004;428:748–751. [PubMed]
  • Vogel EK, McCollough AW, Machizawa MG. Neural measures reveal individual differences in controlling access to working memory. Nature. 2005;438:500–503. [PubMed]
  • Wilken P, Ma WJ. A detection theory account of change detection. Journal of Vision. 2004;4:1120–1135. [PubMed]
  • Xu Y, Chun MM. Dissociable neural mechanisms supporting visual short-term memory for objects. Nature. 2006;440:91–95. [PubMed]
  • Zhang W, Luck SJ. Discrete fixed-resolution representations in visual working memory. Nature. 2008;453:233–235. [PMC free article] [PubMed]
  • Zhang W, Luck SJ. The number and quality of representations in working memory. Psychological Science. 2011;22:1434–1441. [PMC free article] [PubMed]
  • Zokaei N, Gorgoraptis N, Bahrami B, Bays PM, Husain M. Precision of working memory for visual motion sequences and transparent motion surfaces. Journal of Vision. 2011;11 [PMC free article] [PubMed]