During the M/EEG recordings, the subjects performed a delayed-match-to-sample-VWM task (Suppl. Fig. 1A
) (Luck and Vogel, 1997
, Palva et al., 2010a
, Palva et al., 2010b
) wherein they memorized a 0.1 s Sample stimulus containing one to six colored squares. A Test stimulus appeared one second after the offset of the Sample and in 50 % of the trials, one square in the Test had a different color than that square in the Sample. The subjects indicated with a forced-choice left- or right-hand thumb twitch whether or not the Sample was identical to the Test. As reported for earlier (Luck and Vogel, 1997
) and the present (Palva et al., 2010a
) data, the behavioral accuracy decreases with increasing memory load (Suppl. Fig. 1B
) while the reaction times become longer (Suppl. Fig. 1C
). Thus, this VWM task yields quantitatively similar psychophysical data during repeated M/EEG recordings as obtained outside the neuroimaging context.
Stimulus processing and memory encoding period
The aim of this study was to identify the cortical sources and dynamics of VWM-related neuronal activities and to address their functional significance in VWM by finding the components that are correlated with memory load and/or predict individual psychophysical performance. The processing of the Sample and Test stimuli was examined first by evaluating absolute-valued evoked responses (ERs) with two broad-band FIR band-pass filters (1–45 Hz and 0.01–45 Hz) across the six memory load conditions. The data were summarized into waveforms indicating the fraction of brain regions where the ERs were significantly (p < 0.001, FDR corrected) above the baseline level. Both Sample and Test stimuli were associated with ERs lasting roughly the first 400 ms from stimulus onset (). Interestingly, the ERs obtained with the lower high-pass filter (0.01–45 Hz) remained above the baseline throughout the retention period and thus contained a slow baseline-shift-like component. We then estimated the effect of memory load to the evoked responses by using Spearman’s rank correlation test (p < 0.001, FDR corrected), which revealed that the ER components from 60 to 200 ms, but not those after 200 ms from stimulus onset, were strongly modulated by the number of objects in Sample stimuli (). One should note that in the present experiment, the number of objects covaried with the overall complexity of the stimulus and hence the strengthening of ERs with increasing memory load may be at least partly attributed to a concurrent change in physical stimulus properties.
ERs are generally thought to be generated by additive evoked activity but may also be influenced by event-related amplitude changes of non-zero mean oscillations (Nikulin et al., 2007
) (see also de Munck and Bijma, 2010
) and phase resetting (Makeig et al., 2002
) of ongoing activity. To consider the contributions of these mechanisms into the generation of the ERs in the present VWM task, we quantified the phase locking of ongoing activity to the Sample and Test stimulus onsets (hereafter called “Stimulus Locking” (SL)) and evaluated the peri-stimulus amplitude dynamics by using the same broad-band filters that were used for ERs (Palva et al., 2005
). The dynamics of SL was similar to that of ERs but SL was significant in a greater fraction of cortical regions (). The amplitude of ongoing cortical broad-band activity, on the contrary, was only briefly strengthened above the baseline level at around 100 ms after the Sample stimulus onset and thereafter suppressed below the baseline level for the rest of trial. Like ERs, both SL and the early amplitude enhancement were robustly correlated with memory load ().
We next identified the cortical regions involved in the processing of Sample stimuli. The first significant ER components (p
< 0.001, FDR corrected) were found at around 70 ms. As the broad-band filter provided a good temporal resolution, we identified the ER sources in 10–20 ms steps (Suppl. Figs. 2, 3
). The evoked activity spread rapidly from V1/V2 to higher-level extrastriate, occipito-temporal and parietal regions. One should note here, however, that the inverse modeling method used here (MNE) provides a distributed source estimate that is likely to be more widespread than the true cortical source. Hence the source reconstructions contain cross-talk between neighboring cortical patches in spatial scales of a few centimeters (Hauk et al., 2011
). The P1 peak at 120-ms was observed in distributed visual areas in the occipital and inferior temporal cortices including the lateral occipito-temporal cortex (LOT), inferotemporal gyrus (ITG), and the intraparietal sulcus (IPS), as well as in the insular and cingulate cortices (). The N1 peak at 190-ms was observed also in distributed regions in the posterior parietal cortex (PPC) as well as in central and superior temporal sulci. This constellation of occipital-, temporal-, and parietal regions was sustained for roughly 50 ms after which it gradually degraded. Broad-band SL was strongest in the same network of brain regions as the evoked response, but with a more smeared temporal evolution. This was expected because the phase estimates of ongoing activity, by definition, incorporate information across a time window that is dependent on the characteristic time scales of the activity itself. This may, in part, also explain the greater sensitivity of SL compared to ERs. It is interesting to also note that SL revealed early stimulus processing in several frontal cortical regions including the frontal eye fields (FEF) and lateral prefrontal cortex (LPFC) as well as in anterior insula (Suppl. Figs. 2, 3
). In contrast, the brief broad-band amplitude enhancement was limited to striate cortex and extrastriate regions surrounding the occipital pole (Suppl. Figs. 2, 3
VWM load strengthened ERs, amplitudes, and SL mostly in the same regions that were identified in the average condition (, Suppl. Figs 4, 5
). While the memory load effect at P1 peak was limited to occipital, occipito-temporal and posterior parietal cortices, the VWM load strengthened N1 peak additionally in superior temporal, superior and orbito-frontal, insular, and cingulate cortices. The correlation of amplitude with memory load was robust throughout the posterior cortical networks and well co-localized with the spatio-temporal evolution of ERs (Suppl. Figs 4, 5
). The SL, in contrast, was strongly correlated with memory load also in several more anterior structures including the anterior cingulate and insula as well as paracentral and superior precentral sulci (supplementary motor area (SMA) and FEF, respectively). These results hence suggest that in these data in posterior brain regions, ERs could have been generated by additive stimulus-evoked activity but in the anterior brain regions phase resetting of ongoing oscillations contributed to the generation of ERs.
Finally, we also localized the sustained slow evoked component that was salient in the ER obtained with 0.01–45 Hz filter (see ). After the termination of the transient ER and in the time window from 600 to 1100 ms, the slow component was strongest in the lateral prefrontal (LPFC) and cingulate cortices as well as in para-, pre-, and post-central areas (). More posteriorly, the slow component was also observed in PPC and lingual gyrus.
Moreover, while the slow retention period ER component was largely co-localized with frontal load-dependent oscillation amplitude modulations (see below), it was uncorrelated with memory load, which suggests that this component might not be related to the amplitude changes of α or faster oscillations through the non-zero mean mechanism. The insensitivity to VWM load of the slow ER component suggests that it could reflect arousal- or alertness-related fluctuations (Schroeder and Lakatos, 2009
, Monto et al., 2008
Early evoked responses are correlated with VWM performance
To shed light on the functional significance of neuronal activity reflected in the ERs, we characterized the early components in more detail with post hoc
analyses separately for each memory load condition in those cortical regions wherein the ERs were significantly correlated with memory load (see ). We first averaged the absolute-valued-ER time-series across this selection () and then used peak detection to estimate the mean latencies and amplitudes of the P1 and N1 peaks (at around 120 ms and 190 ms, respectively) across the subject population (). The peak latencies were relatively stable across the memory loads and while N1 appeared to be earlier for higher VWM loads, this tendency was not significant. The peak amplitudes were enhanced with VWM load (). We then focused the analysis into two anatomically defined regions of interest: one roughly containing V1 and V2 (calcarine fissure, cuneus, occipital pole, lingual gyrus) and another comprising the remaining extrastiate visual regions in lateral and inferior occipital (LO / IO) cortices ( and ). The strength of P1 was larger in the V1/V2 than in the LO/IO selection () but in both regions of interest, the strength of P1 and N1 were increased with the VWM load (). Interestingly, while the amplitude of the P1 was essentially linearly dependent on the number of objects in the Sample stimulus, the amplitude of N1 increased from one to three objects and plateaued thereafter. This plateau onset is near the mean behavioral memory capacity in these data (4.1 ± 0.2, mean ± SEM; Palva et al., 2010a
) and we hence investigated whether this plateau onset in individual subjects’ ERs would be correlated with their VWM capacity. We estimated the plateau onset by detecting the first positive peak in the difference of third and second order polynomials that were least-squares fitted to the series of ER peak amplitudes for the six memory loads (see Methods). The correlation of this ER-predicted capacity value with the real behavioral capacity value was then evaluated with Spearman’s correlation test. Intriguingly, this analysis showed that strength of N1 was, indeed, positively correlated with memory capacity in V1/V2 but not in LO/IO (). P1 was uncorrelated with behavioral capacity in both V1/V2 and LO/IO.
Because early ER components are well known to reflect such physical stimulus properties, the memory-load dependence of P1 in these data cannot be used to infer a role for in VWM encoding. However, the correlation of N1 with individual behavioral memory capacity indicates that ERs in this latency range already reflect VWM-related processing and are not fully determined by the physical properties of the stimuli.
Oscillation dynamics in VWM
The principal aim of this study was to identify the cortical regions where non-stimulus locked ongoing oscillations are correlated with the VWM retention, memory load, or with behavioral performance. ERs, reflecting phasic stimulus-locked brain activity, lasted until ~400 ms from the Sample-stimulus onset and were VWM-capacity sensitive already at around ~200 ms suggesting that memory encoding could be completed before 400 ms. We thus consider the VWM-retention period in these data to be from 400 to 1100 ms from the Sample-stimulus onset. The majority of studies on the role of oscillations in VWM have used retention periods of up to three seconds. It has, however, been suggested that the mechanism of memory maintenance may change after first second of memory retention (Todd and Marois, 2004
). The present study explores the neuronal substrates of this one-second memory retention with an experimental paradigm that has been psychophysically well validated (Luck and Vogel, 1997
, Todd and Marois, 2004
, Vogel and Machizawa, 2004
, Vogel et al., 2005
). To grasp the temporal and spectral evolution of event-related ongoing activity, we first computed time-frequency (TF) representations of SL and oscillation amplitudes by using a bank of 36 Morlet wavelets from 3 to 90 Hz. Similarly to the analyses of fMRI data (Kriegeskorte et al., 2009
), TF-analyses of MEG data are susceptible to statistical inflation arising from circular data inspection and ROI selection. To have a non-circular discovery strategy, we first pooled the statistical TF data across all brain regions by plotting for each TF element the fraction of statistically significant patches (or M/EEG sensors). To control for multiple comparisons, we removed the number of significant TF elements predicted to be false discoveries at the chosen alpha level, A
. Because VWM is known to involve a widely distributed cortical network, this approach was expected to reveal those TF regions where large-scale cortical activity was modulated by VWM processing.
We first evaluated the changes from baseline level with Wilcoxon-signed rank test for data averaged across the six memory loads. TFR for SL showed that frequencies up to 40 Hz were phase-locked to the stimulus onset for several hundreds of milliseconds from stimulus onset without a clear oscillatory component (A
= 0.001, FDR corrected, ). The TFR of oscillation amplitudes revealed an amplitude increase in the θ/α band during the first 400 ms after stimulus onset, which coincided with the period of strong stimulus-locked activity (A
= 0.01, FDR corrected, ). After this transient, the oscillation amplitudes were suppressed below the baseline level throughout the analyzed frequency range from θ/α- to γ-frequency bands. This suppression was also evident in broad-band filtered data (see 1E). We also confirmed that the recording method would not affect the observed pattern of SL or oscillation amplitudes by computing the TFRs separately for MEG gradiometers, magnetometers, and EEG electrodes. All three recording methods converged with source modeling on a similar result for both SL and amplitude dynamics. The fraction of significant sensors was greater than the fraction of significant source patches suggesting that the source modeling was successful in disentangling cortical sources from their mixed sensor level signals (cf
. , Suppl. Fig. 6
To investigate the impact of memory load to the oscillatory dynamics, we identified the brain regions in which SL or amplitude dynamics were correlated with memory load by using the Spearman’s rank correlation test with the null hypothesis that the SL or amplitude values in the six individual memory-load conditions were uncorrelated among the memory loads from 1 to 6 objects. Prior to testing, the individual subjects’ data were rank transformed to ensure comparability across the population. The TFR of SL revealed an early positive correlation with memory load (A = 0.001, FDR corrected, ). The TFR of memory-load dependent amplitude changes showed an interesting pattern. Similarly to SL and ERs, the amplitudes of θ/α- and h-α-frequency bands were positively correlated with memory load during the first 200 ms after stimulus onset (A = 0.01, FDR corrected, ). The early positive effect thereafter changed into a negative correlation in the θ/α, h-α, and β bands, which indicates that the amplitude suppression was even deeper at higher memory loads. The negative correlation, however, was sustained through the VWM retention period only in the θ/α band. The amplitudes of h-α and β bands were positively correlated with memory load from 400 ms to the end of the VWM retention period even though they were was suppressed below the baseline level in the Average-condition.
On the basis of the TF-mapping, we designed a set of broad-band FIR filters to capture the θ/α-, h-α-, β-, γ-, and h-γ-frequency bands with a temporal resolution greater than that provided by the wavelets and we also used a patch collection with a spatial resolution higher than was used in the wavelet-based analysis (see Methods). The amplitude time series obtained with these filters reproduced very well the phenomena discovered with wavelets (A = 0.01, FDR corrected, ), which shows that the results presented here are not qualitatively affected by the filtering or cortical parcellation approaches. However, the dynamics obtained with these broad-band filters revealed that also γ- and h-γ-band amplitudes were strengthened with increasing memory load.
Cortical sources of retention period modulations of oscillation amplitudes
We used the FIR filtered θ/α-, h-α, β-, γ- and h-γ-band data to identify the cortical regions underlying the amplitude modulations during VWM retention. In data averaged across all six memory loads, the amplitudes were suppressed in distributed visual regions in striate, extrastriate, temporal, and parietal cortices (A
= 0.01, ). In the β and γ bands, the suppression was most pronounced in medial occipital and parietal cortices, but also observed in occipito-temporal, and temporal visual areas. In addition, the suppression was observed in the same cortical regions during the early (400–700 ms from Sample stimulus onset) and late (700–1000 ms from Sample stimulus onset) parts of the retention period (Suppl. Fig. 7
Increasing memory load was associated with a further suppression of amplitudes in the θ/α- and h-α bands in the occipital-, occipito-temporal-, and parietal regions (). The amplitudes in the h-α, β, γ, h-γ bands, and very weakly in the θ/α band, were strengthened in widespread fronto-parietal cortical regions including the LPFC, pre- and post-central regions, and cingulate, insular, and orbito-frontal cortices. Interestingly, the β- and γ-band amplitudes were, in addition, strengthened in several visual regions in the occipital and occipito-temporal, and temporal cortices that are known to play a central role in the formation of visual object representations (Konen and Kastner, 2008
, Riesenhuber and Poggio, 2002
, Grill-Spector and Malach, 2004
). While the anterior positive correlations remained very similar at A
= 0.001, the posterior positive correlations in β and γ bands did not exceed this higher alpha level. High-γ-band amplitudes were positively correlated with memory load in LPFC, posterior parietal, temporal, insular cortical regions. Importantly, the network of load-dependent h-γ oscillations was spatially stable throughout the retention period (see Suppl. Fig. 7
The Morlet-wavelet-analysis approach revealed essentially the same fronto-parietal pattern indicating that the parcellation scheme did not affect the primary results (A
= 0.01, FDR corrected, Suppl. Fig. 8
). However, the memory-load dependent strengthening of β- and γ-band amplitudes in visual regions did not exceed the significance threshold, which is likely to be caused by a greater spectral variability in these bands across the subjects.
Correlation of oscillation amplitudes with memory load and behavioral capacity
The results have so far clearly indicated that the amplitudes of h-α-, β-, γ-, and h-γ-band oscillations were positively correlated with memory load in a fronto-parietal network (see ). To further characterize this phenomenon with post-hoc
analyses, we estimated the amplitudes in all frequency bands separately for each memory load and individual subject as an average across the cortical regions that were positively correlated with memory load (see ). These data corroborated that the amplitudes were indeed memory-load dependent during the memory retention period as a function of memory load (). Evaluation of mean amplitudes averaged over the later-half of the retention period (700–1000 ms) showed that oscillation amplitudes were systematically increased with memory load, but with a biphasic shape and a plateau at 3–4 objects (). The analysis for the Morlet-wavelet filtered data showed similarly that both the strengths and fraction of significant cortical regions were systematically modulated by the increasing memory load (Suppl. Fig. 9A–D
To address the link between the retention period amplitude modulations and individual behavioral VWM capacity, we used the same approach as above for ERs (). In the bilateral fronto-parietal regions, the plateau onsets of the h-α-band amplitudes were negatively correlated with memory capacity () implying that in high-capacity subjects, the load-dependent alpha strengthening leveled off at lower memory loads than in low-capacity subjects. On the other hand, despite a similar negative trend, neither β- nor γ-band amplitudes were significantly correlated with the memory capacity. Interestingly, although the h-γ-band amplitudes were only weakly correlated with the VWM load, the plateau onsets in h-γ band were clearly positively correlated with the individual VWM capacity.
We also investigated the correlation of the load-dependent amplitude suppression with memory capacity. We first estimated the relative amplitudes from those cortical regions that were negatively correlated with the memory load (Supplementary Figure 10A
). The amplitudes were then averaged over the retention period (Supplementary Figure 10B
). Lastly, we estimated the correlation between the predicted and real capacity values (Supplementary Figure 10C
). The amplitude suppression was not, however, significantly correlated with the individual memory capacity in any of the studied frequency bands.
Finally, to corroborate the group analysis approach, we asked whether low- and high-capacity subject populations were associated with qualitatively and phenomenologically similar retention-period oscillation dynamics. The subject group was divided into two halves by their individual memory capacity and cortical regions where oscillation amplitudes were correlated memory load were identified with Spearman’s rank correlation test (p < 0.01, FDR corrected) as above (). The left-lateralized but bilaterally highly significant fronto-parietal h-α-, β- and γ-band oscillations were observed in both subject populations although they were more pronounced for the high capacity subjects. On the other hand, only the high-capacity group was associated with a significant memory-load correlation in the high-γ band. The apparent absence of memory-load dependent high-γ in the low-capacity group may, however, be associated the overall poor signal-to-noise-ratio in this frequency band. Interestingly, the amplitude suppression in the h-α-band was less pronounced for the high- than the low-capacity subjects.