3.1. Behavioral Results
Subjects were able to recall the target image category (either “specific” or “general” memory for the image) on the majority of retrieval trials (M = 79.2%); hereinafter ‘Remembered’ items. ‘Forgotten’ items corresponded to trials on which subjects responded “Don't Know” (M = 13.2%) or responded with the incorrect category (M = 5.7%). Trials for which subjects failed to respond (M = 2.0%) were excluded from subsequent memory analyses. The percentage of items Remembered did not differ for Face vs. Scene trials (M = 80.1% vs. M = 78.2%, respectively, t(17) = 1.14, p = .27).
3.2. Category Information during Encoding
3.2.1. Classification of Image Category
Across subjects, mean classification accuracy for the category of the encoded image (Face vs. Scene) was above chance (50%) in each of the temporal and prefrontal ROIs (all t(17)'s > 4.40, all p's < .001 and significant following Bonferroni correction; ). Classification based on temporal ROIs (averaged across HIPP, FG, and PHG) yielded markedly higher accuracy than classification based on prefrontal ROIs (averaged across IFG, MFG, SFG, mPFC, and OFC) (M = 91.1% vs. M = 69.6%, respectively, t(17) = 19.77, p < .001). It is of note that this difference was in spite of the fact that the temporal ROIs were generally much smaller (number of voxels) than the prefrontal ROIs. To visualize the distribution of voxels that positively contributed to Face vs. Scene classification we ran a separate classification analysis using a single meta-ROI that combined each of the eight temporal and prefrontal ROIs (mean accuracy = 94.7%) and generated an importance map from this classification. As can be seen in , within prefrontal cortex, voxels that positively contributed to Face classification were most prevalent in IFG and mPFC, and, to a lesser extent, in SFG and OFC. Voxels that positively contributed to Scene classification were more prevalent in MFG and, to a lesser extent in SFG and anterior portions of IFG. In the temporal lobes, voxels that positively contributed to Face classification were evident in posterior and anterior FG, as well as anterior HIPP. Voxels that positively contributed to Scene classification were evident in posterior PHG, posterior FG, and posterior HIPP.
Figure 2 (A) Classification accuracy for image category (Face vs. Scene) across temporal and prefrontal ROIs. (B) Classification accuracy for image sub-category (Male vs. Female for Face trials; Manmade vs. Natural for Scene trials) across temporal and prefrontal (more ...) 3.2.2. Classification of Image Sub-categories
We next tested for evidence of sub-category representation in prefrontal and temporal ROIs (Male vs. Female for Faces; Manmade vs. Natural for Scenes). Collapsing across Face and Scene sub-categories, classification accuracy was significantly above chance for temporal ROIs (t(17) = 3.51, p < .005) and prefrontal ROIs (t(17) = 3.51, p < .005) (). However, sub-category classification differed robustly across temporal ROIs (F(2,34) = 24.73, p < .001), with accuracy markedly higher for FG (M = 59.6%) than PHG (M = 53.1%) or HIPP (49.9%). Accuracy also differed across prefrontal ROIs (F(4,68) = 4.32, p < .005), with accuracy tending to be higher in lateral prefrontal ROIs (IFG, MFG, SFG) than mPFC or OFC. For the temporal ROIs, an interaction was observed between ROI and image category (F(2,34) = 6.36, p < .01), reflecting greater classification of Face sub-category in HIPP and PHG but better classification of Scene sub-category in FG (). For the prefrontal ROIs, there was no interaction between ROI and image category (F < 1; ).
Classification accuracy for image sub-categories: Faces (Male vs. Female); Scenes (Manmade vs. Natural). SEM, standard error of the mean.
3.3. Relationship between Category Information at Encoding and Subsequent Memory
3.3.1. Trial-by-Trial Variability
To assess the relationship between trial-by-trial variability in the representation of category information during encoding and subsequent memory, we separated all encoding trials according to whether the target image was later Remembered vs. Forgotten. Classification accuracy and classifier evidence were then considered for Remembered vs. Forgotten items. This was done separately for Face and Scene trials, so that image category was not confounded with subsequent memory, but all data reported below were averaged across image category.
To first address the relationship between classification accuracy at encoding and subsequent memory, two ANOVA's were generated: one for the temporal ROIs and one for the prefrontal ROIs. Each ANOVA contained two factors: ROI and subsequent memory. For the ANOVA on the temporal ROIs, the subsequent memory effect was not significant (F(1,17) = 1.37, p = .26; ), nor did subsequent memory interact with ROI (F < 1). Thus, there was no evidence from the temporal ROIs that subsequently Remembered items were better classified as Faces vs. Scenes, relative to subsequently Forgotten items. Rather, both mnemonic classes of items were classified with extremely high accuracy; this was particularly evident for FG, where subsequently Remembered and Forgotten items were each classified with near-perfect accuracy (M = 98.5% and M = 98.9%, respectively).
Figure 3 (A) Difference in classification accuracy for items subsequently Remembered vs. subsequently Forgotten across temporal and prefrontal ROIs. (B) Difference in continuous measure of classifier evidence for items subsequently Remembered vs. subsequently (more ...)
For the ANOVA on the prefrontal ROIs, the main effect of subsequent memory was significant, reflecting greater classification accuracy for Remembered vs. Forgotten items (F(1,17) = 8.13, p < .05; ). This subsequent memory effect did not interact with ROI (F < 1). Thus, in contrast to classification accuracy based on temporal ROIs, Face vs. Scene classification accuracy based on the prefrontal ROIs was significantly higher for items that would later be Remembered. A separate ANOVA indicated that the difference in the subsequent memory effect for prefrontal vs. temporal ROIs was marginally significant (F(1,17) = 4.21, p < .06).
The preceding analyses relating classification accuracy to subsequent memory outcomes point to a potential dissociation in terms of how diagnostic the distributed encoding activity within prefrontal vs. temporal cortex is of memory outcomes. On the one hand, these data suggest that category information is highly discriminable in ventral temporal regions, with classification accuracy approaching ceiling levels (), whereas the representation of category information in prefrontal cortex is more variable and, critically, predictive of memory outcomes. On the other hand, it is important to note that our measure of classification accuracy only reflects whether neural evidence on a given trial favored the target image category or not, but does not capture potential gradations in the strength of evidence. It is possible that Forgotten items were associated with weaker temporal lobe representations than Remembered items, but that these weaker representations were nevertheless sufficient to allow for very high classification accuracy.
To address whether more subtle differences in representational strength in temporal regions were related to memory outcomes, we replicated the analyses described above—generating one ANOVA for temporal ROIs and one for prefrontal ROIs—except that, instead of considering the binary measure of classification accuracy, we now considered the continuous measure of classifier evidence. Critically, for the ANOVA on the temporal ROIs, we now observed a significant main effect of subsequent memory, reflecting greater evidence for Remembered vs. Forgotten items (F(1,17) = 5.84, p < .05; ). This effect did not interact with ROI (F < 1). For the ANOVA on the prefrontal ROIs, the effects were consistent with those based on classification accuracy: there was a main effect of subsequent memory (F(1,17) = 6.76, p < .05; ) and this effect did not interact with ROI (F < 1). While the effect was, numerically, larger for prefrontal than temporal ROIs, a separate ANOVA indicated that the difference was not significant (F(1,17) = 1.73, p = .21). Thus, these data indicate that the continuous measure of classifier evidence captured differences in category information in temporal regions that were not reflected in the categorical measure of classification accuracy (the latter null result may have partially stemmed from a restricted range due to ceiling effects).
3.3.2. Individual Differences
In the preceding section, we presented evidence that trial-by-trial differences in classifier-based measures of category information were related to subsequent memory outcomes. We next asked whether cross-subject variability in the strength of category information at encoding was related to individual differences in retrieval success. Specifically, we tested for a correlation between mean classifier evidence based on prefrontal and temporal ROIs for the Face vs. Scene discrimination at encoding and the percentage of Remembered items for each subject. This was done separately for Face trials (i.e., correlating mean classifier evidence across Face trials with mean retrieval success for Face trials) and Scene trials.
Using classifier evidence from temporal ROIs, we observed a positive, but nonsignificant relationship between classifier evidence at encoding and subsequent memory for Faces and Scenes (p's > .1; ). For classifier evidence from the prefrontal ROIs, the correlations for Face and Scene trials were each significant (p's < .05; ). These correlations reflected a positive relationship between the discriminability of Faces vs. Scenes at encoding and later retrieval success. While statistical significance was only considered for data averaged across prefrontal ROIs vs. data averaged across temporal ROIs (to avoid excessive hypothesis testing), correlation coefficients for each ROI within the temporal and prefrontal groups are reported in . It also is worth noting that while the correlations were significant across the prefrontal ROIs, but not the temporal ROIs, the former correlations were not significantly greater than the latter (William's test: t's < 1.4; p's > .20).
Figure 4 Across-subject correlations showing relationship between mean classifier evidence at encoding and mean success rate at retrieval. Correlations are separately shown for prefrontal ROIs (top row; classifier evidence averaged across the five prefrontal ROIs) (more ...)
Table 2 Correlation coefficients (r) representing across-subject relationship between classifier evidence for Face vs. Scene discrimination during encoding and the percentage of items later Remembered. Results are reported separately for Face trials (i.e., the (more ...)
3.4. Correlations between Temporal and Prefrontal Category Information
The data presented thus far indicate that, during encoding, category information was robustly represented in distributed patterns in both temporal and prefrontal structures. While these representations were clearly more robust in temporal regions, they were more predictive of memory outcomes in prefrontal regions. However, while it is possible that prefrontal and temporal regions represent independent forms of information, extant evidence suggests that these regions interact, with perceptual representations feeding forward from temporal to prefrontal regions (e.g., Simons & Spiers, 2003
) and prefrontal regions influencing temporal representations via top-down control (e.g., Miller & Cohen, 2001
; Miller, Vytlacil, Fegen, Pradhan, & D'Esposito, 2010
; Tomita, Ohbayashi, Nakahara, Hasegawa, & Miyashita, 1999
; Zanto, Rubens, Thangavel, & Gazzaley, 2011
). To the extent that such interactions occur, trial-by-trial variability in the strength of category information within temporal regions should be correlated with variability in the strength of category information within prefrontal cortex.
To test the hypothesis that category information within temporal regions is correlated with such information within prefrontal cortex, we used the continuous measure of classifier evidence, generating a within-subject correlation coefficient for each pairing of temporal vs. prefrontal ROIs. Correlation coefficients were separately generated for Face and Scene trials and transformed to z-scores (Fischer's z). The z-scores were then considered across subjects and across prefrontal-temporal pairings for group-level statistical analyses. An ANOVA was generated with three levels: image category (Face vs. Scene trials), prefrontal ROI (IFG, MFG, SFG, mPFC, OFC), and temporal ROI (HIPP, FG, PHG).
Collapsing across image category, individual t tests confirmed that each of the frontal-temporal correlations (15 pairings total) was significantly greater than 0 (t's > 5.11, p's < .001, significant following Bonferroni correction), reflecting a general positive relationship between evidence within temporal and prefrontal ROIs (). The main effect of image category was not significant (F < 1), indicating that the correlations between prefrontal and temporal ROIs were not, overall, different for Face vs. Scene trials. The main effect of prefrontal ROI was significant (F(4,68) = 6.84, p < .001), with MFG displaying the strongest correlations with temporal ROIs and OFC displaying the weakest. The main effect of temporal ROI was marginally significant (F(2,34) = 2.75, p = .08), with HIPP displaying somewhat stronger correlations with prefrontal ROIs relative to FG and PHG. Additionally, the interaction between prefrontal ROI and temporal ROI was significant (F(8,136) = 4.36, p < .001), indicating that the strength of correlations across temporal ROIs varied as a function of the prefrontal ROI with which it was correlated. Moreover, an interaction between image category and prefrontal ROI (F(4,68) = 2.93, p < .05) indicated that the strength of correlations with temporal structures differed across prefrontal ROIs as a function of the type of image being encoded. For example, during Scene encoding, MFG displayed the strongest correlations with temporal ROIs, whereas during Face encoding mPFC displayed the strongest correlations with temporal ROIs. The interaction between image category and temporal ROI was not significant (F(2,34) = 2.51, p = .10), nor was the three-way interaction between image category, prefrontal ROI, and temporal ROI (F(8, 136) = 1.63, p = .12). Individual ANOVAs with factors of temporal ROI and image category were also applied to each prefrontal ROI to test whether any of the prefrontal ROIs displayed correlations that varied across temporal ROIs as a function of the category of image being encoded. For OFC, a robust interaction between image category and temporal ROI was observed (F(2,34) = 7.22, p < .005). This interaction reflected stronger correlations between OFC and HIPP/PHG during Scene encoding, relative to Face encoding, and stronger correlations between OFC and FG during Face encoding, relative to Scene encoding. No other prefrontal ROI displayed a significant interaction between temporal ROI and image category (all p's > .23).
Figure 5 Correlation matrices for Face and Scene encoding trials showing the mean strength of correlations between trial-level classifier evidence in individual prefrontal and temporal ROIs. Correlation coefficients were transformed to Fisher's z prior to averaging. (more ...)
3.5. Univariate Analyses of Subsequent Memory Effects
To assess the relationship between the preceding MVPA analyses and more typical univariate subsequent memory analyses, we conducted two univariate analyses on the present data. First, we contrasted encoding trials that were subsequently Remembered vs. those subsequently Forgotten (collapsing across image category). At a standard threshold (p < .001, 5-voxel extent) we did not observe any clusters positively associated with subsequent memory that overlapped with the prefrontal or temporal ROIs. At a very liberal threshold (p < .01, 5-voxel extent) the only clusters that overlapped with the prefrontal ROIs were in bilateral IFG; no clusters overlapped with the temporal ROIs. To more closely parallel the subsequent memory analysis applied to the classifier data, we also tested for an interaction between the subsequent memory effects for Faces and Scenes [(Face Remembered > Face Forgotten) > (Scene Remembered > Scene Forgotten)]. At a standard threshold (p < .001, 5-voxel extent) there were no clusters, either from the positive or negative tail of the contrast, that overlapped with the prefrontal or temporal ROIs. At a very liberal threshold (p < .01, 5-voxel extent) there were no clusters from the positive tail of the contrast that overlapped with the prefrontal or temporal ROIs; for the negative tail, a few small clusters of activation, bilaterally, overlapped with the PHG and FG ROIs.
3.6. Pattern similarity analysis
To complement the main classification analyses reported above, we also conducted a pattern similarity analysis (e.g., Kriegeskorte, Mur, & Bandettini, 2008) for which the pattern of activity elicited during each encoding trial was correlated with the pattern of activity elicited on every other encoding trial. This analysis allowed us to consider how correlations varied across trials as a function of subsequent memory (Remembered vs. Forgotten) and visual category (within-category vs. between-category). This was separately performed for each of the temporal and prefrontal ROIs. All correlations were transformed to z-scores and then averaged according to subsequent memory status and visual category. Subjects with five or fewer trials in one or more of the four relevant bins (Face-Remember, Face-Forget, Scene-Remember, Scene-Forget) were excluded to reduce the influence of small samples on the correlations.
Consistent with the general success of our pattern classifier in discriminating Face vs. Scene trials, within-category correlations (e.g., Face trials correlated with other Face trials) were significantly higher than between-category correlations, both in prefrontal (t(10) = 4.99, p < .001) and temporal regions (t(10) = 10.74, p < .001) (). Notably, for the temporal ROIs, within-category correlations were greater among Remembered items than among Forgotten items (t(10) = 2.89, p < .05). Indeed, Forgotten items were more positively correlated with within-category Remembered items than other within-category Forgotten items (t(10) = 3.10, p < .05). Between-category correlations were numerically, but not significantly more negative for Remembered items (e.g., Face-Remember to Scene-Remember) than Forgotten items (e.g., Face-Forget to Scene-Forget) (t(10) = -1.03, p = .32). There was, however, a significant interaction between subsequent memory group (Remembered-Remembered vs. Forgotten-Forgotten) and category (within vs. between) (F(1,10) = 7.98, p < .05), reflecting the tendency for Remembered items to be associated with greater within-category similarity and greater between-category dissimilarity than Forgotten items.
Figure 6 Pattern similarity analysis. Correlation coefficients were computed for all pairs of encoding trials, reflecting the similarity of the neural response across voxels for each pair of trials. The resulting r values were z-transformed and averaged according (more ...)
For the prefrontal ROIs, correlations did not differ among Remembered items and Forgotten items either within categories (p = .58) or between categories (p = .83). The correlations in prefrontal ROIs were, however, much weaker than the temporal ROIs (see ), likely reflecting a lower proportion of category-selective voxels in the prefrontal ROIs.