Subjects viewed one of eight possible stimulus orientations while activity was monitored in early visual areas (V1–V4 and MT+) using standard fMRI procedures (3T MRI scanner, spatial resolution 3×3×3 mm; see Methods). For each 16-s “trial” or stimulus block, a square-wave annular grating () was presented at the specified orientation (0, 22.5, …, 157.5°), and flashed on and off every 250 ms with a randomized spatial phase to ensure that there was no mutual information between orientation and local pixel intensity. Subjects maintained steady fixation throughout each fMRI run, during which each of the eight stimulus orientations was presented in randomized order. Subjects viewed a total of 20–24 trials for each orientation.
Fig. 1 Orientation decoder and ensemble orientation selectivity. (a) The orientation decoder predicts stimulus orientation based on fMRI activity patterns. The cubes depict an input fMRI activity pattern obtained while the subject viewed gratings of a given (more ...)
Orientation decoder and ensemble orientation selectivity
We constructed an “orientation decoder” to classify ensemble fMRI activity on individual trials according to stimulus orientation, based on the orientation-selective activity pattern in visual cortex that was obtained from a training data set (). The input consisted of the average response amplitude of each fMRI voxel in the visual area(s) of interest, for each 16-s stimulus trial. In the next layer, “linear ensemble orientation detectors” for each of the eight orientations received weighted inputs from each voxel and calculated the linearly weighted sum as output. Individual voxel weights for each orientation detector were determined by using a statistical learning algorithm applied to an independent training data set15
. Voxel weights were optimized so that each detector’s output became larger for its preferred orientation than for other orientations (see Methods). The final output prediction was made by selecting the most active linear ensemble orientation detector as the most likely orientation to be present.
We first trained the orientation decoder using 400 voxels from areas V1/V2 for individual subjects (). Individual voxels showed poor response selectivity for different orientations (see also Supplementary Figs. 1
online). Nonetheless, the output of the linear ensemble orientation detectors, which reflect the weighted sum of many individual voxel responses, revealed well-tuned responses centered around the preferred orientation of each detector. Furthermore, the detectors showed a graded response that increased according to the similarity of stimulus orientation to their preferred orientation. Because the similarity among orientations was not explicitly specified in the learning procedure, this graded response indicates that similar orientations give rise to more similar patterns of fMRI activity. These results suggest that the ensemble pattern of fMRI activity contains orientation information that greatly exceeds the selectivity of individual voxels.
Accuracy of orientation decoding across human visual areas
We evaluated if fMRI activity patterns in the human visual cortex are sufficiently reliable to predict what stimulus orientation the subject is viewing on individual trials. We performed a cross-validation analysis in which the orientation of each fMRI sample was predicted after the orientation decoder was trained with the remaining samples. Therefore, independent samples were used for training and test. Ensemble fMRI activity in areas V1/V2 led to remarkably precise decoding of which of the eight orientations was seen by the subject on individual stimulus trials (). Decoded orientation responses peaked sharply at the true orientation, with errors, which were infrequent, occurring primarily at neighboring orientations and rarely at orthogonal orientations. The accuracy of these decoded orientation responses was quantified for all four subjects, by calculating the root mean squared error (RMSE) between the true and the predicted orientations (, orientation responses aligned to vertical axis). The RMSEs for subjects S1–S4 were 17.9°, 21.0°, 22.2°, 31.2°, respectively.
Fig. 2 Decoding stimulus orientation from ensemble fMRI activity in the visual cortex. (a) Decoded orientation responses for eight orientations. Polar plots indicate the distribution of predicted orientations for each of eight orientations (S2, 400 voxels from (more ...)
In general, orientation decoding performance progressively improved with increasing numbers of voxels, as long as voxels were selected from the retinotopic region corresponding to the stimulated visual field. Voxels that displayed stronger orientation preference were confined well within retinotopic boundaries of the annular stimulus (see Supplementary Fig. 3
online), suggesting the retinotopic specificity of these orientation-selective signals. Consistent with this notion, fMRI activity patterns from unstimulated regions around the foveal representation in V1 and V2 led to chance levels of orientation decoding performance. Although our subjects were well trained at maintaining stable fixation, it is conceivable that different orientations might elicit small but systematic eye movements that could alter the global pattern of cortical visual activity. However, additional control experiments revealed that when independent gratings (45° or 135°) were presented simultaneously to each hemifield, visual activity in each hemisphere could accurately decode the orientation of the contralateral stimulus but not the ipsilateral stimulus (96.1% vs. 54.9% correct using 200 voxels from V1/V2 for each hemisphere; chance level, 50%; see also Supplementary Fig. 4
online). The independence of orientation information in the two hemispheres cannot be explained in terms of eye movements or any other factors that would lead to global effects on cortical activity.
We further investigated the physiological reliability of these orientation signals in the human visual cortex by testing generalization across separate sessions in two subjects. This was done by training the orientation decoder with fMRI activity patterns from one day and using it to predict perceived orientation with the fMRI data from another day (). The RMSEs for the across-session generalization were 18.9° (31 days apart) and 21.7° (40 days apart) for subjects S1 and S2, respectively, almost as small as those for within-session generalization (17.9° and 21.0°, respectively). The results indicate that these orientation-selective activity patterns reflect physiologically stable response preferences across the visual cortex.
The ability to extract robust orientation information from ensemble fMRI activity allowed us to compare orientation selectivity across different human visual areas. Orientation selectivity was most pronounced in early areas V1 and V2, and declined in progressively higher visual areas (). All four subjects showed this same trend of diminishing orientation selectivity across retinotopic visual areas (RMSEs [mean ± SD] for ventral V1, V2, V3, and V4 were 31 ± 4º, 29 ± 4º, 40 ± 8º, and 46 ± 4º, respectively). This pattern of orientation selectivity is consistent with monkey data showing poorer orientation selectivity and weaker columnar organization in higher visual areas10
, but has never been revealed in the human visual cortex. Unlike areas V1 through V4, human area MT+ showed no evidence of orientation selectivity (53.2 ± 3.7º; chance level, 52.8º), consistent with the notion that this region may be more sensitive to motion than to stimulus form.
Fig. 3 Orientation selectivity across the human visual pathway. Decoded orientation responses are shown for individual visual areas from V1 through V4 and MT+ (S3, 100 voxels per area). The color map indicates t-values associated with the responses to the visual (more ...)
Source of orientation information
Additional analyses confirmed that this orientation information obtained from the human visual cortex reflects actual orientation-dependent responses. Our linear ensemble orientation detectors were unable to discriminate the orientation of the phase-randomized stimulus gratings based on pixel intensity values alone because orientation is a higher-order property that cannot be expressed by a linearly weighted sum of inputs16
. However, the decoder composed of linear detectors could classify these images if intervening orientation filters with nonlinearity, analogous to V1 neurons, served as input (). The results indicate that ensemble orientation selectivity does not arise from the retinotopic projection of bitmap grating images on the cortex but rather from the orientation information inherent in individual voxels, which can then be pooled together.
Fig. 4 Pairwise decoding performance as a function of orientation difference (all pairs from eight orientations), for (I) grating images (pixel intensities), (II) fMRI images (voxel intensities), and (III) transformed grating images. The gratings (I) were 20×20 (more ...)
What is the pattern of orientation preferences among these voxels that is responsible for such precise ensemble selectivity? We plotted the orientation preference of individual voxels on flattened cortical representations, by coloring voxels according to the orientation detector for which each voxel provided the largest weight (). Voxel orientation preferences revealed a scattered pattern that was variable and idiosyncratic across subjects. Although some local clustering of the same orientation preference was observed, much of this may reflect spatial blurring resulting from subtle head motion, data reinterpolation required for the Talaraich transformation, and the blurred nature of BOLD hemodynamic responses12,13
. There were no significant differences in the proportion of preferred orientations across different visual areas or different visual field quadrants. Overall, orientation preference maps revealed considerable local variability, indicating that global bias effects due to eye movements or other factors cannot account for the high degree of orientation-selective information that resides in these activity patterns.
Fig. 5 Orientation preference map on flattened cortical surface. The color maps depict the orientation preference of individual voxels on the flattened surface of left ventral V1 and V2 for subjects S2 and S3 (scale bar, 1 cm). Each cell delineated by thick (more ...)
Additional analyses showed that orientation selectivity remained equally robust even when the fMRI data underwent normalization to remove differences in mean activity levels across individual activity patterns. Also, differences in mean activity level were small (Supplementary Fig. 1
online), even between canonical and oblique orientations (the oblique effect17
). Therefore, gross differences in response amplitudes were not a critical source of orientation information for our decoding analysis. We also tested if a global bias for radial orientations might account for our results, as some studies have reported evidence of a bias for orientations radiating outward from the fovea in retinal ganglion cells and V1 neurons of the monkey18,19
. We removed the global response modulation along the radial dimension from each activity pattern, by dividing voxels into the 16 polar-angle sections obtained from retinotopic mapping, and then normalizing the mean response within each set of iso-polar voxels to the same value for every stimulus trial. After this normalization procedure, orientation selectivity diminished only slightly. The mean RMSEs of four subjects for the original and the normalized data were 22.7 ± 6.2° and 26.7 ± 7.1°, respectively (200 voxels from V1/V2). Although global factors, such as radial bias, might account for a small degree of the extracted orientation information, local variations in orientation preference seem to provide the majority of the orientation content in these fMRI activity patterns.
The scattered distribution and local variability of orientation preferences across cortex () are consistent with the notion that random variations in the distribution or response strength of individual orientation columns can lead to small local orientation biases that remain detectable at voxel-scale resolutions. To evaluate the viability of this hypothesis, we performed simulations using one-dimensional arrays of orientation columns, which were sampled by coarse-scale voxels and analyzed by our orientation decoder. The array of voxels was allowed to jitter randomly from trial to trial to mimic the effects of small amounts of brain motion. We compared two types of column arrays, one with regularly shifting preferred orientations () and the other with small random variations in the shifted orientation () as can be observed in columnar structures in animals. While the regular array showed poor orientation decoding performance, the array with random variation resulted in very similar performance to what was found from actual fMRI activity patterns in the human visual cortex (). These results support the hypothesis that a small amount of random variability in the spatial distribution of orientation columns could lead to small local biases in individual voxels that allow for robust decoding of ensemble orientation selectivity.
Fig. 6 Simulation of one-dimensional array of columns and voxels. Each column was assumed to respond to orientation input according to a Gaussian-tuning function peaking at its preferred orientation (SD, 45°; noise was added to the output). The preferred (more ...)
Mind-Reading of attended orientation
Finally, we asked if the ability to characterize brain states corresponding to different stimulus orientations can be extended to the problem of mind-reading, that is, determining a subject’s mental state given knowledge of his or her brain state. Specifically, we tested if the activity patterns evoked by unambiguous single orientations can be used to decode which of two competing orientations is dominant in a person’s mind under conditions of perceptual ambiguity. We hypothesized that activity in early human visual areas, which was found here to represent unambiguous stimulus orientations, may subserve a common role in representing the subjective experience of orientation when two competing orientations are viewed. If so, then when subjects are instructed to attend to one of two overlapping gratings (i.e., plaid), activity patterns in early visual areas should be biased towards the attended grating.
First, subjects viewed single gratings of 45° or 135° orientation ( top; black and white counterbalanced), and the resulting fMRI activity patterns were used to train the orientation decoder. Then in separate test runs, subjects viewed a plaid stimulus consisting of both overlapping gratings ( bottom). In each 16-s trial, subjects were required to attend to one oriented grating or the other by monitoring for small changes in the width of the bars in the attended grating while ignoring changes in the unattended grating. The fMRI data obtained while subjects viewed the two competing orientations were analyzed using the decoder trained on fMRI activity patterns evoked by the single gratings.
Fig. 7 Mind-reading of attended orientation. (a) Mind-reading procedure. First, a decoder was trained using fMRI activity evoked by single gratings to discriminate 45° vs. 135°. Black and white gratings (equal contrast) were counterbalanced across (more ...)
Orientation signals in early human visual areas were strongly biased towards the attended orientation during viewing of the ambiguous stimulus (). Even though the same overlapping gratings were presented in the two attentional conditions, fMRI activity patterns in the visual cortex reliably predicted which of the two orientations the subject was attending to on a trial-by-trial basis at overall accuracy levels approaching 80% (P < 0.0005 for 4/4 subjects using 800 voxels from V1–V4, Chi square test). Analyses of individual visual areas revealed that ensemble activity was significantly biased toward the attended orientation in all four subjects for area V1 and also V2 (P < 0.05), and in 3/4 subjects for area V3 and areas V3a/V4v combined.
We conducted an additional control experiment to address whether eye movements, orthogonal to the attended orientation, might account for the enhanced responses to the attended grating by inducing retinal motion. The visual display was split into left and right halves, and activity from corresponding regions of the contralateral visual cortex was used to decode the attended orientation in each visual field (Supplementary Fig. 4
online). Even when the subject was instructed to pay attention to different orientations in the plaids of the left and right visual fields simultaneously, cortical activity led to accurate decoding of both attended orientations. Since eye movements would bias only one orientation in the whole visual field, these results indicate that the attentional bias effects in early visual areas are not due to retinal motion induced by eye movements.
The robust effects found in V1 and V2 suggest that top-down voluntary attention acts very early in the processing stream to bias orientation-selective signals. Although previous studies have reported evidence of feature-based attentional modulation in the visual cortex20–25
, our results provide novel evidence that top-down attention can bias orientation signals at the earliest stage of cortical processing when two competing stimuli are entirely overlapping. These results suggest that feedback signals to V1 and V2 may have an important role in voluntary feature-based attentional selection of orientation signals.