Search tips
Search criteria 


Logo of nihpaAbout Author manuscriptsSubmit a manuscriptHHS Public Access; Author Manuscript; Accepted for publication in peer reviewed journal;
Neuroimage. Author manuscript; available in PMC 2010 July 15.
Published in final edited form as:
PMCID: PMC2702152

Mapping the Bilateral Visual Integration by EEG and fMRI


In the human visual system, the internal representation of the left and right visual hemifields is split at the midline of the two cerebral hemispheres. The present study aims to address the questions of when and where the lateralized cortical visual representations are merged to form an intact percept by using a multimodal neuroimaging approach. Visual evoked potential (VEP) and functional magnetic resonance imaging (fMRI) data were acquired from a group of healthy subjects presented with unilateral versus bilateral visual stimuli. Cortical activities involved in processing bilateral visual information are expected to be equally responsive to ipsilateral and contralateral stimuli, and demonstrate spatial nonlinearity in the response to bilateral stimuli. Utilizing these features, we performed integrative as well as separate analyses for both VEP and fMRI data. The present results suggest that i) the majority of cortical activity that integrates visual information across hemifields takes place at extrastriate areas during late visual processing, and that ii) the lateral occipito-temporal (LOT) regions (likely the MT+ complex) and the medial occipital cortex (i.e. V1) may contribute to bilateral visual integration during early visual processing. Our findings are generally in agreement with the bottom-up visual hierarchy, with the exception of the evidence suggesting an early activation of the higher-tier LOT areas and the influence from ipsilateral visual inputs upon the V1 response.

Keywords: MT+, Multimodal Neuroimaging, Visual Evoked Potential, Functional Magnetic Resonance Imaging, Contextual Effect, Global Integration, Visual Hierarchy


Humans readily perceive a unified visual world in spite of its fragmented and discontinuous internal representation at the primary visual cortex (V1). Retinal inputs from the left and right hemifields are separately projected to contralateral V1 areas. Consequently, the V1 retinotopy is split at the midline between the two cerebral hemispheres (Lavidor and Walsh, 2004). This fact points to an important question of how the lateralized visual representations are integrated into an intact percept, namely the bilateral visual integration (BVI).

The traditional visual hierarchy theory suggests a serial processing through a bottom-up cascade of discrete visual areas (Felleman and van Essen, 1991). According to this theory, the integrative processing of bilateral visual information is initiated at certain higher-tier functional areas in which the receptive fields (RF) of neurons are large enough to cover both the left and right visual fields (LVF and RVF, respectively). It also implies that these higher-tier areas are activated later than lower-tier areas with unilateral RF. However, among the areas with bilateral RF, it still remains unclear which area contains the first cortical substrate that merges bilateral visual information while other areas successively act upon the whole visual scene to perform more sophisticated processing (e.g. visual object recognition). To address this question, it is necessary to investigate not only the receptive field properties of all visual areas but also the temporal sequence of their responses to bilateral visual inputs.

The traditional visual hierarchy theory is challenged by more recent theories proposing a parallel top-down and bottom-up information flow (Lamme and Roelfsema, 2000). Evidence has increasingly demonstrated the existence of functional as well as anatomical feedback connections from higher-tier areas targeting lower-tier areas (Salin and Bullier 1995; Hupé et al. 1998; Lamme et al. 1998). As a result, lower-tier cortical areas with traditionally contralateral RF may respond indirectly to ipsilateral visual inputs, through a top-down stream stemming from higher-tier areas with bilateral RF (Tootell et al., 1998; Ban et al., 2006). Cortical responses at higher-tier areas may also precede the activations or re-activations of areas at lower levels of the hierarchy (Buchner et al., 1997; Hupé et al., 2001; Barnikol et al., 2006). These challenges to the traditional hierarchical view amount to further uncertainties with regard to the timing and localization of the cortical activity underlying the integrative processing of bilateral visual information (Ban et al., 2006; Vanni et al., 2004).

Since distributed visual areas in all hierarchical levels may likely contribute to BVI, it is necessary to simultaneously monitor neural responses from the entire visual system, which covers many regions within the occipital, parietal and temporal lobes. As such, high spatiotemporal resolution is desirable. However, existing functional neuroimaging modalities all have limitations in either the spatial or temporal aspect (Liu et al., 2006a). For example, functional magnetic resonance imaging (fMRI) (Kwong et al., 1992; Ogawa et al., 1992; Bandettini et al., 1992), by measuring hemodynamic and/or metabolic responses, is capable of revealing spatial details of neural activations but limited by its low temporal resolution in the order of seconds. Conversely, electroencephalography (EEG) (or magnetoencephalography, MEG) can detect rapid electrophysiological responses, but the EEG/MEG source imaging often suffers from limited spatial resolution due to its ill-posed nature (Baillet et al., 2001; He and Lian, 2002). In light of their complementary strengths and limitations, many methodological developments have been focused upon the integration of fMRI and EEG/MEG (Dale and Halgren, 2001; Gotman et al., 2006; Liu et al., 2006a, 2006b; Liu and He, 2008), which has potential to significantly advance our knowledge in understanding sensory and cognitive neurosciences (Dale et al., 2000; Eichele et al., 2005).

Such a multimodal approach was employed in the present study to address when and where bilateral visual inputs are processed in an integrative manner on the cortex. We acquired the blood oxygenation level dependent (BOLD) fMRI and scalp EEG (visual evoked potential, VEP) signals induced by the same visual stimuli. The stimuli consisted of a pair of identical pattern-reversal (i.e. counter-phase flicking) checkerboards presented separately, simultaneously or in an interleaved manner at vertically symmetric positions within the lower visual field. Such stimuli have been shown to activate the elementary components of the visual system without substantial involvement of high-level psychological processes (Tobimatsu and Celesia, 2006; Miniussi et al., 1998; Skrandies, 2007). Hence, it is well suited to study the fundamental process of bilateral visual integration.

There are two hypotheses with respect to the features of the cortical activity responsible for the processing of the bilateral visual fields. In the first hypothesis, we speculate that BVI arises from the anatomical convergence of two separate cortical pathways in which visual inputs from LVF and RVF are processed respectively. As illustrated in Fig. 1.A, this implies that the responses to stimuli from different hemifields take place within different sets of cortical regions until both pathways converge to allow for the integration across the visual hemifields. The regions of convergence are expected to have bilateral RF and exhibit invariant responses regardless of the stimulus location. Such features allow us to localize these regions by measuring the degree of the invariance of the BOLD fMRI responses to ipsilateral as well as contralateral stimuli. With the same rationale, we may also detect the timing of such anatomical convergence using the VEP. Assuming an equal speed of information flow via both LVF and RVF pathways, a unilateral stimulus from either hemifield activates a region of bilateral convergence (or integration) at the same time. When comparing the LVF- and RVF-elicited VEP responses, identical (or closely similar) scalp potential maps are expected when the bilateral convergence occurs. This is opposed to dissimilar scalp potential maps arising from separate cortical processes responsible for the processing of LVF and RVF inputs.

Fig. 1
(A) Cortical areas involved in the processing of the bilateral visual stimuli are the regions of convergence for two separate cortical pathways specialized to LVF and RVF, respectively. (B) Cortical areas responsible for the integration of bilateral visual ...

In the second hypothesis, cortical regions involved in BVI may also be characterized by a spatially nonlinear response, as illustrated in Fig. 1.B. In line with the previous hypothesis, the responses at regions exclusively specialized to one visual hemifield (i.e. with unilateral RF) are independent of any stimulus from the other hemifield. Within such regions, the sum of responses to separate LVF and RVF stimuli should strictly equal the response to a bilateral stimulus combining both unilateral components. In contrast, a failure of such linear additivity is expected for regions engaged in the integrative processing of bilateral stimuli, due to a lack of unilateral specialization within these regions. It follows that the nonlinear component can be obtained by subtracting the response to a bilateral stimulus from the sum of the individual responses to the unilateral components. As the former is often found to be less than the latter (Vanni et al., 2004; Miniussi et al., 1998; Murray et al., 2001; Steger et al., 2001; Supek et al., 1999), the response nonlinearity may be specifically characterized by an under-summed response to bilateral stimuli. With this rationale, we may identify the cortical regions involved in BVI as those that demonstrate significant nonlinear BOLD fMRI responses to bilateral stimuli, and detect the timing of their activations by searching for the post-stimulus latencies at which the spatial linearity of the VEP responses fails.

Materials and Methods


Eight healthy right-handed subjects (age 24±6 years, 6 male and 2 female) participated in the EEG study. Six of them also participated in the fMRI study. All of the subjects had normal or corrected-to-normal vision and gave written, informed consent in accordance with a protocol approved by the institutional review board at the University of Minnesota.


Visual stimuli consisted of one or two rectangular black-and-white pattern-reversal checkerboards (size: 12° horizontal and 10° vertical; reversing frequency: 2 Hz; spatial frequency: 1.0 cycle/degree; mean luminance: 20 cd/m2) displayed on a dark gray background (luminance: 5 cd/m2) with a yellow central fixation point. Each rectangle was presented 2° below the horizontal meridian and 4° left or right of the fixation point (measured from the near edge). In the two unilateral conditions, a single pattern-reversal checkerboard was presented in either the lower-left or lower-right quadrant of the visual field. In the two bilateral conditions, a pair of checkerboards presented within both lower quadrants were reversed either simultaneously or in an interleaved manner.

Data Acquisition

For each subject, an EEG experiment was conducted in an electrically shielded room. For six out of the eight subjects, separate fMRI measurements were obtained in a 3-T/90-cm bore magnet equipped with an eight-channel phase array head volume coil (Siemens Trio, Siemens, Germany). All of the subjects were trained and instructed to maintain sustained visual gaze upon the central fixation point during both EEG and fMRI experiments.

The EEG experiment included six repeated 3.5-min runs. During each run, the unilateral stimuli and the simultaneous bilateral stimuli were presented in a mixed sequence interspersed with six 4-sec periods with only the central fixation point on the gray background. In total, each type of stimuli was presented for about 6 minutes, yielding around 700 trials. Scalp potentials from 64 electrodes (referenced to FCz and placed according to the extended international 10/20 system) were recorded at 1000 Hz and filtered (0.3~70 Hz) through a pair of amplifiers (BrainAmp MR 64 Plus, BrainProducts, Germany). Eye blinks and movements were monitored with horizontal and vertical electrooculographic (EOG) electrodes. The locations of the electrodes and three anatomical landmarks (left/right preauricular points and nasion) were digitized using a three-dimensional (3-D) RF localizer (Polhemus Fastrak, Colchester, VT).

In the fMRI experiment, the whole-head anatomy was first acquired with 256 sagittal T1-weighted MR images (matrix size: 256×256; in-plane resolution: 1×1 mm2; slice thickness: 1 mm; no gap between slices; TR/TE = 20/5 ms) using a TurboFLASH sequence (Haas, 1990). The functional study included six repeated 4.5-min runs. Each run consisted of four 30-sec task blocks (with the left/right unilateral stimuli and the simultaneous/interleaved bilateral stimuli respectively) separated by five 30-sec control periods with only the central fixation point on the gray background. The BOLD fMRI data was acquired with 16 axial T2*-weighted images (matrix size: 64×64; in-plane resolution: 4×4 mm2; slice thickness: 5 mm; no gap between slices; TR/TE = 1000/35 ms) covering both the occipital and parietal lobes using a conventional gradient-echo echo-planar imaging (EPI) sequence.

Data Analysis

Visual Evoked Potentials

We used BrainVision Analyzer (BrainProducts, Gilching, Germany) to extract the VEP signals from the EEG recordings. The EEG raw data were sequentially preprocessed through ocular artifact rejection (by visual inspection), band-pass filtering (0.3 – 40 Hz), segmentation from −100 to 500 ms around the onsets of specific stimuli, linear trend removal and pre-stimulus baseline correction. The segmented responses were averaged to obtain the stimulus-specific VEP signals. For each subject, we measured three sets of VEP signals in response to the LVF, RVF and bilateral stimuli, respectively, and derived an additional set of VEP signals from the summation of the LVF- and RVF-elicited VEP signals. For the sake of simplicity, we denote the response to the bilateral stimuli as BOTH, and the sum of the individual responses to separate unilateral stimuli as SUM.

Based on the multi-channel VEP signals, we computed global field power (GFP) at every time point. The GFP, defined as the spatial standard deviation of instantaneous scalp potentials, served as a global measure of the VEP response strength (Lehmann and Skrandies, 1980). The GFP waveforms were compared pair-wisely between LVF and RVF or between SUM and BOTH to assess their difference in the temporal behavior and global strength. Statistical significance was evaluated through a paired t-test with a 0.05 significance level.

We computed the correlation coefficient (CC) (Brandeis et al. 1992; Murray et al. 2008) and relative difference (RD) to quantify the topographic similarity and dissimilarity, respectively, between two scalp potential maps at any time instant. Note that CC and RD are related but complementary measures. CC measures the similarity between two spatial patterns regardless of their amplitudes. RD reflects the difference in both the pattern and magnitude. Specifically, the CC and RD between the VEP responses to the LVF and RVF stimuli (denoted as ΦL(t) and ΦR(t) respectively) were computed using the following equations.



Similarly, we computed the CC and RD between SUM and BOTH to assess the response linearity or nonlinearity. We further subtracted BOTH from SUM to obtain the spatially nonlinear component of the bilaterally evoked VEP response, according to the scenario illustrated in Fig. 1.B.

The VEP responses were averaged across subjects. The resulting scalp topographies were visualized as a series of 2-D maps using a standard 64-channel montage.

BOLD-fMRI Analysis

The fMRI data was processed using BrainVoyager (Brain Innovation, Maastricht, Netherlands). The EPI volumes underwent preprocessing steps including head motion correction, slice scan time correction, linear trend removal and high-pass filtering (3 cycles per scan). The preprocessed data were then averaged across six functional runs. One subject with considerable head motion artifact was rejected from subsequent fMRI data analysis. The functional volumes were aligned to the subjects’ anatomical images and re-sampled to a voxel size of 3×3×3 mm3 with a trilinear interpolation.

For each individual subject, the fMRI data was analyzed using a general linear model (GLM) (Friston et al. 1995; Worsley and Friston 1995; Liu and He 2008). In our GLM analysis, the design matrix was derived from the convolution of the stimulus-specific train of spikes, indexing the occurrences of transient visual stimuli, with a canonical hemodynamic impulse response function (HRF). Such a model differs from those used in the conventional GLM analysis (Friston et al., 1995; Worsley and Friston, 1995), wherein the regressors are defined by convolving the “box-car” stimulus functions with the HRF. As the theoretical basis and details are described elsewhere (Liu and He, 2008), the BOLD-to-VEP relationship suggests that the BOLD effect size (i.e. the regression coefficient associated with each stimulus condition) should be proportional to the time integral of the power of the corresponding current source activity that generates the measured VEP signals. Moreover, statistical parametric maps contrasting a stimulus condition vs. the control condition or multiple stimulus conditions were obtained by visualizing the corresponding t statistic thresholded at p<0.003.

The cortical regions responding to either the LVF or RVF stimulus were identified by using a “LVF plus RVF” contrast. Among these regions, those showing significant difference between the LVF and RVF conditions were further identified using a “LVF minus RVF” contrast. Combining these two contrasts, we were able to identify the visual areas that predominantly responded to only one visual hemifield, as opposed to those that responded equally to contralateral and ipsilateral visual hemifields.

The spatial nonlinearity of BOLD responses was evaluated in two different ways: 1) contrasting the sum of BOLD responses to both the LVF and RVF unilateral stimuli vs. the response to their combined bilateral stimuli presented simultaneously, denoted as “LVF+RVF–Both”, and 2) contrasting the BOLD responses to the interleaved vs. simultaneous bilateral stimuli, denoted as “L2R–Both”. According to a previous theoretical study (Liu and He, 2008), both contrasts in principle reflect the same quantity proportional to the time integral of the source power underlying the nonlinear VEP responses.

For the fMRI group analysis, we first transformed the individual subjects’ fMRI images into a common Talairach space. In the Talairach space, the fMRI data were averaged across subjects and sessions before statistical parametric mapping.

Cortical source imaging

The VEP signals were re-referenced to the grand average across channels. A boundary element model consisting of triangulated surfaces of the scalp, skull and brain was built for each individual subject after segmenting the subject’s anatomical MRI. Distributed cortical current density (CCD) was modeled by around 7,000 dipoles placed on the cortical surface (i.e. the boundary between the white matter and the gray matter) with dipole orientations along the outer-normal directions of local cortical patches. A boundary element method (Hämäläinen and Sarvas, 1989) was employed to compute the scalp potential distribution arising from each individual unitary dipole, which collectively sets up a spatial linear system describing the source-to-signal transformation. The minimum norm algorithm (Hämäläinen and Ilmoniemi, 1984) was used to reconstruct the spatiotemporal CCD distribution from the measured scalp potential maps.

The spatiotemporal CCD distribution was also reconstructed by using an advanced multimodal neuroimaging approach integrating both fMRI and VEP data (Liu and He, 2008). Briefly, the central strategy of this approach was to use the BOLD effect sizes, quantified voxel by voxel from the aforementioned GLM analysis, to constrain the time integral of the power of corresponding current source estimates during the post-stimulus period (0~400 ms) of the VEP.

The imaged CCD distribution obtained from either the EEG alone or the fMRI-EEG integration was visualized on the folded cortical surfaces.


Equal Responses to Unilateral Stimuli

We compared the temporal behaviors of the VEP responses to the LVF and RVF stimuli. The GFP of the LVF- and RVF-elicited VEP signals were computed instant by instant for every individual subject. After averaging the GFP values across all subjects, we plotted the group means, as well as the standard errors of the mean (s.e.m), as functions of time as shown in Fig. 2.A. For both the LVF and RVF conditions, the group-averaged GFP waveforms showed a generally consistent morphology, and three GFP peaks were observed at almost identical latencies (75, 108 and 200 ms). This suggests that the retinal information from LVF and RVF is processed with an equal speed via their respective cortical pathways.

Fig. 2
Comparison between the LVF- and RVF-elicited VEP responses (A) Group-averaged GFP waveforms in response to the LVF (blue) and RVF (red) stimuli. (B) Group-averaged VEP topographies at three GFP peak latencies (75, 108 and 200 ms). (C) Group-averaged CC ...

According to the scenario illustrated in Fig. 1.A, we proceeded to examine when the separate LVF and RVF processes reached the regions of anatomical convergence. That is to seek the post-stimulus latencies in which the LVF- and RVF-elicited VEP signals exhibit highly similar topographies. At all three peak latencies identified from Fig. 2.A, the group-averaged scalp potential maps are shown in Fig. 2.B. At the two early latencies (75 and 108 ms), the LVF- and RVF-elicited VEP responses were spatially distinct yet symmetric (CC=−0.09 and 0.41, respectively). Such spatial patterns demonstrate a retinotopic relationship of the underlying cortical generators symmetrically arranged within the right and left hemispheres, respectively. In contrast, both VEP responses at 200 ms were highly consistent in terms of their spatial patterns (CC=0.98) and absolute amplitudes (RD=0.02), and exhibited a bilaterally extended distribution of negative potentials.

We further measured the CC and RD between the LVF- and RVF-elicited VEP responses at every time instant and for every individual subject. The CC and RD values were averaged across subjects and plotted as functions of time, as shown in Fig. 2.C. Both the maximum CC and the minimum RD were found around 200 ms. From 185 to 245 ms, the CC and RD values were sustained at 0.88±0.03 and 0.05±0.01, respectively. These results suggest that the spatiotemporal cortical activities responsible for the separate processing of the LVF and RVF information are virtually identical during late visual processing. Interestingly, we also observed an early increase in CC together with a decrease in RD around 50 ms.

We also analyzed the fMRI data to identify which regions are equally responsive to both LVF and RVF, as opposed to those predominantly responsible for the contralateral hemifield. For this purpose, the GLM analysis was performed based on the group-averaged fMRI data. We used the “LVF plus RVF” contrast to identify the regions responding to either LVF or RVF. As shown in the left panel of Fig. 3, such regions included bilateral areas located within the lateral geniculate nuclei (LGN), striate (V1) and extrastriate cortex (higher-tier visual cortex). Using the “LVF minus RVF” contrast, we found that only the striate cortex exhibited a statistically significant difference in the BOLD fMRI response between the LVF and RVF conditions, and that the striate cortex within both hemispheres demonstrated a dominant response to contralateral stimuli (or contralateral dominance), as shown in the right panel of Fig. 3. Combining the above fMRI mapping results, we can conclude that the bilateral extrastriate visual areas equally respond to both contralateral and ipsilateral visual inputs.

Fig. 3
Group-averaged fMRI statistical maps contrasting “LVF+RVF” vs. rest (left) and LVF vs. RVF (right). Color indicates the corresponding t statistic.

Nonlinear Response to Bilateral Stimuli

We analyzed the VEP response linearity (or nonlinearity) by comparing the response to the bilateral stimulus (denoted as BOTH) with the sum of the individual responses to the unilateral stimulus components (denoted as SUM). We computed the GFP of BOTH and SUM for each individual subject, and then averaged the GFP values across subjects. Fig. 4.A illustrates the comparison between BOTH and SUM in terms of the group-averaged GFP waveforms. In general, the GFP of BOTH was smaller than that of SUM. Such an under-summed response to bilateral stimuli was most apparent after 200 ms since the stimulus onset. A paired t-test was employed to further assess the statistical significance of the difference in GFP between BOTH and SUM. As shown in Fig. 4.B, the t-test results indicate a sustained and significant difference (p<0.05) from 185 to 350 ms. During an early period from 35 to 60 ms, the GFP of BOTH was also significantly smaller than that of SUM.

Fig. 4
Comparison between the VEP signals of SUM and BOTH (A) Group-averaged GFP waveforms of SUM and BOTH. (B) Results of the paired t-test applied to the difference in GFP between SUM and BOTH. Red dashed line indicates the significance level of 0.05. (C) ...

At all three peak latencies (75, 112 and 200 ms) identified from Fig. 4.A, the group-averaged scalp potential maps are displayed in Fig. 4.C. At the two early latencies (75 and 112 ms), the topographies of BOTH and SUM were significantly correlated (CC=0.98) with almost identical signal amplitudes. At the late latency (200 ms), BOTH and SUM still exhibited a similar spatial pattern (CC=0.99); but the signal amplitudes of SUM were considerably larger than those of BOTH. The group-averaged VEP topographies of BOTH and SUM were quantitatively compared for every time point, resulting in the time courses of CC and RD as shown in Fig. 4.D. From 30 to 60 ms, we observed an increase in RD together with a decrease in CC. Smaller changes in CC and RD were also observed around 95 ms and 130 ms. From 200 to 300 ms, RD increased considerably even though CC remained close to 1 with a subtle decrease. After 300 ms, both RD and CC fluctuated over time. Collectively, these results suggest that the spatial nonlinearity of the VEP response to bilateral stimuli occurs primarily during late visual processing (after 200 ms), as well as in a relatively short period (before 60 ms) during early visual processing.

To pinpoint such response nonlinearity to specific regions, we further extracted the nonlinear VEP component by subtracting BOTH from SUM for every individual subject, and then averaged the results across subjects. Fig. 5 shows the group-averaged results. We observed two early components with peak latencies at 60 and 96 ms respectively, in addition to a sustained late component from 160 to 290 ms with its peak at 235 ms. The scalp topographies indicate focal positive potentials at the lateral occipital and posterior temporal regions (PO7 and PO8) around 60 ms; focal negative potentials at the medial occipital and posterior parietal regions (Oz and POz) around 96 ms; and bilaterally extended negative potentials around 235 ms.

Fig. 5
Nonlinear component of the VEP response to bilateral stimuli averaged across subjects. The curve shows the GFP of the group-averaged nonlinear VEP response. The 2-D topographic maps are shown at places close to several representative peak latencies.

Similarly, the nonlinear fMRI responses to bilateral stimuli were examined through two contrasts: “LVF+RVF–Both” and “L2R–Both”, where “L2R” stands for the interleaved presentation of bilateral stimuli. Fig. 6 shows the t-statistic maps resulting from both contrasts. Both maps exhibited consistent spatial distributions (as shown in Fig. 6.A), with a considerable overlap at the medial, posterior and lateral occipital and inferior temporal areas (as shown in Fig. 6.B). According to the scenario illustrated Fig. 1.B, these regions in overlap are likely involved in the cortical processing of bilateral visual information. In light of our previous theoretical results (Liu and He, 2008), both of these two contrasts led to the same quantification of the nonlinear BOLD fMRI response to bilateral visual stimuli. Such a BOLD quantification, in turn, reflects the time integral of the power of the nonlinear component in the neural response to the bilateral visual stimuli.

Fig. 6
(A) For a representative subject (S1), the t-statistic maps of two different contrasts [“LVF+RVF–Both” (left) and “L2R–Both” (right)] are shown in red-to-yellow and blue-to-green respectively. (B) Two maps ...

Spatiotemporal Cortical Source Imaging

We further imaged the spatiotemporal cortical activities responsible for the unilateral visual processing and bilateral visual integration through cortical current density (CCD) reconstruction. The CCD reconstruction allowed us to pinpoint the VEP response at every time point to specific brain locations. The fMRI response provided additional spatial constraints to further improve the spatial specificity (Liu and He, 2008).

Fig. 7.A shows the CCD distributions reconstructed from the VEP responses to a LVF stimulus, with or without incorporating the corresponding fMRI data. The imaged cortical activity at early latencies was contralateral to the stimulus (i.e. within the right hemisphere). The ipsilateral activations were mainly found at late latencies. During late visual processing, the cortical activity was located more bilaterally rather than solely on the contralateral hemisphere. Such a trend was consistently observed in both the EEG-alone and the fMRI-EEG-combined imaging results, whereas the fMRI-EEG integrated analysis resulted in much higher spatial specificity.

Fig. 7
Spatiotemporal cortical activity underlying the unilateral visual processing (A) and the bilateral visual integration (B), respectively. For both panels, images in the 1st row visualize the cortical current density (CCD) estimates obtained by using the ...

Fig. 7.B shows the CCD estimates based on the nonlinear VEP (or VEP-fMRI) responses. Similar to Fig. 7.A, a general agreement was found between the CCD images obtained from the EEG alone and the fMRI-EEG integration, whereas the latter ended up with more focal activations and higher spatial resolution. During early processing of bilateral visual information, cortical activities were found at the lateral occipito-temporal (LOT) areas around 50~60 ms and at V1 around 70~80 ms. After about 150 ms, the cortical activity spread out onto more visual areas within the medial, dorsal and lateral occipital cortex as well as the posterior parietal lobe. These results suggest the involvement of an extended network of visual areas during late visual processing.

It is also noticeable that at late latencies, the cortical activities for processing bilateral stimuli (Fig. 7.B) were similar to those for processing a unilateral stimulus (Fig. 7.A). This observation confirms that the late visual processing acts upon the whole visual field and produces invariant electrical responses independent of the stimulus location.


Hierarchical Visual Processing

The results suggest that the integrative processing of bilateral visual information primarily takes place within the extrastriate regions during late visual processing. This finding is in line with the hierarchical visual organization originally inferred from the intracranial recordings in primates (Felleman and van Essen 1991). The hierarchical theory generally agrees with the present study with noninvasive and multimodal imaging data obtained from healthy humans. Specifically, the fMRI data suggest that the higher-tier extrastriate regions respond equally to both LVF and RVF, as opposed to spatially specialized lower-tier striate regions that predominantly respond to the contralateral hemifield. The VEP data suggest that the spatially invariant and nonlinear responses within the higher-tier bilateral extrastriate regions primarily occur during late visual processing (after 200 ms), whereas the retinotopic responses within the lower-tier extrastriate regions primarily take place during early visual processing. Although such a hierarchical visual processing is also implied by previous studies using fMRI data alone (Tootell et al. 1998; Nelles et al. 2002), the present study with additional VEP data provides more concrete evidence about the timing of the responses within multiple higher-tier and lower-tier visual area.

Early Activations of the LOT Regions

The present study also reports early activations of the LOT regions, which may play an important role in the integrative processing of bilateral visual inputs during early visual processing. The spatially nonlinear VEP response to bilateral stimuli occurs as early as before 60 ms. This early nonlinear VEP response manifests itself as bilateral and focal potential distributions around PO7 and PO8 on the scalp surface, and as bilateral and focal current density distributions within the LOT cortex. Importantly, such LOT regions may represent the first cortical substrates that merge the information across the visual hemifields, since the nonlinear responses within the LOT regions take place earlier than most other higher-tier visual areas, or even V1. According to the results reported in the present study as well as many previous studies (e.g. Vanni et al. 2004; Di Russo et al. 2005), the visual-evoked V1 activity accounts for the VEP component at 75 ms, which is later than the LOT response observed before 60 ms.

These early activated LOT regions may be further identified as the MT+ complex, which encompasses both middle temporal (MT) and medial superior temporal (MST) areas (Hupé et al. 1998; Buchner et al. 1997; Hupé et al. 2001; Barnikol et al. 2006; Vanni et al. 2004; Watson et al. 1993; Dukelow et al. 2001; Nelles et al. 2002; Di Russo et al. 2005; Ffytche et al. 1995; Nowak and Bullier 1997). The MT+ complex is located at the juncture of the ascending limb of the inferior temporal sulcus and the lateral occipital sulcus (Watson et al. 1993; Dukelow et al. 2001). Such an anatomical landmark generally agrees with the LOT locations identified in the present study. In addition, neurons in the MT+ have been reported to have bilateral RF and respond to some elementary visual stimuli similar as what we used in the present study (Tootell et al. 1998; Buchner et al. 1997; Barnikol et al. 2006; Nelles et al. 2002; Di Russo et al. 2005).

In fact, retinal outputs can project to bilateral MT+ areas by bypassing the V1 area via direct connections from subcortical structures (Lamme and Roelfsema 2000; Barnikol et al. 2006; Nowak and Bullier, 1997). Alternatively, the early MT+ activation may result from a very fast feed-forward input from V1 through the magnocellular (Vanni et al. 2004) or koniocellular pathway (Morand et al. 2000).

Nonlinear Response within V1

Both our fMRI and VEP data indicate a nonlinear V1 response to bilateral stimuli. In addition, we also observed an ipsilateral positive BOLD response at V1, with the absence of stimuli from the contralateral hemifield but the presence of stimuli from the ipsilateral hemifield. These results suggest that the neural activity within V1 may be influenced by ipsilateral visual inputs in addition to inputs from the contralateral receptive field, even though the V1 response is dominated by the contralateral visual field.

Several previous studies have found similar but not identical effects. For instance, Ban et al. demonstrated using fMRI that the contextual information outside the traditional receptive field may facilitate the responses at early visual areas including V1 (Ban et al. 2006). Tootell et al. demonstrated a negative BOLD response to ipsilateral stimuli at V1 (Tootell et al. 1998). Regardless of the facilitation (Ban et al. 2006, and the present study) or inhibition (Tootell et al. 1998), all of these studies provide evidence against a seemingly over-simplified role of V1 as a lower-tier processing unit exclusively responsible to the contralateral visual field.

Furthermore, we may posit that the V1 activity modulated by visual context or involved in global integration might arise from the feedback connections from the LOT regions (likely MT+), which show responses (50~60 ms) preceding the V1 response (around 75 ms) in the present study. Hupé et al. also demonstrated such a possibility, by showing that neural responses at V1, V2 and V3 can be modulated by the activity level at the MT+ area (Hupé et al. 1998; Hupé et al. 2001).

Response Linearity or Nonlinearity

Extracting the spatially nonlinear components of visual evoked responses allows for the isolation of brain signals associated with bilaterally integrative process from other independent processes specialized for processing unilateral information. This provides an effective means to investigate the locations and latencies of the cortical responses underlying the BVI. Without performing this computation, the MT+ activity is much less observable, because it is temporally overlapped with the emerging V1 activation.

Furthermore, the use of nonlinear VEP components for imaging the source activity related to the BVI has the benefit of excluding both spatial and temporal interferences from other irrelevant electrical activities. The ill-posedness of the EEG inverse problem results in cross-talk among the current source estimates, particularly at neighboring locations (Dale et al. 2000; Liu et al. 2006a). Since most visual areas are closely clustered at the occipital lobe, the accompanying unilaterally responsive cortical activities almost unavoidably introduce spurious source estimates and false positive temporal correlation at other cortical regions. Alternatively, linearly separating the under-summed VEP components removes the non-integrative activities in the signal space. Since the head volume conductor model is a linear quasi-static system, this procedure precludes the possibility of confounding cross-interference among source estimates without affecting the locations and latencies of the cortical BVI process. This procedure also points to an important distinction from other related electrical source imaging studies with single vs. paired visual stimuli (Vanni et al. 2004; Steger et al. 2001).

fMRI-EEG Integration and Recording

In the present study, we use an fMRI-EEG integrated approach (Liu and He 2008) to image the spatiotemporal activity underlying the bilateral visual integration. This approach is based on a principled fMRI-EEG cross-modal relationship, and provides a mechanism to utilize the information from both modalities.

Simultaneous fMRI-EEG recording is desirable in some studies when the activity of interest is not simply reproducible during separate sessions. This is true for studying epilepsy, sleep, resting-state activities, or some high-level cognitive tasks. However, it is not always necessary for studying some passive sensory evoked responses, as in the present study. One has to base his/her choice on the reproducibility of the task (or stimulus) vs. the possible risk of dealing with largely contaminated EEG data if recorded simultaneously with fMRI. This is in light of the fact that regardless of the theoretical efficacy of artifact correction algorithms for post-processing simultaneously recorded fMRI-EEG data, the outcome of these algorithms is “artificial” and inevitably “worse” than the clear EEG data recorded in a shielded EEG room. With the above considerations, we recorded EEG and fMRI signals through separate sessions while using the same stimuli.


The cortical activity responsible for the integration of bilateral visual fields is characterized by an equal response to both LVF and RVF and a nonlinear response to bilateral visual inputs. These two features provide important clues to the timing and localization of the bilaterally integrative cortical processes. In the present study, we analyzed the VEP and fMRI responses to unilateral vs. bilateral stimuli. Based on the converging evidence from both the VEP and fMRI data, we conclude that 1) the integrative processing of bilateral visual information primarily occurs within the extrastriate regions during late visual processing (200 ms after the stimulus onset), and that 2) the LOT and V1 areas may contribute to BVI during early visual processing (around 60 ms and 100 ms, respectively).


We thank Cameron Sheikholeslami, Han Yuan, Xiaoxiao Bai & Christopher Wilke for technical assistance. This work was supported in part by NIH RO1 EB007920, EB00178 and EB00329, NSF BES-0411898, and a grant from the Institute of Engineering in Medicine of the University of Minnesota. The 3T MRI scanner was partially supported by NIH P41RR008079 and P30NS057091.


  • Baillet S, Mosher JC, Leahy RM. Electromagnetic brain mapping. IEEE Sign. Proc. Mag. 2001;18:14–30.
  • Ban H, Yamamoto H, Fukunaga M, Nakagoshi A, Umeda M, Tanaka C, Ejima Y. Toward a common circle: interhemispheric contextual modulation in human early visual areas. J Neurosci. 2006;26:8804–8809. [PubMed]
  • Bandettini PA, Wong EC, Hinks RS, Tikofsky RS, Hyde JS. Time course EPI of human brain function during task activation. Magn. Reson. Med. 1992;25:390–397. [PubMed]
  • Barnikol UB, Amunts K, Dammers J, Mohlberg H, Fieseler T, Malikovic A, Zilles K, Niedeggen M, Tass PA. Pattern reversal visual evoked responses of V1/V2 and V5/MT as revealed by MEG combined with probabilistic cytoarchitectonic maps. NeuroImage. 2006;31:86–108. [PubMed]
  • Brandeis D, Naylor H, Halliday R, Callaway E, Yano L. Scopolamine effects on visual information processing, attention, and event-related potential map latencies. Psychophys. 1992;29:315–336. [PubMed]
  • Buchner H, Gobbelé R, Wagner M, Fuchs M, Waberski TD, Beckmann R. Fast visual evoked potential input into human area V5. NeuroReport. 1997;8:2419–2422. [PubMed]
  • Dale AM, Liu AK, Fischl BR, Buckner RL, Belliveau JW, Lewine JD, Halgren E. Dynamic statistical parametric mapping: combining fMRI and MEG for high-resolution imaging of cortical activity. Neuron. 2000;26:55–67. [PubMed]
  • Dale AM, Halgren E. Spatiotemporal mapping of brain activity by integration of multiple imaging modalities. Curr. Opin. Neurobiol. 2001;11:202–207. [PubMed]
  • Di Russo F, Pitzalis S, Spitoni G, Aprile T, Patria F, Spinelli D, Hillyard SA. Identification of the neural sources of the pattern-reversal VEP. NeuroImage. 2005;24:874–886. [PubMed]
  • Dukelow SP, Desouza JFX, Culham JC, Van Den Berg AV, Menon RS, Tutis V. Distinguishing subregions of the human MT+ complex using visual fields and pursuit eye movements. J. Neurophysiol. 86:1991–2000. [PubMed]
  • Eichele T, Specht K, Moosmann M, Jongsma MLA, Quiroga RQ, Nordby H, Hugdahl K. Assessing the spatiotemporal evolution of neuronal activation with single-trial event-related potentials and functional MRI. Proc. Natl. Acad. Sci. USA. 2005;102:17798–17803. [PubMed]
  • Felleman DJ, van Essen DC. Distributed hierarchical processing in the primate cerebral cortex. Cereb. Cortex. 1991;1:1–47. [PubMed]
  • Friston KJ, Holmes AP, Poline JB, Grasby PJ, William SCR, Frackowiak RSJ, Turner R. Analysis of fMRI time-series revisited. NeuroImage. 1995;2:45–53. [PubMed]
  • Gotman J, Kobayashi E, Bagshaw AP, Bénar CG, Dubeau F. Combining EEG and fMRI: a multimodal tool for epilepsy research. J. Magn. Reson. Imaging. 2006;23:906–920. [PubMed]
  • Haase A. Snapshot FLASH MRI. Applications to T1, T2, and chemical-shift imaging. Magn. Reson. Med. 1990;13:77–89. [PubMed]
  • Hämäläinen MS, Sarvas J. Realistic conductivity geometry model of the human head for interpretation of neuromagnetic data. IEEE. Trans. Biomed. Eng. 1989;36:165–171. [PubMed]
  • Hämäläinen MS, Ilmoniemi J. Interpreting measured magnetic fields of the brain: estimates of current distributions. Helsinki University of Technology, Department of Technical Physics Technical Report TKK-F-A559. 1984
  • He B, Lian J. High-resolution spatio-temporal functional neuroimaging of brain activity. Crit. Rev. Biomed. Eng. 2002;30:283–306. [PubMed]
  • Hupé JM, James AC, Payne BR, Lomber SG, Girard P, Bullier J. Cortical feedback improves discrimination between figure and background by V1, V2 and V3 neurons. Nature. 1998;394:784–787. [PubMed]
  • Hupé JM, James AC, Girard P, Lomber S, Payne BR, Bullier J. Feedback connections act on the early part of the responses in monkey visual cortex. J. Neurophysiol. 2001;85:134–145. [PubMed]
  • Kwong KK, Belliveau JW, Chesler DA, Goldberg IE, Weisskoff RM, Poncelet BP, Kennedy DN, Hoppel BE, Cohen MS, Turner R. Dynamic magnetic resonance imaging of human brain activity during primary sensory stimulation. Proc. Natl. Acad. Sci. USA. 1992;89:5675–5679. [PubMed]
  • Lamme VAF, Supèr H, Spekreijse H. Feedforward, horizontal and feedback processing in the visual cortex. Curr. Opinion Neurobiol. 1998;8:529–535. [PubMed]
  • Ffytche DH, Guy CN, Zeki S. The parallel visual motion inputs into areas V1 and V5 of human cerebral cortex. Brain. 1995;118:1375–1394. [PubMed]
  • Fischl B, Sereno MI, Dale AM. Cortical surface-based analysis II: Inflation, flattening, and a surface-based coordinate system. NeuroImage. 1999;2:195–207. [PubMed]
  • Friston KJ, Fletcher P, Josephs O, Holmes AP, Rugg MD, Turner R. Event-related fMRI: characterizing differential responses. NeuroImage. 1998;7:30–40. [PubMed]
  • Lamme VAF, Roelfsema PR. The distinct modes of vision offered by feedforward and recurrent processing. Trends Neurosci. 2000;23:571–579. [PubMed]
  • Lavidor M, Walsh V. The nature of foveal representation. Nat. Rev. Neurosci. 2004;5:729–735. [PubMed]
  • Lehmann D, Skrandies W. Reference-free identification of components of checkerboard-evoked multichannel potential fields. Electroencephalogr Clin Neurophysiol. 1980;48:609–621. [PubMed]
  • Liu Z, Ding L, He B. Integration of EEG/MEG with MRI and fMRI in Functional Neuroimaging. IEEE Eng. Med. Biol. Mag. 2006a;25:46–53. [PMC free article] [PubMed]
  • Liu Z, Kecman F, He B. Effects of fMRI-EEG mismatches in cortical current density estimation integrating fMRI and EEG: A simulation study. Clin. Neurophysiol. 2006b;117:1610–1622. [PMC free article] [PubMed]
  • Liu Z, He B. FMRI-EEG integrated cortical source imaging by use of time-variant spatial constraints. NeuroImage. 2008;39:1198–1214. [PMC free article] [PubMed]
  • Miniussi C, Girelli M, Marzi CA. Neural site of the redundant target effect: electrophysiological evidences. J Cogn. Neurosci. 1998;10:216–230. [PubMed]
  • Morand S, Thut G, Grave de Peralta R, Clarke S, Asid K, Landis T, Michel CM. Electrophysiological evidence for fast visual processing through the human koniocellular pathway when stimuli move. Cereb. Cortex. 2000;10:817–825. [PubMed]
  • Murray MM, Foxe JJ, Higgins BA, Javitt DC, Schroeder CE. Visuo-spatial neural response interactions in early cortical processing during a simple reaction time task: a high-density electrical mapping study. Neuropsychol. 2001;39:828–844. [PubMed]
  • Murray MM, Brunet D, Michel CM. Topographic ERP analysis: a step-by-step tutorial review. Brain. Topo. 2008;20:249–264. [PubMed]
  • Nelles G, Widman G, de Greiff A, Meistrowitz A, Dimitrova A, Weber J, Forsting M, Esser J, Diener C. Brain representation of hemifield stimulation in poststroke visual field defects. Stroke. 2002;33:1286–1293. [PubMed]
  • Nowak LG, Bullier J. The timing of information transfer in the visual system. In: Rockland KS, Kaas JH, Peters A, editors. Cerebral Cortex. London: Plenum; 2002. pp. 205–241.
  • Ogawa S, Tank DW, Menon R, Ellermann JM, Kim SG, Merkle H, Ugurbil K. Intrinsic signal changes accompanying sensory stimulation: functional brain mapping with magnetic resonance imaging. Proc. Natl. Acad. Sci. USA. 1992;89:5951–5955. [PubMed]
  • Salin P-A, Bullier J. Corticocortical connections in the visual system: structure and function. Physiol. Rev. 1995;75:107–154. [PubMed]
  • Skrandies W. The effect of stimulation frequency and retinal stimulus location on visual evoked potential topography. Brain Topo. 2007;20:15–20. [PubMed]
  • Steger J, Imhof K, Denoth J, Pascual-Marqui RD, Steinhausen HC, Brandeis D. Brain mapping of bilateral visual interactions in children. Psychophysiol. 2001;38:243–253. [PubMed]
  • Supek S, Aine CJ, Ranken D, Best E, Flynn ER, Wood CC. Single vs. paired visual stimulation: superposition of early neuromagnetic responses and retinotopy in extrastriate cortex in humans. Brain Res. 1999;830:43–55. [PubMed]
  • Tobimatsu S, Celesia GC. Studies of human visual pathophysiology with visual evoked potentials. Clin. Neurophysiol. 2006;117:1414–1433. [PubMed]
  • Tootell RBH, Mendola JD, Hadjikhani NK, Liu AK, Dale AM. The representation of the ipsilateral visual field in human cerebral cortex. Proc. Natl. Acad. Sci. 1998;95:818–824. [PubMed]
  • Vanni S, Dojat M, Warnking J, Delon-Martin C, Segebarth C, Bullier J. Timing of interactions across the visual field in the human cortex. NeuroImage. 2004;21:818–828. [PubMed]
  • Watson JD, Myers R, Frackowiak RS, Hajnal JV, Woods RP, Mazziotta JC, Shipp S, Zeki S. Area V5 of the human brain: evidence from a combined study using positron emission tomography and magnetic resonance imaging. Cereb. Cortex. 1993;3:79–94. [PubMed]
  • Worsley KJ, Friston KJ. Analysis of fMRI time-series revisited again. NeuroImage. 1995;2:173–181. [PubMed]