Search tips
Search criteria 


Logo of nihpaAbout Author manuscriptsSubmit a manuscriptHHS Public Access; Author Manuscript; Accepted for publication in peer reviewed journal;
Neuroimage. Author manuscript; available in PMC 2011 August 1.
Published in final edited form as:
PMCID: PMC3129373

Discriminating Schizophrenia and Bipolar Disorder by Fusing FMRI and DTI in A Multimodal CCA+ Joint ICA Model


Diverse structural and functional brain alterations have been identified in both schizophrenia and bipolar disorder, but with variable replicability, significant overlap and often in limited number of subjects. In this paper, we aimed to clarify differences between bipolar disorder and schizophrenia by combining fMRI (collected during an auditory oddball task) and diffusion tensor imaging (DTI) data. We proposed a fusion method, “multimodal CCA+ joint ICA’, which increases flexibility in statistical assumptions beyond existing approaches and can achieve higher estimation accuracy. The data collected from 164 participants (62 healthy controls, 54 schizophrenia and 48 bipolar) were extracted into “features” (contrast maps for fMRI and fractional anisotropy (FA) for DTI) and analyzed in multiple facets to investigate the group differences for each pair-wised groups and each modality. Specifically, both patient groups shared significant dysfunction in dorsolateral prefrontal cortex and thalamus, as well as reduced white matter (WM) integrity in anterior thalamic radiation and uncinate fasciculus. Schizophrenia and bipolar subjects were separated by functional differences in medial frontal and visual cortex, as well as WM tracts associated with occipital and frontal lobes. Both patients and controls showed similar spatial distributions in motor and parietal regions, but exhibited significant variations in temporal lobe. Furthermore, there were different group trends for age effects on loading parameters in motor cortex and multiple WM regions, suggesting brain dysfunction and WM disruptions occurred in identified regions for both disorders. Most importantly, we can visualize an underlying function-structure network by evaluating the joint components with strong links between DTI and fMRI. Our findings suggest that although the two patient groups showed several distinct brain patterns from each other and healthy controls, they also shared common abnormalities in prefrontal thalamic WM integrity and in frontal brain mechanisms.

Keywords: multimodal canonical correlation analysis (mCCA), independent component analysis (ICA), functional MRI (fMRI), diffusion tensor imaging (DTI), schizophrenia, bipolar disorder, fractional anisotropy (FA)


Schizophrenia (SZ) is a psychotic disorder characterized by altered perception, thought processes, and behaviors ; and bipolar (BP) illness is a mood disorder involving prolonged states of depression and mania (Goodwin and Jamison 2007). These two brain diseases have overlapping symptoms (for example reports suggest that 60% of bipolar 1 patients have psychotic features (Goes, et al. 2007; Guze, et al. 1975)), persistent neurocognitive deficits (Glahn, et al. 2004), similar risk genes (Bahn 2002), and co-occurrence within relatives (Lichtenstein, et al. 2009) ; however, the common and distinct neural mechanisms underlying these disorders remain unclear.

Many brain imaging techniques including functional MRI (fMRI), structural MRI (sMRI), EEG/MEG and diffusion tensor imaging (DTI) provide information on different aspects of the brain (e.g. blood flow, integrity of gray matter, integrity of white matter tracts). FMRI can assess functional differences related to rest or neurocognition and DTI can additionally provide information on structural connectivity among brain networks. Numerous studies have been published directly comparing schizophrenia to bipolar disorder in one imaging modality, such as fMRI (Calhoun, et al. 2007; Hamilton, et al. 2009; Pearlson 1997), sMRI (Altshuler, et al. 2000; McIntosh, et al. 2005; Strasser, et al. 2005), and genetics (Bousman, et al. 2009; McIntosh, et al. 2009). Diverse brain alterations have been identified in these two patient groups, but with limited replicability (Fornito, et al. 2009) and generally small sample sizes.

Although functional and structural brain studies have identified quantitative alterations in schizophrenia and bipolar disorder, such findings have not yet provided a diagnostic measure that is both sensitive and specific. It is likely in part that the lack of consistent findings is because most models favor only one data type or do not combine data from different imaging modalities effectively, thus missing potentially important differences which are only partially detected by each modality (Calhoun, et al. 2006a). Combining modalities may thus uncover previously hidden relationships that can unify disparate findings in brain imaging.

Existing multimodal studies in these two patient groups have combined EEG with MRI data (McCarley, et al. 2008; McIntosh, et al. 2006); however, to our knowledge no report has combined fMRI and DTI data to investigate both commonalities and differences between SZ and BP; although within each illness, these modality combinations have been examined. Specifically, Schlosser observed a direct correlation in schizophrenia between frontal fractional anisotropy (FA) reduction and fMRI activation in regions in prefrontal and occipital cortices (Schlosser, et al. 2007). Wang identified abnormal perigenual anterior cingulate and amygdala functional connectivity in bipolar disorder when combining fMRI and DTI data (Wang, et al. 2009), suggesting that disrupted white matter connectivity may contribute to disturbances in coordinated functional processing. All of the latter applications were based on univariate analysis in regions of interest. Although these initial studies combined DTI and fMRI data in a rather simplistic fashion, they demonstrated the feasibility and potential of a multimodal fusion approach in addressing various theoretical and applied issues of structure-function relationships (Rykhlevskaia, et al. 2008).

The complexities of SZ and BP make these diseases an ideal test bed for exploration using joint information derived from functional-structural data fusion, although more advanced statistical and analytical models are required to do so. Data from different imaging methods are typically analyzed separately; however such approaches do not enable examination of cross-information between modalities. An alternative approach, called data integration, combines modalities only at the moment of interpreting the result, thus not allowing for any interaction between the data-types (Olesen, et al. 2003; Seghier, et al. 2004), e.g. using seed areas. Such methods based on voxel-wise comparison of structural and functional maps cannot be statistically informative, hence limiting inferences based on these data.

Another strategy based on simultaneous analysis of multimodal data is data fusion, which includes “blind” data-driven approaches such as joint independent component analysis (jICA) (Calhoun, et al. 2006b), multimodal canonical correlation analysis (mCCA) (Correa, et al. 2010a; Correa, et al. 2010c), partial least squares (PLS) (Grady, et al. 2006; Martinez-Montes, et al. 2004), linked ICA (Groves, et al. 2010) and other adaptive approaches such as parallel ICA (Liu, et al. 2009) and coefficients-constrained ICA (Sui, et al. 2009). All of the above are based on linear mixture models and provide complimentary perspectives on data fusion via different optimization assumptions.

According to many previous findings in brain connectivity studies which also combined function and structure (Camara, et al. 2010; Olesen, et al. 2003; Rykhlevskaia, et al. 2008), it is plausible to assume the components decomposed from each modality have some degree of correlation between their mixing profiles among subjects. Therefore, we are motivated to propose a data-driven model that is optimized for this situation and also to have excellent performance for achieving both flexible modal association and source separation. Among the existing joint analysis models, joint ICA maximizes the independence of joint sources of two or more datasets, thus the decomposed source maps are distinct from each other, but it only allows one mixing matrix for all modalities (e.g. modalities are highly correlated, r=1); On the other hand, mCCA allows a different mixing matrix for each modality and links two datasets by maximizing the correlation of the inter-subject mixing profile, but the associated source maps may not differ sufficiently in some cases, especially when the canonical correlation coefficients are close in value (Sui, et al. 2010a). Hence, we are motivated to combine the complementary mCCA and jICA approaches to improve the performance of the joint source extraction. In other words, mCCA first relates two datasets with flexible linkage (correlation), thus giving us more confidence to perform joint ICA on the associated maps, which allows both highly and weakly correlated joint components. Figure 1 depicts four blind data fusion methods, including the above three and our proposed scheme: mCCA+jICA. Note that the proposed mCCA+jICA approach is distinct from our previous work in (Sui, et al. 2010b), where spatial CCA (sCCA) +jICA is used to analyze datasets of the same type, such as multitask fMRI data. The difference between sCCA and mCCA is illustrated in Figure 1 for clarification.

Figure 1
presents a summary of various blind data-driven methods for data fusion. Note that both mCCA and sCCA belong to CCA, but maximize the correlation in different parts of the data. In mCCA+jICA, mCCA automatically links two datasets via correlation between ...

The basic strategy of mCCA+jICA is as follows: mCCA is first adopted to link two or more modalities by correlation maximization of the mixing profiles, see Figure 1. The two mixing matrices are so-called canonical variants (CVs) and their associated maps (Ci, i=1, 2) may still contain mixtures of real independent sources. Next we perform a joint ICA on the concatenated maps [C1, C2] and further decompose the remained mixtures to [S1, S2] by maximizing the joint independence. Hence, mCCA and jICA are complementary to one another and if used together can relax the limitations of each. In our work we extract two mixing matrices (A1, A2), with variable A1–A2 correlation for each component, which provides both shared (highly-correlated) and unique (weakly-correlated) information between modalities. In DTI-fMRI fusion, the structure-function link is reflected by this A1–A2 correlation, which can be seen as a multivariate generalization of published reports which correlate an FA value in one ROI for each subject to all the fMRI contrast voxels (Avants, et al. 2010; Olesen, et al. 2003)”.

In order to reduce the redundancy of the large scale brain data and facilitate the identification of relationships between modalities, each modality is first reduced to a “feature” for each subject, e.g. an fMRI contrast map from the general linear model and DTI measures such as fractional anisotropy (FA). The fMRI contrast map depicts the cortical regions representing the task effect in relation to an experimental baseline. FA refers to the coherence of the orientation of water diffusion, independent of direction. It is calculated as the fraction of total diffusion that can be attributed to anisotropic diffusion, with higher values corresponding to a more consistent diffusion orientation (McIntosh, et al. 2008a). Therefore, a breakdown in white matter integrity results in a lower FA. Our goal was to simultaneously extract the group-discriminating brain activations in fMRI contrasts as well as the abnormal white matter tracts reflected in the FA maps.

Therefore, in this paper we highlighted both similarities and differences between two modalities across three diagnostic groups, which had not yet been attempted for a relatively large number of subjects. Our aim was to better understand the complex distinctions at a brain level between bipolar disorder and schizophrenia. We applied the proposed model to 164 subjects comprising 62 healthy controls (HCs), 54 SZs and 48 BPs, who were recruited at the Olin Neuropsychiatric Research Center, Hartford and were scanned by both fMRI (while performing an auditory oddball task) and DTI. Results were examined from several perspectives. We found not only group-discriminating regions for each modality and each pair-wise groups, but also spatial variations in temporal lobe and different group relationships with regard to age effects on our dependent measures. Most importantly, we are able to visualize an underlying function-structure network by evaluating the joint components which have strong DTI-fMRI links. Note that some of the specific findings we report were not obtainable from separate analysis of each modality and required a joint analysis. The potential benefits of our approach were demonstrated both in a simulation and an application to multimodal human brain data.



We assume that the multimodal dataset Xk is a linear mixture of sources Sk and the nonsingular mixing matrix Ak, k = 1, 2,


where Xk is in form of subjects by voxels, as shown in Figure 2; Ak is in dimension of subjects by number of sources Mk. The columns of A1 and A2 (mixing profile of each modality) have higher correlation only on their corresponding indices,

E{A1TA2}=R1,2diag(r1,r2rM)   M=min(M1,M2)

where r1, r2rM can be either common or different from each other. This assumption is more flexible compared to that of both joint ICA and mCCA. Because joint ICA requires A1 and A2 to be exactly the same; and mCCA achieves complete source separation only when r1, r2rM are sufficiently distinct (Li, et al. 2009a). This constraint is not always easily satisfied, especially when the number of components M are large (e.g. >10) or the number of datasets used in the analysis is small (e.g., two as in this case). Furthermore, due to the potential common correlation values among r1, r2rM, applying individual ICA within each dataset may introduce ambiguity in feature matching via cross-correlation, as we shown in (Sui, et al. 2010b). By contrast, the proposed mCCA+jICA model alleviates these limitations: because mCCA makes the jICA job more reliable by providing a closer initial match via correlation; while jICA further decomposes the remaining mixtures in associated maps and relaxes the constraint that mCCA has on the distinctiveness of r1, r2rM.

Figure 2
Flow chart of the mCCA+jICA analysis process for the clinical fMRI and DTI data fusion

Theory Development

Given that there are N subjects, typically, the number of voxels L in Xk is much larger than N. Due to the high dimensionality and high noise levels in the brain imaging data, order selection is critical to avoid overfitting the data. Using the minimum description length (MDL) criterion as in (Li, et al. 2007), Mk, the number of independent components are estimated for each dataset, Mk < N. Dimension reduction is then performed on Xk using singular value decomposition (SVD), a scheme where small singular values of the matrix are treated as noise/redundancy are discarded, given

XkT=EkUkFk=[EkEk]UkFk,   k=1,2

where Ek in dimension of L × Mk, contain the eigenvectors corresponding to the significant eigenvalues sorted from high to low in Uk; E″ contain the eigenvectors which are treated as noise and hence omitted from the next steps of the analysis. CCA is thus performed on the dimension-reduced datasets Yk in size of N × Mk, given by Yk=XkEk, generating two transformed variants, D1 D2, the so-called canonical variants (CV) (Correa, et al. 2008). D1 and D2 have maximum correlation between each other, while the vectors within each are uncorrelated and whitened. Their correlation vector c1, c2cM is called the canonical correlation coefficient (CCC), given by

E{D1TD2}=E{D2TD1}=diag(c1,c2cM)   M=min(M1,M2)


Based on the linear mixture model, we simultaneously obtain the associated components C1 and C2 via

Xk=Dk·Ck   k=1,2

However, the performance of mCCA for blind source separation (BSS) may suffer when c1, c2cM are very close in values, which might occur frequently in applications using real brain data, since the multimodal connection among components usually are not very high and could be similar in value(Correa, et al. 2010b). Therefore, C1 and C2 will typically be two sets of sources that do not represent a completely decomposed set of components.

Joint ICA is then implemented on the concatenated maps, [C1, C2], to maximize the independence among joint components by reducing their second and higher order statistical dependencies, as in equation (7). ICA as a central tool for BSS has been studied extensively and many algorithms have been developed based on different patterns of cost function (Bell and Sejnowski 1995; Cichocki A, et al. 2007; Hyvarinen 2001). We utilized COMBI (Tichavsky, et al. 2008) in our work due to its flexibility in adapting to different source distributions and its fast speed.


Thus two joint independent components S1 and S2 are achieved, with their corresponding mixing profile linked via correlation. According to (1), (6), and (7), the proposed mCCA+jICA scheme can be summarized as

Xk=(Dk·W1)·Sk,   Ak=Dk·W1

We then correlate A1 with A2, generating a correlation profile, i.e. r1, r2rM in (2), which could contain identical values.

Simulated data

By simulating two types of image sources as two features, we investigate the joint BSS performance of mCCA+jICA on simulated data and compare it to that of joint ICA and mCCA. As shown in Figure 4, eight sources are generated for each feature to simulate images (256×256) and one-dimensional signals (1×2000) respectively, resulting in true sources S1 (in dimension of 8×65536) and S2 (in dimension of 8×2000). Note that the source type in 1D, 2D or 3D does not matter, the relevant factor is the final vector length. Here S1 and S2 are designed on purpose with greatly different vector length (65536 vs. 2000). This has multiple benefits over our previous multitask fusion paper (Sui, et al. 2010b) in which we used spatial CCA instead of multimodal CCA which requires S1 and S2 to have the same length. The mixing matrices of each feature, i.e., A1 and A2 (in dimension of 100×8), have diverse correlations between their corresponding columns, A1A2 correlation = [0.99 0.07 0.36 0.63 0.20 0.23 0.79 0.36], as the ground truth listed in Figure 3. One hundred noisy mixed images Xk are generated for each modality under each of the 11 noisy conditions via Xk = Ik + Nk = Ak Sk + Nk, k=1, 2; where Ik is pure signal mixture and Nk is random Gaussian noise. The corresponding mean peak signal-to-noise ratios (PSNR) are in range of [−1 20] dB. The PSNR is a most commonly used measure of image quality after corruption or recovery, which is defined as (9) for the jth mixture of feature k at every noisy condition. Typical PSNR value for the acceptable image quality is about 30 dB; the lower the value, the more degraded the image (Thomos, et al. 2006).

PSNR(k,j)=10log10[maxi[1,l](Ik(i))21li=1l|Xk(i)Ik(i)|2]   j=1,2100,l={65536,k=12000,  k=2

Figure 3
illustrates the whole simulation results for 3 factors and in 2 ways. The 3 factors are source estimation (Ŝ), mixing matrix estimation (Â) and modal linkage shown by correlation between mixing coefficients of each modality(corr(A1(:, ...
Figure 4
simulation results of comparing separation performance of 3 methods 8 sources for each simulated modality: S1 (left) and S2 (right) are designed on purpose with greatly different final vector length (65536 vs. 2000, implying multimodal data); 100 mixtures ...

Three joint BSS models: jICA, mCCA and mCCA+jICA were implemented on simulated datasets respectively under every PSNR for 10 runs. The decomposed components are paired with the true sources via cross-correlation automatically within each feature. We adopted three metrics to estimate the joint BSS performance:

  1. the average correlation of the estimated components Ŝ with true sources S;
  2. the average correlation of the estimated mixing profiles  with the ground truth A;
  3. the mean square error of the estimated A1A2 correlation compared to the truth.

For each metric, we compared the three algorithms in two ways, i.e., under different noise conditions and at different source indices.

Human brain data

Participants 62 healthy controls (HC, age 38±17, 30 females), 54 patients with schizophrenia (SZ, age 37±12, 22 females) and 48 patients with bipolar disorder (BP, age 37±14, 26 females) were recruited at the Olin Neuropsychiatric Research Center and were scanned by both fMRI and DTI. All subjects gave written, informed, Hartford hospital IRB approved consent. Schizophrenia or bipolar disorder was diagnosed according to DSM-IV-TR criteria in the on the basis of a structured clinical interview (First 1995) administered by a research nurse and review of the medical file. All patients were stabilized on medication prior to the scan session in this study. Healthy participants were screened to ensure they were free from DSM-IV Axis I or Axis II psychopathology (assessed using the SCID (Spitzer 1996)) and also interviewed to determine that there was no history of psychosis in any first-degree relatives). All subjects were urine-screened to eliminate those who were positive for abused substances. Patients and controls were age and gender matched, with no significant differences among 3 groups, where age: p=0.93, F=0.07, DF=2. Sex: p=0.99, χ2 =0.017, DF=2. All participants had normal hearing, and were able to perform the oddball task successfully during practice prior to the scanning session.

The Auditory oddball task involved subjects encountering three frequencies of sounds: target (1200 Hz with probability, p=0.09), novel (computer generated complex tones, p = 0.09), and standard (1000 Hz, p=0.82) presented through a computer system via sound insulated, MR-compatible earphones. Stimuli were presented sequentially in pseudorandom order for 200 ms each with inter-stimulus interval (ISI) varying randomly from 500 to 2050 ms. Subjects were asked to make a quick button-press response with their right index finger upon each presentation of each target stimulus; no response was required for the other two stimuli. Two runs of 244 stimuli were presented (Kiehl, et al. 2001).

Imaging parameters Scans were acquired at the Institute of Living, Hartford, CT on a 3T dedicated head scanner (Siemens Allegra) equipped with 40mT/m gradients and a standard quadrature head coil. The functional scans were acquired using gradient-echo echo planar imaging (EPI) with the following parameters: repeat time (TR) = 1.5 sec, echo time (TE) = 27 ms, field of view = 24 cm, acquisition matrix = 64×64, flip angle = 70°, voxel size = 3.75×3.75×4 mm3, slice thickness = 4mm, gap = 1 mm, number of slices = 29; ascending acquisition. Six dummy scans were carried out at the beginning to allow for longitudinal equilibrium, after which the paradigm was automatically triggered to start by the scanner. DTI images were acquired via a single-shot spin-echo echo planar imaging (EPI) with a twice-refocused balance echo sequence to reduce eddy current distortions, TR/TE= 5900/83 ms, FOV= 20 cm, acquisition matrix=128×96, reconstruction matrix 128×128, 8 averages, b=0, and 1000 s/mm2 along 12 non-collinear directions, 45 contiguous axial slices with 3 mm slice thickness.

FMRI preprocessing fMRI data were preprocessed using the software package SPM5 (, et al. 2005). Images were realigned using INRIalign, a motion correction algorithm unbiased by local signal changes (Freire, et al. 2002). Data were spatially normalized into the standard MNI space (Friston, et al. 1995), smoothed with a 12 mm3 full width at half-maximum Gaussian kernel. The data, originally 3.75×3.75×4 mm, were slightly subsampled to 3×3×3 mm, resulting in 53×63×46 voxels.

FMRI Feature Extraction a GLM approach using SPM5 was used to find task-associated brain regions, labeled as contrast maps which were then used as fMRI features within our mCCA+jICA analysis. Specifically, a GLM analysis consisted of a univariate multiple regression of each voxel's time course with an experimental design matrix, generated by the convolution of the task onset times with a hemodynamic response function. This resulted in a set of beta-weight maps associated with each parametric regressor. The subtraction of target beta-weight map with the standard beta weight map was referred to as a “target contrast” map, which represented the task effect in relation to an experimental baseline. We utilized the average target effect for the AOD task.

DTI preprocessing consists of the following steps:

  1. DTI Quality Check. The DTI data quality is checked for a) signal dropout due to subject motion, producing striated artifacts on images; b) excessive background noise in the phase encoding direction, due to external RF leakage in the MRI scan room or to subject motion; and c) large amounts of motion in the absence of signal dropout. If for a specific gradient direction any slice was found to have a problem, we decided to exclude the whole volume rather than some specific slices. Our assumption was that the presence of a large head movement in one slice compromised the entire volume. This resulted in different number of gradients being used for different subjects. The average number of gradient directions dropped was 12% for healthy controls and 14% for patients with schizophrenia and bipolar disorder (being similar for the two patient groups). FA may tend to increase as number of gradient directions is reduced, but the bias is negligible if the reduction in the number of gradient directions is no more than 15%, as reported in (Ling, et al. 2010), which examined this problem via simulations. In our study, the motion artifacts were not significantly larger in patients group.
  2. Motion and eddy current correction. After the above data pruning we had one 4D DTI volume and a table of corresponding b-values and gradient direction vectors. Next we registered all the images to a b=0 s/mm2 image. Twelve degrees of freedom, affine transformation with mutual information cost function was used for image registration.
  3. Adjusting the diffusion gradient direction. Two corrections were applied to the diffusion gradients. The nominal diffusion gradients directions are prescribed in the magnet axis frame. We rotated them to correspond to the image slice orientation. No correction is required if the imaging slice is pure axial. A second correction accounted for any image rotation during the previous motion and eddy current correction step. The rotation part of the transformation found previously was extracted, and each gradient direction vector corrected accordingly. All the image registration and transformations were done with the FLIRT (FMRIB’s Linear Image Registration Tool) program (FMRIB Software Library [FSL];, and the detection of outliers and data pruning was done with a custom program written in IDL (

DTI Feature Extraction. We used Dtifit, a tool in FSL to calculate the diffusion tensor and the fractional anisotropy (FA) maps. The FA image was aligned to a MNI FA template with a nonlinear registration algorithm FNIRT (FMRIB’s Nonlinear Image Registration Tool; FSL) and was co-registered with fMRI contrast via SPM, resulting in a final 53×63×46 matrix with the voxel size of 3×3×3 mm. The FA maps were then smoothed using SPM5 with a 12 mm3 full width at half-maximum Gaussian kernel (Silver, et al. 2011).

Analysis Pipeline After feature extraction, the 3D image of each subject was reshaped into a one-dimensional non-zero vector and stacked one by one, forming a matrix with dimensions of 164×[number of voxels] for each modality. Then the feature matrix was normalized to have the same average sum-of-squares (computed across all subjects and all voxels for each modality). The normalization was needed because the AOD and FA data had different ranges. A single normalization factor was used for each data type; thus, following normalization, the relative scaling within a given data type was preserved, but the units between data types were the same (in a least-squares sense). After normalization, the data was processed via the pipeline shown in Figure 2, i.e., dimension reduction-> multimodal CCA-> joint ICA-> component analysis.



Figure 3 illustrates the simulation results for three evaluation metrics on the whole, displayed in three rows. For each metric, we compared three algorithms in different noisy conditions (PSNRs, left column) and source distributions (source index, right column). Under each noise condition (PSNR), we illustrate the averaged estimation accuracy on sources, mixing profiles and the modal linkage respectively in figure (a), (c) and (e). It is evident that mCCA+jICA is quite robust to noise, and its BSS performance is consistently the best in all noise conditions. Consequently, joint ICA is the second best in source estimation and mCCA is the second best in mixing profile estimation; mCCA+jICA also overperforms mCCA on estimation of modal connection and their estimation errors were not influenced much by the noise. Note that when PSNR=−1dB, i.e., noise is bigger than signal, all three methods can still have the estimation accuracy of Ŝ and  higher than 0.55.

For each specific joint source, we plotted the mean correlation and its standard derivation across all noisy conditions via bars in figure 3 (b) and (d). In figure 3 (f), the true A1A2 correlation is given via a yellow bar for every source, while the mean square error and its standard derivation of the link estimation were plotted in red for mCCA and in green for mCCA+jICA. Note that both very high (0.99) and low (0.07) correlation values exist in modal connection, representing both shared and distinct parts of two features. Some sources have very close (5 and 6, r =0.20, 0.23) or the same (3 and 8, r=0.36) low correlation values, which is quite ordinary in real applications of brain data fusion and mCCA+jICA consistently performs better.

We next focus on one noisy case (PSNR=6dB) in order to directly investigate the joint BSS performance in this context. Results are presented in Figure 4, where true sources and true modal connection are shown on the left. Joint ICA separates almost all sources accurately especially for sources 1, 4, and 7 since their A1A2 correlation >0.6, but failed to decompose the 3rd source for feature 2 whose A1A2 correlation is lower (r=0.36). Multimodal CCA can track the modal connection more precisely than jICA, but cannot completely decompose image sources in feature 1, particularly sources 3–6. The proposed mCCA+jICA combine advantages of both methods and improve the performance substantially. It succeeds in separating sources and linking them correctly in a less-constrained condition, where there is no stringent requirement for the A1A2 correlation.

Human brain data

Next, mCCA+jICA was applied to real fMRI and DTI data collected from 164 subjects including three groups. Our goal was to identify the aberrant functional-structural brain relationships in schizophrenia and bipolar disorder. Ten components were estimated for each feature according to an improved MDL criterion (Li, et al. 2007). We summarize the results in the following four aspects:

1) Group differences in loading parameters

Two sample t-tests were performed for each IC on its mixing coefficients between each of the possible pair-wised groups, as shown in Figure 5, where the components with significant group differences were summarized and the PT means both SZ and BP. For each component, we chose 4 representative slices to display and use different color frames for discrimination. If two independent components have the same frame color in both modalities they are joint components. The p values displayed with yellow text passed a Bonferroni correction for multiple comparison (p<0.005), others passed an uncorrected significance level (p<0.05).

Figure 5
Summary of components with significant group differences in four pair-wise group combinations. Two ICs with the same frame color in two modalities represent joint ICs. Those p values displayed with yellow text pass the Bonferroni correction for multiple ...

The specific identified regions of the above group-discriminating components and their abbreviations are summarized in Table 1 for fMRI (Talairach labels) and Table 2 for DTI (white matter tracts) respectively, where the ICs with # were joint group-discriminating components, i.e., it can differentiate groups in both AOD task and FA. The ICs with * had highly significant p values, which passed a Bonferroni correction for multiple comparison. To summarize the white matter results, we used the Johns Hopkins white-matter tractography atlas (from FSL), from which 20 structures were identified (Hua, et al. 2008); mostly these are large bundles. In Table 2, the WM tract labels, the identified volume (cc) and the percentage that indicates the overlap of the identified regions in each WM tract are listed in detail.

Table 1
MNI Table of group-discriminating fMRI components (R/L)
Table 2
The white matter tract label of the group-discriminating FA components

2) Group differences in spatial maps

The back-reconstruction of spatial maps for each group was performed based on the linear mixture model as in (10),


where Sk denotes the IC maps derived by mCCA+jICA for all subjects; k represents the modality, [·]−1 represents the pseudo-inverse of the matrix; SHC,k, SSZ,k and SBP,k are the back-reconstructed spatial maps for HC, SZ and BP respectively. To investigate the group variation in spatial maps, we correlated SHC,k/SSZ,k/SBP,k with Sk individually for both modalities. The FA maps did not show large differences among 3 groups with relatively higher correlations (r>0.8); while the fMRI maps exhibited more variations. We listed all correlation values between the reconstructed group maps with the IC maps for fMRI in Table 3. For display, we chose the IC that had correlation values higher than 0.5 for all groups (to make sure that the spatial maps of all groups were consistent to some degree) and controls were obviously different from patients in correlation values (±0.1). Hence IC 9 was selected based on the above criteria, which shows strong activation in both superior temporal gyrus and motor cortex.

Table 3
Similarity between the reconstructed group-maps with the fMRI IC map

Figure 6 displays the variation of functional spatial maps among groups. IC9 derived from mCCA+jICA for all groups is shown on the left. On the right, the back-reconstructed group sources SBP, SSZ, SHC and Sg common (the common activation of SBP, SSZ and SHC) are overlaid one by one from bottom to top. For display, the components are converted to Z-values (by dividing by the standard deviation of the source) and thresholded at |Z|>2.

Figure 6
Variation of functional spatial maps (AOD_IC9) among groups. Left: The AOD_IC9 derived from mCCA+jICA. Right: the back-reconstructed group sources SBP, SSZ and Sg common (the common activation of SBP, SSZ and SHC) are overlaid one by one from bottom to ...

3) Group differences in age effects

The derived mixing coefficients also provide a way to investigate the relationship between the identified regions and subjects’ behavioral data, such as age. We correlated participants’ ages with the mixing profile for each IC and three were found to be significant: AOD_IC1, FA_IC1 and FA_IC9. The correlation values (r) and significance level (p) were listed in Table 4 for all subjects and each group. We also computed the 95% BCa (Bias corrected and accelerated percentile method) bootstrap confidence interval(ci) for all the correlations by using 100 bootstrapping samples for each group. Multiple comparisons for 40 correlations were performed using FDR correction and the significant p values were marked as bold in Table 4. Figure 7 demonstrated the scatter plot of age versus ICA weights, and the group linear trends in different colors and markers for AOD_IC1 and FA_IC9; specifically, HC trend in red line, SZ trend in blue line, BP trend in green line and trend of all subjects in black line. For FA_IC1, all groups showed significant anti-correlation between age and FA weights, thus were plotted using one marker.

Figure 7
demonstrates the scatter plots and linear trends between subjects’ age and loading parameters. Specifically, HC in red line, SZ in blue line, BP in green line and trend of all subjects in black line. For FA_IC1, All groups have very significant ...
Table 4
Correlation with ages and the corresponding p values and confidence intervals (ci)

4) Functional-structural network

We calculated the A1–A2 correlations for all components, as listed in Table 5, which provide a measure that describes the modal linkage, i.e., the functional-structural connection. Note that most of the correlation values are quite significant with p<=0.005, except for IC4, which was a ventricular CSF artifact. We choose 3 representative ICs for display, whose A1–A2 correlations are moderate to high. Among these, IC 8 had the highest A1–A2 correlation (r=0.37, p=1e-6) and was the only component showing SZ versus BP differences in their loadings. FA_IC7 shows the most group-discriminative loadings and FA_IC9 shows a significant correlation with age. In addition, both AOD_IC7 (default mode network) and AOD_IC9 (temporal and motor activation) include brain regions that are likely activated by the auditory oddball task.

Table 5
The functional-structural mixing profile correlation

Figure 8 shows the AOD component maps on the left, the FA component in the middle and a high-level functional-structural network diagram on the right. Our goal is to evaluate the intersection of the results with 1) for FA, known tracts, and 2) for fMRI, known gray matter regions, and to identify which known tracts are both intersected by the regions of FA changes and touch the regional fMRI changes. The functional lobes with large volume activations are bordered by red solid frames; by contrast, the lobe with small volume activations is bordered dotted blue line. Only the FA fiber tracts with more than 1 cm3 volume identified (R+L, positive+negative) were displayed. The specific Talairach and WM labels of the three joint components are listed in table 6 for reference.

Figure 8
This figure attempts to verify that the “linked”(joint) components do correspond to FA changes in known tracts and functional changes in distant regions connected to that tract. Note that we did not perform actual fiber tractography but ...
Table 6
Validation of linkage----labels of three joint components


In this paper, we highlighted patterns of both similarities and differences in two modalities (fMRI and DTI) across three diagnostic groups (HC, SZ, BP), which have not been attempted previously for a relatively large number of subjects. Our aim was to clarify differences between bipolar disorder and schizophrenia using functional-structural data fusion employing an advanced analytical model. We propose a multimodal fusion method called “mCCA+jICA”, which can work with less stringent assumptions than the previously proposed approaches. It can also achieve higher decomposition accuracy and identify valid links between features. Note that the mCCA+jICA approach does not increase the computational load appreciably; however it achieves the best performance in simulations designed to be similar to real-world brain imaging data fusion.

One of the major strengths of this model is that the combination of mCCA and joint ICA improves the performance of joint BSS substantially under a less-constrained condition. Specifically, in Figure 3 and and4,4, sources 1, 4 and 7 have higher A1A2 correlation values, thus joint ICA works well, in accordance with its hypothesis; consequently, the performance of mCCA suffers from ambiguity and misinterpretation in sources 3&8, 5&6 due to the requirement of sufficiently distinct canonical correlations; by contrast, the proposed model mitigates the performance deficits of mCCA by using a further ICA decomposition. In addition, the use of mCCA makes the job of jICA easier and more reliable by providing a closer initial match. We can also define links among data via the A1–A2 correlation values. As shown in Figure 3 (f), for sources with lower or closer A1–A2 correlations (sources 2,3,6,8), which could be the practical cases in brain imaging data fusion, mCCA+jICA consistently performs better, and has no stringent requirements imposed on the A1–A2 correlation.

Based on the merits of the proposed model, we applied it to a real brain data set, with 3 groups and 2 modalities. There are at least three major advantages to combining DTI and fMRI data:

  1. FMRI as a well-established neuroimaging technology, can act as a reference framework verifying the validity of conclusions derived from the relatively newer DTI method (Reinges, et al. 2004).
  2. Such a combination may provide insights into the relationships/connectivity between brain structure and function (Guye, et al. 2008; Skudlarski, et al. 2010).
  3. Combining structural and functional brain imaging data allows for constructing more informative, biologically plausible models and may have utility for neurosurgical applications.

Therefore, the large dataset and the novel method together demonstrated the potential to detect relevant brain differences between bipolar disorder and schizophrenia.

Amplitude difference

Three joint components (IC 1, 2, 8) and three modality-specific components (AOD_IC6, FA_IC7 and FA_IC9) were identified as group-discriminating ICs, as shown in Figure 5. We next discuss the findings for each of these in detail.

AOD_IC2 is the most significant functional component discriminating both SZ and BP from HC, which includes dorsolateral prefrontal cortex (DLPFC), thalamus and occipital cortex. DLPFC plays an important role in the integration of sensory and mnemonic information, executive function, planning and regulation of cognitive function and action. Researchers have frequently reported dysfunction and lack of functional connectivity of this region in patients with schizophrenia (Badcock, et al. 2005; Hamilton, et al. 2009) and bipolar disorder (Curtis, et al. 2001; Glahn, et al. 2010). The thalamus has multiple functions and is believed to act as a relay station between many subcortical areas and the cerebral cortex. It was also reported to be impaired in both SZ and BP (Cerullo, et al. 2009; Danos 2004). Our results are consistent with the above findings and suggest that these deficits might be related to shared risk factors and disease mechanisms common to both disorders. The joint FA_IC2 indicates an FA reduction in BP versus HC in the anterior thalamic radiation (ATR) that projects from anterior and medial regions of the thalamus to frontal lobe, in agreement with (Anand, et al. 2009).

FA_IC7 is the most significant DTI component where the identified WM fiber tracts mostly connect frontal and occipital lobes, including ATR, inferior fronto-occipital fasciculus (IFO), cingulum and superior longitudinal fasciculus (SLF). This finding is consistent with several previous reports of DTI abnormalities in schizophrenia (Kubicki, et al. 2005; Kubicki, et al. 2003; Schlosser, et al. 2007) and bipolar disorder (Wang, et al. 2009; Yurgelun-Todd, et al. 2007), suggesting that disruptions in white matter connectivity may contribute to coordinated brain dysfunction, especially in the frontal lobe, which is frequently thought of as “disconnected” from other brain regions in both disorders (Heng, et al. 2010; Williams, et al. 2004).

AOD_IC8 and FA_IC8 are joint components that can distinguish two patient groups, though not at a high level of significance (p=0.04). In functional maps, the parahippocampal gyrus (PH), medial frontal cortex, thalamus and visual cortex are emphasized, confirming the results in (Hall, et al. 2009), where two brain disorders were separated by differences in hippocampal and prefrontal cortical activation. Moreover, visual cortex impairments in SZ versus HC are also observed, consistent with (Brenner, et al. 2009; Demirci, et al. 2009). In FA maps, fibers associated with occipital and frontal lobes were identified, connecting the identified regions in AOD_IC8 in an organized network (see Figure 8), especially SLF, forceps minor (FMIN, that connects lateral and medial frontal lobes via the genu of the corpus callosum), and forceps major (FMAJ, that connects the occipital lobes via the splenium of the corpus callosum). These patterns of white matter pathology offer a potential diagnostic distinction between the two disorders. Furthermore, a similar function–structure relationship in the visual system was reported in healthy controls by (Toosy, et al. 2004), where the mean FA of the optic radiations correlated significantly with fMRI measures of visual cortex activity. Together with our results, these findings support the hypothesis that cortical fMRI response may be constrained by the external anatomical connections of white matter fibers.

Spatial variation

The back-reconstructed components present a direct view of the spatial distribution for each group (which can vary) and the intensities of the image (Z scores) provide a relative strength of the degree to which the group source contributes to the overall data (Beckmann and Smith 2005). If SHC,k, SSZ,k and SBP,k are more similar to each other, they are likely to have more overlaps with Sk; if there are deviations, it likely means spatial group variability exist in such regions. In practice, it is hard to determine exactly to what extent the deviations between group sources can occur, however, simulations tell us that general back-reconstruction is quite capable of handling variations between the groups in the regions which contribute to the extracted source, an important feature given that we don’t expect the groups to have the same pattern. (Calhoun, et al. 2001; Erhardt, et al. 2010)

As shown in Figure 6, all three groups show higher consistence in parietal lobe (BA 7 19 39 40), primary and supplementary motor cortex (BA 3 4 6), and middle frontal gyrus (BA 10 32 46), implying that the spatial distribution of the identified regions implicated in movement and sensory integration is similar for all subjects (Andersen and Buneo 2002).

On the other hand, more variations were observed in temporal gyrus (BA 21 22 41 42) which is responsible for processing of auditory information, suggesting that aberrant patterns of coherence in temporal lobe may be a cardinal abnormality in both schizophrenia and bipolar disorder (Calhoun, et al. 2008; Chance, et al. 2008; Pearlson 1997). SZ activated more in superior and middle temporal gyri, supported by (Calhoun, et al. 2004). BP exhibits additional activations in insula, which has a role in emotional regulation, in accordance with findings in (Kempton, et al. 2009; McIntosh, et al. 2008c; Pessoa and Adolphs 2011), where insula also differentiates BP from HC and SZ. We also observed spatial deviations occurred between individual group map and IC map. For example, BP alone shows clear bilateral thalamic activation, while the left panel shows only the left thalamus for all subjects, suggesting that there may be diminished prefrontal modulation of subcortical and medial temporal structures within the anterior limbic network (e.g., amygdala and thalamus) that results in mood dysregulation, such as bipolar illness (Blumberg, et al. 2003; Strakowski, et al. 2005).

Age effects

Three ICs show significant correlation between subjects’ age and loadings. AOD_IC1 presents activations mainly in motor cortex (BA 1–6), accompanied by a functional asymmetry with left dominance, see Figure 7(a), consistent with the fact that AOD task needs participants push the button with their right fingers. Controls have a very significant correlation r = 0.5, p=4e-5, while patient groups do not (p does not pass multiple comparison), implying that the motor regions of HC are normally more involved in the task with age increased (Bennett, et al. 2010), whereas patients have no such trend due to motor system deficits (Rogowska, et al. 2004). FA_IC1, as a joint component of AOD_IC1, also demonstrates significant (p=2e-08), but anti-correlation with age. Note that all groups have low p values, suggesting all subjects have a general age-related decrease in FA values, representing a breakdown in white matter integrity, as shown in Figure 7 (b) and Table 4. This finding is in agreement with multiple prior reports from DTI aging research (Grieve, et al. 2007; Sullivan and Pfefferbaum 2003). Furthermore, as illustrated in Figure 7(c) and Table 4, SLF, corticospinal tract (CST), ATR and uncinate fasciculus (UF) are indentified in FA_IC9; interestingly, both patient groups revealed negative correlations with age r<−0.4 while HC do not, suggesting that white matter density in these tracts decreases faster in patients, especially for schizophrenia. Similar abnormalities in SZ were reported by (McIntosh, et al. 2008b).

Function-structure network

Imaging structural and functional brain connectivity has the potential to reveal the complex network of brain organization, which may reveal the pathway of information segregation and integration, and may also underlie the clinical consequences of alterations in mental illnesses. In Figure 8 and Table 6, we selected 3 joint components for illustration, attempting to verify that the “linked” components do indeed correspond to FA changes in known tracts and functional changes in distant regions connected to that tract. Note that we did not perform the WM fiber tractography but provided a type of summary statistic. FA and fMRI are showing different things and a strength of our method is that it can detect complicated FA/fMRI relationships without requiring a detected direct link.

The anterior thalamic radiation is identified in all three FA components (IC7,8,9) in Figure 8, constituting a potential anatomical biomarker for these two mental illnesses. ATR carries association fibers from anterior and medial regions of the thalamus to prefrontal cortex. This pathway is a part of several functionally segregated thalamofronto-striatal loops(Alexander, et al. 1990) and implicated in the pathophysiology of both BP and SZ (Sussmann, et al. 2009). These findings are also supported by neuro-pathological studies (Lewis 2000) showing evidence of decreased thalamic projection neurons and prefrontal thalamic inputs.

AOD_IC7 demonstrates negative activations in the typical default mode network (Raichle, et al. 2001) and positive activations in STG and IPL, which are potentially related by the tracts shown in FA_IC7. In FA_IC7, except ATR, IFO passes backward from frontal lobe into occipital and lateral temporal lobes; CST originates from the motor cortex to lower motor neurons in the spinal cord, cingulum projects from the cingulate gyrus to the entorhinal cortex(an important memory-related area in brain, that is a major input to the hippocampus) and SLF connects all brain lobes with each other.

AOD_IC9 reflects main fMRI activations in temporal and frontal lobe, consequently, FA_IC9 localize the breakdown of WM integrity in UF (carrying association fibers between the medial prefrontal cortex and the anterior temporal lobe, including amygdala) and ATR for patients, more so for bipolar. Our results support the DTI findings in (McIntosh, et al. 2008a) and (Sussmann, et al. 2009), where patients with SZ and BP show consistent reduced integrity in the UF and ATR, implying a considerable overlap in white matter pathology, possibly relating to risk factors common to both disorders.

Future extension

We would like to highlight additional aspects of our study. Besides the application to fMRI-DTI data fusion, mCCA+jICA can be used to analyze various combinations of multimodal data too, such as fMRI with EEG/MEG, or genetic data. Furthermore, this approach is not limited to two-way fusion, but can potentially be extended to 3-way or N-way fusion of multiple data types by replacing the multimodal CCA with multi-set CCA (Li, et al. 2009b); e.g., combining fMRI, sMRI and genetic data to construct a broad function-structure-genetics network, thus arguing for widespread utilization of the proposed method.

Another interesting future direction is to further examine the subgroup differences within patients, e.g., BPs with/without psychotic features, by applying the proposed model, which might optimally separate a larger-sample size of BPs, though in current case we did not find very significant differences between these two subgroups.

A possible limitation is that mCCA+jICA works on extracted features of the original imaging data. In our case, a “feature” is a distilled dataset representing the interesting part of each modality and tends to be more tractable than working with the large-scale original data due to the reduced dimension, e.g. 4D fMRI data (Calhoun and Adali 2009). Hence the main reason to use features is to provide a simpler space in which to link the data. The trade-off is that some information may be lost. However there is considerable evidence that the use of features is quite useful. For example, for task-related fMRI data, the contrast maps depict the cortical regions representing the task effect in relation to an experimental baseline, which already utilize the temporal information. In a recent PNAS paper (Smith, et al. 2009), which used ICA on rest data and on task blobs (in the case of task blobs, even more information was lost than we used) and they found nearly identical resting fMRI patterns from the task blobs, motivating the use of features.

Our approach is also suitable for resting-state fMRI, we can perform an ICA on the original 4D data and then use the ICA components as input. This has been done for default mode in the joint ICA context in multiple papers including (Franco, et al. 2008). Similar work has also been done by (Smith, et al. 2009).

In conclusion, studies featuring combination of DTI and fMRI information prove to be fruitful for a more informative understanding of bipolar disorder and schizophrenia. In this paper, we developed an mCCA+jICA model that takes advantage of two multivariate approaches and enables more flexibility in statistical assumptions, promising for wider applications in neuroimaging community. Our application highlighted data from two illnesses and two modalities, and was analyzed carefully to uncover multiple types of group differences in the brain, including amplitude, spatial distribution, age effects and functional-structural relationships. We identified several group-discriminating aspects that replicated previous reports, implying that while SZ and BP show differences in regions including parahippocampal gyrus and visual cortex, they also share several common abnormalities in frontal brain mechanisms and in prefrontal thalamic white matter tracts. Such observations add to our understanding of the neural correlates of both disorders and may have utility as potential brain illness biomarkers.

Supplementary Material

Supplementary material


This work was supported by the National Institutes of Health grants R01MH072681-01 (to Kiehl, K.A.) R01EB 006841 and R01EB 005846 (to Calhoun VD), DOE grant DE-FG02-08ER64581 (to Sui J) and R01MH43775, R01MH074797 and R01MH077945 (to Pearlson GD). We thank the research staff at the Olin Neuropsychiatric Research Center who helped collect and check the data. We also appreciate the advices given by Yi-Ou Li at UCSF and Nicolle Correa at UMBC.


  • Alexander GE, Crutcher MD, DeLong MR. Basal ganglia-thalamocortical circuits: parallel substrates for motor, oculomotor, "prefrontal" and "limbic" functions. Prog Brain Res. 1990;85:119–146. [PubMed]
  • Altshuler LL, Bartzokis G, Grieder T, Curran J, Jimenez T, Leight K, Wilkins J, Gerner R, Mintz J. An MRI study of temporal lobe structures in men with bipolar disorder or schizophrenia. Biol Psychiatry. 2000;48(2):147–162. [PubMed]
  • Anand A, Li Y, Wang Y, Lowe MJ, Dzemidzic M. Resting state corticolimbic connectivity abnormalities in unmedicated bipolar disorder and unipolar depression. Psychiatry Res. 2009;171(3):189–198. [PMC free article] [PubMed]
  • Andersen RA, Buneo CA. Intentional maps in posterior parietal cortex. Annu Rev Neurosci. 2002;25:189–220. [PubMed]
  • Avants BB, Cook PA, Ungar L, Gee JC, Grossman M. Dementia induces correlated reductions in white matter integrity and cortical thickness: a multivariate neuroimaging study with sparse canonical correlation analysis. Neuroimage. 2010;50(3):1004–1016. [PMC free article] [PubMed]
  • Badcock JC, Michiel PT, Rock D. Spatial working memory and planning ability: contrasts between schizophrenia and bipolar I disorder. Cortex. 2005;41(6):753–763. [PubMed]
  • Bahn S. Gene expression in bipolar disorder and schizophrenia: new approaches to old problems. Bipolar Disord. 2002;4 Suppl 1:70–72. [PubMed]
  • Beckmann CF, Smith SM. Tensorial extensions of independent component analysis for multisubject FMRI analysis. Neuroimage. 2005;25(1):294–311. [PubMed]
  • Bell AJ, Sejnowski TJ. An information-maximization approach to blind separation and blind deconvolution. Neural Comput. 1995;7(6):1129–1159. [PubMed]
  • Bennett IJ, Madden DJ, Vaidya CJ, Howard DV, Howard JH. Age-related differences in multiple measures of white matter integrity: A diffusion tensor imaging study of healthy aging. Human Brain Mapping. 2010;31(3):378–390. [PMC free article] [PubMed]
  • Blumberg HP, Martin A, Kaufman J, Leung HC, Skudlarski P, Lacadie C, Fulbright RK, Gore JC, Charney DS, Krystal JH, et al. Frontostriatal abnormalities in adolescents with bipolar disorder: preliminary observations from functional MRI. Am J Psychiatry. 2003;160(7):1345–1347. [PubMed]
  • Bousman CA, Chana G, Glatt SJ, Chandler SD, Lucero GR, Tatro E, May T, Lohr JB, Kremen WS, Tsuang MT, et al. Preliminary evidence of ubiquitin proteasome system dysregulation in schizophrenia and bipolar disorder: Convergent pathway analysis findings from two independent samples. Am J Med Genet B Neuropsychiatr Genet. 2009 [PMC free article] [PubMed]
  • Brenner CA, Krishnan GP, Vohs JL, Ahn WY, Hetrick WP, Morzorati SL, O'Donnell BF. Steady state responses: electrophysiological assessment of sensory function in schizophrenia. Schizophr Bull. 2009;35(6):1065–1077. [PMC free article] [PubMed]
  • Calhoun VD, Adali T. Feature-based fusion of medical imaging data. IEEE Trans Inf Technol Biomed. 2009;13(5):711–720. [PMC free article] [PubMed]
  • Calhoun VD, Adali T, Giuliani NR, Pekar JJ, Kiehl KA, Pearlson GD. Method for multimodal analysis of independent source differences in schizophrenia: combining gray matter structural and auditory oddball functional data. Hum Brain Mapp. 2006a;27(1):47–62. [PubMed]
  • Calhoun VD, Adali T, Kiehl KA, Astur R, Pekar JJ, Pearlson GD. A method for multitask fMRI data fusion applied to schizophrenia. Hum Brain Mapp. 2006b;27(7):598–610. [PMC free article] [PubMed]
  • Calhoun VD, Adali T, Pearlson GD, Pekar JJ. A method for making group inferences from functional MRI data using independent component analysis. Hum Brain Mapp. 2001;14(3):140–151. [PubMed]
  • Calhoun VD, Kiehl KA, Liddle PF, Pearlson GD. Aberrant localization of synchronous hemodynamic activity in auditory cortex reliably characterizes schizophrenia. Biol Psychiatry. 2004;55(8):842–849. [PMC free article] [PubMed]
  • Calhoun VD, Maciejewski PK, Pearlson GD, Kiehl KA. Temporal lobe and "default" hemodynamic brain modes discriminate between schizophrenia and bipolar disorder. Hum Brain Mapp. 2007 [PMC free article] [PubMed]
  • Calhoun VD, Maciejewski PK, Pearlson GD, Kiehl KA. Temporal lobe and "default" hemodynamic brain modes discriminate between schizophrenia and bipolar disorder. Hum Brain Mapp. 2008;29(11):1265–1275. [PMC free article] [PubMed]
  • Camara E, Rodriguez-Fornells A, Munte TF. Microstructural brain differences predict functional hemodynamic responses in a reward processing task. J Neurosci. 2010;30(34):11398–11402. [PubMed]
  • Cerullo MA, Adler CM, Delbello MP, Strakowski SM. The functional neuroanatomy of bipolar disorder. Int Rev Psychiatry. 2009;21(4):314–322. [PubMed]
  • Chance SA, Casanova MF, Switala AE, Crow TJ. Auditory cortex asymmetry, altered minicolumn spacing and absence of ageing effects in schizophrenia. Brain. 2008;131(Pt 12):3178–3192. [PMC free article] [PubMed]
  • Cichocki A, Amari SSK, Tanaka T, AH P. ICALAB Toolboxes. 2007
  • Correa N, Adali T, Li YO, Calhoun VD. Canonical correlation analysis for fusion and analysis of multi-modal medical imaging data. IEEE Signal Proc. Magazine. 2010a In Press.
  • Correa NM, Adali T, Li YO, Calhoun VD. Canonical Correlation Analysis for Data Fusion and Group Inferences: Examining applications of medical imaging data. IEEE Signal Process Mag. 2010b;27(4):39–50. [PMC free article] [PubMed]
  • Correa NM, Eichele T, Adali T, Li YO, Calhoun VD. Multi-set canonical correlation analysis for the fusion of concurrent single trial ERP and functional MRI. Neuroimage. 2010c [PMC free article] [PubMed]
  • Correa NM, Li YO, Adali T, Calhoun VD. Canonical Correlation Analysis for Feature-Based Fusion of Biomedical Imaging Modalities and Its Application to Detection of Associative Networks in Schizophrenia. IEEE J Sel Top Signal Process. 2008;2(6):998–1007. [PMC free article] [PubMed]
  • Curtis VA, Dixon TA, Morris RG, Bullmore ET, Brammer MJ, Williams SC, Sharma T, Murray RM, McGuire PK. Differential frontal activation in schizophrenia and bipolar illness during verbal fluency. J Affect Disord. 2001;66(2–3):111–121. [PubMed]
  • Danos P. Pathology of the thalamus and schizophrenia--an overview. Fortschr Neurol Psychiatr. 2004;72(11):621–634. [PubMed]
  • Demirci O, Stevens MC, Andreasen NC, Michael A, Liu J, White T, Pearlson GD, Clark VP, Calhoun VD. Investigation of relationships between fMRI brain networks in the spectral domain using ICA and Granger causality reveals distinct differences between schizophrenia patients and healthy controls. Neuroimage. 2009;46(2):419–431. [PMC free article] [PubMed]
  • Erhardt EB, Rachakonda S, Bedrick EJ, Allen EA, Adali T, Calhoun VD. Comparison of multi-subject ICA methods for analysis of fMRI data. Hum Brain Mapp. 2010 in press. [PMC free article] [PubMed]
  • First MB, Spitzer RL, Gibbon M, Williams JB. Structured Clinical Interview for DSM-IV axis I disorders. New York: Biometrics Research Department, New York State Psychiatric Institute; 1995.
  • Fornito A, Yucel M, Pantelis C. Reconciling neuroimaging and neuropathological findings in schizophrenia and bipolar disorder. Curr Opin Psychiatry. 2009;22(3):312–319. [PubMed]
  • Franco AR, Ling J, Caprihan A, Calhoun VD, Jung RE, Heileman GL, Mayer AR. Multimodal and Multi-tissue Measures of Connectivity Revealed by Joint Independent Component Analysis. IEEE J Sel Top Signal Process. 2008;2(6):986–997. [PMC free article] [PubMed]
  • Freire L, Roche A, Mangin JF. What is the best similarity measure for motion correction in fMRI time series? IEEE Trans Med Imaging. 2002;21(5):470–484. [PubMed]
  • Friston KJ, Ashburner J, Frith CD, Poline JP, Heather JD, Frackowiak RS. Spatial registration and normalization of images. Hum Brain Mapp. 1995;2:165–189.
  • Friston KJ, Ashburner J, Kiebel SJ, Nichols TE, Penny WD. Statistical parametric mapping. 2005
  • Glahn DC, Bearden CE, Niendam TA, Escamilla MA. The feasibility of neuropsychological endophenotypes in the search for genes associated with bipolar affective disorder. Bipolar Disord. 2004;6(3):171–182. [PubMed]
  • Glahn DC, Robinson JL, Tordesillas-Gutierrez D, Monkul ES, Holmes MK, Green MJ, Bearden CE. Fronto-temporal dysregulation in asymptomatic bipolar I patients: A paired associate functional MRI study. Hum Brain Mapp. 2010 In press. [PMC free article] [PubMed]
  • Goes FS, Zandi PP, Miao K, McMahon FJ, Steele J, Willour VL, Mackinnon DF, Mondimore FM, Schweizer B, Nurnberger JI, Jr, et al. Mood-incongruent psychotic features in bipolar disorder: familial aggregation and suggestive linkage to 2p11-q14 and 13q21-33. Am J Psychiatry. 2007;164(2):236–247. [PubMed]
  • Goodwin, Jamison Manic-Depressive Illness. 2007:54–59.
  • Grady CL, Springer MV, Hongwanishkul D, McIntosh AR, Winocur G. Age-related changes in brain activity across the adult lifespan. J Cogn Neurosci. 2006;18(2):227–241. [PubMed]
  • Grieve SM, Williams LM, Paul RH, Clark CR, Gordon E. Cognitive aging, executive function, and fractional anisotropy: a diffusion tensor MR imaging study. AJNR Am J Neuroradiol. 2007;28(2):226–235. [PubMed]
  • Groves AR, Beckmann CF, Smith SM, Woolrich MW. Linked independent component analysis for multimodal data fusion. Neuroimage. 2010 in press. [PubMed]
  • Guye M, Bartolomei F, Ranjeva JP. Imaging structural and functional connectivity: towards a unified definition of human brain organization? Curr Opin Neurol. 2008;21(4):393–403. [PubMed]
  • Guze SB, Woodruff RA, Jr, Clayton PJ. The significance of psychotic affective disorders. Arch Gen Psychiatry. 1975;32(9):1147–1150. [PubMed]
  • Hall J, Whalley HC, Marwick K, McKirdy J, Sussmann J, Romaniuk L, Johnstone EC, Wan HI, McIntosh AM, Lawrie SM. Hippocampal function in schizophrenia and bipolar disorder. Psychol Med. 2009;40(5):761–770. [PubMed]
  • Hamilton LS, Altshuler LL, Townsend J, Bookheimer SY, Phillips OR, Fischer J, Woods RP, Mazziotta JC, Toga AW, Nuechterlein KH, et al. Alterations in functional activation in euthymic bipolar disorder and schizophrenia during a working memory task. Hum Brain Mapp. 2009 [PMC free article] [PubMed]
  • Heng S, Song AW, Sim K. White matter abnormalities in bipolar disorder: insights from diffusion tensor imaging studies. J Neural Transm. 2010;117(5):639–654. [PubMed]
  • Hua K, Zhang J, Wakana S, Jiang H, Li X, Reich DS, Calabresi PA, Pekar JJ, van Zijl PC, Mori S. Tract probability maps in stereotaxic spaces: analyses of white matter anatomy and tract-specific quantification. Neuroimage. 2008;39(1):336–347. [PMC free article] [PubMed]
  • Hyvarinen A, Karhunen J, Oja E. Independent Component Analysis. John Wiley & Sons; 2001.
  • Kempton MJ, Haldane M, Jogia J, Grasby PM, Collier D, Frangou S. Dissociable brain structural changes associated with predisposition, resilience, and disease expression in bipolar disorder. J Neurosci. 2009;29(35):10863–10868. [PubMed]
  • Kiehl KA, Laurens KR, Duty TL, Forster BB, Liddle PF. Neural sources involved in auditory target detection and novelty processing: an event-related fMRI study. Psychophysiology. 2001;38(1):133–142. [PubMed]
  • Kubicki M, Park H, Westin CF, Nestor PG, Mulkern RV, Maier SE, Niznikiewicz M, Connor EE, Levitt JJ, Frumin M, et al. DTI and MTR abnormalities in schizophrenia: analysis of white matter integrity. Neuroimage. 2005;26(4):1109–1118. [PMC free article] [PubMed]
  • Kubicki M, Westin CF, Nestor PG, Wible CG, Frumin M, Maier SE, Kikinis R, Jolesz FA, McCarley RW, Shenton ME. Cingulate fasciculus integrity disruption in schizophrenia: a magnetic resonance diffusion tensor imaging study. Biol Psychiatry. 2003;54(11):1171–1180. [PMC free article] [PubMed]
  • Lewis DA. Is there a neuropathology of schizophrenia? Recent findings converge on altered thalamic-prefrotnal cortical connectivity. The Neuroscientist. 2000;6:208–218.
  • Li Y-O, Adali T, Wang W, Calhoun VD. Joint Blind Source Separation by Multiset Canonical Correlation Analysis. IEEE Trans Signal Process. 2009a;57(10):3918–3929. [PMC free article] [PubMed]
  • Li YO, Adali T, Calhoun VD. Estimating the number of independent components for functional magnetic resonance imaging data. Hum Brain Mapp. 2007;28(11):1251–1266. [PubMed]
  • Li YO, Adali T, Wang W, Calhoun VD. Joint Blind Source Separation by Multi-set Canonical Correlation Analysis. IEEE Trans Signal Process. 2009b;57(10):3918–3929. [PMC free article] [PubMed]
  • Lichtenstein P, Yip BH, Bjork C, Pawitan Y, Cannon TD, Sullivan PF, Hultman CM. Common genetic determinants of schizophrenia and bipolar disorder in Swedish families: a population-based study. Lancet. 2009;373(9659):234–239. [PMC free article] [PubMed]
  • Ling J, Merideth F, Caprihan A, Peña A, Teshiba T, Mayer AR. Head Injury or Head Motion? Assessment and Quantification of Motion Artifacts in Diffusion Tensor Imaging Studies. Hum Brain Mapp. 2010 In press. [PubMed]
  • Liu J, Pearlson G, Windemuth A, Ruano G, Perrone-Bizzozero NI, Calhoun V. Combining fMRI and SNP data to investigate connections between brain function and genetics using parallel ICA. Hum Brain Mapp. 2009;30(1):241–255. [PMC free article] [PubMed]
  • Martinez-Montes E, Valdes-Sosa PA, Miwakeichi F, Goldman RI, Cohen MS. Concurrent EEG/fMRI analysis by multiway Partial Least Squares. Neuroimage. 2004;22(3):1023–1034. [PubMed]
  • McCarley RW, Nakamura M, Shenton ME, Salisbury DF. Combining ERP and structural MRI information in first episode schizophrenia and bipolar disorder. Clin EEG Neurosci. 2008;39(2):57–60. [PMC free article] [PubMed]
  • McIntosh AM, Hall J, Lymer GK, Sussmann JE, Lawrie SM. Genetic risk for white matter abnormalities in bipolar disorder. Int Rev Psychiatry. 2009;21(4):387–393. [PubMed]
  • McIntosh AM, Job DE, Moorhead TW, Harrison LK, Lawrie SM, Johnstone EC. White matter density in patients with schizophrenia, bipolar disorder and their unaffected relatives. Biol Psychiatry. 2005;58(3):254–257. [PubMed]
  • McIntosh AM, Job DE, Moorhead WJ, Harrison LK, Whalley HC, Johnstone EC, Lawrie SM. Genetic liability to schizophrenia or bipolar disorder and its relationship to brain structure. Am J Med Genet B Neuropsychiatr Genet. 2006;141B(1):76–83. [PubMed]
  • McIntosh AM, Maniega SM, Lymer GK, McKirdy J, Hall J, Sussmann JE, Bastin ME, Clayden JD, Johnstone EC, Lawrie SM. White matter tractography in bipolar disorder and schizophrenia. Biol Psychiatry. 2008a;64(12):1088–1092. [PubMed]
  • McIntosh AM, Moorhead TW, Job D, Lymer GK, Munoz Maniega S, McKirdy J, Sussmann JE, Baig BJ, Bastin ME, Porteous D, et al. The effects of a neuregulin 1 variant on white matter density and integrity. Mol Psychiatry. 2008b;13(11):1054–1059. [PubMed]
  • McIntosh AM, Whalley HC, McKirdy J, Hall J, Sussmann JE, Shankar P, Johnstone EC, Lawrie SM. Prefrontal function and activation in bipolar disorder and schizophrenia. Am J Psychiatry. 2008c;165(3):378–384. [PubMed]
  • Olesen PJ, Nagy Z, Westerberg H, Klingberg T. Combined analysis of DTI and fMRI data reveals a joint maturation of white and grey matter in a fronto-parietal network. Brain Res Cogn Brain Res. 2003;18(1):48–57. [PubMed]
  • Pearlson GD. Superior temporal gyrus and planum temporale in schizophrenia: a selective review. Prog Neuropsychopharmacol Biol Psychiatry. 1997;21(8):1203–1229. [PubMed]
  • Pessoa L, Adolphs R. Emotion processing and the amygdala: from a 'low road' to 'many roads' of evaluating biological significance. Nat Rev Neurosci. 2011;11(11):773–783. [PMC free article] [PubMed]
  • Raichle ME, MacLeod AM, Snyder AZ, Powers WJ, Gusnard DA, Shulman GL. A default mode of brain function. Proc Natl Acad Sci U S A. 2001;98(2):676–682. [PubMed]
  • Reinges MH, Krings T, Kranzlein HH, F J, Thron A, Gilsbach JM. Functional and diffusion-weighted magnetic resonance imaging for visualization of the postthalamic visual fiber tracts and the visual cortex. Minimally Invasive Neurosurgery. 2004;47(160–164) [PubMed]
  • Rogowska J, Gruber SA, Yurgelun-Todd DA. Functional magnetic resonance imaging in schizophrenia: cortical response to motor stimulation. Psychiatry Res. 2004;130(3):227–243. [PubMed]
  • Rykhlevskaia E, Gratton G, Fabiani M. Combining structural and functional neuroimaging data for studying brain connectivity: a review. Psychophysiology. 2008;45(2):173–187. [PubMed]
  • Schlosser RG, Nenadic I, Wagner G, Gullmar D, von Consbruch K, Kohler S, Schultz CC, Koch K, Fitzek C, Matthews PM, et al. White matter abnormalities and brain activation in schizophrenia: a combined DTI and fMRI study. Schizophr Res. 2007;89(1–3):1–11. [PubMed]
  • Seghier ML, Lazeyras F, Zimine S, Maier SE, Hanquinet S, Delavelle J, Volpe JJ, Huppi PS. Combination of event-related fMRI and diffusion tensor imaging in an infant with perinatal stroke. Neuroimage. 2004;21(1):463–472. [PubMed]
  • Silver M, Montana G, Nichols TE. False positives in neuroimaging genetics using voxel-based morphometry data. Neuroimage. 2011;54(2):992–1000. [PMC free article] [PubMed]
  • Skudlarski P, Jagannathan K, Anderson K, Stevens MC, Calhoun VD, Skudlarska BA, Pearlson G. Brain Connectivity Is Not Only Lower but Different in Schizophrenia: A Combined Anatomical and Functional Approach. Biol Psychiatry. 2010 [PMC free article] [PubMed]
  • Smith SM, Fox PT, Miller KL, Glahn DC, Fox PM, Mackay CE, Filippini N, Watkins KE, Toro R, Laird AR, et al. Correspondence of the brain's functional architecture during activation and rest. Proc Natl Acad Sci U S A. 2009;106(31):13040–13045. [PubMed]
  • Spitzer RL, Williams JB, Gibbon M. Structured Clinical interview for DSM-IV: Non-patient edition (SCID-NP) New York: Biometrics Research Department, New York State Psychiatric Institute; 1996.
  • Strakowski SM, Delbello MP, Adler CM. The functional neuroanatomy of bipolar disorder: a review of neuroimaging findings. Mol Psychiatry. 2005;10(1):105–116. [PubMed]
  • Strasser HC, Lilyestrom J, Ashby ER, Honeycutt NA, Schretlen DJ, Pulver AE, Hopkins RO, Depaulo JR, Potash JB, Schweizer B, et al. Hippocampal and ventricular volumes in psychotic and nonpsychotic bipolar patients compared with schizophrenia patients and community control subjects: a pilot study. Biol Psychiatry. 2005;57(6):633–639. [PubMed]
  • Sui J, Adali T, Li YO, Yang H, Calhoun DC. A review of multivariate methods in brain imaging data fusion. SPIE Medical Imaging Conference; Biomedical Applications in Molecular, Structural, and Functional Imaging; 13 – 18 February, 2010; San Diego, CA, USA. 2010a.
  • Sui J, Adali T, Pearlson G, Yang H, Sponheim SR, White T, Calhoun VD. A CCA+ICA based model for multi-task brain imaging data fusion and its application to schizophrenia. Neuroimage. 2010b;51(1):123–134. [PMC free article] [PubMed]
  • Sui J, Adali T, Pearlson GD, Calhoun VD. An ICA-based method for the identification of optimal FMRI features and components using combined group-discriminative techniques. Neuroimage. 2009;46(1):73–86. [PMC free article] [PubMed]
  • Sullivan EV, Pfefferbaum A. Diffusion tensor imaging in normal aging and neuropsychiatric disorders. Eur J Radiol. 2003;45(3):244–255. [PubMed]
  • Sussmann JE, Lymer GK, McKirdy J, Moorhead TW, Maniega SM, Job D, Hall J, Bastin ME, Johnstone EC, Lawrie SM, et al. White matter abnormalities in bipolar disorder and schizophrenia detected using diffusion tensor magnetic resonance imaging. Bipolar Disord. 2009;11(1):11–18. [PubMed]
  • Thomos N, Boulgouris NV, Strintzis MG. Optimized Transmission of JPEG2000 Streams Over Wireless Channels. IEEE Trans Image Process. 2006;15(1) [PubMed]
  • Tichavsky P, Koldovsky Z, Yeredor A, Gomez-Herrero G, Doron E. A Hybrid Technique for Blind Separation of Non-Gaussian and Time-Correlated Sources Using a Multicomponent Approach. IEEE Transactions on Neural Networks. 2008;19(3):421–430. [PubMed]
  • Toosy AT, Ciccarelli O, Parker GJ, Wheeler-Kingshott CA, Miller DH, Thompson AJ. Characterizing function-structure relationships in the human visual system with functional MRI and diffusion tensor imaging. Neuroimage. 2004;21(4):1452–1463. [PubMed]
  • Wang F, Kalmar JH, He Y, Jackowski M, Chepenik LG, Edmiston EE, Tie K, Gong G, Shah MP, Jones M, et al. Functional and structural connectivity between the perigenual anterior cingulate and amygdala in bipolar disorder. Biol Psychiatry. 2009;66(5):516–521. [PMC free article] [PubMed]
  • Williams LM, Das P, Harris AW, Liddell BB, Brammer MJ, Olivieri G, Skerrett D, Phillips ML, David AS, Peduto A, et al. Dysregulation of arousal and amygdala-prefrontal systems in paranoid schizophrenia. Am J Psychiatry. 2004;161(3):480–489. [PubMed]
  • Yurgelun-Todd DA, Silveri MM, Gruber SA, Rohan ML, Pimentel PJ. White matter abnormalities observed in bipolar disorder: a diffusion tensor imaging study. Bipolar Disord. 2007;9(5):504–512. [PubMed]