Search tips
Search criteria 


Logo of nihpaAbout Author manuscriptsSubmit a manuscriptHHS Public Access; Author Manuscript; Accepted for publication in peer reviewed journal;
Biol Psychiatry Cogn Neurosci Neuroimaging. Author manuscript; available in PMC 2017 May 1.
Published in final edited form as:
PMCID: PMC4917230

Multimodal fusion of brain imaging data: A key to finding the missing link(s) in complex mental illness


It is becoming increasingly clear that combining multi-modal brain imaging data is able to provide more information for individual subjects by exploiting the rich multimodal information that exists. However, the number of studies that do true multimodal fusion (i.e. capitalizing on joint information among modalities) is still remarkably small given the known benefits. In part, this is because multi-modal studies require broader expertise in collecting, analyzing, and interpreting the results than do unimodal studies. In this paper, we start by introducing the basic reasons why multimodal data fusion is important and what it can do, and importantly how it can help us avoid wrong conclusions and help compensate for imperfect brain imaging studies. We also discuss the challenges that need to be confronted for such approaches to be more widely applied by the community. We then provide a review of the diverse studies that have used multimodal data fusion (primarily focused on psychosis) as well as provide an introduction to some of the existing analytic approaches. Finally, we discuss some up-and-coming approaches to multi-modal fusion including deep learning and multimodal classification which show considerable promise. Our conclusion is that multimodal data fusion is rapidly growing, but it is still underutilized. The complexity of the human brain coupled with the incomplete measurement provided by existing imaging technology makes multimodal fusion essential in order to mitigate against misdirection and hopefully provide a key to finding the missing link(s) in complex mental illness.

Keywords: data fusion, psychosis, connectivity, brain function, schizophrenia, independent component analysis

Introduction and motivation

“Multimodal” is a widely used phrase in the context of brain imaging studies. Collecting multiple modalities of magnetic resonance imaging (MRI) data from the same individual has been popular in brain imaging studies. There is increasing evidence that multimodal brain imaging studies can help provide a more complete understanding of the brain and its disorders, for example it can inform us about how brain structure shapes brain function, in which way they are impacted by psychopathology and which functional or structural aspects of physiology could drive human behavior and cognition.

In this paper we first provide some basic motivation regarding the benefits of multimodal imaging and also introduce some basically terminology for characterizing multimodal data analysis. Next, we review a large class of multivariate approaches for performing multimodal data fusion, the most powerful type of multimodal analysis. Followed by this we survey some of the existing articles that have applied multimodal imaging to study psychopathology. Finally, we discuss some exciting emerging trends and approaches and conclude.

2.1. Terminology

We now present some basic terminology with which to describe existing multimodal imaging work. On one end of the spectrum is visual inspection which is basically inferring the multimodal information by separately visualizing results from essentially unimodal analyses. This is the least informative, but is used quite extensively, and can highlight the different results that are provided by each modality in a qualitative manner. An alternative approach—which we call data integration(1-3)—is to analyze each data type separately and overlay them—thereby not allowing for an examination of interaction among data-types. For example, a data integration approach would not detect a change in gray matter concentration between patients and controls that is related to fMRI activation maps, as shown in the example. A third approach, called one-sided or asymmetric data fusion is the use of one data set to constrain another as in diffusion MRI (dMRI)(4-6) or magneto/electroencephalography (M/EEG)(7-9) being constrained by structural MRI (sMRI) or fMRI data. While these techniques are powerful, a restriction is that they impose potentially unrealistic assumptions upon the dMRI or EEG data, which are of an essentially different nature than fMRI data. Finally, symmetric data fusion, utilizes and treats multiple image-types equally to take fully advantage of the joint information in multiple data sets. The approaches just described are shown in Figure 1. The use of joint information is only qualitatively used on the far left, and maximally used on the far right.

Figure 1
A spectrum of data fusion approaches. Fusion, in in increasing order of joint information provided, can range from simple visual inspection of two modalities (red and yellow circles), to overlaying them (e.g. PET/CT fusion), to jointly analyzing in series ...

Surprisingly, the maximal use of multimodal data via data fusion is still not a universally accepted way to study brain function. This is true, despite the common understanding that any brain-imaging modality alone provides only a limited view into brain function; despite growing availability of multimodal datasets; and despite general acceptance of the complementary nature of information hidden in various modalities.

2.2. The benefit of joint information [multitask & multimodal]

Currently, a large number of studies are collecting multimodal brain imaging data and information from the same participants. These imaging data types should be leveraged to extract the complementary information. For example, fMRI measures the hemodynamic response related to neural activity in the brain dynamically; sMRI enables us to estimate the tissue type for each voxel in the brain [gray matter (GM), white matter (WM), cerebrospinal fluid (CSF)]. Diffusion MRI can additionally provide information on the integrity of white matter tracts and structural connectivity. A key motivation for jointly analyzing multimodal data is to leverage the cross-information in the existing data, thereby revealing important relationships that cannot be detected by using a single modality.

But, practically speaking, why should we analyze multimodal imaging data jointly, instead of just analyzing each domain separately? Consider identifying a single relevant feature from one modality (say volume of the hippocampus) and correlating it with all brain voxels across subjects in the other modality (say a default mode connectivity map), then testing for group differences in this correlation. This is not the same as separately evaluating which brain volumes show group-related activity changes and which regions in the default mode network show group-related differences. The former analysis can be considered a type of data fusion, because both data sets are used to estimate a joint result. Such approaches have in many cases enhanced our ability to distinguish patients versus controls (see Figure 2)(10).

Figure 2
The benefit of a joint analysis is we can capitalize on the joint distribution of the multimodal imaging data which can improve our ability to discriminate health and disease. When we have two data sets, each with numerous variables, we could compute ...

Capturing joint information from multiple data sets can be done using univariate approaches (such as correlation) or multivariate approaches (such as independent component analysis (ICA)). Multivariate approaches(11-16) have a unique advantage as their focus is on inter-related patterns rather than unrelated points (see Figure 3). This makes them ideally suited for detecting complicated, and potentially weak, effects hidden in a high-dimensional data sets. In contrast to univariate analyses, multivariate approaches estimate all the variables jointly. This provides a couple major benefits. First, it helps with interpretation, as one can accurately assume the region in a given component covary together. Secondly, it can provide robustness to noise. For example, correlation-based approaches can be ‘tricked’ by phenomena such as phase randomized noise which can appear to represent real signal(17), whereas approaches that focus on identifying ‘patterns’ are better able to distinguish randomized noise from real signal. This does not mean that multivariate methods are impervious to noise, but they do tend to be more robust than univariate correlation in many cases, because they are working with patterns instead of just paired relationships.

Figure 3
Univariate vs multivariate approaches for capturing information among multiple data types. The figure on the left shows a cloud of points from two datasets which are analyzed using a univariate approach, essentially an analysis of each set of points pairwise. ...

The carrot and the stick

Among the multitude of reasons preventing researchers from practicing multimodal analysis, arguably, two reasons contribute the most: 1) a doubt that modalities that are very different contain any common information to reinforce their signal, and 2) an assumption that what is learned from one modality is at worst, incomplete, but not incorrect. It is quite easy to show specific examples where both reasons are incorrect. For example, in Figure 4(left) shows an example where a combined EEG (high temporal information) and fMRI (localized spatial information) analysis results in overlapping intrinsic brain activity(18). We can also easily demonstrate a case in which unimodal analysis can be misleading without any way for one to know it. Figure 4(right) shows how separate analysis of MEG and fMRI collected from the same subjects in identical experiments leads to strictly opposite conclusions about effective connectivity and thus brain function. Both of these examples motivate the importance of analysis frameworks that can correctly account for the properties of each of the data sources and also for taking multiple data sources into account whenever possible.

Figure 4
(left) Results high showing similarity between brain networks extracted from EEG and fMRI. (right) Graph results computed from MEG and fMRI data collected from the same subjects for two tasks show dramatically different answers.

Is there a down side?

One of the issues to consider is whether there is a downside to a focus on multimodal data fusion. We focus on two areas here, the first being data collection and the second being data analysis. For the first, one obviously needs to collect multimodal data, which can add time and effort to the data collection process. However this particular down side is in many cases not a barrier as virtually all imaging studies collect at least two modalities, an often many more than two (e.g. sMRI, rest fMRI, task fMRI, dMRI). Regarding the analytic aspect, there are real barriers here. In terms of the analytic approach, understanding the various assumptions of a given algorithms (among many choices) is important but not trivial. Fortunately, the requirement for number of subjects is similar to that for a unimodal approach, as most approaches first reduce the data down into a smaller number of joint components, however the investigator would also need to be familiar with how to preprocess multiple modalities (or just rest fMRI and task fMRI which are less different). This may require learning different types of software tools and expanding the overall workload on the team. Secondly, once the data are preprocessed and analyzed, interpretation involves a learning curve. For example, one might need to consider multiple regions in two or more data sets, the relationship between the covariation of the data within these regions, and ultimately, what this can tell us about the particular problem being studied. This can take time, and unfortunately, there are not many multimodal training courses available, thus most of the knowledge is gained by interacting with labs that have expertise or learning from the beginning within a given lab. However, as we hope to show in this paper, we believe the advantages to be gained by taking a multimodal approach to one's data are many and worth the initial investment to learn how to perform such analyses.

2.3. A framework for multimodal data fusion

Approaches to data fusion can be conceptualized as having a place on an analytic spectrum with detailed large-scale computational modeling at one end and highly distilled data at the other end(19,20). In between are methods that attempt to perform direct data fusion on high-dimensional summary measures(3,21-23). In this intermediate approach one extracts features by preprocessing the data, and employs these to examine the inter-modality relationships at the group level (i.e., variations among individuals). Here a “feature” is a distilled dataset representing the interesting part of each modality(3,24) and it is used as the input to the fusion analysis for each modality and each subject. Examples of features include a component image such as the default mode network resulting from a group ICA(25), a fractional amplitude of low frequency fluctuations (fALFF) map calculated from resting-state fMRI, a fractional anisotropy (FA) from dMRI measures, or segmented gray matter (GM) from sMRI data. The main reason to use features is to provide a more concise/focused, but still informative, space in which to link the data. This approach has several advantages in that it 1) allows us to take advantage of the ‘cross’-information among data types(1,2) and 2) enables indirect or direct associations to be inferred on putative structure-function relationships(26) in a way that does not require these modalities to have been measured simultaneously. The trade-off is that some information may be lost, e.g., GM does not directly measure cortical thickness or volume, and FA does not provide directional information; however, one key advantage is with features we can directly leverage the extensive amount of work focused on unimodal analysis(3,27,28), for example widely replicated patterns of gray matter differences in schizophrenia including temporal lobe and medial frontal cortex(29) and consistent replication of the widely studied default mode network including demonstration of heritability(30). Figure 5 provides a direct view of several multivariate voxel-wise data fusion approaches.

Figure 5
Summary of several multivariate voxel-wise data fusion approaches covering the various discussed approaches of asymmetric/symmetric and blind/semi-blind.

We start with the division of multivariate fusion approaches into two main classes: model-driven and data-driven. Model-driven approaches are those that incorporate specific knowledge about the problem, for example specific influence of one brain region upon another. They include approaches based on the general linear model(31,32), dynamic causal modeling (DCM), and confirmatory structural equation modeling (SEM)(33). Model-driven approaches have the advantage of: 1) enabling testing of particular hypotheses about interaction among the identified networks/regions; 2) simultaneous assessment of multiple connectivity links, moving beyond a one-by-one assessment of covariance(34). However, such approaches may miss important relationships that are not included in the prior hypotheses, and typically do not enable examination of full inter-voxel relationships(35,36).

Data-driven approaches include, but are not limited to, principal component analysis (PCA), ICA, canonical correlation analysis (CCA), and partial least squares (PLS). These methods do not require a priori hypotheses about the inter-relationships and thus are useful for exploring the entire data set. They typically work with the entire set of voxels rather than a selected set of regions. It is also possible to use a data-driven approach to test hypotheses if one can design a hypothesis about the output that is independent of the data. For example, one can certainly hypothesize that patients will show a breakdown in the link between function and structure estimated from a joint ICA algorithm.

A model-based approach is most useful if you know enough about the problem being studied to incorporate this information as assumptions in the algorithm. To the degree the assumptions made are correct, model-based approaches typically perform better and to the degree the assumptions are incorrect, the model-based method will perform worse. Data-driven approaches make fewer assumptions about the structure of the problem, and thus are most useful when one does not want to commit to strong assumptions about the data. Given our knowledge about the human brain and complex mental illness is so incomplete, there is great benefit in making fewer strong assumptions up front, but the two approaches are of course quite complementary to one another.

Given the wide array of approaches that have been applied, any subdivision is limited, however, we can divide multivariate approaches based on the incorporation of priori knowledge and the dimension of the MRI data used:

  • 1)
    blind methods that typically use second-level fMRI data (e.g. a 3D contrast image), including joint ICA (jICA)(37), multimodal canonical correlation analysis (mCCA)(38), linked ICA(39), independent vector analysis (IVA) and mCCA+jICA(40,41).
  • 2)
    blind methods that have been used with first-level fMRI data (e.g. 4D fMRI data), including PLS(42,43), group ICA(44), and multiset CCA(45).
  • 3)
    semi-blind methods that use second-level fMRI data, e.g., parallel ICA(46), coefficient-constrained ICA (CC-ICA)(47-49), PCA with reference (PCA-R)(50,51) and informed multimodal PLS(42).
  • 4)
    Other multimodal fusion applications using 4D fMRI (typical with regions of interest) and EEG data, include GLM-based approaches, SEM(33) or DCM(8).

The optimization strategies of several of the above mentioned multivariate models are displayed in Figure 6. We provide a slightly more detailed summary of several blind fusion models in the supplemental. We also compare several multivariate multimodal fusion methods including their statistical assumptions, strengths and limitations, and multi-modal neuroimaging applications.

Figure 6
Summary of 7 blind and semi-blind data-driven methods for multimodal fusion. Figure modified and reprinted with permission from Sui et al(21).

In the next sections we review some psychosis-related data fusion studies, starting first from those that focus more on spatial overlap and then moving to those that perform symmetric data fusion. A brief comment on our review methodology, we searched pubmed for the terms ‘multimodal fusion’, ‘multimodal’, ‘multimodal modalities’, and then narrowed these to studies that actually used one of the afore mentioned fusion-based approach. We also narrowed the topics to studies that focused on specific disorders.

2. Other multimodal applications based on various MRI measures

Though a data-driven symmetric approach is the most informative and makes fewer assumptions about the specific relationships among data sets, the spatial overlap approach has been one of the most widely used to date. As such, we review some of the relevant work in the context of spatial overlap between function and structure including cross-modal connectivity.

3.1. Spatial overlap between modalities

A central assumption of systems neuroscience is that the structure of the brain can predict and/or is related to brain function. The findings of (52) support this hypothesis, which generally show that each single structural component derived from ICA usually corresponds to several resting-state functional components. On the other hand, functional information can help improve the correspondence of functional boundaries across subjects compared to standard structural normalization, as reported in (53), Many psychopathological studies have already indicated the spatial overlap between brain structure and function in mental disorders. Specifically, Salgado-Pineda P. et al (54) found three regions including the thalamus, the anterior cingulate and the inferior parietal that showed both structural and functional impairments associated with attentional processing in schizophrenia. A follow up study of the same group (55) also found both functional alterations in a facial emotion task and GM volume reductions in the DMN in schizophrenia. Phillips ML et al(56) described a novel noninvasive approach for relating brain structure and function with diffusion spectrum imaging (DSI) and fMRI, and revealed co-active areas of task-relevant functional brain activity are anatomically connected by WM tractography, creating a “circuit diagram” in different cognitive tasks. Sasamoto A. et al. (57) postulated a global association between pathologies of GM cortical thickness and FA in schizophrenia and found the mean of both measures were significantly lower in patients with schizophrenia (SZ). Moreover, only in patient group the mean cortical thickness and mean FA showed significant positive correlations in both hemispheres, suggesting GM and WM pathologies of schizophrenia are intertwined at the global level. In a combined dMRI-GM study in medicated-naïve chronic schizophrenia(58), Liu X. et al. observed that patients possessed lower FA values in the left inferior fronto-occipital fasciculus and left inferior longitudinal fasciculus, along with smaller GM volume and cortical thinning in temporal lobe than healthy controls, which reflected the interdependent WM and GM disruption that contributed to the disease. In sum, while it is too early to identify a systematic and replicable link between brain function and structure, it is clear that there is a strong inter-relationship which merits additional study.

3.2. Disruption of brain connectivity

Inter-hemispheric disconnection or misconnection has been implicated in the disconnection hypothesis in psychosis(59-61). Recent advances in dMRI allow subtle white matter abnormalities in schizophrenia to be captured, which cannot be detected by structural MRI alone. Combining dMRI tractography with sMRI, Miyata et al(62) tested the inter-hemispheric disconnection hypothesis by parcellating the corpus callosum into functionally-anatomically relevant sub-regions and discovered that inter-frontal commissural fibers are specifically reduced in SZ. This is not only consistent with the disconnection hypothesis, but also specifies the locus of disconnection in a functionally-anatomically relevant way. A recent study of Koch et al(63) showed that increased radial diffusivity of the left superior temporal gyrus is associated with reduced neuronal activation in lateral frontal and cingulate cortex, suggesting a key role for white matter connectivity in determining the pattern and intensity of functional activation related to decision making.

FMRI has also been used to examine the functional disconnection hypothesis of schizophrenia(64). However, the number of studies that have combined fMRI with other modalities is limited. One study using rsfMRI and dMRI found decreased functional connectivity in regions including the right thalamus, which was associated with decreased structural connectivity in the left superior cerebellar peduncle(65). Another study found concordant reductions in functional and anatomical connectivity in the medial frontal and anterior cingulate regions in SZ compared with healthy controls (HC)(66). Schlosser et al. observed a direct correlation in schizophrenia between frontal FA reduction and fMRI activation in regions of the prefrontal and occipital cortices(67). This finding highlights a potential relationship between anatomical changes in a frontal-temporal anatomical circuit and functional alterations in the prefrontal cortex. Staempfli et al. discovered that dMRI- and fMRI-derived topologies are similar, and the combination of fMRI and dMRI can provide extra information to help select appropriate seed regions for identifying functionally relevant networks, and validate reconstructed WM fibers(68). Such studies combining functional and structural connectivity provide clear evidence of a complex interaction between modalities, though in most cases the directionality is similar in that decreased structural connectivity is associated with decreased functional connectivity.

3. Review of fusion studies focused on psychopathology

We now review a number of studies, mostly using data-driven approaches to study associations among modalities in the context of psychopathology. The multimodal studies reviewed are summarized and classified into different categories in Table 1. Generally speaking, most of the studies we reviewed demonstrate congruent effects across modalities and multimodal fusion almost always provided more power to differentiate disease than unimodal approaches.

Table 1
Summary of Studies Combining Measures of Functional and Structural MRI

There has been a rapid growth in the use of multimodal fusion approaches. Figure 7 shows a summary of multiple Pubmed searches on various terms including 2-way and N-way multimodal fusion (2-way means two modalities or tasks are analyzed, N-way means more than two modalities or tasks are analyzed). In all cases there has been a rapid increase in the past few years, though, as we argue, more of these type of approaches are needed.

Figure 7
Summary of multimodal fusion studies found via Pubmed. Note the rapid increase in all categories including both 2-way and N-way fusion.

4.1. Schizophrenia

Multimodal MRI can reveal insightful information about key clinical aspects of schizophrenia(69,70). As the most widely studied psychosis, schizophrenia has served as the test bed for almost all above mentioned fusion approaches(71). Specifically, Sui et al examined the linked cognitive biomarkers of schizophrenia by combining fALFF, GM and FA measures from 3 MRI modalities via MCCA(72), suggesting linked functional and structural deficits in distributed cortico-striato-thalamic circuits may be closely related to cognitive impairments measured via the MATRICS battery(73). Using a similar method, Correa et al.(74) identified differences in the co-variation of fMRI and EEG data in SZ versus HC during an auditory oddball task(75). Significant group differences were found in the bilateral temporal lobe/middle anterior cingulate region in fMRI, associated with the N2 and P3 peak in EEG.

Sugranyes et al.(76) examined interactions between fMRI contrast maps from a working memory task and dMRI data by joint ICA, and characterized linked functional and WM changes related to working memory dysfunction, including fMRI hypoactivation of SZ in anterior cingulate and ventrolateral prefrontal cortex and reduced FA localized in the splenium and posterior cingulum. Similarly, Stephen et al.(77) employed joint ICA to investigate the link between magnetoencephalography (MEG) and dMRI data, pinpointing to dysfunction in a posterior visual processing network in schizophrenia, with reduced MEG amplitude, reduced FA and poorer overall cognitive performance. Xu et al also identified four linked gray–white matter networks that were significantly associated with SZ using joint ICA(78), included: 1) temporal-corpus callosum 2) occipital/frontal-inferior fronto-occipital fasciculus 3) frontal/ parietal/occipital/temporal-superior longitudinal fasciculus and 4) parietal/frontal-thalamus, reflecting the widespread nature of the disease.

N-way aberrant brain alterations in SZ(79,80) were also investigated by MCCA+jICA, a tool optimized for identifying correspondence across modalities/tasks, which has been applied to multitask fMRI, fMRI-sMRI-dMRI or fMRI-sMRI-methylation combinations. Given the complexity of mental illness, such combinations are essential evidence to help us better understand psychopathology, but at this point investigations are limited to a relatively spares set of combinations. This is very much a ‘big data’ problem as well, as we need approaches to utilize all available data while also helping us to summarize the complexity in order to find the key relationships that are impacted by mental illness. Another method to identify inter-correlations among gray matter and fMRI voxels within the whole brain was introduced by Michael et al.(81) for schizophrenia by reducing the cross-correlation matrix to histograms. Results showed that the linkage between gray matter and task-related functional activation in both an auditory sensorimotor task(82) and a working memory task(83) was weaker in SZ than HC. A multimodal voxel-based meta-analysis of structural and functional MRI studies at high genetic risk of developing schizophrenia(84) further showed that SZ relatives had decreased GM with functional hyper-activation in the left inferior frontal gyrus/amygdale, and decreased GM with hypo-activation in the thalamus.

4.2. Mood disorders

In order to probe abnormalities in brain circuits underpinning episodic memory deficits in bipolar disorder (BD) one recent study jointly analyzed, fMRI, sMRI and dMRI(85). Multimodal changes in frontal and parietal areas were revealed and associated with poorer episodic memory. This group also conducted a multi-modal assessment using resting state fMRI, GM volume, WM fiber integrity, and neurobehavioral measures to examine possible shared alterations in BD and SZ patients in the hippocampus(86). Results imply that two disorders may share common alterations in all 3 modalities, but SZ patients showed more severe structural alterations within hippocampus than BD. Similarly, an fMRI-dMRI fusion study comparing SZ and BD(87), showed distinct brain patterns for both clinical groups but interesting they also revealed shared abnormalities in prefrontal thalamic WM integrity and in frontal brain mechanisms.

In major depression disorder (MDD), Vasic N. et al.(88) suggest that while changes of cerebral blood flow (brain perfusion) and GM volume co-occur in MDD patients, they appear to reflect distinct levels of neuropathology. Dutt A et al.(89) reported P300 latency, a measure of the speed of neural transmission, appeared to relate to the size of left hippocampus in schizophrenia, but not in psychotic bipolar disorder. The specificity of this brain structure-function association for schizophrenia opens the scope for further research using integration of multimodal biological data for objective categorization of psychosis. De Kwaasteniet B. et al.(90) concluded structural abnormalities in MDD are associated with increased functional connectivity between subgenual anterior cingulate cortex (ACC) and medial temporal lobe. In addition, a negative structure-function relation in MDD was positively associated with depression severity. Han KM et al.(91) demonstrated structural alteration in both gray and white matter in medication-naïve first episode MDD patients, such as reduced cortical volume of the caudal ACC and decreased WM integrity in the body of the corpus callosum. One of the most interesting aspects of these studies is that it shows that in some cases the directionality in modalities is not always the same (e.g. decreased structure can lead to increased function and vice versa).

4.3. Other psychopathology

One recent study(92) combined GM and dMRI to investigate obsessive-compulsive disorder (OCD) and discovered significant alterations of interrelated gray and white matter networks over occipital and parietal cortices, frontal inter-hemispheric connections and cerebellum. Additionally, white matter networks adjacent to basal ganglia correlated with obsessive-compulsive symptoms. Another study(93) reported linked alterations in GM and WM morphology in adults with high-functioning autism spectrum disorder (ASD) via linked ICA, ASD patients showed decreased GM volumes in bilateral fusiform gyri, orbitofrontal cortices, and pre-/post-central gyri, which were linked with a pattern of decreased FA in tracts of inferior longitudinal fasciculi, inferior fronto-occipital fasciculi, and corticospinal tracts, all bilaterally.

4. Emerging approaches

Although recent multimodal imaging results are promising(21,34), much work remains to be done. As the field of multimodal data fusion is still relatively new, many of the studies represent novel findings by using a variety of data combinations; however, replication is needed to draw general conclusions about structure-function relationships. Secondly, despite the many successes of multimodal fusion, fusing as many modalities/features as possible in the training sample does not guarantee the optimal discrimination or classification between groups, as reported in (3,94); thus it can be useful to incorporate a mixture of uni-modal and multimodal results, as done in (95). This work can be pursued in future by making use of larger data sets and various modalities. More complex models, such as those that can handle N-way multimodal fusion, are being introduced and may become one of the leading directions in future neuroimaging research given the predominance of multi-modal data acquisition(22).

Classification using multimodal data

Regarding classification, there are multiple studies demonstrating the combination of structural and functional data can improve brain disease classification. A strong demonstration of this is found in a recent multimodal classification challenge for schizophrenia versus controls using GM and rest fMRI connectivity received over 2000 submissions, most of which were able to achieve greater than 80% accuracy(96). Dai et al.(97) proposed an automatic classification framework which integrated multimodal image features using multi-kernel learning (MKL) for predicting attention deficit/hyperactivity disorder versus controls finding a very high classification performance. Using an ensemble feature selection strategy and an advanced support vector machine approach, Sui et al.(98) combined resting-state fMRI, EEG and sMRI data to classify schizophrenia from healthy controls and achieved the best performance with 91% accuracy compared to using a single modality. By adopting Gaussian process classifiers to evaluate the prognostic value of neuroimaging data and clinical characteristics, Schmaal et al.(99) discovered that prediction of the naturalistic course of depression over 2 years is improved by considering different task contrasts or data sources, especially those derived from neural responses to emotional facial expressions. Finally, Pettersson-Yeo et al.(100) used a multimodal SVM approach to examine the ability of sMRI, fMRI, dMRI and cognitive data to differentiate between ultra-high-risk (UHR) and first-episode (FEP) psychosis at the single-subject level, supporting clinical development of SVM to help inform identification of FEP and UHR. These findings strongly suggest that multi-modal classification facilitated by advanced modeling techniques can provide more accurate and early detection of brain abnormalities beyond approaches that use only a single modality.

Deep learning

Major advances in performance have been obtained in multiple domains, including brain imaging, via deep (multilayered) learning algorithms to capture nonlinear/higher order relationships. Recent work has shown the potential for such models in neuroimaging data(101-106) and provide a framework to extend promising approaches such as linear ICA(107-109). One potential issue is that training of deep models requires extensive amounts of data. However this issue can in part be overcome by training the models with realistic simulation data(29,103). Deep models have recently made significant advances, outperforming shallow models in multiple problem domains such as image classification(110). Our work shows that class separation improves with deep belief network (DBN) depth while DBNs uncover hidden relations within data and thus facilitate discovery(102,111). Specifically, we investigated if classification rates improve with depth by sequentially investigating DBNs of 3 depths. Figure 8a displays 2D maps of the raw data, as well as the depth 1, 2, and 3 activations: the deeper networks place schizophrenia patients and healthy control groups further apart for both training and validation data (for more details see (102)). Another benefit of deep learning models is their ability to automatically discover high level representation(102), which is especially important for multimodal analysis incorporating different data types (e.g., fMRI, sMRI, and EEG) that are unlikely to have a simple linear correspondence. An example of a multimodal deep learning architecture is shown in Figure 8b. There are many interesting emerging models, for example, motivated by the concept of brain function and structure representing static images (sMRI) annotated by sequential captions of the brain (fMRI), one can build a model to translate the relationship between brain structure and brain function using a recurrent neural network(112). Many more fusion approaches based on deep learning will emerge in the near future.

Figure 8
A) Two dimensional display of deep learning analysis of brain imaging data for multiple models ranging from raw data to 3 levels (each dot represents an individual). Results show separation between patients and controls (both testing and training data) ...

5. Conclusions

In sum, we are just beginning to unlock the potential of multimodal imaging, which offers unprecedented opportunities to further deepen our understanding of the brain disorders(113) based on various brain imaging measures. The most promising avenues for the future may lie in developing better models that can complement and exploit the richness of our data(114). We are able to image the brain from living humans with multiple modalities, each providing a unique perspective. In order to minimize incorrect conclusions about mental illness, or even perhaps enabling us to identify the missing links between the brain and mental illness, multimodal data fusion is not only important, it is necessary.

Supplementary Material


6. Acknowledgements

The authors thank Sergey Plis for helpful input on the manuscript. The work was in part funded by NIH via a COBRE grant P20GM103472 and grants R01EB005846 and 1R01EB006841; the “100 Talents Plan” of the Chinese Academy of Sciences, the Chinese National Science Foundation grant No. 81471367, the State High-Tech Development Plan (863) No. 2015AA020513 and the Strategic Priority Research Program of the Chinese Academy of Sciences (XDB02060005).


Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

7. Financial disclosures

The authors report no biomedical financial interests or potential conflicts of interest.


1. Savopol F, Armenakis C. Proc.ISPRS. Buenos Aires; Argentina: 2002. Merging of Heterogeneous Data for Emergency Mapping: Data integration or Data Fusion? pp. 615–620.
2. Ardnt C. Information gained by data fusion. SPIE Proc. 1996
3. Calhoun VD, Adalı T. Feature-based Fusion of Medical Imaging Data. IEEE Transactions on Information Technology in Biomedicine. 2009;13:1–10. PMC2737598. [PMC free article] [PubMed]
4. Kim DS, Ronen I, Formisano E, Kim KH, Kim M, van Zijl P, Ugurbil K, Mori S, Goebel R. Proc.HBM. Sendai; Japan: 2003. Simultaneous mapping of functional maps and axonal connectivity in cat visual cortex.
5. Ramnani N, Lee L, Mechelli A, Phillips C, Roebroeck A, Formisano E. Exploring brain connectivity: a new frontier in systems neuroscience. Functional Brain Connectivity, 4-6 April 2002, Dusseldorf, Germany. Trends Neurosci. 2002;25:496–497. [PubMed]
6. Goldberg-Zimring D, Mewes AU, Maddah M, Warfield SK. Diffusion tensor magnetic resonance imaging in multiple sclerosis. J Neuroimaging. 2005;15:68S–81S. [PubMed]
7. Dale AM, Sereno MI. Improved Localizadon of Cortical Activity by Combining EEG and MEG with MRI Cortical Surface Reconstruction: A Linear Approach. J Cogn Neurosci. 1993;5:162–176. Spring. [PubMed]
8. Henson RN, Flandin G, Friston KJ, Mattout J. A Parametric Empirical Bayesian Framework for fMRI-Constrained MEG/EEG Source Reconstruction. Hum Brain Mapp. 2010 Oct;31:1512–1531. [PMC free article] [PubMed]
9. Lemieux L. Electroencephalography-correlated functional MR imaging studies of epileptic activity. Neuroimaging Clinics of North America. 2004 Aug;14:487. [PubMed]
10. Pearlson GD, Liu J, Calhoun VD. An Introductory Review of Parallel Independent Component Analysis (p-ICA) and a Guide to Applying p-ICA to Genetic Data and Imaging Phenotypes to Identify Disease-Associated Biological Pathways and Systems in Common Complex Disorders. Frontiers in Genetics: Statistical Genetics and Methodology. in press, PMC Journal - in process. [PMC free article] [PubMed]
11. Purcell SM, Wray NR, Stone JL, Visscher PM, O'Donovan MC, Sullivan PF, Sklar P. Common polygenic variation contributes to risk of schizophrenia and bipolar disorder. Nature. 2009 Aug 6;460:748–752. 3912837. [PMC free article] [PubMed]
12. Le Floch E, Guillemot V, Frouin V, Pinel P, Lalanne C, Trinchera L, Tenenhaus A, Moreno A, Zilbovicius M, Bourgeron T, Dehaene S, Thirion B, Poline JB, Duchesnay E. Significant correlation between a set of genetic polymorphisms and a functional brain network revealed by feature selection and sparse partial least squares. Neuroimage. 2012 Oct 15;63:11–24. [PubMed]
13. Hardoon DR, Ettinger U, Mourao-Miranda J, Antonova E, Collier D, Kumari V, Williams SC, Brammer M. Correlation-based multivariate analysis of genetic influence on brain volume. Neurosci Lett. 2009 Feb 6;450:281–286. [PubMed]
14. Calhoun VD, Liu J, Adalı T. A Review of Group ICA for fMRI Data and ICA for Joint Inference of Imaging, Genetic, and ERP data. NeuroImage. 2009;45:163–172. PMC2651152. [PMC free article] [PubMed]
15. Vounou M, Nichols TE, Montana G, Alzheimer's Disease Neuroimaging I. Discovering genetic associations with high-dimensional neuroimaging phenotypes: A sparse reduced-rank regression approach. Neuroimage. 2010 Nov 15;53:1147–1159. [PMC free article] [PubMed]
16. Liu J, Calhoun V. A review of multivariate analyses in imaging genetics. Frontiers in Neuroinformatics. 2014;8:1–11. PMC Journal - In Process. [PubMed]
17. Handwerker DA, Roopchansingh V, Gonzalez-Castillo J, Bandettini PA. Periodic changes in fMRI connectivity. Neuroimage. 2012 Nov 15;63:1712–1719. [PMC free article] [PubMed]
18. Plis SM, Bridwell D, Huster R, Damaraju E, Calhoun VD. Annual Meeting of the Organization for Human Brain Mapping. Honolulu, HI: 2015. EEG independent topographies match to electrode-space projections of fMRI default mode networks.
19. Horwitz B, Poeppel D. How can EEG/MEG and fMRI/PET data be combined? Hum. Brain Map. 2002;17:1–3. [PubMed]
20. Hampton T. European-led project strives to simulate the human brain. JAMA. 2014 Apr 23-30;311:1598–1600. [PubMed]
21. Sui J, Adalı T, Yu Q, Calhoun VD. A Review of Multivariate Methods for Multimodal Fusion of Brain Imaging Data. Journal of Neuroscience Methods. 2012;204:68–81. PMC3690333. [PMC free article] [PubMed]
22. Sui J, Huster R, Yu QB, Segall JM, Calhoun VD. Function-structure associations of the brain: Evidence from multimodal connectivity and covariance studies. Neuroimage. 2014 Nov 15;102:11–23. PMC3969780. [PMC free article] [PubMed]
23. Wang F, Kalmar JH, He Y, Jackowski M, Chepenik LG, Edmiston EE, Tie K, Gong G, Shah MP, Jones M, Uderman J, Constable RT, Blumberg HP. Functional and structural connectivity between the perigenual anterior cingulate and amygdala in bipolar disorder. Biol Psychiatry. 2009 Sep 1;66:516–521. [PMC free article] [PubMed]
24. Blum AL, Langley P. Selection of relevant features and examples in machine learning. Artificial Intelligence. 1997;97:245–271.
25. Garrity A, Pearlson GD, McKiernan K, Lloyd D, Kiehl KA, Calhoun VD. Aberrant ‘default mode’ functional connectivity in schizophrenia. Am.J.Psychiatry. 2007;164:450–457. [PubMed]
26. Schultz CC, Fusar-Poli P, Wagner G, Koch K, Schachtzabel C, Gruber O, Sauer H, Schlosser RG. Multimodal functional and structural imaging investigations in psychosis research. Eur Arch Psychiatry Clin Neurosci. 2012 Nov;262(Suppl 2):S97–106. [PubMed]
27. Smith SM, Fox PT, Miller KL, Glahn DC, Fox PM, Mackay CE, Filippini N, Watkins KE, Toro R, Laird AR, Beckmann CF. Correspondence of the brain's functional architecture during activation and rest. Proceedings of the National Academy of Sciences of the United States of America. 2009 Aug 4;106:13040–13045. 2722273. [PubMed]
28. Calhoun VD, Allen E. Extracting intrinsic functional networks with feature-based group independent component analysis. Psychometrika. 2013 Apr;78:243–259. PMC Journal -In Process. [PubMed]
29. Meda S, Giuliani N, Calhoun VD, Jagannathan K, Schretlen D, Pulver A, Cascella N, Keshavan M, Kates W, Buchanan RJ, Sharma T, Pearlson G. A large scale (N=400) investigation of gray matter differences in schizophrenia using optimized voxel-based morphometry. Schizophrenia Research. 2008;101:95–105. PMC pending #163012. [PMC free article] [PubMed]
30. Glahn DC, Winkler AM, Kochunov P, Almasy L, Duggirala R, Carless MA, Curran JC, Olvera RL, Laird AR, Smith SM, Beckmann CF, Fox PT, Blangero J. Genetic control over the resting brain. Proc Natl Acad Sci U S A. 2010 Jan 19;107:1223–1228. 2824276. [PubMed]
31. De Martino F, Valente G, de Borst AW, Esposito F, Roebroeck A, Goebel R, Formisano E. Multimodal imaging: an evaluation of univariate and multivariate methods for simultaneous EEG/fMRI. Magn Reson Imaging. 2010 Oct;28:1104–1112. [PubMed]
32. Eichele T, Specht K, Moosmann M, Jongsma ML, Quiroga RQ, Nordby H, Hugdahl K. Assessing the spatiotemporal evolution of neuronal activation with single-trial event-related potentials and functional MRI. Proc Natl Acad Sci U S A. 2005 Dec 6;102:17798–17803. [PubMed]
33. Astolfi L, Cincotti F, Mattia D, Salinari S, Babiloni C, Basilisco A, Rossini PM, Ding L, Ni Y, He B, Marciani MG, Babiloni F. Estimation of the effective and functional human cortical connectivity with structural equation modeling and directed transfer function applied to high-resolution EEG. Magn Reson Imaging. 2004 Dec;22:1457–1470. [PubMed]
34. Rykhlevskaia E, Gratton G, Fabiani M. Combining structural and functional neuroimaging data for studying brain connectivity: a review. Psychophysiology. 2008 Mar;45:173–187. [PubMed]
35. Oakes TR, Fox AS, Johnstone T, Chung MK, Kalin N, Davidson RJ. Integrating VBM into the General Linear Model with voxelwise anatomical covariates. Neuroimage. 2007 Jan 15;34:500–508. 2586764. [PMC free article] [PubMed]
36. Schlosser R, Gesierich T, Kaufmann B, Vucurevic G, Hunsche S, Gawehn J, Stoeter P. Altered effective connectivity during working memory performance in schizophrenia: a study with fMRI and structural equation modeling. Neuroimage. 2003;19:751–763. [PubMed]
37. Calhoun VD, Adalı T, Giuliani N, Pekar JJ, Pearlson GD, Kiehl KA. A Method for Multimodal Analysis of Independent Source Differences in Schizophrenia: Combining Gray Matter Structural and Auditory Oddball Functional Data. Hum.Brain Map. 2006;27:47–62. [PubMed]
38. Correa N, Adalı T, Calhoun VD. Performance of Blind Source Separation Algorithms for fMRI Analysis. Mag.Res.Imag. 2007;25:684. PMC2358930. [PMC free article] [PubMed]
39. Groves AR, Beckmann CF, Smith SM, Woolrich MW. Linked independent component analysis for multimodal data fusion. Neuroimage. 2011 Feb 1;54:2198–2217. [PubMed]
40. Sui J, He H, Pearlson GD, Adali T, Kiehl KA, Yu Q, Clark VP, Castro E, White T, Mueller BA, Ho BC, Andreasen NC, Calhoun VD. Three-way (N-way) fusion of brain imaging data based on mCCA+jICA and its application to discriminating schizophrenia. Neuroimage. 2013 Feb 1;66:119–132. 3897558. [PMC free article] [PubMed]
41. Sui J, Pearlson GD, Adalı T, Kiehl KA, Caprihan A, Liu J, Yamamoto J, Calhoun VD. Discriminating Schizophrenia and Bipolar Disorder by Fusing FMRI and DTI in A Multimodal CCA+Joint ICA Based Model. NeuroImage. 2011;57:839–855. PMC Pending #297883. [PMC free article] [PubMed]
42. Chen K, Reiman EM, Huan Z, Caselli RJ, Bandy D, Ayutyanont N, Alexander GE. Linking functional and structural brain images with multivariate network analyses: a novel application of the partial least square method. Neuroimage. 2009 Aug 15;47:602–610. 2700206. [PMC free article] [PubMed]
43. Martinez-Montes E, Valdes-Sosa PA, Miwakeichi F, Goldman RI, Cohen MS. Concurrent EEG/fMRI analysis by multiway Partial Least Squares. NeuroImage. 2004;22:1023–1034. [PubMed]
44. Cetin MS, Christensen F, Abbott CC, Stephen JM, Mayer AR, Canive JM, Bustillo JR, Pearlson GD, Calhoun VD. Thalamus and posterior temporal lobe show greater inter-network connectivity at rest and across sensory paradigms in schizophrenia. Neuroimage. 2014 Aug 15;97:117–126. 4087193. [PMC free article] [PubMed]
45. Correa N, Eichele T, Adalı T, Li Y, Calhoun VD. Multi-set canonical correlation analysis for the fusion of concurrent single trial ERP and functional MRI. NeuroImage. 2010;50:1438–1445. PMC pending #180189. [PMC free article] [PubMed]
46. Liu J, Pearlson GD, Windemuth A, Ruano G, Perrone-Bizzozero NI, Calhoun VD. Combining fMRI and SNP data to investigate connections between brain function and genetics using parallel ICA. Hum.Brain Map. 2009;30:241–255. PMC2668960. [PMC free article] [PubMed]
47. Sui J, Adalı T, Clark VP, Pearlson G, Calhoun VD. A Method for Accurate Group Difference Detection by Constraining the Mixing Coefficients in an ICA Framework. Human Brain Mapping. 2009;30:2953–2970. PMC2733923. [PMC free article] [PubMed]
48. Sui J, Adalı T, Pearlson G, Calhoun VD. An ICA-based Method for the Identification of Optimal FMRI Features and Components Using Combined Group-Discriminative Techniques. NeuroImage. 2009;46:73–86. PMC pending #95972. [PMC free article] [PubMed]
49. Sui J, Li Y, Adalı T, Calhoun VD. Proc. HBM. San Francisco, CA: 2009. A New Joint Blind Source Separation Model for Two Datasets and Its Application to Second-level FMRI Group Analysis.
50. Caprihan A, Pearlson GD, Calhoun VD. Application of Principal Component Analysis to Distinguish Patients with Schizophrenia from Healthy Controls Based on Fractional Anisotropy Measurements. NeuroImage. 2008;42:675–682. PMC2566788. [PMC free article] [PubMed]
51. Liu J, Xu L, Caprihan A, Calhoun V. Extracting Principle Components for Discriminant Analysis of fMRI Images. Proc. ICASSP. 2008 [PMC free article] [PubMed]
52. Segall JM, Allen EA, Jung RE, Erhardt EB, Arja SK, Kiehl K, Calhoun VD. Correspondence between structure and function in the human brain at rest. Frontiers in neuroinformatics. 2012;6:10. 3313067. [PMC free article] [PubMed]
53. Khullar S, Michael AM, Cahill ND, Kiehl KA, Pearlson G, Baum SA, Calhoun VD. ICA-fNORM: Spatial Normalization of fMRI Data Using Intrinsic Group-ICA Networks. Front Syst Neurosci. 2011;5:93. 3218372. [PMC free article] [PubMed]
54. Salgado-Pineda P, Junque C, Vendrell P, Baeza I, Bargallo N, Falcon C, Bernardo M. Decreased cerebral activation during CPT performance: structural and functional deficits in schizophrenic patients. Neuroimage. 2004 Mar;21:840–847. [PubMed]
55. Salgado-Pineda P, Fakra E, Delaveau P, McKenna PJ, Pomarol-Clotet E, Blin O. Correlated structural and functional brain abnormalities in the default mode network in schizophrenia patients. Schizophr Res. 2011 Feb;125:101–109. [PubMed]
56. Phillips ML, Swartz HA. A critical appraisal of neuroimaging studies of bipolar disorder: toward a new conceptualization of underlying neural circuitry and a road map for future research. Am J Psychiatry. 2014 Aug;171:829–843. 4119497. [PMC free article] [PubMed]
57. Sasamoto A, Miyata J, Kubota M, Hirao K, Kawada R, Fujimoto S, Tanaka Y, Hazama M, Sugihara G, Sawamoto N, Fukuyama H, Takahashi H, Murai T. Global association between cortical thinning and white matter integrity reduction in schizophrenia. Schizophr Bull. 2014 Mar;40:420–427. 3932083. [PMC free article] [PubMed]
58. Liu X, Lai Y, Wang X, Hao C, Chen L, Zhou Z, Yu X, Hong N. A combined DTI and structural MRI study in medicated-naive chronic schizophrenia. Magn Reson Imaging. 2014 Jan;32:1–8. [PubMed]
59. Bullmore E, Sporns O. The economy of brain network organization. Nat Rev Neurosci. 2012 May;13:336–349. [PubMed]
60. van den Heuvel MP, Sporns O, Collin G, Scheewe T, Mandl RC, Cahn W, Goni J, Hulshoff Pol HE, Kahn RS. Abnormal rich club organization and functional brain dynamics in schizophrenia. JAMA Psychiatry. 2013 Aug;70:783–792. [PubMed]
61. Crow TJ. Schizophrenia as a transcallosal misconnection syndrome. Schizophr Res. 1998 Mar 10;30:111–114. [PubMed]
62. Miyata J, Hirao K, Namiki C, Fukuyama H, Okada T, Miki Y, Hayashi T, Murai T. Interfrontal commissural abnormality in schizophrenia: tractography-assisted callosal parcellation. Schizophr Res. 2007 Dec;97:236–241. [PubMed]
63. Koch K, Wagner G, Schachtzabel C, Schultz CC, Gullmar D, Reichenbach JR, Sauer H, Schlosser RG. Neural activation and radial diffusivity in schizophrenia: combined fMRI and diffusion tensor imaging study. Br J Psychiatry. 2011 Mar;198:223–229. [PubMed]
64. Friston KJ. The disconnection hypothesis. Schizophr.Res. 1998;30:115–125. [PubMed]
65. Liu H, Fan G, Xu K, Wang F. Changes in cerebellar functional connectivity and anatomical connectivity in schizophrenia: a combined resting-state functional MRI and diffusion tensor imaging study. J Magn Reson Imaging. 2011 Dec;34:1430–1438. 3221764. [PMC free article] [PubMed]
66. Camchong J, MacDonald AW, 3rd, Bell C, Mueller BA, Lim KO. Altered functional and anatomical connectivity in schizophrenia. Schizophr Bull. 2011 May;37:640–650. 3080691. [PMC free article] [PubMed]
67. Schlosser RG, Nenadic I, Wagner G, Gullmar D, von Consbruch K, Kohler S, Schultz CC, Koch K, Fitzek C, Matthews PM, Reichenbach JR, Sauer H. White matter abnormalities and brain activation in schizophrenia: A combined DTI and fMRI study. Schizophr Res. 2007;89:1–11. [PubMed]
68. Staempfli P, Reischauer C, Jaermann T, Valavanis A, Kollias S, Boesiger P. Combining fMRI and DTI: a framework for exploring the limits of fMRI-guided DTI fiber tracking and for verifying DTI-based fiber tractography results. Neuroimage. 2008 Jan 1;39:119–126. [PubMed]
69. Isobe M, Miyata J, Hazama M, Fukuyama H, Murai T, Takahashi H. Multimodal neuroimaging as a window into the pathological physiology of schizophrenia: Current trends and issues. Neurosci Res. 2015 Jul 30; [PubMed]
70. Skudlarski P, Jagannathan K, Anderson K, Stevens MC, Calhoun VD, Skudlarska BA, Pearlson G. Brain connectivity is not only lower but different in schizophrenia: a combined anatomical and functional approach. Biol Psychiatry. 2010 Jul 1;68:61–69. 2900394. [PMC free article] [PubMed]
71. Sui J, Yu Q, He H, Pearlson GD, Calhoun VD. A selective review of multimodal fusion methods in schizophrenia. Front Hum Neurosci. 2012;6:27. 3285795. [PMC free article] [PubMed]
72. Sui J, Pearlson GD, Du Y, Yu Q, Jones TR, Chen J, Jiang T, Bustillo J, Calhoun VD. In Search of Multimodal Neuroimaging Biomarkers of Cognitive Deficits in Schizophrenia. Biol Psychiatry. 2015 Feb 24; [PMC free article] [PubMed]
73. Marder SR, Fenton W. Measurement and Treatment Research to Improve Cognition in Schizophrenia: NIMH MATRICS initiative to support the development of agents for improving cognition in schizophrenia. Schizophr Res. 2004 Dec 15;72:5–9. [PubMed]
74. Correa NM, Eichele T, Adali T, Li YO, Calhoun VD. Multi-set canonical correlation analysis for the fusion of concurrent single trial ERP and functional MRI. Neuroimage. 2010 May 1;50:1438–1445. 2857695. [PMC free article] [PubMed]
75. Kiehl KA, Liddle PF. An event-related functional magnetic resonance imaging study of an auditory oddball task in schizophrenia. Schizophr Res. 2001 Mar 30;48:159–171. [PubMed]
76. Sugranyes G, Kyriakopoulos M, Dima D, O'Muircheartaigh J, Corrigall R, Pendelbury G, Hayes D, Calhoun VD, Frangou S. Multimodal analyses identify linked functional and white matter abnormalities within the working memory network in schizophrenia. Schizophr Res. 2012 Jul;138:136–142. [PMC free article] [PubMed]
77. Stephen JM, Coffman BA, Jung RE, Bustillo JR, Aine CJ, Calhoun VD. Using joint ICA to link function and structure using MEG and DTI in schizophrenia. Neuroimage. 2013 Dec;83:418–430. 3815989. [PMC free article] [PubMed]
78. Xu L, Pearlson G, Calhoun VD. Joint source based morphometry identifies linked gray and white matter group differences. Neuroimage. 2009 Feb 1;44:777–789. 2669793. [PMC free article] [PubMed]
79. Sui J, He H, Liu J, Yu Q, Adali T, Pearlson GD, Calhoun VD. Three-way FMRI-DTI-methylation data fusion based on mCCA+jICA and its application to schizophrenia. Conf Proc IEEE Eng Med Biol Soc. 20122012:2692–2695. [PubMed]
80. Sui J, He H, Yu Q, Chen J, Mayer A, Calhoun VD. Combination of Resting state fMRI, DTI & sMRI Data to Discriminate Schizophrenia by N-way MCCA+jICA. Organization for Human Brain Mapping Annual Meeting Seattle; USA. 2013. [PMC free article] [PubMed]
81. Michael AM, Baum SA, White T, Demirci O, Andreasen NC, Segall JM, Jung RE, Pearlson G, Clark VP, Gollub RL, Schulz SC, Roffman JL, Lim KO, Ho BC, Bockholt HJ, Calhoun VD. Does function follow form?: methods to fuse structural and functional brain images show decreased linkage in schizophrenia. Neuroimage. 2010 Feb 1;49:2626–2637. 2911118. [PMC free article] [PubMed]
82. Schroder J, Essig M, Baudendistel K, Jahn T, Gerdsen I, Stockert A, Schad LR, Knopp MV. Motor dysfunction and sensorimotor cortex activation changes in schizophrenia: A study with functional magnetic resonance imaging. NeuroImage. 1999;9:81–87. [PubMed]
83. Manoach DS, Press DZ, Thangaraj V, Searl MM, Goff DC, Halpern E, Saper CB, Warach S. Schizophrenic subjects activate dorsolateral prefrontal cortex during a working memory task, as measured by fMRI. Biol Psychiatry. 1999 May 1;45:1128–1137. [PubMed]
84. Cooper D, Barker V, Radua J, Fusar-Poli P, Lawrie SM. Multimodal voxel-based meta-analysis of structural and functional magnetic resonance imaging studies in those at elevated genetic risk of developing schizophrenia. Psychiatry Res. 2014 Jan 30;221:69–77. [PubMed]
85. Oertel-Knochel V, Reinke B, Feddern R, Knake A, Knochel C, Prvulovic D, Pantel J, Linden DE. Episodic memory impairments in bipolar disorder are associated with functional and structural brain changes. Bipolar Disord. 2014 Dec;16:830–845. [PubMed]
86. Knochel C, Stablein M, Storchak H, Reinke B, Jurcoane A, Prvulovic D, Linden DE, van de Ven V, Ghinea D, Wenzler S, Alves G, Matura S, Kroger A, Oertel-Knochel V. Multimodal assessments of the hippocampal formation in schizophrenia and bipolar disorder: evidences from neurobehavioral measures and functional and structural MRI. Neuroimage Clin. 2014;6:134–144. 4215399. [PMC free article] [PubMed]
87. Sui J, Pearlson G, Caprihan A, Adali T, Kiehl KA, Liu J, Yamamoto J, Calhoun VD. Discriminating schizophrenia and bipolar disorder by fusing fMRI and DTI in a multimodal CCA+ joint ICA model. Neuroimage. 2011 Aug 1;57:839–855. 3129373. [PMC free article] [PubMed]
88. Vasic N, Wolf ND, Gron G, Sosic-Vasic Z, Connemann BJ, Sambataro F, von Strombeck A, Lang D, Otte S, Dudek M, Wolf RC. Baseline brain perfusion and brain structure in patients with major depression: a multimodal magnetic resonance imaging study. J Psychiatry Neurosci. 2015 Jun 30;40:140246.
89. Dutt A, Ganguly T, Shaikh M, Walshe M, Schulze K, Marshall N, Constante M, McDonald C, Murray RM, Allin MP, Bramon E. Association between hippocampal volume and P300 event related potential in psychosis: support for the Kraepelinian divide. Neuroimage. 2012 Jan 16;59:997–1003. [PubMed]
90. de Kwaasteniet B, Ruhe E, Caan M, Rive M, Olabarriaga S, Groefsema M, Heesink L, van Wingen G, Denys D. Relation between structural and functional connectivity in major depressive disorder. Biol Psychiatry. 2013 Jul 1;74:40–47. [PubMed]
91. Han KM, Choi S, Jung J, Na KS, Yoon HK, Lee MS, Ham BJ. Cortical thickness, cortical and subcortical volume, and white matter integrity in patients with their first episode of major depression. J Affect Disord. 2014 Feb;155:42–48. [PubMed]
92. Kim SG, Jung WH, Kim SN, Jang JH, Kwon JS. Alterations of Gray and White Matter Networks in Patients with Obsessive-Compulsive Disorder: A Multimodal Fusion Analysis of Structural MRI and DTI Using mCCA+jICA. PLoS One. 2015;10:e0127118. 4454537. [PMC free article] [PubMed]
93. Itahashi T, Yamada T, Nakamura M, Watanabe H, Yamagata B, Jimbo D, Shioda S, Kuroda M, Toriizuka K, Kato N, Hashimoto R. Linked alterations in gray and white matter morphology in adults with high-functioning autism spectrum disorder: a multimodal brain imaging study. Neuroimage Clin. 2015;7:155–169. 4299973. [PMC free article] [PubMed]
94. Zhang H, Liu L, Wu H, Fan Y. OHBM. Beijing, China: 2012. Feature selection and SVM classification of multiple modality images for predicting MCI.
95. Kim D, Sui J, Rachakonda S, White T, Manoach DS, Clark VP, Ho BC, Schulz SC, Calhoun VD. Identification of imaging biomarkers in schizophrenia: A coefficient-constrained independent component analysis of the Mind multi-site schizophrenia study. Journal of NeuroInformatics. 2010;8:213–229. PMC Pending #278802. [PMC free article] [PubMed]
96. Silva R, Castro E, Gupta N, Cetin M, Arbabshirani M, Potluru V, Plis SM, Calhoun VD. IEEE International Workshop on Machine Learning for Signal Processing. Reims, France: 2014. The Tenth Annual MLSP Competition: Schizophrenia Classification Challenge.
97. Dai D, Wang J, Hua J, He H. Classification of ADHD children through multimodal magnetic resonance imaging. Front Syst Neurosci. 2012;6:63. 3432508. [PMC free article] [PubMed]
98. Sui J, Castro E, Hao H, Bridwell D, Du Y, Pearlson GD, Calhoun VD. the 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC'14) Chicago, Illinois, USA: 2014. Combination of FMRI-SMRI-EEG Data Improves Discrimination of Schizophrenia Patients by Ensemble Feature Selection. [PubMed]
99. Schmaal L, Marquand AF, Rhebergen D, van Tol MJ, Ruhe HG, van der Wee NJ, Veltman DJ, Penninx BW. Predicting the Naturalistic Course of Major Depressive Disorder Using Clinical and Multimodal Neuroimaging Information: A Multivariate Pattern Recognition Study. Biol Psychiatry. 2015 Aug 15;78:278–286. 4449319. [PMC free article] [PubMed]
100. Pettersson-Yeo W, Benetti S, Marquand AF, Joules R, Catani M, Williams SC, Allen P, McGuire P, Mechelli A. An empirical comparison of different approaches for combining multimodal neuroimaging data with support vector machine. Front Neurosci. 2014;8:189. 4097812. [PMC free article] [PubMed]
101. Plis SM, Hjelm RD, Salakhutdinov RR, Calhoun VD. Proceedings of the Organization of Human Brain Mapping. Hamburg, Germany: 2014. Deep learning models for brain imaging: model depth enhances discovery power.
102. Plis SM, Hjelm DR, Salakhutdinov R, Allen EA, Bockholt HJ, Long JD, Johnson HJ, Paulsen JS, Turner JA, Calhoun VD. Deep learning for neuroimaging: a validation study. Front Neurosci. 2014 Aug 20;8:229. 4138493. [PMC free article] [PubMed]
103. Castro E, Ulloa A, Plis SM, Turner J, Calhoun VD. IEEE International Symposium on Biomedical Imaging. New York, NY: 2015. Simulation of structural magnetic resonance images for deep learning pre-training.
104. Brosch T, Tam R, N. Initiative for the Alzheimers Disease Manifold learning of brain MRIs by deep learning. Med Image Comput Comput Assist Interv. 2013;16:633–640. [PubMed]
105. Wu G, Kim M, Wang Q, Gao Y, Liao S, Shen D. Unsupervised deep feature learning for deformable registration of MR brain images. Med Image Comput Comput Assist Interv. 2013;16:649–656. 4073478. [PMC free article] [PubMed]
106. Kim J, Calhoun VD, Shim E, Lee J-H. Deep neural network with weight sparsity control and pre-training extracts hierarchical features and enhances classification performance: Evidence from whole-brain resting-state functional connectivity patterns of schizophrenia. NeuroImage. in press, PMC Journal - In Process. [PMC free article] [PubMed]
107. Hjelm D, Calhoun VD, Salakhutdinov RR, Allen E, Adalı T, Plis SM. Restricted Bolzmann machines for neuroimaging: an application in identifying intrinsic networks. NeuroImage. in press, PMC Journal - In Process. [PMC free article] [PubMed]
108. Hjelm D, Castro E, Plis SM, Calhoun VD. Annual Meeting of the Organization for Human Brain Mapping. Honolulu, HI: 2015. Nonlinear ICA of structural MRI data: A deep learning approach.
109. Castro E, Hjelm D, Plis S, Dinh L, Turner J, Calhoun VD. Independent component estimation of simulated structural magnetic resonance imaging data using deep learning. IEEE Machine Learning for Signal Processing Workshop; Boston, MA. 2015.
110. Deng L, Yu D. Deep learning: methods and applications. Tech. Rep. MSR-TR-2014-21. 2014
111. Srivastava N, Salakhutdinov RR. Multimodal learning with deep Boltzmann machines. Advances in Neural Information Processing Systems. 2012:2231–2239.
112. Amin F, Plis S, Damaraju E, Hjelm D, Cho K, Calhoun VD. Pattern Recognition in NeuroImaging (PRNI) Palo Alto, CA: 2015. A deep-learning approach to translate between brain structure and brain function.
113. Sui J, Yu Q, He H, Calhoun VD. A Selective Review of Multimodal Fusion Methods in Schizophrenia. Frontiers in Human Neuroscience. 2013;6 PMC3285795. [PMC free article] [PubMed]
114. Friston KJ. Modalities, modes, and models in functional neuroimaging. Science. 2009 Oct 16;326:399–403. [PubMed]
115. Calhoun VD, Pearlson GD, Kiehl KA. Neuronal Chronometry of Target Detection: Fusion of Hemodynamic and Event-related Potential Data. NeuroImage. 2006;30:544–553. [PubMed]
116. Xu L, Groth K, Pearlson G, Schretlen D, Calhoun V. Source Based Morphometry: The Use of Independent Component Analysis to Identify Gray Matter Differences with Application to Schizophrenia. Human Brain Mapping. 2009;30:711–724. PMC2751641. [PMC free article] [PubMed]
117. Franco AR, Ling J, Caprihan A, Calhoun VD, Jung R, Heileman GL, Mayer AR. Multimodal and Multi-tissue Measures of Connectivity Revealed by Joint Independent Component Analysis. IEEE JSTSP. 2008;2:986–997. PMC2748354. [PMC free article] [PubMed]
118. Teipel SJ, Bokde AL, Meindl T, Amaro E, Jr., Soldner J, Reiser MF, Herpertz SC, Moller HJ, Hampel H. White matter microstructure underlying default mode network connectivity in the human brain. Neuroimage. 2010 Feb 1;49:2021–2032. [PubMed]
119. Eichele T, Calhoun VD, Debener S. Mining EEG-fMRI using independent component analysis. Int. J. Psych. 2009;73:53–61. PMC2693483. [PMC free article] [PubMed]
120. Calhoun VD, Adalı T, Kiehl KA, Astur RS, Pekar JJ, Pearlson GD. A Method for Multi-task fMRI Data Fusion Applied to Schizophrenia. Human Brain Mapping. 2006;27:598–610. PMC2751648. [PMC free article] [PubMed]
121. Correa N, Li Y, Adalı T, Calhoun VD. Canonical correlation analysis for feature-based fusion of biomedical imaging modalities to detect associative networks in Schizophrenia. IEEE JSTSP. 2008;2:998–1007. PMC2761661. [PMC free article] [PubMed]
122. Sui J, Adalı T, Pearlson G, Yang H, Sponheim S, White T, Calhoun VD. A CCA+ICA Based Model for Multi-Task Brain Imaging Data Fusion And Its Application to Schizophrenia. NeuroImage. 2010;51:123–134. PMC pending #180309. [PMC free article] [PubMed]
123. Li Y, Adalı T, Wang W, Calhoun VD. Joint Blind Source Separation by Multi-set Canonical Correlation Analysis. IEEE Trans. Signal Processing. 2009;57:3918–3929. PMC pending #110331. [PMC free article] [PubMed]
124. Lin FH, McIntosh AR, Agnew JA, Eden GF, Zeffiro TA, Belliveau JW. Multivariate analysis of neuronal interactions in the generalized partial least squares framework: simulations and empirical studies. Neuroimage. 2003 Oct;20:625–642. [PubMed]
125. Krishnan A, Williams LJ, McIntosh AR, Abdi H. Partial Least Squares (PLS) methods for neuroimaging: A tutorial and review. Neuroimage. 2010 May 15;56:455–475. [PubMed]
126. Correa NM, Adali T, Li YO, Calhoun VD. Canonical Correlation Analysis for Data Fusion and Group Inferences: Examining applications of medical imaging data. IEEE Signal Process Mag. 2010;27:39–50. 2919827. [PMC free article] [PubMed]
127. McIntosh AR, Lobaugh NJ. Partial least squares analysis of neuroimaging data: applications and advances. Neuroimage. 2004;23(Suppl 1):S250–263. [PubMed]
128. Camara E, Rodriguez-Fornells A, Munte TF. Microstructural brain differences predict functional hemodynamic responses in a reward processing task. J Neurosci. 2010 Aug 25;30:11398–11402. [PubMed]
129. Sui J, He H, Yu Q, Chen J, Rogers JT, Pearlson GD, Mayer AR, Bustillo J, Canive J, Calhoun VD. Combination of Resting state fMRI, DTI and sMRI Data to Discriminate Schizophrenia by N-way MCCA+jICA. Frontiers in Human Neuroscience. 2013;7 PMC3666029. [PMC free article] [PubMed]
130. Beckmann CF, Smith SM. Tensorial extensions of independent component analysis for multisubject FMRI analysis. NeuroImage. 2005;25:294–311. [PubMed]
131. Liu J, Calhoun VD. A review of multivariate analyses in imaging genetics. Front Neuroinform. 2014;8:29. 3972473. [PMC free article] [PubMed]
132. Liu J, Pearlson G, Windemuth A, Ruano G, Perrone-Bizzozero NI, Calhoun V. Combining fMRI and SNP data to investigate connections between brain function and genetics using parallel ICA. Hum Brain Mapp. 2009 Jan;30:241–255. [PMC free article] [PubMed]
133. Vergara VM, Ulloa A, Calhoun VD, Boutte D, Chen J, Liu J. A three-way parallel ICA approach to analyze links among genetics, brain structure and brain function. Neuroimage. 2014 Sep;98:386–394. 4141686. [PMC free article] [PubMed]
134. Chen J, Calhoun VD, Pearlson GD, Perrone-Bizzozero N, Sui J, Turner JA, Bustillo JR, Ehrlich S, Sponheim SR, Canive JM, Ho BC, Liu J. Guided exploration of genomic risk for gray matter abnormalities in schizophrenia using parallel independent component analysis with reference. Neuroimage. 2013 Dec;83:384–396. 3797233. [PMC free article] [PubMed]
135. Chen J, Calhoun VD, Ulloa AE, Liu J. Parallel ICA with multiple references: a semi-blind multivariate approach. Conf Proc IEEE Eng Med Biol Soc. 20142014:6659–6662. [PMC free article] [PubMed]
136. Tian L, Meng C, Yan H, Zhao Q, Liu Q, Yan J, Han Y, Yuan H, Wang L, Yue W, Zhang Y, Li X, Zhu C, He Y, Zhang D. Convergent evidence from multimodal imaging reveals amygdala abnormalities in schizophrenic patients and their first-degree relatives. PLoS One. 2011;6:e28794. 3234284. [PMC free article] [PubMed]
137. Kim J, Lee JH. Integration of structural and functional magnetic resonance imaging improves mild cognitive impairment detection. Magn Reson Imaging. 2012 Dec 20; [PubMed]
138. Segall J, Allen EA, Jung RE, Erhardt E, Arja S, Kiehl KA, Calhoun VD. Correspondence between Structure and Function in the Human Brain at Rest. Frontiers in Neuroinformatics. 2012;6 PMC3313067. [PMC free article] [PubMed]
139. Casey BJ, Tottenham N, Liston C, Durston S. Imaging the developing brain: what have we learned about cognitive development? Trends Cogn Sci. 2005 Mar;9:104–110. [PubMed]
140. Smieskova R, Allen P, Simon A, Aston J, Bendfeldt K, Drewe J, Gruber K, Gschwandtner U, Klarhoefer M, Lenz C, Scheffler K, Stieglitz RD, Radue EW, McGuire P, Riecher-Rossler A, Borgwardt SJ. Different duration of at-risk mental state associated with neurofunctional abnormalities. A multimodal imaging study. Hum Brain Mapp. 2012 Oct;33:2281–2294. [PubMed]
141. Rasser PE, Johnston P, Lagopoulos J, Ward PB, Schall U, Thienel R, Bender S, Toga AW, Thompson PM. Functional MRI BOLD response to Tower of London performance of first-episode schizophrenia patients using cortical pattern matching. Neuroimage. 2005 Jul 1;26:941–951. [PubMed]
142. Fusar-Poli P, Broome MR, Woolley JB, Johns LC, Tabraham P, Bramon E, Valmaggia L, Williams SC, McGuire P. Altered brain function directly related to structural abnormalities in people at ultra high risk of psychosis: longitudinal VBM-fMRI study. Journal of psychiatric research. 2011 Feb;45:190–198. [PubMed]
143. Michael A, Baum S, White T, Demirci O, Andreasen NC, Segall JM, Jung RE, Pearlson GD, Clark VP, Gollub RL, Schulz SC, Roffmann J, Lim KO, Ho BC, Bockholt HJ, Calhoun VD. Does Function Follow Form?: Methods to Fuse Structural and Functional Brain Images Show Decreased Linkage in Schizophrenia. Human Brain Mapping. 2010;49:2626–2637. PMC pending #184511. [PMC free article] [PubMed]
144. Michael A, King MD, Ehrlich S, Pearlson G, White T, Holt D, Andreasen NC, Sakoglu U, Ho BC, Schulz SC, Calhoun VD. A data-driven investigation of gray matter–function correlations in schizophrenia during a working memory task. Frontiers in Human Neuroscience. 2011;5:1–14. PMC Journal - In Process. [PMC free article] [PubMed]
145. Rektorova I, Mikl M, Barrett J, Marecek R, Rektor I, Paus T. Functional neuroanatomy of vocalization in patients with Parkinson's disease. J Neurol Sci. 2012 Feb 15;313:7–12. [PubMed]
146. Smieskova R, Allen P, Simon A, Aston J, Bendfeldt K, Drewe J, Gruber K, Gschwandtner U, Klarhoefer M, Lenz C, Scheffler K, Stieglitz RD, Radue EW, McGuire P, Riecher-Rossler A, Borgwardt SJ. Different duration of at-risk mental state associated with neurofunctional abnormalities. A multimodal imaging study. Hum Brain Mapp. 2011 Oct;33:2281–2294. [PubMed]
147. Harms MP, Wang L, Csernansky JG, Barch DM. Structure-function relationship of working memory activity with hippocampal and prefrontal cortex volumes. Brain Struct Funct. 2012 Jan;218:173–186. [PMC free article] [PubMed]
148. Choi K, Yang Z, Hu X, Mayberg H. A Combined Functional-Structural Connectivity Analysis of Major Depression Using Joint Independent Components Analysis. ISMRM, Toronto,Cananda. 2008:3555.
149. Wee CY, Yap PT, Zhang D, Denny K, Browndyke JN, Potter GG, Welsh-Bohmer KA, Wang L, Shen D. Identification of MCI individuals using structural and functional connectivity networks. Neuroimage. 2011 Feb 1;59:2045–2056. 3254811. [PMC free article] [PubMed]
150. Olesen PJ, Nagy Z, Westerberg H, Klingberg T. Combined analysis of DTI and fMRI data reveals a joint maturation of white and grey matter in a fronto-parietal network. Brain Res Cogn Brain Res. 2003 Dec;18:48–57. [PubMed]
151. Matthews SC, Strigo IA, Simmons AN, O'Connell RM, Reinhardt LE, Moseley SA. A multimodal imaging study in U.S. veterans of Operations Iraqi and Enduring Freedom with and without major depression after blast-related concussion. NeuroImage. 2011 Jan;54(Suppl 1):S69–75. [PubMed]
152. Skudlarski P, Jagannathan KA, Anderson K, Stevens MC, Calhoun VD, Pearlson GD. Brain connectivity is not only lower but also different in schizophrenia: a combined anatomical and functional approach. Biological Psychiatry. 2010;68:61–69. PMC Pending #193102. [PMC free article] [PubMed]
153. Zhou Y, Shu N, Liu Y, Song M, Hao Y, Liu H, Yu C, Liu Z, Jiang T. Altered resting-state functional connectivity and anatomical connectivity of hippocampus in schizophrenia. Schizophr Res. 2008 Mar;100:120–132. [PubMed]
154. Yan H, Tian L, Yan J, Sun W, Liu Q, Zhang YB, Li XM, Zang YF, Zhang D. Functional and anatomical connectivity abnormalities in cognitive division of anterior cingulate cortex in schizophrenia. PLoS One. 2012;7:e45659. 3458074. [PMC free article] [PubMed]
155. Soldner J, Meindl T, Koch W, Bokde AL, Reiser MF, Moller HJ, Burger K, Hampel H, Teipel SJ. [Structural and functional neuronal connectivity in Alzheimer's disease: a combined DTI and fMRI study] Nervenarzt. 2011 Jul;83:878–887. [PubMed]
156. Schonberg T, Pianka P, Hendler T, Pasternak O, Assaf Y. Characterization of displaced white matter by brain tumors using combined DTI and fMRI. Neuroimage. 2006 May 1;30:1100–1111. [PubMed]
157. Voss HU, Schiff ND. MRI of neuronal network structure, function, and plasticity. Prog Brain Res. 2009;175:483–496. [PubMed]
158. Palacios EM, Sala-Llonch R, Junque C, Roig T, Tormos JM, Bargallo N, Vendrell P. White matter integrity related to functional working memory networks in traumatic brain injury. Neurology. 2012 Mar 20;78:852–860. [PubMed]
159. Jacobson S, Kelleher I, Harley M, Murtagh A, Clarke M, Blanchard M, Connolly C, O'Hanlon E, Garavan H, Cannon M. Structural and functional brain correlates of subclinical psychotic symptoms in 11-13 year old schoolchildren. Neuroimage. 2009 Jan 15;49:1875–1885. [PubMed]
160. Supekar K, Uddin LQ, Prater K, Amin H, Greicius MD, Menon V. Development of functional and structural connectivity within the default mode network in young children. Neuroimage. 2010 Aug 1;52:290–301. 2976600. [PMC free article] [PubMed]
161. Pomarol-Clotet E, Canales-Rodriguez EJ, Salvador R, Sarro S, Gomar JJ, Vila F, Ortiz-Gil J, Iturria-Medina Y, Capdevila A, McKenna PJ. Medial prefrontal cortex pathology in schizophrenia as revealed by convergent findings from multimodal imaging. Mol Psychiatry. 2010 Aug;15:823–830. 2927029. [PMC free article] [PubMed]
162. Sexton CE, Allan CL, Le Masurier M, McDermott LM, Kalu UG, Herrmann LL, Maurer M, Bradley KM, Mackay CE, Ebmeier KP. Magnetic resonance imaging in late-life depression: multimodal examination of network disruption. Arch Gen Psychiatry. 2012 Jul;69:680–689. [PubMed]
163. Qiu MG, Ye Z, Li QY, Liu GJ, Xie B, Wang J. Changes of brain structure and function in ADHD children. Brain Topogr. 2011 Oct;24:243–252. [PubMed]
164. Zhang D, Wang Y, Zhou L, Yuan H, Shen D. Multimodal classification of Alzheimer's disease and mild cognitive impairment. Neuroimage. 2011 Apr 1;55:856–867. 3057360. [PMC free article] [PubMed]
165. Groves AR, Smith SM, Fjell AM, Tamnes CK, Walhovd KB, Douaud G, Woolrich MW, Westlye LT. Benefits of multi-modal fusion analysis on a large-scale dataset: life-span patterns of inter-subject variability in cortical morphometry and white matter microstructure. Neuroimage. 2012 Oct 15;63:365–380. [PubMed]
166. Eichele T, Calhoun VD, Moosmann M, Specht K, Jongsma ML, Quiroga RQ, Nordby H, Hugdahl K. Unmixing concurrent EEG-fMRI with parallel independent component analysis. Int J Psychophysiol. 2008 Mar;67:222–234. 2649878. [PMC free article] [PubMed]
167. Haller S, Xekardaki A, Delaloye C, Canuto A, Lovblad KO, Gold G, Giannakopoulos P. Combined analysis of grey matter voxel-based morphometry and white matter tract-based spatial statistics in late-life bipolar disorder. J Psychiatry Neurosci. 2011 Nov;36:391–401. 3201993. [PMC free article] [PubMed]
168. Chen Z, Cui L, Li M, Jiang L, Deng W, Ma X, Wang Q, Huang C, Wang Y, Collier DA, Gong Q, Li T. Voxel based morphometric and diffusion tensor imaging analysis in male bipolar patients with first-episode mania. Prog Neuropsychopharmacol Biol Psychiatry. 2011 Mar 30;36:231–238. [PubMed]
169. Correa N, Adalı T, Li Y, Calhoun VD. Canonical Correlation Analysis for Data Fusion and Group Inferences: Examining applications of medical imaging data. IEEE Signal Proc. Magazine. 2010;27:39–50. PMC Pending #202222. [PMC free article] [PubMed]
170. Meda S, Jagannathan K, Gelernter J, Calhoun VD, Liu J, Stevens M, Pearlson GD. A pilot multivariate parallel ICA study to investigate differential linkage between neural networks and genetic profiles in schizophrenia. NeuroImage. 2010;53:1007–1015. PMC pending #161905. [PMC free article] [PubMed]
171. Jagannathan K, Calhoun VD, Gelernter J, Stevens M, Liu J, Bolognani F, Windemuth A, Ruano G, Pearlson GD. Genetic associations of brain structural networks in schizophrenia: a preliminary study using parallel ICA. Biological Psychiatry. 2010;68:657–666. PMC pending #211358. [PMC free article] [PubMed]
172. Jamadar S, Powers NR, Meda SA, Gelernter J, Gruen JR, Pearlson GD. Genetic influences of cortical gray matter in language-related regions in healthy controls and schizophrenia. Schizophr Res. 2010 Apr 18; [PMC free article] [PubMed]
173. Hao X, Xu D, Bansal R, Dong Z, Liu J, Wang Z, Kangarlu A, Liu F, Duan Y, Shova S, Gerber AJ, Peterson BS. Multimodal magnetic resonance imaging: The coordinated use of multiple, mutually informative probes to understand brain structure and function. Hum Brain Mapp. 2011 Feb;34:253–271. [PMC free article] [PubMed]
174. Xu L, Pearlson G, Calhoun V. Joint Source Based Morphometry Identifies Linked Gray and White Matter Group Differences. NeuroImage. 2009;44:777–789. PMC2669793. [PMC free article] [PubMed]
175. Meda S, Gill A, Stevens MC, Lorenzoni RP, Glahn DC, Calhoun VD, Sweeney JA, Tamminga CA, Keshavan M, Thaker G, Pearlson GG. Differences in resting-state fMRI functional network connectivity between schizophrenia and psychotic bipolar probands and their unaffected first-degree relatives. Biological Psychiatry. 2012;71:881–889. PMC Journal: In Process. [PMC free article] [PubMed]