In the biomedical domain, there is a real need for developing data integration strategies for combining discriminatory features from multiple modalities to develop multi-modal meta-classifiers for improved disease detection, diagnosis, and prognosis (1
). While data integration methods have been proposed for quantitatively combining multiple imaging modalities (2
), these tools are not readily extensible to integration of imaging and non-imaging data on account of differences in scale and feature dimensionality. For instance, consider the difficulties in quantitatively combining, at the voxel-level, T2-weighted (T2-w) magnetic resonance (MR) imaging (reflecting structural attributes) acquired as scalar intensity values, with MR spectroscopy (MRS) acquired as a vector (or spectrum) of metabolite concentrations; each modality encoding a different type (structural or metabolic) and dimensionality of information. Nonetheless, both modalities reflect information regarding the same region of interest they are captured from, and consequently, examining them in conjunction is crucial. Manual diagnosis of prostate cancer (CaP) on T2-w MRI and MRS involves first visually identifying hypo-intense regions on T2-w MRI, followed by inspection of spectrum at those corresponding spatial locations for changes in metabolite (choline, creatine, citrate) ratios (5
). However, some studies have shown that manual interpretation and visual integration of multi-modal data is subjective and thus prone to inter-and intra-observer variability (6
). It is hence desirable to build a data integration (DI) based computerized support system (DI-CSS) meta-classifier that can accurately extract and combine relevant information from both imaging and non-imaging data channels for improved disease classification (3
). Such a CSS could then be integrated into a clinical setting to assist radiologists in accurate characterization, staging, as well as directing and evaluating disease treatment and treatment response.
One of the major challenges in developing a CSS for quantitative integration of imaging and non-imaging data (hereafter referred to as data level integration) is to represent the different data channels within a common, unified representation prior to data integration. In this work, we introduce a novel computerized framework to address this problem. We refer to this framework as Multimodal Wavelet Embedding Representation for data Combination (MaWERiC), where the aim is to enable seamless quantitative data-level integration of imaging and non-imaging data while overcoming the differences in scales and dimensionalities. Additionally, MaWERiC provides the ability to build meta-classifiers from the different modalities once the data representation and data integration issues have been resolved. While MaWERiC could be applied to a number of different domains and applications, in this work we demonstrate the applicability of MaWERiC in building meta-classifiers for improved prostate cancer detection using T2-w MRI and MRS.
Prostate cancer (CaP) is the second leading cause of cancer related deaths amongst men with 217,730 new cases and 32,050 estimated deaths in United States in 2010 (9
). The current gold standard for CaP detection is a sextant trans-rectal ultrasound examination which is known to have a low detection sensitivity (20–25%) due to the poor image resolution of ultrasound (10
). In the last decade, T2-w MRI has shown great potential for characterizing disease presence as well as staging of CaP (11
). MRS has recently emerged as a complement to traditional T2-w MRI for improved CaP diagnosis (12
). While clinical studies have shown that the use of structural and metabolic MR information yields greater CaP detection accuracy compared to diagnosis based off any individual modality (5
), few attempts have been made to quantitatively combine the different information channels (17
). While a few groups have proposed CSS classifiers for combining multiple MR imaging protocols (T2-w, dynamic contrast enhanced (DCE), line-scan diffusion, diffusion weighted imaging (DWI), T2-mapping MRI) (3
), to the best of our knowledge no previous methods for quantitative integration of T2-w MRI and spectroscopy (imaging and non-imaging modalities) for CaP detection have been presented.
In this work we aim to leverage MaWERiC to build an integrated structural, metabolic meta-classifier which will assign a probability of CaP presence at every spatial location on in-vivo
prostate T2-w-MRI/MRS studies. illustrates the organization of our MaWERiC strategy. Our scheme is based on combining wavelet (Gabor and Haar filters) features (19
) extracted from both the T2-w MRI and MRS modalities. The advantage of using wavelet transform (19
) is that it can provide multi-resolution discriminatory information from different data modalities, including but not limited to signals and images (20
). The 2D-Gabor wavelet filter is defined as the convolution of a 2D Gaussian function with a sinusoid (23
). A Gabor filter bank is then generated by variation of the associated scale and orientation parameters. This filter bank provides a means for multi-scale, multi-orientation texture characterization and representation of an image. Haar wavelet decomposition is a commonly employed signal filtering technique (19
) which provides a way of extracting the class discriminating frequency components that may yield higher classifier accuracy compared to the original signal (24
). An advantage of the Haar wavelet (24
) is that it preserves features that are representative of abrupt changes in signals, dominant spectral peaks such as those that correspond to the most significant metabolites on MRS, while simultaneously eliminating spectral noise. Both the Gabor and Haar wavelet filters have been previously used in conjunction with CSS classifiers to distinguish between different data classes for various biomedical applications (20
). In the context of this work, multi-resolution features for T2-w MRI and MRS are obtained via use of the Gabor and Haar wavelet filters respectively.
Figure 1 Flowchart showing various components and methodological overview of our data integration scheme. Wavelet features are first extracted individually from T2-w MRI and MRS, followed by dimensionality reduction using PCA. The reduced dimensional vectors, (more ...)
While the wavelet based representations of the MRS, T2-w MRI channels provide a uniform, feature representation of the data, the feature vectors obtained via the application of the Gabor and Haar wavelet filters are of a very high and dimensionality and hence subject to the curse of dimensionality
). Consequently a subsequent step is application of Principal Component Analysis (PCA) (27
) to the high dimensional feature vectors obtained (via wavelet decomposition) from each individual data modality (T2-w MRI, MRS) to obtain a reduced dimensional representation of the data and thus make the data representation amenable to the application of the classifier. The representation of the MRS and T2-w MRI data in terms of the Eigenvectors (obtained via PCA) also allows for the overcoming of the scale (resolution) and dimensionality differences between the two modalities, since the wavelet representations obtained from each individual modality are of varying dimensionality, and can be reduced to the same number of Eigenvectors. Data level fusion is then performed by concatenating the principal Eigenvectors corresponding to each modality. This fused Eigenvector representation is then used to train a Random forest (RF) classifier. RF is a commonly used ensemble classifier that combines predictions from several weak classifiers to generate a more accurate and stable classifier (28
). The RF classifier has previously been successfully employed for various biomedical classification applications (29
). Advantages of RF include, (1) ability to integrate a large number of input variables, (2) robustness to noise in the data, and (3) relatively few tuning-parameters.