Home | About | Journals | Submit | Contact Us | Français |

**|**Int J Biomed Imaging**|**v.2017; 2017**|**PMC5339635

Formats

Article sections

Authors

Related links

Int J Biomed Imaging. 2017; 2017: 3020461.

Published online 2017 February 21. doi: 10.1155/2017/3020461

PMCID: PMC5339635

*Yin Fei: Email: nc.tpo@iefniy

Academic Editor: Jyh-Cheng Chen

Received 2016 August 31; Revised 2017 January 1; Accepted 2017 January 10.

Copyright © 2017 Yin Fei et al.

This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

As a novel multiscale geometric analysis tool, sparse representation has shown many advantages over the conventional image representation methods. However, the standard sparse representation does not take intrinsic structure and its time complexity into consideration. In this paper, a new fusion mechanism for multimodal medical images based on sparse representation and decision map is proposed to deal with these problems simultaneously. Three decision maps are designed including structure information map (SM) and energy information map (EM) as well as structure and energy map (SEM) to make the results reserve more energy and edge information. SM contains the local structure feature captured by the Laplacian of a Gaussian (LOG) and EM contains the energy and energy distribution feature detected by the mean square deviation. The decision map is added to the normal sparse representation based method to improve the speed of the algorithm. Proposed approach also improves the quality of the fused results by enhancing the contrast and reserving more structure and energy information from the source images. The experiment results of 36 groups of CT/MR, MR-T1/MR-T2, and CT/PET images demonstrate that the method based on SR and SEM outperforms five state-of-the-art methods.

Medical imaging attracts more and more attention due to the increasing requirements of clinic investigation and disease diagnosis. Owing to different imaging mechanisms, medical images of different modals provide a variety of complementary information about the human body in a limited domain. For example, the computed tomography (CT) images provide better information on dense tissue, the positron emission tomography (PET) images supply better information on blood flow and tumor activity with low space resolution, and the magnetic resonance (MR) images show better information on soft tissue. Moreover, the MR-T1 images give more detailed information about anatomical structures, whereas the MR-T2 images contain a greater contrast between the normal and abnormal tissues [1–4]. However, single multiple modality cannot satisfy the demand of images with high resolution and visualization for disease diagnosis.

In this regard, medical image fusion is a useful and powerful technique for integrating complementary information from multimodality images to improve the diagnostic accuracy. Besides, the fused images are more suitable for assisting the doctors in diagnosis and treatment planning [5]: fusing MR and CT images can generate the images which can describe the soft tissue and bone in order to concurrently represent anatomical and physiological features of the human body [6, 7]. MR-T1 and MR-T2 images are fused to segment white matter lesions and guide neurosurgical resection of epileptogenic lesions [7, 8]. In oncology, the combined PET/CT imaging is helpful to view the anatomical, physiological characteristics and the tumor activity [9, 10]. More than that, medical image fusion not only helps in diagnosing diseases but also reduces the storage cost [8].

As the most popular technique of the image fusion, the multiscale decomposition methods have developed quickly in recent years, such as discrete wavelet transform (DWT) [3, 7], framelet transform [9], contourlet transform [10], and nonsubsampled contourlet transform (NSCT) [1, 4, 6]. Unfortunately, transform-based methods produce poor fusion results in the presence of noise and it is difficult to choose the decomposition levels [11, 12].

Sparse representation (SR) has proven to be an exceedingly powerful tool for analyzing the signals of high dimensionality [13], so more and more researchers adapt SR to the field of image fusion for the purpose of getting better fused results [14–19]. However, the standard SR does not take the intrinsic structure [14] and the time complexity [17] into consideration. Therefore, adding them into the SR model is a reasonable strategy to improve the performance of SR [14], but it is complicated to find the relationship between the intrinsic structure information and the sparse coefficients. Reference [14] proposed a dictionary learning method combining with the geometrical structure by group sparse coding, but it did not talk about the time complexity of the algorithms based on sparse representation. Image fusion methods based on joint sparse representation (JSR) [11, 12, 20] need much more iterations to realize image vectors sparse representation for the trained dictionary with bigger size. Some researchers proposed some novel methods combining multiscale transform and SR [15, 18, 21], to put the structure information of the source images into the fused images, which make methods much more complex and time-consuming.

In that way, how to realize image fusion based on SR with local structure information in shorter time became the chief task. The decision map can help us achieve this goal by extracting the local structure feature of the image blocks [22–27]. Unfortunately, most of the methods combining with decision map are only appropriate for multifocus image fusion [22–25]. References [26, 27] apply the decision map into infrared and visible image fusion, which demonstrate that the decision map can be suitable for other type image fusion. In fact, for the methods based on SR, almost all the sparse coefficients fusion rules depend on different blocks feature values, which means they all belong to the methods based on decision map [28]. To realize the medical image fusion based on SR with decision map, we add the local structure and energy information of source images into the decision map to improve the speed of the algorithm and the quality of the fused results.

The main contribution of this paper is as follows:

- To add the local structure and energy information of the source images into the SR algorithm for medical image fusion, we design three decision maps to extract the local energy and structure features of the source images.
- It is good to use the decision to reduce the number of image blocks to sparse representation, so that we can get the results in much shorter time. Using the maps to remain more structure and energy information in fused images will also improve the quality of the results.

There is the framework of the proposed approach based on SR and feature extraction as shown in Figure 1.

Firstly, we divide all source images *A* and *B* with the size of *m* × *n* into patches *y*_{1}^{i,j} and *y*_{2}^{i,j} through a sliding window with the size of ×*w*(*i* ≤ *m* − *w*, *j* ≤ *n* − *w*). All patches are arranged into vectors *V*_{1}^{(i−1)(n−w)+j} and *V*_{2}^{(i−1)(n−w)+j} from left to right and from top to bottom.

Secondly, we group these vectors into vector pairs according to corresponding positions of original patches and design the decision map according to their features separately.

Thirdly, we use the decision map to determine which one vector of each group as the result when the map is marked as 1 or 2. It means that these groups are regarded as the input of the sparse representation system when the map is marked 0.

Fourthly, we fuse the other vector pairs by the SR method.

Finally, the system can generate the fused results according to the decision map. The overlaps of the patches are averaged.

In SR algorithms, a signal can be expressed as a sparse combination of the fewest possible atoms of an overcomplete dictionary [29]. Let *V* *R*^{m} denote a signal vector from the source images and let *D* *R*^{m×k} (*k* > *m*) denote an overcomplete dictionary whose column vectors are its atoms. The signal vector can be represented as *V* = *Dθ*, where *θ* *R*^{k} is a sparse coefficient vector. The sparse coefficient vector is acquired by solving the following question:

$$\begin{array}{c}\widehat{\theta}=\mathrm{a}\mathrm{r}\mathrm{g}\hspace{0.17em}\underset{\theta}{\mathrm{m}\mathrm{i}\mathrm{n}}\hspace{1em}{\Vert \phantom{\rule[-0.12pt]{0ex}{7.08pt}}\theta \phantom{\rule[-0.12pt]{0ex}{7.08pt}}\Vert}_{\mathrm{0}}\\ \\ \mathrm{s}.\mathrm{t}.\hspace{1em}{\Vert \phantom{\rule[-0.23pt]{0ex}{7.08pt}}V-D\theta \phantom{\rule[-0.23pt]{0ex}{7.08pt}}\Vert}_{2}^{2}\le \epsilon ,\\ \end{array}$$

(1)

where*ε*is error tolerance parameter. An image vector can be represented as a superposition of the smallest possible number of atoms in the dictionary. We can solve (1) by OMP, BP, or other algorithms [12], because it is an NP-hard problem. It is important to find an appropriate dictionary for SR. There are two main approaches to build a dictionary. One is to use a fixed dictionary such as the Gabor dictionary [30], the discrete cosine transform (DCT) dictionary [31], and the Haar dictionary [32]. Another is to train a dictionary from a large number of training image patches, like* K*-SVD [32], which usually shows better performance in image processing compared to the fixed dictionary methods [11].

Let us regard ${\stackrel{-}{y}}_{\mathrm{1}}^{i,j}$ and ${\stackrel{-}{y}}_{\mathrm{2}}^{i,j}$ as mean values of *y*_{1}^{i,j} and *y*_{2}^{i,j}, respectively. We use ‖*y*_{1}^{i,j}‖_{2} as the sign of energy and the mean square deviation $\left(\mathrm{1}/\mathrm{64}\right){\Vert {y}_{\mathrm{1}}^{i,j}-{\stackrel{-}{y}}_{\mathrm{1}}^{i,j}\Vert}_{\mathrm{2}}$ as the sign of energy distribution for *y*_{1}^{i,j}, which are similar to *y*_{2}^{i,j}. We design the first decision map EM *R*^{M×N} (where *M* = *m* − *w* and *N* = *n* − *w*) which called the energy map by

$$\begin{array}{c}t\mathrm{1}=\frac{\mathrm{1}}{\mathrm{64}}\left(\phantom{\rule[-4.72096pt]{0ex}{9.87196pt}}{\Vert \phantom{\rule[-2.95702pt]{0ex}{9.87196pt}}{y}_{\mathrm{1}}^{i,j}\phantom{\rule[-2.95702pt]{0ex}{9.87196pt}}\Vert}_{\mathrm{2}}-{\Vert \phantom{\rule[-2.95702pt]{0ex}{9.87196pt}}{y}_{\mathrm{2}}^{i,j}\phantom{\rule[-2.95702pt]{0ex}{9.87196pt}}\Vert}_{\mathrm{2}}\phantom{\rule[-4.72096pt]{0ex}{9.87196pt}}\right),\\ \\ t\mathrm{2}=\frac{\mathrm{1}}{\mathrm{64}}\left(\phantom{\rule[-4.72096pt]{0ex}{9.87196pt}}{\Vert \phantom{\rule[-2.95702pt]{0ex}{9.87196pt}}{y}_{\mathrm{1}}^{i,j}-{\stackrel{-}{y}}_{\mathrm{1}}^{i,j}\phantom{\rule[-2.95702pt]{0ex}{9.87196pt}}\Vert}_{\mathrm{2}}-{\Vert \phantom{\rule[-2.95702pt]{0ex}{9.87196pt}}{y}_{\mathrm{2}}^{i,j}-{\stackrel{-}{y}}_{\mathrm{2}}^{i,j}\phantom{\rule[-2.95702pt]{0ex}{9.87196pt}}\Vert}_{\mathrm{2}}\phantom{\rule[-4.72096pt]{0ex}{9.87196pt}}\right),\\ \\ \mathrm{E}{\mathrm{M}}_{}^{i,j}=\left\{\begin{array}{cc}\mathrm{1},\hfill & t\mathrm{1}>\mathrm{0},\hspace{0.17em}\hspace{0.17em}t\mathrm{2}>\mathrm{0},\hfill \\ \mathrm{2},\hfill & t\mathrm{1}<\mathrm{0},\hspace{0.17em}\hspace{0.17em}t\mathrm{2}<\mathrm{0},\hfill \\ \mathrm{0},\hfill & \mathrm{o}\mathrm{t}\mathrm{h}\mathrm{e}\mathrm{r}\mathrm{s}.\hfill \end{array}\right.\\ \end{array}$$

(2)

In this way, our map contains the energy and energy distribution information of the vector pairs. However, this map does not contain enough image structure information. So we use the Laplacian of a Gaussian (LOG) to detect the structure information of the source images [33]. For noise cleaning, we smooth the image by convolving it with a digital mask corresponding to Gaussian function. The Gaussian function is expressed by (3)–(5) and we can get the local normalized structure information by (6). One has

$$\begin{array}{c}h\left(\phantom{\rule[-2.59pt]{0ex}{4.53pt}}x,y\phantom{\rule[-2.59pt]{0ex}{4.53pt}}\right)h\left(\phantom{\rule[-0.12pt]{0ex}{4.53pt}}r\phantom{\rule[-0.12pt]{0ex}{4.53pt}}\right)=-{e}^{-{r}^{\mathrm{2}}/\mathrm{2}{\sigma}^{\mathrm{2}}},\\ \end{array}$$

(3)

$$\begin{array}{c}{\nabla}^{\mathrm{2}}\left(\phantom{\rule[-2.59pt]{0ex}{7.08pt}}g\left(\phantom{\rule[-0.12pt]{0ex}{4.53pt}}r\phantom{\rule[-0.12pt]{0ex}{4.53pt}}\right)\ast h\left(\phantom{\rule[-0.12pt]{0ex}{4.53pt}}r\phantom{\rule[-0.12pt]{0ex}{4.53pt}}\right)\phantom{\rule[-2.59pt]{0ex}{7.08pt}}\right)g\left(\phantom{\rule[-0.12pt]{0ex}{4.53pt}}r\phantom{\rule[-0.12pt]{0ex}{4.53pt}}\right)\ast {\nabla}^{\mathrm{2}}h\left(\phantom{\rule[-0.12pt]{0ex}{4.53pt}}r\phantom{\rule[-0.12pt]{0ex}{4.53pt}}\right),\\ \end{array}$$

(4)

$$\begin{array}{c}pg\left(\phantom{\rule[-0.12pt]{0ex}{4.53pt}}r\phantom{\rule[-0.12pt]{0ex}{4.53pt}}\right)\ast {\nabla}^{\mathrm{2}}h\left(\phantom{\rule[-0.12pt]{0ex}{4.53pt}}r\phantom{\rule[-0.12pt]{0ex}{4.53pt}}\right)\\ =-g\left(\phantom{\rule[-0.12pt]{0ex}{4.53pt}}r\phantom{\rule[-0.12pt]{0ex}{4.53pt}}\right)\left[\phantom{\rule[-7.18999pt]{0ex}{15.36499pt}}\frac{{r}^{\mathrm{2}}-{\sigma}^{\mathrm{2}}}{{\sigma}^{\mathrm{4}}}\phantom{\rule[-7.18999pt]{0ex}{15.36499pt}}\right]{e}^{-{r}^{\mathrm{2}}/\mathrm{2}{\sigma}^{\mathrm{2}}},\end{array}$$

(5)

$$\begin{array}{c}e\frac{p}{\mathrm{max}\left(p\right)},\\ \end{array}$$

(6)

where *r*^{2} = *x*^{2} + *y*^{2} and *σ* is the mean square deviation. Given an image matrix *g*(*x*, *y*), the LOG of the image function is the second-order partial derivatives along *x* and *y* directions. There is an example of the LOG edge detection of CT and MR images as shown in Figure 2.

(a) The source images, (b) the structure information of LOG, and (c) the local structure information normalization.

*e*
_{1}
^{i,j} and *e*_{2}^{i,j} represent the local normalized structure information of *y*_{1}^{i,j} and *y*_{2}^{i,j}, respectively. Therefore, we design the second map SM *R*^{M×N} named the structure map by

$$\begin{array}{c}s\mathrm{1}=\frac{\mathrm{1}}{\mathrm{64}}\left(\phantom{\rule[-4.72096pt]{0ex}{9.87196pt}}{\Vert \phantom{\rule[-2.95702pt]{0ex}{9.87196pt}}{e}_{\mathrm{1}}^{i,j}\phantom{\rule[-2.95702pt]{0ex}{9.87196pt}}\Vert}_{\mathrm{2}}-{\Vert \phantom{\rule[-2.95702pt]{0ex}{9.87196pt}}{e}_{\mathrm{2}}^{i,j}\phantom{\rule[-2.95702pt]{0ex}{9.87196pt}}\Vert}_{\mathrm{2}}\phantom{\rule[-4.72096pt]{0ex}{9.87196pt}}\right),\\ \\ s\mathrm{2}=\frac{\mathrm{1}}{\mathrm{64}}\left(\phantom{\rule[-4.72096pt]{0ex}{9.87196pt}}{\Vert \phantom{\rule[-2.95702pt]{0ex}{9.87196pt}}{e}_{\mathrm{1}}^{i,j}-{\stackrel{-}{e}}_{\mathrm{1}}^{i,j}\phantom{\rule[-2.95702pt]{0ex}{9.87196pt}}\Vert}_{\mathrm{2}}-{\Vert \phantom{\rule[-2.95702pt]{0ex}{9.87196pt}}{e}_{\mathrm{2}}^{i,j}-{\stackrel{-}{e}}_{\mathrm{2}}^{i,j}\phantom{\rule[-2.95702pt]{0ex}{9.87196pt}}\Vert}_{\mathrm{2}}\phantom{\rule[-4.72096pt]{0ex}{9.87196pt}}\right),\\ \\ \mathrm{S}{\mathrm{M}}_{}^{i,j}=\left\{\begin{array}{cc}\mathrm{1},\hfill & s\mathrm{1}>\mathrm{0.05},\hspace{0.17em}\hspace{0.17em}s\mathrm{2}>\mathrm{0},\hfill \\ \mathrm{2},\hfill & s\mathrm{1}<-\mathrm{0.05},\hspace{0.17em}\hspace{0.17em}s\mathrm{2}<\mathrm{0},\hfill \\ \mathrm{0},\hfill & \mathrm{o}\mathrm{t}\mathrm{h}\mathrm{e}\mathrm{r}\mathrm{s}.\hfill \end{array}\right.\\ \end{array}$$

(7)

Combining the energy, energy distribution, and structure information, we design the third map SEM *R*^{M×N} which we name the structure and energy map by

$$\begin{array}{c}\mathrm{S}\mathrm{E}{\mathrm{M}}_{}^{i,j}=\left\{\begin{array}{cc}\mathrm{0},\hfill & \mathrm{E}{\mathrm{M}}_{}^{i,j}=\mathrm{0},\hspace{0.17em}\hspace{0.17em}\mathrm{S}{\mathrm{M}}_{}^{i,j}=\mathrm{0},\hfill \\ \mathrm{1},\hfill & \mathrm{E}{\mathrm{M}}_{}^{i,j}=\mathrm{1},\hfill \\ \mathrm{1},\hfill & \mathrm{E}{\mathrm{M}}_{}^{i,j}=\mathrm{0},\hspace{0.17em}\hspace{0.17em}\mathrm{S}{\mathrm{M}}_{}^{i,j}=\mathrm{1},\hfill \\ \mathrm{2},\hfill & \mathrm{E}{\mathrm{M}}_{}^{i,j}=\mathrm{2},\hfill \\ \mathrm{2},\hfill & \mathrm{E}{\mathrm{M}}_{}^{i,j}=\mathrm{0},\hspace{0.17em}\hspace{0.17em}\mathrm{S}{\mathrm{M}}_{}^{i,j}=\mathrm{2}.\hfill \end{array}\right.\end{array}$$

(8)

When *V*_{1}^{h} = *Dθ*_{1}^{h} and *V*_{2}^{h} = *Dθ*_{2}^{h}, we can get the fusion vectors by (9) according to the decision map:

$$\begin{array}{c}g\left(\phantom{\rule[-2.9pt]{0ex}{9.75598pt}}{\theta}_{\mathrm{1}}^{h},{\theta}_{\mathrm{2}}^{h}\phantom{\rule[-2.9pt]{0ex}{9.75598pt}}\right)=\left\{\begin{array}{cc}{\theta}_{\mathrm{1}}^{h},\hfill & \left|{\theta}_{\mathrm{1}}^{h}\right|>\left|{\theta}_{\mathrm{2}}^{h}\right|,\hfill \\ {\theta}_{\mathrm{2}}^{h},\hfill & \left|{\theta}_{\mathrm{1}}^{h}\right|\le \left|{\theta}_{\mathrm{2}}^{h}\right|,\hfill \end{array}\right.\\ \\ {F}^{h}=\left\{\begin{array}{cc}{V}_{\mathrm{1}}^{h},\hfill & {M}^{i,j}=\mathrm{1},\hfill \\ {V}_{\mathrm{2}}^{h},\hfill & {M}^{i,j}=\mathrm{2},\hfill \\ D\times g\left({\theta}_{\mathrm{1}}^{h},{\theta}_{\mathrm{2}}^{h}\right),\hfill & {M}^{i,j}=\mathrm{0},\hfill \end{array}\right.\\ \phantom{\rule{39.0865631103515625pt}{0ex}}i=\u230a\frac{h}{n-w}\u230b,\hspace{0.17em}\hspace{0.17em}j=h-i\ast \left(n-w\right),\end{array}$$

(9)

where *M* *R*^{M×N} can be EM, SM, SEM, or other decision maps.

In general, the proposed method at least has three merits in contrast to the normal SR based method. Firstly, it can make the fused results preserve the information of the source images as much as possible and remit the effect of algorithm noise. Secondly, it can get the results more rapidly because we just sparse-represent a part of the vector pairs. Thirdly, our algorithm combines energy, energy distribution, and structure characteristics of the images to enhance contrast of the results. In the abstract, the results fused by proposed have the best contrast information, which is the most important information to locate the position of the abnormal tissue.

To evaluate the performance of the proposed method, three experiments are implemented. All the images are the same size of 256 × 256 pixels. In this paper, we train the dictionary with* K*-SVD using the pictures as shown in Figure 3. The error tolerance *ε* is set to be 0.01. The maximum iterations of the* K*-SVD are set to be 30. The initial dictionary is the DCT dictionary with the size of 64 × 256. We use OMP to estimate the sparse coefficients for simplicity. The moving step of the sliding window is set to be one pixel. We use three kinds of medical image pairs including CT/MR images, MR-T1/MR-T2 images, and CT/PET images to test the performances of those abovementioned methods. The DCT dictionary and trained dictionary are shown in Figure 4. The window size of LOG is set to be 5 × 5, and *σ* is set to be 2.

The source images for fusion and training dictionary, including 12 pairs of CT/MR images, 12 pairs of MR-T1/MR-T2 images, and 12 pairs of CT/PET images.

For comparison, five state-of-the-art methods are evaluated in the experiments, including methods based on NSCT [1, 6], method based on JSR [12], and methods based on NSCT and SR [15, 18]. In this paper, five objective evaluation measurements parameters are adopted to evaluate the fusion performance. There are local quality index (*Q*_{0}) [34], weighted fusion quality index (*Q*_{W}) [34], edge-dependent fusion quality index (*Q*_{E}) [34], *Q*_{AB/F} [35] which measures the transmission of edge and visual information from source images to fused images, and mutual information (MI) [36] which computes the information transformed from the source images to the fused images. For *Q*_{0}, *Q*_{W}, *Q*_{E}, and *Q*_{AB/F}, they all lie in the interval [0,1]. For these parameters, the greater value indicates the better fusion performance. The experiments are carried out in the PC with the Intel i7-3770 CPU 3.40GHz and 4G RAM, operating under MATLAB R2010a.

In the first experiment, the CT and MR images are fused with eight different image fusion methods listed above. We used 12 groups CT and MR images to test the performance of these methods as shown in Figure 3. Two groups of results are shown in Figure 5. It is obviously seen that the results of NSCT are fuzzy in some parts, especially in Figures 5(m) and 5(n), the results of SR + NSCT [15], SR + SM, and SR + SEM can reserve better source image boundary information than the results of the other methods. And these results have no block effects, because all the methods use the sliding window strategy, in which NSCT [1] and NSCT [6] use the window with size of 3 × 3 and the others use the window with size of 8 × 8. And results of NSCT [1], NSCT [6], JSR [12], and SR + NSCT [18] are brighter than all source images, which will lead some dim information to be hidden by light information. As shown in Figures 5(m), 5(n), 5(o), and 5(q), we cannot tell the tissue information between the skull and brain. In a certain extent, the proposed method can ease these problems and meanwhile remain the merits of the SR based methods. Comparatively, the results of SR + SEM can remain better image boundary and energy information, where we can get better anatomical information from CT images and soft tissue information from MR images simultaneously. More than that, the calcified meningioma in Figure 5(f) can be distinguished from background easily in Figures 5(r), 5(s), and 5(t). As for the fused results of CT and MR images, the average scores of quantitative evaluation metrics are listed in Table 1 and the “bold” values indicate the highest values. We can see that the proposed method SR + SEM outperforms other methods in all scores. The results fused by proposed methods are all better than normal methods.

In the second experiment, we used 12 groups MR-T1 and MR-T2 images to test the performance of these methods as shown in Figure 3. To illustrate the proposed fusion method, two sets of results are presented in Figure 6. In general, the results of NSCT [1], NSCT [6], and SR + NSCT [18] look gloomy and bright, demonstrating the grey distortion happens. NSCT [1], NSCT [6], JSR [12], and SR + NSCT [18] create many bad edges and make the fused results too smoothness. Comparatively, the results of SR + NSCT [15] and proposed methods show better boundary information and energy information with fewer artifacts, so that we can get better information on adipose tissue from MR-T1 images and information on vascular and tumor from MR-T2 images. Compared to the other methods, the results of proposed methods contain more information from the source images. The proposed methods preserve both better local edge and texture information, which is the vital information for diagnosis. The subacute premature hematoma is seen clearly in Figure 6(l), and we can see the location and contour of the intracranial hematoma in Figures 6(r), 6(s), and 6(t). For 12 MR-T1 and MR-T2 fused results, the average scores of quantitative evaluation metrics are listed in Table 2 and the “bold” values indicate the highest values. We can see that SR + SEM outperforms other methods in all scores. In general, the results fused by proposed methods are better than the other methods.

In the third experiment, we used 12 CT and PET image pairs to test the performance of these methods as shown in Figure 3. Two sets of results are shown in Figure 7. Comparatively, the results of NSCT [1] and NSCT [6] are best especially in energy information, so that the fused images can capture both more spatial information in the CT images and functional information contents in PET images. However, in clinical applications, doctors need to see the position of bone and tumor to determine pathology and aid in diagnosis. The results fused by SR + SEM contain more detailed information and higher contrast but without information distortion so that we can see the outline of the kidney clearly in Figures 7(r), 7(s), and 7(t). Nasopharyngeal carcinoma can be seen in Figure 7(b), and we can use the result fused by proposed method to locate it in Figure 7 SR + NSCT [15], SR + SM, and SR + SEM easily, where are helpful to view the tumor activity, allowing physicians to better understand the effect of cancer treatment. For 12 CT and PET fused results, the average scores of quantitative evaluation metrics are listed in Table 3 and the “bold” values indicate the highest values. We can see that the SR + SEM outperforms other methods in all scores. It shows that this approach is flexible and stable.

To realize the fusion and reconstruction of 3D medical images, a lot of CT/PET and MR/PET image slices need to be fused firstly [37, 38]. Therefore, there is a need to find a faster and stronger image fusion algorithm. As shown in Figure 8, we record the average time consumption of different methods for 36 different medical image pairs listed. It is evident that the multiscale approaches including NSCT [1] and NSCT [6] are very fast while the SR based approaches (JSR [12], SR + NSCT [15], and SR + NSCT [18]) take much more time. Comparably, the time consuming of SR + SM is about 1/20, SR + EM is about 1/20, and SR + SEM is about 1/50 of the SR based approach. From the above analysis and discussion, we draw the conclusion that SR + SEM outperforms all the others in the field of medical image fusion. Because it contains more original information from source images and better local structure information, our methods are more appropriate for doctors to localize the abnormal masses and tumors in patients.

In this paper, a new medical image fusion approach based on SR and feature extraction is proposed. There are at least three major improvements compared with the conventional SR based fusion methods. Firstly, we put forward three decision maps to improve quality of the SR based image fusion methods in extracting the structure and energy features of the source images. This strategy can help remain the original information from the source images as much as possible. Secondly, we add the decision map into the SR based methods to improve the speed of the algorithm. It takes only 1/50 of the time that the standard SR method needs to realize the image fusion based on proposed approach. Thirdly, adding the structure and energy information of source images into the decision map improve the quality of the fused results a lot. The experiments results indicate that the proposed fusion approach can achieve better results than the conventional fusion methods in both subjective and objective aspects.

The authors declare that there is no conflict of interests regarding the publication of this paper.

1. Gomathi P. S., Kalaavathi B. Multimodal medical image fusion in non-subsampled contourlet transform domain. *Circuits & Systems*. 2016;7(8):1598–1610. doi: 10.4236/cs.2016.78139. [Cross Ref]

2. Valdés Hernández M. D. C., Ferguson K. J., Chappell F. M., Wardlaw J. M. New multispectral MRI data fusion technique for white matter lesion segmentation: method and comparison with thresholding in FLAIR images. *European Radiology*. 2010;20(7):1684–1691. doi: 10.1007/s00330-010-1718-6. [PMC free article] [PubMed] [Cross Ref]

3. Liu Z., Yin H., Chai Y., Yang S. X. A novel approach for multimodal medical image fusion. *Expert Systems with Applications*. 2014;41(16):7425–7435. doi: 10.1016/j.eswa.2014.05.043. [Cross Ref]

4. Yang Y., Tong S., Huang S., Lin P. Log-Gabor energy based multimodal medical image fusion in NSCT domain. *Computational and Mathematical Methods in Medicine*. 2014;2014:12. doi: 10.1155/2014/835481.835481 [PMC free article] [PubMed] [Cross Ref]

5. Calhoun V. D., Adali T. Feature-based fusion of medical imaging data. *IEEE Transactions on Information Technology in Biomedicine*. 2009;13(5):711–720. doi: 10.1109/TITB.2008.923773. [PMC free article] [PubMed] [Cross Ref]

6. Ganasala P., Kumar V. CT and MR image fusion scheme in nonsubsampled contourlet transform domain. *Journal of Digital Imaging*. 2014;27(3):407–418. doi: 10.1007/s10278-013-9664-x. [PMC free article] [PubMed] [Cross Ref]

7. Shen R., Cheng I., Basu A. Cross-scale coefficient selection for volumetric medical image fusion. *IEEE Transactions on Biomedical Engineering*. 2013;60(4):1069–1079. doi: 10.1109/TBME.2012.2211017. [PubMed] [Cross Ref]

8. Wang Z., Ma Y. Medical image fusion using *m*-PCNN. *Information Fusion*. 2008;9(2):176–185. doi: 10.1016/j.inffus.2007.04.003. [Cross Ref]

9. Bhatnagar G., Jonathan Wu Q. M., Liu Z. Human visual system inspired multi-modal medical image fusion framework. *Expert Systems with Applications*. 2013;40(5):1708–1720. doi: 10.1016/j.eswa.2012.09.011. [Cross Ref]

10. Yang L., Guo B. L., Ni W. Multimodality medical image fusion based on multiscale geometric analysis of contourlet transform. *Neurocomputing*. 2008;72(1-3):203–211. doi: 10.1016/j.neucom.2008.02.025. [Cross Ref]

11. Kim M., Han D. K., Ko H. Joint patch clustering-based dictionary learning for multimodal image fusion. *Information Fusion*. 2016;27:198–214. doi: 10.1016/j.inffus.2015.03.003. [Cross Ref]

12. Yao Y., Guo P., Xin X., Jiang Z. Image fusion by hierarchical joint sparse representation. *Cognitive Computation*. 2014;6(3):281–292. doi: 10.1007/s12559-013-9235-y. [Cross Ref]

13. Guha T., Ward R. K. Learning sparse representations for human action recognition. *IEEE Transactions on Pattern Analysis and Machine Intelligence*. 2012;34(8):1576–1588. doi: 10.1109/TPAMI.2011.253. [PubMed] [Cross Ref]

14. Li S. T., Yin H. T., Fang L. Y. Group-sparse representation with dictionary learning for medical image denoising and fusion. *IEEE Transactions on Biomedical Engineering*. 2012;59(12):3450–3459. doi: 10.1109/tbme.2012.2217493. [PubMed] [Cross Ref]

15. Liu Y., Liu S., Wang Z. F. *Pattern Recognition*. Vol. 484. Berlin, Germany: Springer; 2014. Medical image fusion by combining nonsubsampled contourlet transform and sparse representation; pp. 372–381. (Communications in Computer and Information Science). [Cross Ref]

16. Yin H., Li S., Fang L. Simultaneous image fusion and super-resolution using sparse representation. *Information Fusion*. 2013;14(3):229–240. doi: 10.1016/j.inffus.2012.01.008. [Cross Ref]

17. Falco C. M., Jiang X., Yin F., Gao W., Song Z. Image fusion based on group sparse representation. Proceedings of the Eighth International Conference on Digital Image Processing (ICDIP '16); August 2016; Chengu, China. [Cross Ref]

18. Wang J., Peng J., Feng X., He G., Wu J., Yan K. Image fusion with nonsubsampled contourlet transform and sparse representation. *Journal of Electronic Imaging*. 2013;22(4) doi: 10.1117/1.jei.22.4.043019.043019 [Cross Ref]

19. Zhang Q. H., Fu Y. L., Li H. F., Zou J. Dictionary learning method for joint sparse representation-based image fusion. *Optical Engineering*. 2013;52(5):532–543.

20. Yang B., Li S. Visual attention guided image fusion with sparse representation. *Optik*. 2014;125(17):4881–4888. doi: 10.1016/j.ijleo.2014.04.036. [Cross Ref]

21. Liu Y., Liu S., Wang Z. A general framework for image fusion based on multi-scale transform and sparse representation. *Information Fusion*. 2015;24:147–164. doi: 10.1016/j.inffus.2014.09.004. [Cross Ref]

22. Yang Y., Tong S., Huang S., Lin P. Dual-tree complex wavelet transform and image block residual-based multi-focus image fusion in visual sensor networks. *Sensors*. 2014;14(12):22408–22430. doi: 10.3390/s141222408. [PMC free article] [PubMed] [Cross Ref]

23. Yang Y., Tong S., Huang S., Lin P. Multifocus image fusion based on NSCT and focused area detection. *IEEE Sensors Journal*. 2015;15(5):2824–2838. doi: 10.1109/jsen.2014.2380153. [Cross Ref]

24. Liu Y., Liu S., Wang Z. Multi-focus image fusion with dense SIFT. *Information Fusion*. 2015;23:139–155. doi: 10.1016/j.inffus.2014.05.004. [Cross Ref]

25. Nejati M., Samavi S., Shirani S. Multi-focus image fusion using dictionary-based sparse representation. *Information Fusion*. 2015;25:72–84. doi: 10.1016/j.inffus.2014.10.004. [Cross Ref]

26. Saeedi J., Faez K. Infrared and visible image fusion using fuzzy logic and population-based optimization. *Applied Soft Computing Journal*. 2012;12(3):1041–1054. doi: 10.1016/j.asoc.2011.11.020. [Cross Ref]

27. Zhang B., Lu X., Pei H., Zhao Y. A fusion algorithm for infrared and visible images based on saliency analysis and non-subsampled Shearlet transform. *Infrared Physics and Technology*. 2015;73:286–297. doi: 10.1016/j.infrared.2015.10.004. [Cross Ref]

28. Li S., Kang X., Fang L., Hu J., Yin H. Pixel-level image fusion: a survey of the state of the art. *Information Fusion*. 2017;33:100–112. doi: 10.1016/j.inffus.2016.05.004. [Cross Ref]

29. Mohammadi M. R., Fatemizadeh E., Mahoor M. H. PCA-based dictionary building for accurate facial expression recognition via sparse representation. *Journal of Visual Communication and Image Representation*. 2014;25(5):1082–1092. doi: 10.1016/j.jvcir.2014.03.006. [Cross Ref]

30. Agili S., Bjornberg D. B., Morales A. Optimized search over the Gabor dictionary for note decomposition and recognition. *Journal of the Franklin Institute*. 2007;344(7):969–990. doi: 10.1016/j.jfranklin.2007.04.002. [Cross Ref]

31. Yang B., Li S. Multifocus image fusion and restoration with sparse representation. *IEEE Transactions on Instrumentation and Measurement*. 2010;59(4):884–892. doi: 10.1109/TIM.2009.2026612. [Cross Ref]

32. Aharon M., Elad M., Bruckstein A. K-SVD: an algorithm for designing overcomplete dictionaries for sparse representation. *IEEE Transactions on Signal Processing*. 2006;54(11):4311–4322. doi: 10.1109/tsp.2006.881199. [Cross Ref]

33. Singh M., Mandal M. K., Basu A. Gaussian and Laplacian of Gaussian weighting functions for robust feature based tracking. *Pattern Recognition Letters*. 2005;26(13):1995–2005. doi: 10.1016/j.patrec.2005.03.015. [Cross Ref]

34. Piella G., Heijmans H. A new quality metric for image fusion. Proceedings of the International Conference on Image Processing (ICIP '03); September 2003; Catalonia, Spain. pp. 173–176.

35. Xydeas C. S., Petrović V. Objective image fusion performance measure. *Electronics Letters*. 2000;36(4):308–309. doi: 10.1049/el:20000267. [Cross Ref]

36. Qu G., Zhang D., Yan P. Information measure for performance of image fusion. *Electronics Letters*. 2002;38(7):313–315. doi: 10.1049/el:20020212. [Cross Ref]

37. Tian J., Smith M. F., Chinnadurai P., et al. Clinical application of PET-CT fusion imaging for three-dimensional myocardial scar and left ventricular anatomy during ventricular tachycardia ablation. *Journal of Cardiovascular Electrophysiology*. 2009;20(6):597–604. doi: 10.1111/j.1540-8167.2008.01377.x. [PubMed] [Cross Ref]

38. Hausmann D., Bittencourt L. K., Attenberger U. I., et al. Diagnostic accuracy of 18F choline PET/CT using time-of-flight reconstruction algorithm in prostate cancer patients with biochemical recurrence. *Clinical Nuclear Medicine*. 2014;39(3):e197–e201. doi: 10.1097/rlu.0b013e3182a23d37. [PubMed] [Cross Ref]

Articles from International Journal of Biomedical Imaging are provided here courtesy of **Hindawi**

PubMed Central Canada is a service of the Canadian Institutes of Health Research (CIHR) working in partnership with the National Research Council's national science library in cooperation with the National Center for Biotechnology Information at the U.S. National Library of Medicine(NCBI/NLM). It includes content provided to the PubMed Central International archive by participating publishers. |