The first contribution of this work concerns a simple idea for image SR which is the use of an HR reference image to improve the resolution of the LR image. As previously stated, such approach is directly related to the medical context we are interested in. However, we believe that such new image SR approach may have a substantial impact in the image processing research field. Moreover, possible further work can focus on the unification of this framework with other previously proposed approaches such as object-based interpolation and registration-based interpolation.
The second contribution is about the modeling of the SR problem introducing a non-local regularization term. We have shown that example-based approaches such as non-local means can be embedded into the reconstruction process to enhance the performance of super-resolution techniques. If an HR brain image of the patient is available, LR images of the same patient can be enhanced by exploiting non-local pairwise interactions. We believe that this work is a new original way to tackle the problem of image enhancement in MR imaging by exploiting the multimodality aspect of MR data. The key point of this work is the use of a non-local approach to define a new regularization term in the reconstruction process. The proposed methodology is related to example-based SR methods such as the one developed by Baker and Kanade (2000)
but in our case the learning database is reduced to one reference image. Moreover, in this work we exploit non-local interactions between voxels by defining non-local weights w
(also called non-local graphs) which take into account non-local interactions in the reconstructed image and the reference image.
Experimental results show that the developed algorithm compares favorably with interpolation approaches. The two key points of the proposed approach, with respect to interpolation methods, are the use of an observation model (as in SR approaches) and the use of a reference HR image which drives the reconstruction process. Experiments on Brainweb images show that even with the presence of lesions visible only in the LR T2-weighted image, the reconstruction method we propose is able to recover such “outliers” without introducing artefacts which may come from the HR T1-weighted image (where lesions are not clearly visible). In this particular case, we would have thought that the lesions would disappear. Although our experiments tend to show that our approach is robust to such outliers, this is a crucial point that needs to be further investigated (small lesions, tumors, etc.). In general, this point raises the following question: what is the influence of an interpolation or image SR algorithm on image analysis (segmentation, detection, etc.)?
This points out that such image enhancement technique needs to be validated using physical phantoms. Moreover, an evaluation on large database is required to prove that SR based methods can have a significant impact on medical image processing pipelines (specifically for registration, segmentation or change detection). Such validation can be application dependent and thus requires a specific evaluation framework. As it has been shown in Section 4, in addition to PSNR and visual assessment, there is a clear need to develop standard criteria to measure image quality reconstruction of SR technique. This is a key point for image enhancement technique particularly in medical imaging where image artefacts have to be avoided.
On one hand, this work shows that SR based techniques provides higher MR image quality than standard interpolation algorithms. On the other hand, interpolation techniques do not require any rigid registration step and are less time consuming (SR based methods may require several hours per reconstruction). Based on our experiments (not shown in the paper), registration does not introduce artefacts in the reconstructed images using the proposed approach. If images are not correctly registered, the correlation coefficient α tends to zero which means that the reconstruction process is only driven by the T2-weighted image.
High-resolution imaging is a key point for MR brain image analysis in order to study anatomical details. Since few brain image analysis algorithms take into account an observation model such as in Eq. (1)
, the proposed reconstruction approach may have a substantial impact on segmentation or registration. Therefore, such model-based HR image reconstruction algorithm may represent an important step towards multimodal brain analysis at fine scale.
While in this work we are only focused on brain MR image reconstruction, the proposed approach relying intermodality priors might have potential applications to other medical image modalities. Future work would involve studying a similar method for multimodal images (computerized tomography, diffusion MRI, ultrasound, etc.). Depending on the intended medical application, such approach could allow to save time of data acquisition by enabling image quality improvement as a post-acquisition step. The corner stone would concern then the criterion for linking the different modalities (How to calculate the local correlation map? What type of information can be used between the imaging modalities?.)