|Home | About | Journals | Submit | Contact Us | Français|
Tissue contrast and resolution of magnetic resonance neuroimaging data have strong impacts on the utility of the data in clinical and neuroscience tasks such as registration and segmentation. Lengthy acquisition times typically prevent routine acquisition of multiple MR tissue contrast images at high resolution, and the opportunity for detailed analysis using these data would seem to be irrevocably lost. This paper describes an example based approach using patch matching from a multiple resolution multiple contrast atlas in order to change an image's resolution as well as its MR tissue contrast from one pulse-sequence to that of another. The use of this approach to generate different tissue contrasts (T2/PD/FLAIR) from a single T1-weighted image is demonstrated on both phantom and real images.
A principal goal of the collaboration between the neuroscience and medical imaging communities is the accurate segmentation of brain structures, with a view towards offering insight about the normal and abnormal features of a brain. Several methods1, 2 have been proposed to find cortical and sub-cortical structures. These methods are intrinsically dependent on the tissue contrast and resolution of the acquired data. In this paper, we propose a method to alter both the tissue contrast and resolution of a magnetic resonance (MR) image, thereby permitting image analysis techniques that would otherwise be inappropriate or ineffective.
MR tissue contrast and image resolution provided by the application of specific pulse sequences in MR imaging fundamentally determine or limit the performance of tissue classification algorithms3. If several images with different tissue contrasts (e.g., T1-weighted, T2-weighted, and PD-weighted) at high resolution can be obtained then optimal tissue classification solutions can be applied4, 5. But in many scenarios—e.g., routine clinical imaging of patients—it is not feasible for cost and time reasons to obtain such a rich data set, thus losing the opportunity for detailed image analysis. In neuroimaging, it is important to be able to automatically delineate the cortical gray matter and to observe white matter lesions. Yet there are inevitable tradeoffs made in choosing pulse sequences that will provide good contrast between the gray matter (GM) and the surrounding white matter (WM) and cerebro-spinal fluid (CSF) as well as delineating lesions. In particular, the tissue contrast between GM and WM is typically larger in a magnetization prepared rapid gradient echo (MPRAGE) image than in a spoiled gradient (SPGR) acquisition. On the other hand, the tissue contrast between CSF and GM is larger in an SPGR image. As well, white matter lesions can best be seen as hyperintensities on fluid-attenuated inversion recovery (FLAIR) images or6 T2-weighted images rather than either of the (MPRAGE or SPGR) T1-weighted images mentioned above. All of these observations can be seen in Fig. 1, where equivalent brain sections are shown with different MR tissue contrasts.
Delineation of small brain structures and accurate localization of object boundaries is also dependent on the image resolution. In particular, poor resolution yields blurring of boundaries and loss of image contrast in small structures. The technique we develop in this paper is capable of both changing the tissue contrast and improving the resolution of an MR image. The resulting image (or images) can then be used to carry out a more detailed analysis than would otherwise be possible.
Image hallucination7, 8 is often used to generate a high-resolution image from its multiple low-resolution acquisitions. There are two major categories in image hallucination, Bayesian9-11 and example-based12-14. Bayesian approaches are often formulated as a constrained optimization problem where the imaging process is known and the high resolution image is often the maximum likelihood estimator of a cost function given one or more low resolution images. Example based hallucination techniques are learning algorithms that rely on training data from a training set or atlas15 consisting of one or more high resolution images. These methods are primarily patch based, where a patch in the low-resolution image is matched to another patch in the training data. The similarity criteria are often chosen as image gradients, neighborhood information or textures16, 17.
In this paper, we extend the idea of atlas based hallucination by using patch matching to synthesize alternative contrast and high resolution MR images. We demonstrate the performance of the method using two applications. First, we synthesize different (T2/PD/FLAIR) tissue contrasts from a single T1-weighted (T1w) image, keeping the resolution the same, thus enhancing the capability of image analysis techniques that require different tissue contrasts. The utility of synthesizing alternate contrast is shown by generating a T1w MPRAGE image from its T1w SPGR acquisition. Then we convert a low resolution (LR) SPGR image to a high resolution (HR) MPRAGE image which has superior GM-WM contrast. We show that the overall delineation of the inner surface improves by such conversion.
Consider two registered MR images fM1 and fM2 of the same subject having different tissue contrasts (i.e., generated by different pulse sequences). In our experiments, the tissue contrasts can be any two of the following: T1w, T2w, PDw, or FLAIR. The images are related by the imaging process , which depends on underlying T1 and T2 relaxation times and other imaging parameters such as the pulse repetition time and the flip angle. In mathematical terms,
where η is a random noise and Θ comprises the imaging parameters. Ideally, if is known then fM2 can be directly estimated4. But for many studies Θ is not always known precisely and is difficult to model accurately, and it is therefore impractical to try to reconstruct fM2 from fM1 directly. Instead, our approach is to synthesize fM2 from fM1 using an atlas.
Define an atlas as N sets of triplets, , where are two tissue contrasts of the same subject having same resolution, and is a high-resolution image with tissue contrast M2. By assumption, and are registered to , ∀n. Also assume that fM1 and are made of 3D patches , centered at i Ωf, jn Ωgn, Jn ΩGn, respectively. Ωf, Ωgn and ΩGn are the image domains of and . Therefore, and is a low-resolution M1 patch of its high-resolution M2 patch . Defining as a down-sampling operator, if jn Ωgn and Jn Ωgn are corresponding locations in and , respectively, then jn = (Jn).
Assume that all the images f and are normalized in such a way that their WM peaks are the same. Using this definition of an atlas, a synthetic M2 image, having same resolution as fM1, can be generated by,
where is a non-local means operator18 and,
λ is a smoothing weight. makes sure that the patches are chosen such that the boundaries between two neighboring patches in the synthesized M2 remain smooth14. We use the following smoothness function,
where Ni and Njn are neighborhoods of the ith and jnth voxel, respectively, with i Ωf, jn Ωgn, Ni Ωf, Njn Ωgn.
Instead of just taking any patch that maximizes Eqn. 3, an average of the “best matching patches” are used. The “best matching patches” Jn are defined to be those for which the errors from Eqn. 3 are the lowest p% obtained from all the patches. We choose p = 3 in our experiments.
Define Ωi,n as the set of all best matching patches for the patch fM1(i) obtained from Eqn. 3 for nth pair of images . Clearly, Ωi,n Ωgn. Using this definition, the non-local means filtered patch is obtained by,
where β is a smoothing parameter on the NLM operator. In our experiments, β is chosen empirically, although it is possible to estimate it in a optimal way19.
Assuming M1 as T1w and M2 as T2w, the algorithm can be described for one atlas as :
We merge the learning based image hallucination idea12, 17 to our tissue contrast synthesis approach to synthesize different contrast as well as improve resolution. To synthesize a high-resolution M2 image M2 from a low-resolution M1 image fM1, Eqn. 2 is re-written as,
where n is obtained from Eqn. 3. −1 takes the low-resolution patch from Ωgn to the high-resolution domain ΩGn. This approach is similar to the example based super-resolution algorithms12, 14, except that we are using patches from a different tissue contrast image .
We use the Brainweb phantom20 to validate the contrast synthesis method. M1 is chosen as T1w (Fig. 3(d)), and its original T2w and PDw acquisitions shown in Fig. 3(e)-(f). For all our experiments, we choose N = 1. The atlas is another set of phantoms, consisting of one of each of T1w, T2w, and PDw acquisitions, as shown in Fig. 3(a)–(c). Because the image intensities are taken from a codebook, the mean square error between the original and the reconstructed image is not a meaningful measure of performance. Instead, we use normalized mutual information (NMI), a visual information fidelity metric21 (VIF), and a universal image quality index22 (UQI) to quantify similarity between the original and the synthetic images. The NMI between two images A and B is defined as where H(A) is the entropy of the image A and H(A, B) is the joint entropy between A and B, with H(A, A) = H(A). We want low NMI values between the original and the synthetic images. Ideally, the synthetic image should give an accurate representation of the original image, which implies a small joint entropy between them.
The NMI between original T2 and synthetic T2 is 0.7464, while it is 0.7597 between the original PD and the synthetic one. VIF and UQI take two 2D images and return a number between 0 and 1. 1 is achieved only when the images are same or one is a scalar multiple of another. We plot UQI and VIF metrics in Fig. 3(i)-(j) for each slice of the 3D volumes between original and the synthetic T2 and PD. It is observed that the similarity is high on average.
We use our algorithm to synthesize T2 and FLAIR images of a normal subject from its T1w SPGR acquisition for which we also have the actual T2 and FLAIR available for comparison. Fig. 4(a)-(c) show the T1, T2 and FLAIR images of another subject, which is used as the atlas. Fig. 4(g)-(h) show the synthetic T2 and FLAIR of the test subject in Fig. 4(d)-(f). Fig. 4(f) shows that the atlas FLAIR has a better contrast in GM-WM boundary compared to the test FLAIR. This is reflected in the synthetic FLAIR also. This highlights the benefit of our method, where a new contrast is created from the atlas, instead of the contrast reconstruction from the test image.
The NMI between the original and the reconstructed T2 and FLAIR are 0.3697 and 0.3102, respectively. NMI, being a distribution dependent statistic, is sensitive to the actual distribution of the intensities rather than the contrast. The NMI numbers for the phantom validation are larger than those of the real data, because the Brainweb phantoms have widely different histograms while keeping the same contrast, while the real data have similar histograms. The plot of UQI and VIF for each slice is also shown in Fig. 4(i)-(j).
The next experiment on real data consists of synthesizing an MPRAGE image from its SPGR acquisition of the same resolution, because MPRAGE images are of importance for their superior GM-WM contrast. The required atlas consists of a pair of SPGR and MPRAGE acquisitions of another subject. Fig. 5(a)-(b) shows the true SPGR and true MPRAGE acquisitions, with Fig. 5(c) being the same resolution synthetic MPRAGE image. The NMI between the original MPRAGE and the synthetic MPRAGE is 0.3254, while it is 0.3341 between the original MPRAGE and the original SPGR.
We show that using both contrast synthesis and resolution enhancementment leads to improved delineation of cortical surfaces. We use M1 as SPGR and M2 as MPRAGE. MPRAGE images, having superior GM-WM contrast, are a better candidate for the delineation of the inner cortical surface compared to SPGR images. So in the absence of an MPRAGE image, we could synthesize one, thus enabling better delineation. Our data set contains a 1.875×1.875×3 mm low-resolution (LR) SPGR image fM1, its high resolution (HR) 0.9375×0.9375 1.5 mm true MPRAGE acquisition FM2 and true HR SPGR acquistion FM1, shown in Fig. 6(a)-(c), respectively. Using Eqn. 7, a HR MPRAGE image M2 is generated, shown in Fig. 6(d). The cortical inner surfaces are found using CRUISE23. For comparison, “the best available truth” or a “reference” standard of the inner surface is obtained from FM2, with which we compare the surface obtained from M2 and FM1. Fig. 7(a) and (b) show how the lack of GM-WM contrast gives rise to a poor cortical inner surface reconstruction for FM1 when compared to the reference “true” reconstruction from FM2 and the reconstruction from our method, M2. Table 1 shows that the mean differences for four subjects between their inner surfaces as generated from M2 and FM1 as compared to FM2. The smaller differences show our super-resolution approach gives a marginal improvement in the delineation of the inner surface.
The similarity criteria in Eqn 3 is chosen to be the L2 norm, the underlying assumption being the intensities of f and differ by a Gaussian noise, or they follow a similar distribution. If the intensity distributions are different, then L2 norm fails to produce correct matches from Eqn. 3. Also, we used one atlas for our experiments, but we believe using more than one atlas helps in finding more accurate Jn. Involving more complex similarity criteria like gradient or texture into Eqn. 3 will also produce more accurate Jn. Some of the parameters, like λ in Eqn. 3 or β in Eqn. 6 are required to be estimated by cross-validation. Also, the smoothness of the image depends on the amount p% of “best matching patches” used, which also needs to be estimated by cross validation.
In summary, we proposed an atlas based image synthesis technique to generate different MR tissue contrasts from a single image acquisition. It is essentially a patch-matching algorithm where a template patch from a test image is matched onto a multi-modal multi-resolution atlas and patches from the atlas are used to generate a synthetic alternate tissue contrast high resolution image. The contribution of this method is that new MR contrasts can be synthesized from the atlas, and unlike histogram matching, this method uses local contextual information to synthesize images. We have validated our method on Brainweb phantoms, and showed that T2, PD, and FLAIR images can be generated from a single T1w acquisition. We also demonstrated that a synthetic high-resolution MPRAGE image can be generated from its low-resolution SPGR acquisition, which leads to improved cortical segmentation. So far our experiments have been carried out on normal subjects. Future work includes reconstruction of alternate tissue contrasts like T2 and FLAIR on patients with lesions.
This research was supported in part by the Intramural Research Program of the NIH, National Institute on Aging. We are grateful to all the participants of the Baltimore Longitudinal Study on Aging (BLSA), as well as the neuroimaging staff for their dedication to these studies.