Search tips
Search criteria 


Logo of nihpaAbout Author manuscriptsSubmit a manuscriptHHS Public Access; Author Manuscript; Accepted for publication in peer reviewed journal;
Conf Proc IEEE Eng Med Biol Soc. Author manuscript; available in PMC 2017 September 19.
Published in final edited form as:
PMCID: PMC5603287

A framework for informing segmentation of in vivo MRI with information derived from ex vivo imaging: Application in the medial temporal lobe


Automatic segmentation of cortical and subcortical structures is commonplace in brain MRI literature and is frequently used as the first step towards quantitative analysis of structural and functional neuroimaging. Most approaches to brain structure segmentation are based on propagation of anatomical information from example MRI datasets, called atlases or templates, that are manually labeled by experts. The accuracy of automatic segmentation is usually validated against the “bronze” standard of manual segmentation of test MRI datasets. However, good performance vis-a-vis manual segmentation does not imply accuracy relative to the underlying true anatomical boundaries. In the context of segmentation of hippocampal subfields and functionally related medial temporal lobe cortical subregions, we explore the challenges associated with validating existing automatic segmentation techniques against underlying histologically-derived anatomical “gold” standard; and, further, developing automatic in vivo MRI segmentation techniques informed by histological imaging.

I. Introduction

The medial temporal lobe (MTL) is a brain region that has received considerable attention in the neuroimaging literature due to its primary involvement in the declarative memory and its early selective vulnerability to pathologies associated with dementia, epilepsy, and other neurological and psychiatric disorders [1]. The MTL structures primarily involved in memory are the hippocampus and the parahippocampal gyrus. The hippocampus is a complex and heterogeneous structure that is constituted by several anatomically and functionally distinct subfields, including cornu ammonis 1–3 (CA1-3), dentate gyrus (DG), and subiculum. The parahippocampal gyrus is also formed by functionally distinct subregions: the entorhinal cortex (ERC), perirhinal cortex (PRC), and parahippocampal cortex (PHC). Advances in MRI technology have made it possible to visualize different hippocampal layers in routine MRI scans, which led to increased interest in recent years in quantitative analysis of hippocampal subfields and extrahippocampal MTL subregions using high-resolution structural and functional MRI [2]. Quantitative analysis usually requires the structure of interest, in this case MTL subregions, to be segmented. Over the last decade, over 20 protocols for manual segmentation of MTL subregions have been developed [2], [3], and three automatic segmentation techniques have become available [4], [5], [6]. However, currently, the segmentation of MTL subregions on in vivo MRI, either manual or automated, is based on heuristic rules, as many of the actual anatomical boundaries between subregions cannot be observed consistently. Presently, both manual segmentation protocols and automatic segmentation methods are validated in terms of reliability; for manual segmentation the standard test is to measure the agreement (volume, overlap) between segmentations of the same image by two or more experts (inter-rater reliability), or by the same expert at different times (intra-rater reliability). For automatic segmentation, the standard is to report agreement with manual segmentation. Such reliability measures, while not without value, do not reflect the anatomical accuracy of the segmentation. The resulting uncertainty about the accuracy of in vivo segmentation poses challenges for interpretation of the results of MTL neuroimaging studies, and especially for relating these results to broader neuroscience literature.

Therefore, important goals for the MTL neuroimaging field are (1) to use anatomical accuracy, and not only reliability, for validation of in vivo MRI segmentation; and (2) to develop more anatomically accurate manual segmentation protocols and automatic segmentation techniques. We believe that meeting both goals will require extensive use of ex vivo histological imaging and fusion of information between the in vivo MRI and ex vivo histology domains. This paper describes potential strategies for addressing these goals, particularly in the context of automatic segmentation.

II. In Vivo MTL Imaging and Segmentation

In T1-weighted MRI scans with approximately 1mm × 1mm × 1mm resolution, arguably the most common type of scan in brain morphometry studies, the hippocampus appears largely homogeneous. However, much better contrast between hippocampal layers can be attained using T2-weighted imaging with slices oriented along the main axis of the hippocampus, and high in-plane resolution (e.g., 0.4mm × 0.4mm × 2mm). Interestingly, these T2 scans also reveal much better contrast between meninges and the MTL cortex, as shown in Fig. 1. Virtually all manual segmentation protocols for hippocampal subfields and MTL subregions use these high-resolution T2-weighted MRI scans. However, recent comparison of 20 such protocols [3] revealed considerable variation in the placement of subfield boundaries, and a broad community effort is currently underway to develop a harmonized protocol that would combine strong reliability with anatomical accuracy (

Fig. 1
Comparison of high-resolution T2-weighted and standard T1-weighted MRI in the same subject. T2-weighted MRI reveals hippocampal layers and better distinguishes between cortical gray matter and meninges.

A few techniques for automatic segmentation of MTL substructures from in vivo MRI have been developed. In [5], [7], a multi-atlas segmentation algorithm “ASHS” (Fig. 2) is used to propagate subregion labels from a set of 20–30 expert-labeled T2-weighted MRI scans to unlabeled MRI scans. Comparison with manual segmentation yields Dice coefficient of 0.8 and higher for large hippocampal subfields CA1 and DG and 0.75 and higher for subiculum, ERC and PRC. A multi-atlas technique that requires very few expert labeled atlases was separately developed for this problem in [6], but yielded lower Dice coefficients (0.5–0.6 range) in comparison to manual segmentation. Many studies have also quantified hippocampal subfield using 1mm × 1mm × 1mm, using either shape analysis techniques [8] or a recent expectation-maximization segmentation technique with a high-resolution ex vivo atlas which is implemented in the FreeSurfer software [4]. However, given the difficulty in performing manual segmentation in T1-weighted MRI, this segmentation approach has not yet been evaluated in terms of reliability relative to manual segmentation. A particular limitation of these automated techniques is that they have not been validated against the anatomical ground truth, but, at best, against manual segmentation with heuristic protocols.

Fig. 2
Examples of MTL subregion segmentation using ASHS in 3T and 7T in vivo MRI scans. Both examples show the median performance in a set of cross-validation experiments reported in [5] and [7].

III. Strategies and Datasets for Ex Vivo Validation of In Vivo MRI Segmentation

Since in vivo MRI does not offer sufficient features to unequivocally determine MTL subregional boundaries, development of the anatomical ground truth will require histological imaging of ex vivo specimens. Furthermore, it will require datasets in which in vivo MRI can be linked and mapped to the underlying histological imaging.1 This poses two challenges. Firstly, in vivo MRI and ex vivo histology would need to be obtained for the same set of subjects. Such a study can be done prospectively, for example by recruiting patients with terminal disease; or retrospectively, by obtaining specimens from autopsies in patients who had in vivo MRI as part of their clinical care or research study participation during life. Our group has taken the second approach. Over 1000 older adults participating in research studies of dementia, or seen at our memory clinic, have had T2-weighted high-resolution MRI scans of the MTL done in the past five years. Many of these patients and research study participants have consented to autopsy as part of related research studies. As the result, in approximately 30% of autopsies performed at the Penn Center for Neurodegenerative Research, these in vivo T2-weighted MRI are available retrospectively. We have obtained intact MTL specimens in 10 such autopsies in a project that is ongoing.

The second challenge involves finding spatial correspondences between in vivo MRI to histology for each subject. Histology offers virtually unlimited resolution, but it is two-dimensional, has very different contrast characteristics from MRI, and the physical processing of tissue causes shrinking, tearing, and other artifacts that are not present in the MRI. Furthermore, in the interval between when in vivo MRI and histology are acquired, the MTL may undergo changes due to aging, the dying process, brain extraction, and fixation. Directly mapping histology to in vivo MRI would seem to be a daunting challenge. However, high-resolution MRI of the ex vivo specimens prior to histology can offer an intermediate three-dimensional image that shares appearance characteristics with the in vivo MRI and is closer in structure and resolution to the histology. Furthermore, ex vivo MRI not only has the potential to facilitate the mapping of information between in vivo MRI and histology, but also to study the anatomical variability of the MTL at greater detail in 3D.

A. Alignment of In Vivo and Ex Vivo MRI in Same Subjects

We have collected ex vivo MRI of the intact MTL in 10 autopsy cases in which subjects had antemortem in vivo MRI within last 3 years of life. Subjects were diagnosed with either frontotemporal dementia or prodromal or probable AD during life. MRI scans with 0.2 × 0.2 × 0.2mm3 resolution were acquired overnight on a Varian 9.4 tesla scanner. Scans were co-registered by manually tracing the outline of the hippocampus in both sets of scans, and applying rigid followed by deformable registration [10] with large amounts of regularization to the resulting binary images. This resulted in good alignment of hippocampal layers between the in vivo and ex vivo images in all subjects across the whole hippocampus (median distance between the mid-surfaces of the hypointense myelinated hippocampal layers traced manually in the ex vivo and in vivo scans was 0.29 mm after rigid alignment and 0.25 after deformable alignment – less than the in vivo MRI voxel size). Fig. 3 compares in vivo and ex vivo MRI of the same subject side by side.

Fig. 3
Aligned in vivo 3 tesla and ex vivo 9.4 tesla MRI of the hippocampal head (a–d) and hippocampal body (e–h). The shape of the hippocampus and of white matter band (white arrow in a and b) can be appreciated in both the in vivo and ex vivo ...

B. Alignment of Ex Vivo MRI and Histology

Ex vivo MRI and histology was performed on 10 MTL specimens (this is a different set of specimens from the one above, and in vivo MRI is available for only 2 of these 10 cases). For each specimen, an MRI scan covering the whole MTL was obtained with 0.2mm × 0.2mm × 0.2mm resolution, after which the MTL was cut into 1cm thick slabs, each slab again scanned at 0.2mm × 0.2mm × 0.2mm resolution, embedded in paraffin, sectioned at approximately 0.2mm intervals, prepared with the Kluver-Barrera stain [11], and scanned at 0.5μm × 0.5μm resolution.

While previously we employed an automatic registration pipeline [12] for 3D reconstruction of histology stacks and alignment to the 3D MRI volume, current work relies on manual reconstruction using an interactive tool HistoloZee ( The interactive tool allows concurrent visualization of multiple histology slices and MRI, rotation, translation and scaling of histology slices in-plane, 3D rotation and translation of the MRI volume. While much more labor-intensive, we found manual alignment to lead to better overall reconstruction of histology volumes, particularly for slices with tears, partial coverage of the MTL, and poor staining. An example of reconstructed histology is shown in Fig. 4.

Fig. 4
3D reconstruction of histological slices into the MRI space. Top row: axial views of reconstructed histology, MRI of 1cm thick tissue slabs, and MRI of intact MTL. Bottom row: corresponding sagittal views, with hippocampal subfield labels overlaid in ...

Fig. 5 shows a preliminary result comparing ASHS automatic segmentation of the subfields in the anterior hippocampus on in vivo 3 tesla MRI with ex vivo MRI and histology in the same subject. The segmentations show similarities in general, though several boundaries and microstructural transitions cannot be visualized in vivo or reliably segmented. Effort is underway to obtain serial histology in the specimens for which in vivo MRI is available, align histology to the ex vivo MRI using HistoloZee, and validate current automatic segmentation algorithms using 10–20 images.

Fig. 5
Preliminary comparison of automatic segmentation of in vivo MRI to the manual segmentation in histology for the same subject. Ex vivo MRI is used as the intermediate modality to match information between histology and in vivo MRI. It should also be noted ...

IV. Strategies for Leveraging Detailed Anatomical Information from Ex Vivo Imaging for Improving In Vivo Segmentation

Beyond validation against anatomical ground truth, the same-subject in vivo/ex vivo imaging dataset can be used to imbibe the ASHS algorithm with richer anatomical information, which in turn should improve its anatomical accuracy. As the first step, the current set of atlases in ASHS, which consists of 29 MRI scans manually labeled using a heuristic rule-based protocol, should be replaced by the in vivo MRI scans of subjects for whom ex vivo histology is available, with anatomical labels derived from histology and mapped onto the in vivo MRI domain using within-subject deformable registration. However, ex vivo MRI offers additional potential improvements for registration, by providing a high-resolution 3D anatomical reference that can be used as a prior to guide and constrain in vivo segmentation. Toward building such a prior, we have used groupwise deformable registration [10] to construct an unbiased template of the hippocampus from a set of 25 ex vivo MRI scans (Fig. 6). This template captures the average hippocampal shape and the primary modes of variation in hippocampal shape. This information can in turn be used to constrain ASHS segmentation to stay within the range of feasible topology and shape.

Fig. 6
Left: sagittal and coronal views of the unbiased hippocampus template obtained by groupwise registration of 25 ex vivo specimen MRI scans. Right: Hippocampal subfield labels, mapped from 10 histology datasets into the corresponding ex vivo MRI scan and ...

Fig. 7 shows a pilot result illustrating the feasibility of integrating ex vivo data into ASHS. Since histology is not yet available for most of the in vivo/ex vivo dataset, this result was generated using 8 pairs of in vivo and manually labeled ex vivo MRI scans of the same subjects. A target image was first labeled by multi-atlas label fusion [13], with in vivo MRIs used as atlases and anatomical labels propagated from the ex vivo MRI to in vivo MRI. The ex vivo MRI template in Fig. 6 was then deformed to the multi-atlas segmentation, providing a more detailed and topologically consistent segmentation of the target image. The added advantage of the template is that it allows the mapping of other histologically derived information into the target image domain. As a proof of concept, Fig. 7(v) projects a map of cell body density, extracted from a histology dataset and mapped into the ex vivo MRI template, into the target in vivo MRI image.

Fig. 7
Pilot result illustrating the feasibility of integrating ex vivo data into ASHS. See text for details.

V. Discussion and Conclusions

Validating automatic and manual segmentation using ground truth derived from histology is the best way to ensure anatomical accuracy of these techniques. Obtaining and aligning histology, ex vivo MRI and in vivo MRI in the same subjects offers a way to perform such validation, and even to integrate histologically-derived anatomical information into the automatic in vivo segmentation algorithm. However, doing so requires a great deal of data collection, annotation, and registration effort.


This work was supported by NIH grants R01 AG037376 and R01 EB017255.


1One important domain in which there are alternatives to ex vivo validation is epilepsy, where the MTL is frequently surgically removed. In [9], the authors analyzed in vivo MRI and histology data in 15 subjects.


1. Small S, Schobel S, Buxton R, Witter M, Barnes C. A pathophysiological framework of hippocampal dysfunction in ageing and disease. Nature Reviews Neuroscience. 2011;12(10):585–601. [PMC free article] [PubMed]
2. de Flores R, La Joie R, Chételat G. Structural imaging of hippocampal subfields in healthy aging and Alzheimer’s disease. Neuroscience. 2015 Nov;309:29–50. [PubMed]
3. Yushkevich PA, Amaral RSC, Augustinack JC, Bender AR, Bernstein JD, Boccardi M, Bocchetta M, Burggren AC, Carr VA, Chakravarty MM, Chételat G, Daugherty AM, Davachi L, Ding S-L, Ekstrom A, Geerlings MI, Hassan A, Huang Y, Iglesias JE, La Joie R, Kerchner GA, LaRocque KF, Libby LA, Malykhin N, Mueller SG, Olsen RK, Palombo DJ, Parekh MB, Pluta JB, Preston AR, Pruessner JC, Ranganath C, Raz N, Schlichting ML, Schoemaker D, Singh S, Stark CEL, Suthana N, Tompary A, Turowski MM, Van Leemput K, Wagner AD, Wang L, Winterburn JL, Wisse LEM, Yassa MA, Zeineh MM, for the Hippocampal Subfields Group (HSG) Quantitative comparison of 21 protocols for labeling hippocampal subfields and parahippocampal subregions in in vivo MRI: Towards a harmonized segmentation protocol. Neuroimage. 2015 Jan; [PMC free article] [PubMed]
4. Iglesias JE, Augustinack JC, Nguyen K, Player CM, Player A, Wright M, Roy N, Frosch MP, McKee AC, Wald LL, Fischl B, Van Leemput K, Alzheimer’s Disease Neuroimaging Initiative A computational atlas of the hippocampal formation using ex vivo, ultra-high resolution MRI: Application to adaptive segmentation of in vivo MRI. Neuroimage. 2015 Jul;115:117–37. [PMC free article] [PubMed]
5. Yushkevich PA, Pluta JB, Wang H, Xie L, Ding S-L, Gertje EC, Mancuso L, Kliot D, Das SR, Wolk DA. Automated volumetry and regional thickness analysis of hippocampal subfields and medial temporal cortical structures in mild cognitive impairment. Human Brain Mapping. 2014 [Online]. Available: [PMC free article] [PubMed]
6. Pipitone J, Park MTM, Winterburn J, Lett TA, Lerch JP, Pruessner JC, Lepage M, Voineskos AN, Chakravarty MM, the Alzheimer’s Disease Neuroimaging Initiative Multi-atlas segmentation of the whole hippocampus and subfields using multiple automatically generated templates. Neuroimage. 2014 Apr; [PubMed]
7. Wisse LEM, Kuijf HJ, Honingh AM, Wang H, Pluta JB, Das SR, Wolk DA, Zwanenburg JJM, Yushkevich PA, Geerlings MI. Automated hippocampal subfield segmentation at 7T MRI. AJNR Am J Neuroradiol. 2016 Feb; [PMC free article] [PubMed]
8. Wang L, Miller JP, Gado MH, McKeel DW, Rothermich M, Miller MI, Morris JC, Csernansky JG. Abnormalities of hippocampal surface structure in very mild dementia of the Alzheimer type. Neuroimage. 2006 Mar;30(1):52–60. [Online]. Available: [PMC free article] [PubMed]
9. Goubran M, Bernhardt BC, Cantor-Rivera D, Lau JC, Blinston C, Hammond RR, de Ribaupierre S, Burneo JG, Mirsattari SM, Steven DA, Parrent AG, Bernasconi A, Bernasconi N, Peters TM, Khan AR. In vivo mri signatures of hippocampal subfield pathology in intractable epilepsy. Hum Brain Mapp. 2016 Mar;37(3):1103–19. [PubMed]
10. Avants B, Gee JC. Geodesic estimation for large deformation anatomical shape averaging and interpolation. Neuroimage. 2004;23(Suppl 1):S139–S150. [PubMed]
11. Kluever H, Barrera E. A method for the combined staining of cells and fibers in the nervous system*. Journal of Neuropathology & Experimental Neurology. 1953;12(4):400. [PubMed]
12. Adler DH, Pluta J, Kadivar S, Craige C, Gee JC, Avants BB, Yushkevich PA. Histology-derived volumetric annotation of the human hippocampal subfields in postmortem MRI. Neuroimage. 2014 Jan;84:505–23. [PMC free article] [PubMed]
13. Wang H, Suh JW, Das SR, Pluta J, Craige C, Yushkevich PA. Multi-atlas segmentation with joint label fusion. IEEE Trans Pattern Anal Mach Intell. 2013 Mar;35(3):611–623. [PMC free article] [PubMed]