Home | About | Journals | Submit | Contact Us | Français |

**|**HHS Author Manuscripts**|**PMC4854961

Formats

Article sections

- Abstract
- I. Introduction
- II. A Generative Brain Lesion Segmentation Model
- III. Discriminative Extensions
- IV. Experiment 1: Properties and Performance of the Generative Model
- V. Experiment 2: Properties and Performances of the Generative-Discriminative Model Extensions
- VI. Summary and Conclusions
- Supplementary Material
- References

Authors

Related links

IEEE Trans Med Imaging. Author manuscript; available in PMC 2017 April 1.

Published in final edited form as:

Published online 2015 November 20. doi: 10.1109/TMI.2015.2502596

PMCID: PMC4854961

NIHMSID: NIHMS775645

Bjoern H. Menze,^{1,}^{2,}^{3,}^{4,}^{5} Koen Van Leemput,^{6,}^{7} Danial Lashkari,^{1} Tammy Riklin-Raviv,^{8} Ezequiel Geremia,^{2} Esther Alberts,^{4,}^{5} Philipp Gruber,^{9} Susanne Wegener,^{9} Marc-André Weber,^{10} Gabor Székely,^{3} Nicholas Ayache,^{2} and Polina Golland^{1}

Personal use is permitted, but republication/redistribution requires IEEE permission.

The publisher's final edited version of this article is available at IEEE Trans Med Imaging

See other articles in PMC that cite the published article.

We introduce a generative probabilistic model for segmentation of brain lesions in multi-dimensional images that generalizes the *EM segmenter*, a common approach for modelling brain images using Gaussian mixtures and a probabilistic tissue atlas that employs expectation-maximization (EM) to estimate the label map for a new image. Our model augments the probabilistic atlas of the healthy tissues with a latent atlas of the lesion. We derive an estimation algorithm with closed-form EM update equations. The method extracts a latent atlas prior distribution and the lesion posterior distributions jointly from the image data. It delineates lesion areas individually in each channel, allowing for differences in lesion appearance across modalities, an important feature of many brain tumor imaging sequences. We also propose discriminative model extensions to map the output of the generative model to arbitrary labels with semantic and biological meaning, such as “tumor core” or “fluid-filled structure”, but without a one-to-one correspondence to the hypo-or hyper-intense lesion areas identified by the generative model.

We test the approach in two image sets: the publicly available BRATS set of glioma patient scans, and multimodal brain images of patients with acute and subacute ischemic stroke. We find the generative model that has been designed for tumor lesions to generalize well to stroke images, and the generative-discriminative model to be one of the top ranking methods in the BRATS evaluation.

Gliomas are the most frequent primary brain tumors. They originate from glial cells and grow by infiltrating the surrounding tissue. The more aggressive form of this disease is referred to as “high-grade” glioma. The tumor grows fast and patients often have survival times of two years or less, calling for immediate treatment after diagnosis. The slower growing “low-grade” disease comes with a life expectancy of five years or more, allowing the aggressive treatment to be delayed. Extensive neuroimaging protocols are used before and after treatment, mapping different tissue contrasts to evaluate the progression of the disease or the success of a chosen treatment strategy. As evaluations are often repeated every few months, large longitudinal datasets with multiple modalities are generated for these patients even in routine clinical practice. In spite of the need for accurate information to guide decision making process for an treatment, these image series are primarily evaluated using qualitative criteria – indicating, for example, the presence of characteristic hyperintense intensity changes in contrast-enhanced T1 MRI – or relying on quantitative measures that are as basic as calculating the largest tumor diameter that can be recorded in a set of axial images.

While an automated and reproducible quantification of tumor structures in multimodal 3D and 4D volumes is highly desirable, it remains difficult. Glioma is an infiltratively growing tumor with diffuse boundaries and lesion areas are only defined through intensity changes *relative* to surrounding normal tissues. As a consequence, the outlines of tumor structures cannot be easily delineated – even manual segmentations by expert raters show a significant variability [1] – and common MR intensity normalization strategies fail in the presence of extended lesions. Tumor structures show a significant amount of variation in size, shape, and localization, precluding the use of related mathematical priors. Moreover, the mass effect induced by the growing lesion may lead to displacements of the normal brain tissues, as well as a resection cavity that is present after treatment, limits the reliability of prior knowledge available for the healthy parts of the brain. Finally, a large variety of imaging modalities can be used for mapping tumor-related tissue changes, providing different types of biological information, such as differences in tissue water (T_{2}-MRI, FLARI-MRI), enhancement of contrast agents (post-Gadolinium T_{1}-MRI), water diffusion (DTI), blood perfusion (ASL-, DSC-, DCE-MRI), or relative concentrations of selected metabolites (MRSI). A segmentation algorithm must adjust to any of these, without having to collect large training sets, a common limitation for many data-driven learning methods.

Brain tumor segmentation has been the focus of recent research, most of which is dealing with glioma [2], [3]. Few methods have been developed for less frequent and less aggressive tumors [4], [5], [6], [7]. Tumor segmentation methods often borrow ideas from other brain tissue and other brain lesion segmentation methods that have achieved a considerable accuracy [8]. Brain lesions resulting from traumatic brain injuries [9], [10] and stroke [11], [12] are similar to glioma lesions in terms of size and multimodal intensity patterns, but have attracted little attention so far. Most automated algorithms for brain lesion segmentation rely on either generative or discriminative probabilistic models at the core of their processing pipeline. Many encode prior knowledge about spatial regularity and tumor structures, and some offer longitudinal extensions for 4D image volumes to exploit longitudinal image sets that are becoming increasingly available [13], [14].

Generative probabilistic models of spatial tissue distribution and appearance have enjoyed popularity for tissue classification as they exhibit good generalization performance [15], [16], [17]. Encoding spatial prior knowledge for a lesion, however, is challenging. Tumors may be modeled as outliers relative to the expected shape [18], [19] or to the image signal of healthy tissues [16], [20]. In [16], for example, a criterion for detecting outliers is used to generate a tumor prior in a subsequent EM segmentation that treats the tumor as an additional tissue class. Alternatively, the spatial prior for the tumor can be derived from the appearance of tumor-specific markers, such as Gadolinium enhancements [21], [22], or from using tumor growth models to infer the most likely localization of tumor structures for a given set of patient images [23]. All these segmentation methods rely on registration to align images and the spatial prior. As a result, joint registration and tumor segmentation [17], [24] and joint registration and estimations of tumor displacement [25] have been investigated, as well as the direct evaluation of the deformation field for the purpose of identifying the tumor region [7], [26].

Discriminative probabilistic models directly learn the differences between the appearance of the lesion and other tissues from the data. Although they require substantial amounts of training data to be robust to artifacts and variations in intensity and shape, they have been applied successfully to tumor segmentation tasks [27], [28], [29], [30], [31]. Discriminative approaches proposed for tumor segmentation typically employ dense, voxel-wise features from anatomical maps [32] or image intensities, such as local intensity differences [33], [34] or intensity profiles, that are used as input to inference algorithms such as support vector machines [35], decision trees ensembles [32], [36], [37], or deep learning approaches [38], [39]. All methods require the imaging protocol to be exactly the same in the training set and in the novel images to be segmented. Since local intensity variation that is characteristic of MRI is not estimated during the segmentation process, as in most generative mixture models, calibration of the image intensities becomes necessary which is already a difficult task in the absence of lesions [40], [41], [42].

Advantageous properties of generative and discriminative probabilistic models have been combined for a number of applications in medical imaging: generative approaches can be used for model-driven dimensional reduction to form a low-dimensional basis for a subsequent discriminative method, for example, in whole brain classification of Alzheimer’s patients [43]. Vice versa, a discriminative model may serve as a filter to constrain the search space for employing complex generative models in a subsequent step, for example, when fitting biophysical metabolic models to MRSI signals [44], or when fusing evidence across different anatomical regions in the analysis of contrast-enhancing structures [45]. Other approaches improve the output of a discriminative classification of brain scans by adding prior knowledge on the location of subcortical structures [46] or the skull shape [47] through generative models. The latter approach for skull stripping showed superior robustness in particular on images of glioma patients [48]. To the best of our knowledge no generative-discriminative model has been used for tumor analysis so far, although the advantages of employing a secondary discriminative classifier on the probabilistic output of a first level discriminative classifier, which considers prior knowledge on expected anatomical structures of the brain, has been demonstrated in [32].

Spatial regularity and spatial arrangement of the 3D tumor sub-structure is used in most generative and discriminative segmentation techniques, often in a postprocessing step and with extensions along the temporal dimension for longitudinal tasks: Local regularity of tissue labels can be encoded via boundary modeling within generative [16], [49] and discriminative methods [27], [28], [50], [49], or by using Markov random field priors [30], [31], [51]. Conditional random fields help to impose structures on the adjacency of specific labels and, hence, impose constraints on the wider spatial context of a pixel [29], [35]. 4D extensions enforce spatial contiguity along the time dimension either in an undirected fashion [52], or in a directed one when imposing monotonic growth constraints on the segmented tumor lesion acting as a non-parametric growth model [13], [53], [14]. While all these segmentation models act locally, more or less at the pixel level, other approaches consider prior knowledge about the global location of tumor structures. They learn, for example, the relative spatial arrangement of tumor structures such as tumor core, edema, or enhancing active components, through hierarchical models of super-voxel clusters [54], [34], or by relating image patterns with phenomenological tumor growth models that are adapted to the patient [25].

In this paper we address three different aspects of multi-modal brain lesion segmentation, extending preliminary work we presented earlier in [55] [56], [57]:

- We propose a new generative probabilistic model for channel-specific tumor segmentation in multi-dimensional images. The model shares information about the spatial location of the lesion among channels while making full use of the highly specific multimodal, i.e., multivariate, signal of the healthy tissue classes for segmenting normal tissues in the brain. In addition to the tissue type, the model includes a latent variable for each voxel encoding the probability of observing a tumor at that voxel, similar to [49], [50]. The probabilistic model formalizes qualitative biological knowledge about hyper- and hypo-intensities of lesion structures in different channels. Our approach extends the general
*EM segmentation*algorithm [58], [59] using probabilistic tissue atlases [60], [15], [61] for situations when specific spatial structures cannot be described sufficiently through population priors. - We illustrate the excellent generalization performance of the generative segmentation algorithm by applying it to MR images of patient with ischemic stroke, which – to the best of our knowledge – is one of the first automated segmentation algorithms for this major neurological disease.
- We extend the generative model to a joint generative-discriminative method that compensates for some of the shortcomings of both the generative and the discriminative modeling approach. This strategy enables us to predict clinically meaningful tumor tissue labels and not just the channel-specific hyper- and hypo-intensities returned by the generative model. The discriminative classifier uses the output of the generative model, which improves its robustness against intensity variation and changes in imaging sequences. This generative-discriminative model defines the state-of-the-art on the public BRATS benchmark data set [1].

In the following we introduce the probabilistic model (Section II), derive the segmentation algorithm and additional biological constraints, and we describe the discriminative model extensions (Section III). We evaluate the properties and performance of the generative and the generative-discriminative methods on a public glioma dataset (Sections IV and V, respectively), including an experiment on the transfer of the generative model to images from stroke patients. We conclude with a discussion of the results and of future research directions (Section VI).

Generative models consider prior information about the structure of the observed data and exploit such information to estimate latent structure from new data. The *EM segmenter*, for example, models the image of a healthy brain through three tissue classes [60], [15], [61]. It encodes their approximate spatial distribution through a population atlas generated by aligning a larger set of reference scans, segmenting them manually, and averaging the frequency of each tissue class in a given voxel within the chosen reference frame. Moreover, it assumes that all voxels of a tissue class have about the same image intensity which is modeled through a Gaussian distribution. This segmentation method, whose parameters can be estimated very efficiently through the expectation maximization (EM) procedure, treats image intensities as nuisance parameters which makes it robust in the presence of the characteristic variability of the intensity distributions of MR images. Moreover, since the method formalizes the image content explicitly through the probabilistic model, it can be combined with other parametric transformations, for example, for registration [62] or bias field correction [15], and account for the related changes in the observed data. Generative models with tissue atlases used as spatial priors are at the heart of most advanced image segmentation models in neuroimaging [63], [64].

Population atlases cannot be generated for tumors as their location and extensions vary significantly across patients. Still, the tumor location is similar in different MR images of the *same* patient and a patient-specific atlas of the lesion class could be generated. Segmentation and atlas building can be performed simultaneously, in a joint estimation procedure [50]. Here, the key idea is to model the lesions through a separate latent atlas class. Combined with the standard population atlas of the normal tissues and the standard EM segmentation framework, this extends the *EM segmenter* to multimodal or longitudinal images of patients with a brain lesion. The generative model is illustrated in Fig. 1.

Graphical model for the proposed segmentation approach. Voxels are indexed with *i*, channels are indexed by *c*. The known prior *π*_{k} determines the label *k* of the normal healthy tissue. The latent atlas *α* determines the channel-specific presence **...**

We model the *normal* healthy tissue label *k _{i}* of voxel

$$p\left({k}_{i}=k\right)={\mathrm{\pi}}_{\mathit{ki}},$$

(1)

where
${\sum}_{k}{\mathrm{\pi}}_{\mathit{ki}}=1$ for all *i* and
${\mathrm{\pi}}_{\mathit{ki}}\ge 0$ for all *i*, *k*. The tissue label *k _{i}* is shared among all

We model the *tumor* label using a spatially varying latent probabilistic atlas *α* [49], [50] that is specific to the given patient (Fig. 1, red). At each voxel *i*, this atlas contains a scalar parameter *α _{i}* that defines the probability of observing a tumor at that voxel, forming the 3D parameter volume

$$p\left({\mathit{t}}_{i};{\alpha}_{i}\right)={\displaystyle \prod _{c}p\left({t}_{i}^{c};{\alpha}_{i}\right)={\displaystyle \prod _{c}{\alpha}_{i}^{{t}_{i}^{c}}\cdot {\left(1-{\alpha}_{i}\right)}^{1-{t}_{i}^{c}}}}.$$

(2)

Here, we assume tumor occurrence to be independent from the type of the underlying healthy tissue. We will introduce conditional dependencies between the underlying tissue class and the likelihood of observing tumor-induced intensity modifications in Sec. II-C.

The *image observations*
${y}_{i}^{c}$ are generated by Gaussian intensity distributions for each of the *K* tissue classes and the *C* channels, with mean
${\mu}_{k}^{c}$ and variance
${v}_{k}^{c}$, respectively (Fig. 1, purple). In the tumor (i.e., if
${t}_{i}^{c}=1$, the normal observations are replaced by intensities from another set of channel-specific Gaussian distributions with mean
${\mu}_{T}^{c}$ and variance
${v}_{T}^{c}$ representing the tumor class. Letting ** θ** denote the set of mean and variance parameters for normal tissue and tumor classes, and
${\mathit{y}}_{i}={\left[{y}_{i}^{1},\dots ,{y}_{i}^{C}\right]}^{T}$ denote the vector of the intensity observations at voxel

$$\begin{array}{l}p\left({\mathit{y}}_{i}|{\mathit{t}}_{i},{k}_{i};\mathit{\theta}\right)={\displaystyle \prod _{c}p\left({y}_{i}^{c}|{t}_{i}^{c},{k}_{i};\mathit{\theta}\right)}\\ ={\displaystyle \prod _{c}\left[\mathcal{N}{\left({y}_{i}^{c};{\mu}_{{k}_{i}}^{c},{v}_{{k}_{i}}^{c}\right)}^{1-{t}_{i}^{c}}\cdot \mathcal{N}{\left({y}_{i}^{c};{\mu}_{T}^{c},{v}_{T}^{c}\right)}^{{t}_{i}^{c}}\right]},\end{array}$$

(3)

where
$\mathcal{N}\left(\cdot ;\mu ,v\right)$ is the Gaussian distribution with mean *μ* and variance *v*.

Finally, the *joint probability* of the atlas, the latent tumor class, and the observed variables is the product of the components defined in Eqs. (1–3):

$$p\left({\mathit{y}}_{i},{\mathit{t}}_{i},{k}_{i};\mathit{\theta},{\alpha}_{i}\right)=p\left({\mathit{y}}_{i}|{\mathit{t}}_{i},{k}_{i};\mathit{\theta}\right)\cdot p\left({\mathit{t}}_{i};{\alpha}_{i}\right)\cdot p\left({k}_{i}\right).$$

(4)

We let ** Y** denote the set of the

We derive an expectation-maximization scheme that jointly estimates the model parameters {** θ**,

$$\langle \widehat{\mathit{\theta}},\widehat{\mathit{\alpha}}\rangle =\underset{\langle \mathit{\theta},\mathit{\alpha}\rangle}{\text{argmax}}\phantom{\rule{0.2em}{0ex}}p\left(\mathit{Y};\mathit{\theta},\mathit{\alpha}\right)$$

(5)

$$=\underset{\langle \mathit{\theta},\mathit{\alpha}\rangle}{\text{argmax}}\phantom{\rule{0.2em}{0ex}}{\displaystyle \prod _{i=1}^{M}p\left({\mathit{y}}_{i};\mathit{\theta},\mathit{\alpha}\right)},$$

(6)

and

$$p\left({\mathit{y}}_{i};\mathit{\theta},\mathit{\alpha}\right)={\displaystyle \sum _{{\mathit{t}}_{i}}{\displaystyle \sum _{{k}_{i}}p\left({\mathit{y}}_{i},{\mathit{t}}_{i},{k}_{i};\mathit{\theta},\mathit{\alpha}\right)}}$$

(7)

$$={\displaystyle \sum _{{\mathit{s}}_{i}}p\left({\mathit{y}}_{i},{\mathit{s}}_{i};\mathit{\theta},\mathit{\alpha}\right)},$$

(8)

where label vector
${\mathit{s}}_{i}={\left[{s}_{i}^{1},\dots ,{s}_{i}^{C}\right]}^{T}$ indicates tumor
${s}_{i}^{c}=T$ in all channels with
${t}_{i}^{c}=1$, and normal tissue
${s}_{i}^{c}={k}_{i}$ for all other channels. As an example with three channels, illustrated in Fig. 2 (voxel 2), suppose *t** _{i}* = [0, 0, 1] and

In the E-Step of the algorithm, making use of given estimates of the model parameters
$\{\widehat{\mathit{\theta}},\widehat{\mathit{\alpha}}\}$, we compute the posterior probability of all *K* * 2* ^{C}* tissue label vectors

$$p\left({\mathit{s}}_{i}|{\mathit{y}}_{i};\widehat{\mathit{\theta}},\widehat{\mathit{\alpha}}\right)\propto {\mathrm{\pi}}_{ki}{\displaystyle \prod _{c}\phantom{\rule{0.2em}{0ex}}\left[{\left\{{\alpha}_{i}\mathcal{N}\left({y}_{i}^{c};{\widehat{\mu}}_{T}^{c},{\widehat{v}}_{T}^{c}\right)\right\}}^{{t}_{i}^{c}}\cdot {\left\{\left(1-{\alpha}_{i}\right)\mathcal{N}\left({y}_{i}^{c};{\widehat{\mu}}_{k}^{c},{\widehat{v}}_{k}^{c}\right)\right\}}^{1-{t}_{i}^{c}}\right]},$$

(9)

and
${\sum}_{{s}_{i}}p\left({\mathit{s}}_{i}|{\mathit{y}}_{i};\widehat{\mathit{\theta}},\widehat{\mathit{\alpha}}\right)}=1$ for all *i*. Using the tissue label vectors, we can calculate the probability that tumor is visible in channel *c* of voxel *i* by summing over all the configurations *t** _{i}* for which
${s}_{i}^{c}=T$ (or equivalently
${t}_{i}^{c}=1$):

$$p\left({s}_{i}^{c}=T|{\mathit{y}}_{i};\widehat{\mathit{\theta}},\widehat{\mathit{\alpha}}\right)={\displaystyle \sum _{{\mathit{s}}_{i}}\delta \left({s}_{i}^{c},T\right)\phantom{\rule{0.2em}{0ex}}p\left({\mathit{s}}_{i}|{\mathit{y}}_{i};\widehat{\mathit{\theta}},\widehat{\mathit{\alpha}}\right)},$$

(10)

where *δ* is the Kronecker delta that is equal to 1 for
${s}_{i}^{c}=T$ and 0 otherwise. In the same way we can estimate the probability for the healthy tissue classes *k*:

$$p\left({s}_{i}=k|{\mathit{y}}_{i};\widehat{\mathit{\theta}},\widehat{\mathit{\alpha}}\right)={\displaystyle \sum _{{\mathit{s}}_{i}}\underset{c}{max}\left\{\delta \left({s}_{i}^{c},k\right)\right\}\phantom{\rule{0.2em}{0ex}}p\left({\mathit{s}}_{i}|{\mathit{y}}_{i};\widehat{\mathit{\theta}},\widehat{\mathit{\alpha}}\right)},$$

(11)

where
${max}_{c}\left\{\delta \left({s}_{i}^{c},k\right)\right\}$ indicates that one or more of the *C* channels of label vector *s _{i}* contain

In the M-Step of the algorithm, we update the parameter estimates using closed-form update expressions that guarantee increasingly better estimates of the model parameters [65]. The updates are intuitive: the latent tumor prior ${\widehat{\alpha}}_{i}$ is an average of the corresponding posterior probability estimates

$$\begin{array}{c}{\widehat{\alpha}}_{i}\phantom{\rule{0.2em}{0ex}}\leftarrow \phantom{\rule{0.2em}{0ex}}{\displaystyle \sum _{{\mathit{s}}_{i}}p\left({\mathit{s}}_{i}|{\mathit{y}}_{i};\widehat{\mathit{\theta}},\widehat{\mathit{\alpha}}\right)\left[\frac{1}{C}{\displaystyle \sum _{c}\delta \left({s}_{i}^{c},T\right)}\right]}\\ =\frac{1}{C}{\displaystyle \sum _{c}p\left({s}_{i}^{c}=T|{\mathit{y}}_{i};\widehat{\mathit{\theta}},\widehat{\mathit{\alpha}}\right)},\end{array}$$

(12)

and the intensity parameters
${\widehat{\mu}}_{k}^{c}$ and
${\widehat{v}}_{k}^{c}$ are set to the weighted statistics of the data for the healthy tissues (*k* = 1, …*,K*)

$${\widehat{\mu}}_{k}^{c}\phantom{\rule{0.2em}{0ex}}\leftarrow \phantom{\rule{0.2em}{0ex}}\frac{{\displaystyle {\sum}_{i}p\left({s}_{i}^{c}=k|{\mathit{y}}_{i};\widehat{\mathit{\theta}},\widehat{\mathit{\alpha}}\right){y}_{i}^{c}}}{{\displaystyle {\sum}_{i}p\left({s}_{i}^{c}=k|{\mathit{y}}_{i};\widehat{\mathit{\theta}},\widehat{\mathit{\alpha}}\right)}},$$

(13)

$${\widehat{v}}_{k}^{c}\phantom{\rule{0.2em}{0ex}}\leftarrow \phantom{\rule{0.2em}{0ex}}\frac{{\displaystyle {\sum}_{i}p\left({s}_{i}^{c}=k|{\mathit{y}}_{i};\widehat{\mathit{\theta}},\widehat{\mathit{\alpha}}\right){\left({y}_{i}^{c}-{\widehat{\mu}}_{k}^{c}\right)}^{2}}}{{\displaystyle {\sum}_{i}p\left({s}_{i}^{c}=k|{\mathit{y}}_{i};\widehat{\mathit{\theta}},\widehat{\mathit{\alpha}}\right)}}.$$

(14)

Similarly, for the parameters of the tumor class (*T*), we obtain

$${\widehat{\mu}}_{T}^{c}\phantom{\rule{0.2em}{0ex}}\leftarrow \phantom{\rule{0.2em}{0ex}}\frac{{\displaystyle {\sum}_{i}p\left({s}_{i}^{c}=T|{\mathit{y}}_{i};\widehat{\mathit{\theta}},\widehat{\mathit{\alpha}}\right)}{y}_{i}^{c}}{{\displaystyle {\sum}_{i}p\left({s}_{i}^{c}=T|{\mathit{y}}_{i};\widehat{\mathit{\theta}},\widehat{\mathit{\alpha}}\right)}},$$

(15)

$${\widehat{v}}_{T}^{c}\phantom{\rule{0.2em}{0ex}}\leftarrow \phantom{\rule{0.2em}{0ex}}\frac{{\displaystyle {\sum}_{i}p\left({s}_{i}^{c}=T|{\mathit{y}}_{i};\widehat{\mathit{\theta}},\widehat{\mathit{\alpha}}\right){\left({y}_{i}^{c}-{\widehat{\mu}}_{T}^{c}\right)}^{2}}}{{\displaystyle {\sum}_{i}p\left({s}_{i}^{c}=T|{\mathit{y}}_{i};\widehat{\mathit{\theta}},\widehat{\mathit{\alpha}}\right)}}.$$

(16)

We alternate between updating the parameters {** θ**,

Expectation-maximization is a local optimizer. To overcome problems with initialization, we enforce desired properties of the solution by replacing the exact computation with an approximate solution that satisfied additional constraints. These constraints represent our prior knowledge about tumor structure, shape or growth behaviour^{1}.

A possible limitation in the generalization of our probabilistic model is the dimensionality of tissue label vector *s** _{i}* that has

Together, these constraints reduce the total number of label vectors *s** _{i}* to be computed in Equation (9), for a standard glioma imaging sequences with T1c, T1, T2, and FLAIR, from

During the iterations of the EM algorithm we enforce that tumor voxels are hyper- or hypo-intense with respect to the current average image intensity
${\mu}_{k}^{c}$ of the white matter tissue (hypo-intense for T1, hyper-intense for T1c, T2, FLAIR). Similar to [51], we modify the probability that tumor is visible in channel *c* of voxel *i* by comparing the observed image intensity
${y}_{i}^{c}$ with the previously estimated
${\widehat{\mu}}_{k}^{c}$ prior to calculating updates for parameters ** θ** (Eq. 16). We set the probability to zero if the intensity does not align with our expectations:

$$\widehat{p}\left({s}_{i}^{c}-T|{\mathit{y}}_{i};\widehat{\mathit{\theta}},\widehat{\mathit{\alpha}}\right)=\{\begin{array}{ll}p\left({s}_{i}^{c}=T|{\mathit{y}}_{i};\widehat{\mathit{\theta}},\widehat{\mathit{\alpha}}\right)\hfill & \text{if}\phantom{\rule{0.2em}{0ex}}{y}_{i}^{c}>{\widehat{\mu}}_{k}^{c},\hfill \\ 0,\hfill & \text{otherwise}.\hfill \end{array}$$

(17)

For hypo-intensity constraints we modify the posterior probability updates in the same way, using ${y}_{i}^{c}<{\widehat{\mu}}_{k}^{c}$ as a criterion.

Little spatial context is used in the basic model, as we assume the tissue class *s** _{i}* in each voxel to be independent from the class labels of other voxels. Atlas

$$\widehat{p}\left(\mathit{T};\beta ,\mathit{\alpha}\right)\propto {\displaystyle \prod _{c}{\displaystyle \prod _{i}\left[{\alpha}_{i}^{{t}_{i}^{c}}{\left(1-{\alpha}_{i}\right)}^{1-{t}_{i}^{c}}\cdot \mathrm{exp}[-\frac{\beta}{2}{\displaystyle \sum _{j\in {\mathrm{N}}_{i}}\left\{{t}_{i}^{c}\left(1-{t}_{j}^{c}\right)+{t}_{j}^{c}\left(1-{t}_{i}^{c}\right)\right\}}]\right]}}$$

(18)

Here, N* _{i}* denotes the set of the six nearest neighbors of voxel

$${n}_{i}^{c}={\displaystyle \sum _{j\in {\mathrm{N}}_{i}}p\left({\mathit{s}}_{j}^{c}=T|{\mathit{y}}_{j};\widehat{\mathit{\theta}},\widehat{\mathit{\alpha}}\right)}$$

(19)

denote the currently estimated “soft” count of neighbors that contain tumor in channel *c*. The mean-field approximation implies

$$\widehat{p}\left({\mathit{s}}_{i}|{\mathit{y}}_{i};\widehat{\mathit{\theta}},\widehat{\mathit{\alpha}}\right)\propto {\mathrm{\pi}}_{ki}{\displaystyle \prod _{c}\left[{\left\{{\gamma}_{i}^{c}\mathcal{N}\left({y}_{i}^{c};{\widehat{\mu}}_{T}^{c},{\widehat{v}}_{T}^{c}\right)\right\}}^{{t}_{i}^{c}}\cdot {\left\{\left(1-{\gamma}_{i}^{c}\right)\mathcal{N}\left({y}_{i}^{c};{\widehat{\mu}}_{k}^{c},{\widehat{v}}_{k}^{c}\right)\right\}}^{1-{t}_{i}^{c}}\right]}$$

(20)

where
${\gamma}_{i}^{c}={\alpha}_{i}/\left[{\alpha}_{i}+\left(1-{\alpha}_{i}\right)\phantom{\rule{0.2em}{0ex}}\mathrm{exp}\left(-\beta \left(2{n}_{i}^{c}-6\right)\right)\right]$, replacing the previously defined Eq. (9), using a channel-specific
${\gamma}_{i}^{c}$ as a modification of *α _{i}* that features the desired spatial regularity.

High-level context at the organ or lesion level, as well as regional information, is not considered in the segmentation process of the generative model. Although we use different constraints to incorporate local neighbourhood information, the generative model treats each voxel as an independent observation and estimates class labels only from very local information. To evaluate global patterns, such as the presence of characteristic artifacts or tumor sub-structures of specific diagnostic interest, we present two alternative discriminative probabilistic methods that make use of both local and non-local image information. The first one, acting at the regional level, is improving the output of the generative model and maintaining its hyper- and hypo-intense lesion maps, while the second one, acting at the voxel level, is transforming the generative model output to any given set of biological tumor labels.

We employ an algorithm that predicts the probability of label *l* *L* for a given observation *j* which is described by feature vector
${\mathit{x}}_{j}={\left[{x}_{j}^{1},\dots ,{x}_{j}^{P}\right]}^{T}$ derived from the segmentations of the generative model. We seek to address two slightly different problems. In the first task, class labels *L* indicate whether a segmented region *j* is a result of a characteristic artifact rather than of tumor-induced tissue changes, essentially indicating false positive regions in the segmentations of the generative algorithm that should be removed from the output. In the second task, class labels *L* represent dense, voxel-wise labels with a semantic interpretation, for example structural attributes of the tumor that do not coincide with the hyper-and hyper-intense segmentations in the different channels, but labels such as “necrosis”, or “non-enhancing core”. We test both cases in the experimental evaluation, using on channel-wise tumor probabilities
$p\left({s}_{i}^{c}=T|{\mathit{y}}_{i}\right)$ and on normalized intensities *y** _{i}* to derive input features for the discriminative algorithms.

To model relations between *l _{j}* and

As many characteristic artifacts have, at the pixel level, a multimodal image intensity patterns that is similar to the one of a lesion, we design a discriminative probabilistic method that is postprocessing and “filtering” the basic output of the generative model. In addition to the pixel-wise intensity pattern, it evaluates *regional* statistics of each connected tumor area, such as volume, location, shape, signal intensities. It replaces commonly used postprocessing routines for quality control that evaluate hand-crafted rules on lesion size or shape and location by a discriminative probabilistic model, similar to [44].

The discriminative classifier acts at the regional level to remove those lesion areas from the output of the generative model that are not associated with tumor, but that stem from arbitrary biological or imaging variation of the voxel intensities. To this end we identify all *R* isolated regions in the binary tumor map of the FLAIR image (containing voxels *i* with
$p\left({s}_{i}^{\mathit{FLAIR}}=T|{\mathit{y}}_{i}\right)>1/2)$. We choose FLAIR since it is the most inclusive image modality. As artifacts may be connected to lesion areas, we over-segment larger structures using a watershed algorithm, subdividing regions with connections that are less than 5mm in diameter to reduce the number mixed regions containing both tumor pixels and artifact patterns. For each individual region *r* 1 … *R* we calculate a feature vector *x** _{r}* that includes volume, surface area, surface-to-volume ratio, as well as regional statistics that are minimum, maximum, mean and median of the normalized image intensities in the four channels. We scale the image intensities for each channels linearly to match the distribution of intensities in a reference data set. We also determine the absolute and the relative number of voxels

Overall, we construct *P* = 39 features for each region *r* (Fig. 6). To assign labels to each region, we inspect them visually and assign those overlapping well with a tumor area to the true positive “tumor” class *L _{r}* = 1, all other to the false positive “artifact” class

The generative model returns a set of probabilistic maps indicating the presence of hypo- or hyper-intense modifications of the tissue. In most applications and imaging protocols, however, it is necessary to localize arbitrary tumor structures – with biological interpretations and clinically relevant semantic labels, such as “edema”, “active tumor” or “necrotic core”. These structures do not correspond one-by-one to the hypo-and hyper-intense lesions, but have to be inferred by evaluating spatial context and tumor structure as well. We use the probabilistic output of the generative model, together with few secondary features that are derived from the same probabilistic maps and image intensity features, as input to a classifier predicting the posterior probability of the desired semantic labels. The discriminative classifier evaluates local and non-local features to map the output of the generative model to semantic tumor structure and to infer the most likely label *L* for each given voxel, similar to [32].

To predict a dense set of semantic labels *L* we extract the following set of features *x** _{j}* for each voxel

Overall, we construct *P* = 651 image features
${\mathit{x}}_{j}=[{\mathit{x}}_{j}^{k},{\mathit{x}}_{j}^{c},{\mathit{x}}_{j}^{\mathit{im}},{\mathit{x}}_{j}^{\mathit{diff}},{\mathit{x}}_{j}^{\delta},{\mathit{x}}_{j}^{\mathit{edm},\mathit{act}},{\mathit{x}}_{j}^{\mathit{MNI}}]$ for each voxel *j* and, when adapted to the BRATS training data set (Sec. V), five labels *L _{j}* as provided by clinical experts.

In a first experiment, we evaluate the relevance of different components and parameters of the probabilistic model, compare it with related generative approaches,and evaluate the performance on the public BRATS glioma dataset, and test the generalization in a transfer to a related application dealing with stroke lesion segmentation.

We use the public BRATS 2012–2013 dataset that provides a total of 45 annotated multimodal glioma image volumes [1]. Training datasets consist of 10/20 low/high-grade cases with native T1, Gadolinium-enhanced T1 (T1c), T2 and FLAIR MR image volumes. The test dataset contains no labels, but can be evaluated by uploading image segmentations to a server; it includes 4/11 low-grade/high-grade cases. Experts have delineated tumor edema, Gadolinium-enhancing “active” core, non-enhancing solid core, cystic/necrotic core. We co-register the probabilistic MNI tissue atlas of SPM99 with the T1 image of each dataset using the FSL software, and sampled to 1mm^{3} isotropic voxel resolution. We perform a bias field correction using a polynomial spline model (degree 3) together with a multivariate tissue segmentation using an EM segmenter that is robust against lesions^{2} [51]. Image intensities of each channel in each volume are scaled linearly to match the histogram of a reference.

Images are acquired in patients with acute and subacute ischemic stroke. About half of the 18 datasets comprise T1, T2, T1c and FLAIR images in patients in the sub-acute phase, acquired about one or two days after the event; another half comprises T1, T1c, T2 base diffusion and mean diffusivity (MD) images acquired in acute stroke patients within the first few hours after the onset of symptoms. For both groups the imaging sequences return tissue contrasts of normal tissues and lesion areas that are similar to hyper- and hypo-intensities expected in glioma sequences; stroke lesions are characterised here by T1 hypo-, T1c hyper-, T2 hyper-and FLAIR /MD hyper-intense changes. For the quantitative evaluation of the algorithm, we delineate the lesion in every 10th axial, sagittal, and coronal slice, in each of the four modalities. In addition, we annotate about 10% of the 2D slices twice to estimate variability. We register the probabilistic atlas and perform a model based bias field correction as for the glioma data. Image intensities are scaled to the same reference as for the glioma cases.

To measure segmentation performance in the experiments with this dataset, we combine the set of four tumor labels (edema and the three tumor core subtypes) to one binary “complete lesion” label map. We compare this map with the hyper-intense lesion as segmented in T2 and FLAIR. Separately, we compare the “enhancing core” label map with the hyper-intense lesion as segmented in T1c MRI. Quantitatively, we calculate volume overlap between expert annotation *A* and predicted segmentation *B* using the Dice score
$D\left(A,B\right)=2\ast \frac{A\wedge B}{A\vee B}$. We compute Dice scores for whole brain when testing performances on the BRATS data set. We also calculate Dice score within a 3cm distance from the lesion to measure local differences in lesion segmentation rather than in global detection performances.

We compare the proposed generative model against related generative tissue segmentations models and evaluate the relevance of individual components of our approach on the BRATS training data set. We calculate Dice scores in the area containing the lesion and the 3cm margin.

Figure 3A illustrates the benefit of the proposed multivariate tumor and tissue segmentation over a univariate segmentations that treat tumor voxels as intensity outliers similar to Van Leemput’s EM segmentation approach for white matter lesion [51]. On the given data this baseline approach leads to a high number of false positives, either requiring stronger spatial regularization or a more adaptive tuning of the outlier threshold. Figure 3B reports the benefit of enforcing intensity constraints within the proposed generative model. While the benefit for the large hyper-intense regions visible in T2 and FLAIR is minor, the difference for segmenting the enhancing tumor core visible in T1c in high-grade patients is striking: the constraint disambiguates tumor-related hypo-intensities – similar to those visible in native T1, for example, from edema – from hyper-intensities induced by the contrast agent in the active rim. Figure 3C reports a comparison between our approach and Prastawa’s classic tumor EM segmentation approach [22] that models lesions as an additional class with a “flat” global atlas prior. We test different values for the tumor prior *α* in Eq. 2, evaluating result for *α _{flat}* [.005

Our model has a single parameter that has to be set which is the regularization parameter *β* coupling segmentations of neighbouring voxels. Based on our previous study [55], we performed all experiments reported in Figure 3 with weak spatial regularization (*β* = .3). To confirm these preliminary results we test different regularization settings with *β* [0, 2^{−3}, 2^{−2},…, 2^{3}], now also evaluating channel-specific performance in the lesion area (Figure 7 in the online *Supplementary Materials*). We find a strong regularization to be optimal for the large hyper-intense lesions in FLAIR *β* ≥ .5, suppressing small spurious structures, while little or no regularization is best for the hardly visible hypo-intense structures visible in T1 (*β < .*5). Both T2 and T1c are rather insensitive to regularization. We find the previous value of *β* = .3 to work well, but choose *β* = .5 for both low- and high-grade tumors in further experiments, somewhat better echoing the relevance of FLAIR.

We apply our segmentation algorithm to the BRATS test sets that have been used for the comparison of twenty glioma segmentation methods in the BRATS evaluation [1]. We identify the segmentations in FLAIR with the “whole tumor” region of the BRATS evaluation protocol, and the T1c enhancing regions with the “active tumor” region. We evaluate two sets of segmentations: segmentations that are obtained by thresholding the corresponding probabilities at 0.5, and the same segmentations after removing all regions that are smaller than 500mm^{2} in the FLAIR volume. This latter postprocessing approach was motivated by our observation that smaller regions correspond to false positives in almost all cases. We calculate Dice scores for the whole brain.

Table I reports Dice scores for the BRATS test sets with results of about .60 for the whole tumor and about .50 for the active tumor region (‘raw’). As visible from Figure 4, results are heavily affected by false positive regions that have intensity profiles similar to those of the tumor lesions. Applying the basic, size-based postprocessing rule improves results in most cases (‘postproc.’). Most false positives are spatially separated from the real lesion and when calculating Dice scores from a region that contains the FLAIR lesion and a 3cm margin only, results improve drastically to average values of .78 (±.09 std.) for the whole tumor to and .55 (±.27 std) for the active region (not shown in the table) which aligns well with results obtained for T2 and T1c on the training set (Fig. 3).

Exemplary BRATS test sets, with results for generative and generative discriminative models. Shown are axial views through the tumor center for T1, T1c, T2 and FLAIR image (columns from left to right) and the segmented hypo- or hyper-intense areas (red **...**

We test the generalization performance of the generative model by using it for delineating ischemic stroke lesions that are similar in terms of lesion size and clinical image information. We apply the generative model as optimized for the BRATS dataset to the stroke images. As a single modification we allow T1c lesions to be both inside the FLAIR and T2 enhancing area and outside, as bleeding (which leads to the T1c hyper-intensities) may not coincide with the local edema. Stroke images contain cases with both active and chronic lesions with significantly different lesion patterns.

Although datasets, imaging protocol, and even major acquisition parameters differ, we obtain results that are comparable to the tumor data. We calculate segmentations accuracies in the lesion area and observe good agreement between manual delineation and automatic segmentation in all four modalities (Fig. 5). We also observe false positives at the white matter–gray matter interfaces, similar to those we observed for the glioma tests data (Fig. 4). Most false positive regions are disconnected from the lesion and could be removed with little user interaction or postprocessing. Inter-rater differences and performance of the algorithm are comparable to those from the glioma test set, with Dice scores close to .80 for segmenting the edema and around .50–.60 for T1c enhancing structures (Table I). Results on the stroke data underline the versatility of the generative lesion segmentation model and its good generalization performance not only across different imaging sequences, but also across applications. To the best of our knowledge this is one of the first attempts to automatically segment ischemic stroke lesion in multimodal images using a generative model.

Results of the generative model show its robustness and accuracy for delineating lesion structures. Still, it also shows to be sensitive to artifacts that cannot be recognized by evaluating the multimodal intensity pattern at the voxel level, and hypo- and hyper-intense structures can only be matched with selected tumor labels. To this end, we evaluate the two discriminative modeling strategies that are considering non-local features as input and arbitrary labels as output. We first evaluate model properties on the BRATS training set and then compare performances to results of other state-of-the art tumor segmentation algorithms on the BRATS test set.

The random forest classifier handles learning tasks with small sample sizes in high dimensional spaces by only relying on few “strong” variables and ignoring irrelevant features [69]. Still, in order to understand the information used when modeling the class probabilities, we can visualize the importance of the input features used. To this end we evaluate the relevance of the individual features using Breiman’s feature permutation test [67] that compares the test error with the error obtained after the values of a given feature have been randomly permuted throughout all test samples. The resulting decrease in test accuracy, or increase in test error, indicates how relevant the chosen feature is for the overall classification task. Repeated for each feature of all trees in the decision forest, this measure helps to rank the features and to compare the relevance as shown in Figure 6. In our test we augment the dataset with a random feature (random samples from a Gaussian distribution with mean 0 and standard deviation of 1) to establish a lower baseline of the relevance score. For each feature we compare the distribution of changes in classification error against the changes of this random feature in a paired Cox-Wilcoxon test. We analyze feature relevance in a cross-validation on the BRATS training set.

Results for the first discriminative model acting at the regional level are shown in Fig. 6. We find plausible features to be relevant: the relative location of the region with respect to the center of the brain (indicated as *center_x*, *center_y*, *center_z* in the figure), the surface-to-volume ratio (*border*2*area*), the total number of lesions visible for the given patient (*num_lesions*), the ratio of segmented voxels in T1c (*tumor _{T}* 1

For the second discriminative model acting at the pixel level we find about 80% of the features to be relevant, with with some variation across the different classification tasks. The features that rank highest in all tests are those we derived from the probability maps of the generative model: the total number of local edema or active tumor voxels, the geodesic distance to the nearest edema or active tumor pixels, but also the relative anatomical location in the MNI space, and selected image intensities and intensity differences (such as the intensity values of T1 and FLAIR for edema and T1c for active core, and local differences in the T1 image intensities).

Figure 4 displays nine exemplary image volumes of the BRATS test set. Shown are the raw probability maps of the generative model (red and cyan; columns 1–4), those regions that are selected by the regional discriminative model (cyan) and the derived tumor segmentation (column 5), as well as the output of the voxel-level tumor classifier (column 6), together with an expert annotation (column 7).

Quantitative results are reported in Table I, and we find both discriminative models to improve results over those derived from the “raw” probability maps of the generative model. With few exceptions most “false positive” artifact regions are removed (Fig. 4). The voxel-level model shows to be more accurate than the regional-level model, also correcting for “false negative” areas in the center of the tumor (rows 1, 3, 6, and 7). In addition to whole tumor and active tumor areas, the second discriminative model is also predicting the location of necrotic and fluid filled structures, as well as the “tumor core” label (with a Dice score of .58; segmentations not shown in the figure). Sensitivities and specificities for this latter model are balanced with sensitivities of .75/.58/.63 for the three tumor regions (whole tumor/tumor core/active tumor) and a specificities of .86/.71/.56.

The BRATS challenge also allows us to compare the two generative-discriminative modeling approaches with eighteen other state of the art methods including inter-active ones, and we reproduce results of the challenge in Figure 8 in the online *Supplementary Materials* of this manuscript. The generative model with discriminative post-processing at the regional level (indicated by *Menze (G)*) performs comparable to most other approaches in terms of Dice score and robust Hausdorff distance for “whole tumor” and “active tumor”. However, it cannot model the “tumor core” segmentation task as this structure does not have a direct correspondence to any of the segmented hyper- and hypo-intensity regions and, hence, does does not provide competitive results for this tumor sub-structure. The voxel-level generative-discriminative approach (indicated by *Menze (D)*) is able to predict “tumor core” labels. It ranks first among the twenty evaluated methods in terms of average Hausdorff distances for both “tumor core” and in “whole tumor” segmentation, and it is the second best automatic method for the “active tumor” segmentation. In the evaluation of the average Dice scores it is second best for “whole tumor”, it is ranking third among the automated methods for the “tumor core” task, and its result are statistically indistinguishable from the inter-rater variation for “active tumor”. Most notably, the voxel-level generative-discriminative approach is outperforming all discriminative models that are similar in terms of random forest classifier and feature design [37], [32], [2], [34], but that do not rely on the input features derived from the probability maps of the generative model.

In this paper, we extend the atlas-based EM segmenter by a latent atlas class that represents the probability of transition from any of the “healthy” tissues to a “lesion” class. In practice, the latent atlas serves as an adaptive prior that couples the probability of observing tumor-induced intensity changes across different imaging channels for the same voxel. Using the standard brain atlas for healthy tissues together with the highly specific multi-channel information provides us with segmentations of the healthy tissues surrounding the tumor, and enables us to automatically segment the images. The proposed generative algorithm produces outlines of the tumor-induced changes for each channel which makes it independent of the multimodal imaging protocol. We complement the basic probabilistic model with a discriminative model and test two different modeling strategies, both of them addressing shortcomings of the generative model, and find the resulting discriminative-generative model to define the state of the art in tumor segmentation on the BRATS data set [1].

The proposed generative algorithm generalizes the probabilistic model of the standard EM segmenter. As such, it can be improved by combining registration and segmentation [62], or by integrating empirical or physical bias-field correction models [15], [70]. The generative segmentation algorithm that we optimized for glioma images exhibits a good level of generalization when applied to multimodal images from patients with ischemic stroke. The method is likely to also work well for traumatic brain injury with similar hypo- and hyper-intensity patterns, and it can also be adapted to multimodal segmentation tasks beyond the brain. It may be interesting to evaluate relations to multi-channels approaches that do not rely on multiple physical channels, but high-dimensional sets of features extracted from one or few physical images [71]. Analyzing feature relevance indicated that the location of a voxel or region within the MNI space helped in removing false positives, as most of them appeared at white matter–gray matter interfaces or in areas that are often subject to B-field inhomogeneities. Extensions of the generative model may use a location prior that lowers the expectation of tumor occurrences in these areas. Moreover, preliminary findings suggest that results may improve by using non-Gaussian intensity models for the lesion classes.

Some tumor structures – such as necrotic or cystic regions, or the solid tumor core – cannot easily be associated with local channel-specific intensity modifications, but are rather identified based on the wider spatial context and their relation with other tumor compartments. We addressed the segmentation of such secondary structures by combining our generative model with discriminative model extensions evaluating additional non-local features. As an alternative, relations between visible tumor structures can be enforced locally using MRF as proposed by [35], or in a non-local fashing following the hierarchical approach following [54]. Future work may also aim at integrating image segmentation with tumor growth models enforcing spatial or temporal relations as in [53], [14]. Tumor growth models – often described through partial differential equations [72] – offer a formal description of the lesion evolution, and could be used to describe the propagation of channel-specific tumor outlines in longitudinal series [73], as well as a shape and location prior for various tumor structures [23]. This could also promote a deeper integration of underlying functional models of disease progression and formation of image patterns in the modalities that are used to monitor this process [74].

This research was supported by the The National Alliance for Medical Image Analysis (NIH NIBIB NAMIC U54-EB005149), The Neuroimaging Analysis Center (NIH NIBIB NAC P41EB015902), NIH NIBIB (R01EB013565), the Lundbeck Foundation (R141-2013-13117), the European Research Council through the ERC Advanced Grant MedYMA 2011-291080 (on Biophysical Modeling and Analysis of Dynamic Medical Images), the Swiss NSF project Computer Aided and Image Guided Medical Interventions (NCCR CO-ME), the German Academy of Sciences Leopoldina (Fellowship Programme LPDS 2009-10), the Technische Universität München – Institute for Advanced Study (funded by the German Excellence Initiative and the European Union Seventh Framework Programme under grant agreement n 291763), the Marie Curie COFUND program of the the European Union (Rudolf Mössbauer Tenure-Track Professorship to BHM).

^{1}An implementation of the generative tumor segmentation algorithm in Python is available from http://ibbm.in.tum.de.

^{2}available from http://www.medicalimagecomputing.com/downloads/ems.php

To support the further use and analysis of our generative segmentation algorithm, we make an implementation available in Python from http://ibbm.in.tum.de, also illustrating its use on reference data from the BRATS challenge.

1. Menze B, et al. The Multimodal Brain Tumor Image Segmentation Benchmark (BRATS) IEEE Transactions on Medical Imaging. 2014:33. [PMC free article] [PubMed]

2. Bauer S, Wiest R, Nolte L-P, Reyes M. A survey of MRI-based medical image analysis for brain tumor studies. Phys Med Biol. 2013 Jul;58(13):R97–R129. [PubMed]

3. Angelini ED, Clatz O, Mandonnet E, Konukoglu E, Capelle L, Duffau H. Glioma dynamics and computational models: A review of segmentation, registration, and in silico growth algorithms and their clinical applications. Curr Medl Imaging Rev. 2007;3:262–276.

4. Kaus M, Warfield SK, Nabavi A, Chatzidakis E, Black PM, Jolesz FA, Kikinis R. Segmentation of meningiomas and low grade gliomas in MRI. Proc MICCAI. 1999:1–10.

5. Tsai Y-F, Chiang I-J, Lee Y-C, Liao C-C, Wang K-L. Automatic MRI meningioma segmentation using estimation maximization. Proc IEEE Eng Med Biol Soc. 2005;3:3074–3077. [PubMed]

6. Konukoglu E, Wells WM, Novellas S, Ayache N, Kikinis R, B PM, Pohl KM. Monitoring slowly EVolving tumors. Proc ISBI. 2008:1–4.

7. Cuadra BBach, Pollo C, Bardera A, Cuisenaire O, Thiran JP. Atlas-based segmentation of pathological brain MR images using a model of lesion growth. IEEE T Med Imag. 2004;23:1301–14. [PubMed]

8. Styner M, Lee J, Chin B, Chin M, Commowick O, Tran H, Markovic-Plese S, Jewells V, Warfield S. 3D segmentation in the clinic: A grand challenge ii: MS lesion segmentation. MIDAS Journal. 2008:1–5.

9. Irimia A, Chambers MC, Alger JR, Filippou M, Prastawa MW, Wang B, Hovda DA, Gerig G, Toga AW, Kikinis R, Vespa PM, Van Horn JD. Comparison of acute and chronic traumatic brain injury using semi-automatic multimodal segmentation of MR volumes. J Neurotrauma. 2011;28:2287–2306. [PMC free article] [PubMed]

10. Shenton M, Hamoda H, Schneiderman J, Bouix S, Pasternak O, Rathi Y, Vu M-A, Purohit M, Helmer K, Koerte I, et al. A review of magnetic resonance imaging and diffusion tensor imaging findings in mild traumatic brain injury. Brain imaging and behavior. 2012;6:137–192. [PMC free article] [PubMed]

11. Farr TD, Wegener S. Use of magnetic resonance imaging to predict outcome after stroke: a review of experimental and clinical evidence. J Cerebr Blood Flow Metab. 2010;30:703–717. [PMC free article] [PubMed]

12. Rekik I, Allassonnière S, Carpenter TK, Wardlaw JM. Medical image analysis methods in MR/CT-imaged acute-subacute ischemic stroke lesion: Segmentation, prediction and insights into dynamic evolution simulation models. a critical appraisal. NeuroImage: Clinical. 2012 [PMC free article] [PubMed]

13. Tarabalka Y, Charpiat G, Brucker L, Menze BH. Enforcing monotonous shape growth or shrinkage in video segmentation. Proc BMCV (British Machine Vision Conference) 2013

14. Alberts E, Charpiat G, Tarabalka Y, Weber MA, Zimmer C, M BH. Proc MICCAI Brain Lesions Workshop (BrainLes) Springer; 2015. A nonparametric growth model for estimating tumor growth in longitudinal image sequences. (LNCS).

15. Van Leemput K, Maes F, Vandermeulen D, Suetens P. Automated model-based bias field correction of MR images of the brain. IEEE T Med Imaging. 1999;18:885–896. [PubMed]

16. Prastawa M, Bullitt E, Ho S, Gerig G. A brain tumor segmentation framework based on outlier detection. Med Image Anal. 2004;8:275–283. [PubMed]

17. Pohl KM, Fisher J, Levitt JJ, Shenton ME, Kikinis R, Grimson WEL, Wells WM. A unifying approach to registration, segmentation, and intensity correction. LNCS 3750, Proc MICCAI. 2005:310–318. [PMC free article] [PubMed]

18. Zacharaki EI, Shen D, Davatzikos C. ORBIT: A multiresolution framework for deformable registration of brain tumor images. IEEE T Med Imag. 2008;27:1003–17. [PMC free article] [PubMed]

19. Cuadra B Bach, Pollo C, Bardera A, Cuisenaire O, Thiran JP. Atlas-based segmentation of pathological brain MR images using a model of lesion growth. IEEE T Med Imag. 2004;23:1301–14. [PubMed]

20. Gering D, Grimson W, Kikinis R. Recognizing deviations from normalcy for brain tumor segmentation. Lecture Notes In Computer Science. 2002 Sep;2488:388–395.

21. Moon N, Bullitt E, Van Leemput K, Gerig G. Model-based brain and tumor segmentation. Proc ICPR. 2002:528–31.

22. Prastawa M, Bullitt E, Moon N, Leemput KV, Gerig G. Automatic brain tumor segmentation by subject specific modification of atlas priors. Acad Radiol. 2003;10:1341–48. [PMC free article] [PubMed]

23. Gooya A, Pohl KM, Bilello M, Cirillo L, Biros G, Melhem ER, Davatzikos C. GLISTR: glioma image segmentation and registration. IEEE Trans Med Imag. 2012;31:1941–1954. [PMC free article] [PubMed]

24. Parisot S, Duffau H, Chemouny S, Paragios N. Joint tumor segmentation and dense deformable registration of brain MR images. Proc MICCAI. 2012:651–658. [PubMed]

25. Gooya A, Biros G, Davatzikos C. Deformable registration of glioma images using em algorithm and diffusion reaction modeling. IEEE Trans Med Imag. 2011;30:375–390. [PMC free article] [PubMed]

26. Lorenzi M, Menze Bjoern H, Niethammer M, Ayache N, Pennec X. Sparse Scale-Space Decomposition of Volume Changes in Deformations Fields. Proc MICCAI. 2013;8150:328–335. [PubMed]

27. Cobzas D, Birkbeck N, Schmidt M, Jagersand M, Murtha A. 3D variational brain tumor segmentation using a high dimensional feature set. Proc ICCV. 2007:1–8.

28. Lefohn A, Cates J, Whitaker R. Interactive, GPU-based level sets for 3D brain tumor segmentation. Proc MICCAI. 2003:564–572.

29. Gorlitz L, Menze BH, Weber M-A, Kelm BM, Hamprecht FA. Proc DAGM. 2007. Semi-supervised tumor detection in magnetic resonance spectroscopic images using discriminative random fields; pp. 224–233. (LNCS).

30. Lee C, Wang S, Murtha A, Greiner R. Segmenting brain tumors using pseudo conditional random fields. LNCS 5242, Proc MICCAI. 2008:359–66. [PubMed]

31. Wels M, Carneiro G, Aplas A, Huber M, Hornegger J, Comaniciu D. A discriminative model-constrained graph cuts approach to fully automated pediatric brain tumor segmentation in 3D MRI. LNCS 5241, Proc MICCAI. 2008:67–75. [PubMed]

32. Zikic D, Glocker B, Konukoglu E, Criminisi A, Demiralp C, Shotton J, Thomas OM, Das T, Jena R, P SJ. Decision forests for tissue-specific segmentation of high-grade gliomas in multi-channel MR. Proc MICCAI. 2012 [PubMed]

33. Geremia E, Clatz O, Menze BH, Konukoglu E, Criminisi A, Ayache N. Spatial decision forests for MS lesion segmentation in multi-channel magnetic resonance images. Neuroimage. 2011;57:378–90. [PubMed]

34. Geremia E, Menze BH, Ayache N. Spatially adaptive random forests. Proc IEEE ISBI. 2013

35. Bauer S, May C, Dionysiou D, Stamatakos GS, Büchler P, Reyes M. Multi-Scale Modeling for Image Analysis of Brain Tumor Studies. IEEE Trans Bio-Med Eng. 2011 Aug; [PubMed]

36. Wu W, Chen AY, Zhao L, Corso JJ. Brain tumor detection and segmentation in a conditional random fields framework with pixel-pairwise affinity and superpixel-level features. International journal of computer assisted radiology and surgery. 2013:1–13. [PubMed]

37. Tustison NJ, Shrinidhi KL, Wintermark M, Durst CR, Kandel BM, Gee JC, Grossman MC, Avants BB. Optimal symmetric multimodal templates and concatenated random forests for supervised brain tumor segmentation (simplified) with antsr. Neuroinformatics. 2015 Apr;13(2):209–225. [Online]. Available: http://dx.doi.org/10.1007/s12021-014-9245-2. [PubMed]

38. Dvorak P, Menze BH. Structured prediction with convolutional neural networks for multimodal brain tumor segmentation. Proc MICCAI MCV (Medical Computer Vision Workshop) 2015

39. Urban G, Bendszus M, Hamprecht FA, Kleesiek J. Multi-modal Brain Tumor Segmentation using Deep Convolutional Neural Networks. Proc MICCAI BRATS (Brain Tumor Segmentation Challenge) 2014:31–35.

40. Iglesias JE, Konukoglu E, Zikic D, Glocker B, Van Leemput K, Fischl B. Is synthesizing MRI contrast useful for inter-modality analysis? Proc MICCAI. 2013 in press. [PMC free article] [PubMed]

41. Roy S, Carass A, Prince J. A compressed sensing approach for MR tissue contrast synthesis. Proc IPMI. 2011:371–383. [PMC free article] [PubMed]

42. Roy S, Carass A, Shiee N, Pham DL, Calabresi P, Reich D, Prince JL. Longitudinal intensity normalization in the presence of multiple sclerosis lesions. Proc ISBI. 2013 [PMC free article] [PubMed]

43. Batmanghelich NK, Taskar B, Davatzikos C. Generative-discriminative basis learning for medical imaging. IEEE Trans Med Imaging. 2012 Jan;31(1):51–69. [Online]. Available: http://dx.doi.org/10.1109/TMI.2011.2162961. [PMC free article] [PubMed]

44. Menze BH, Kelm BM, Bachert P, Weber M-A, Hamprecht FA. Mimicking the human expert: Pattern recognition for an automated assessment of data quality in MR spectroscopic images. Magn Reson Med. 2008;1466:1457–66. [Online]. Available: http://www.ncbi.nlm.nih.gov/pubmed/18421692. [PubMed]

45. Criminisi A, Juluru K, Pathak S. A discriminative-generative model for detecting intravenous contrast in ct images. Med Image Comput Comput Assist Interv. 2011;14(Pt 3):49–57. [PubMed]

46. Tu Z, Narr KL, Dollar P, Dinov I, Thompson PM, Toga AW. Brain anatomical structure segmentation by hybrid discriminative/generative models. IEEE Trans Med Imaging. 2008 Apr;27(4):495–508. [Online]. Available: http://dx.doi.org/10.1109/TMI.2007.908121. [PMC free article] [PubMed]

47. Iglesias JE, Liu C-Y, Thompson PM, Tu Z. Robust brain extraction across datasets and comparison with publicly available methods. IEEE Trans Med Imaging. 2011 Sep;30(9):1617–1634. [Online]. Available: http://dx.doi.org/10.1109/TMI.2011.2138152. [PubMed]

48. Speier W, Iglesias JE, El-Kara L, Tu Z, Arnold C. Robust skull stripping of clinical glioblastoma multiforme data. Med Image Comput Comput Assist Interv. 2011;14(Pt 3):659–666. [PubMed]

49. Riklin-Raviv T, Menze BH, Van Leemput K, Stieltjes B, Weber MA, Ayache N, Wells WM, Golland P. Joint segmentation via patient-specific latent anatomy model. Proc MICCAI-PMMIA (Workshop on Probabilistic Models for Medical Image Analysis) 2009:244–255.

50. Riklin-Raviv T, Van Leemput K, Menze BH, Wells WM, 3rd, Golland P. Segmentation of image ensembles via latent atlases. Med Image Anal. 2010;14:654–665. [PMC free article] [PubMed]

51. Van Leemput K, Maes F, Vandermeulen D, Colchester A, Suetens P. Automated segmentation of multiple sclerosis lesions by model outlier detection. IEEE T Med Imaging. 2001;20:677–688. [PubMed]

52. Bauer S, Tessier J, Krieter O, Nolte L, Reyes M. Integrated spatio-temporal segmentation of longitudinal brain tumor imaging studies. Proc MICCAI-MCV, Springer LNCS. 2013

53. Tarabalka Y, Charpiat G, Brucker L, Menze BH. Spatio-temporal video segmentation with shape growth or shrinkage constraint. IEEE Transactions in Image Processing. 2014;23:3829–3840. [PubMed]

54. Corso JJ, Sharon E, Dube S, El-Saden S, Sinha U, Yuille A. Efficient multilevel brain tumor segmentation with integrated Bayesian model classification. IEEE T Med Imag. 2008;9:629–40. [PubMed]

55. Menze BH, Van Leemput K, Lashkari D, Weber M-A, Ayache N, Golland P. Proc MICCAI. 2010. A generative model for brain tumor segmentation in multi-modal images; pp. 151–159. (LNCS 751). [PMC free article] [PubMed]

56. Menze BH, Van Leemput K, Lashkari D, Weber M-A, Ayache N, Golland P. Segmenting glioma in multi-modal images using a generative model for brain lesion segmentation. Proc MICCAI-BRATS (Multimodal Brain Tumor Segmentation Challenge) 2012:7p.

57. Menze BH, Geremia E, Ayache N, Szekely G. Segmenting glioma in multi-modal images using a generative-discriminative model for brain lesion segmentation. Proc MICCAI-BRATS (Multimodal Brain Tumor Segmentation Challenge) 2012:8.

58. Wells W, Grimson W, Kikinis R, Jolesz F. Adaptive segmentation of MRI data. Proc Computer Vision, Virtual Reality and Robotics in Medicine. 1995:57–69.

59. Wells WM, Grimson WEL, Kikinis R, Jolesz FA. Adaptive segmentation of MRI data. IEEE T Med Imaging. 1996;15:429–442. [PubMed]

60. Ashburner J, Friston K. Multimodal image coregistration and partitioning–a unified framework. Neuroimage. 1997 Oct;6(3):209–217. [Online]. Available: http://dx.doi.org/10.1006/nimg.1997.0290. [PubMed]

61. Pohl KM, Fisher J, Grimson W, Kikinis R, Wells W. A Bayesian model for joint segmentation and registration. Neuroimage. 2006;31:228–239. [PubMed]

62. Pohl KM, Warfield SK, Kikinis R, Grimson WEL, Wells WM. Coupling statistical segmentation and -PCA- shape modeling. Proc MICCAI. 2004:151–159.

63. Van Leemput K. Encoding probabilistic brain atlases using Bayesian inference. IEEE TMI. 2009;28:822–837. [PMC free article] [PubMed]

64. Iglesias JE, Sabuncu MR, Leemput KV. Improved inference in Bayesian segmentation using Monte Carlo sampling: Application to hippocampal subfield volumetry. Med Image Anal. 2013 in press. [PMC free article] [PubMed]

65. Dempster A, et al. Maximum likelihood from incomplete data via the em algorithm. Journal of the Royal Statistical Society. 1977;39:1–38.

66. Jordan MI, Gaharamani Z, Jaakola TS, Saul LK. An introduction to variational methods for graphical models. Machine Learning. 1999;37:183–233.

67. Breiman L. Random forests. Mach Learn J. 2001;45:5–32.

68. Menze BH, Kelm BM, Splitthoff DN, Koethe U, Hamprecht FA. On oblique random forests. Proc ECML (European Conference on Machine Learning) 2011

69. Breiman L. Consistency for a simple model of random forests. UC Berkeley, Tech Rep. 2004:670.

70. Poynton C, Jenkinson M, Wells W., III Atlas-based improved prediction of magnetic field inhomogeneity for distortion correction of EPI data. Proc MICCAI. 2009:951–959. [PMC free article] [PubMed]

71. Toews M, Zollei L, W W., III Invariant feature-based alignment of volumetric multi-modal images. Proc IPMI. 2013 [PMC free article] [PubMed]

72. Menze BH, Stretton E, Konukoglu E, Ayache N. Image-based modeling of tumor growth in patients with glioma. In: Garbe CS, Rannacher R, Platt U, Wagner T, editors. Optimal control in image processing. Springer; Heidelberg/Germany: 2011.

73. Konukoglu E, Clatz O, Menze BH, Stieltjes B, Weber M-A, Mandonnet E, Delingette H, Ayache N. Image guided personalization of reaction-diffusion type tumor growth models using modified anisotropic eikonal equations. IEEE T Med Imag. 2010;29:77–95. [PubMed]

74. Menze BH, Van Leemput K, Honkela A, Konukoglu E, Weber MA, Ayache N, Golland P. A generative approach for image-based modeling of tumor growth. Proc IPMI (Inform Proc Med Imag) 2011 [PMC free article] [PubMed]

PubMed Central Canada is a service of the Canadian Institutes of Health Research (CIHR) working in partnership with the National Research Council's national science library in cooperation with the National Center for Biotechnology Information at the U.S. National Library of Medicine(NCBI/NLM). It includes content provided to the PubMed Central International archive by participating publishers. |