|Home | About | Journals | Submit | Contact Us | Français|
We propose an active mask segmentation framework that combines the advantages of statistical modeling, smoothing, speed and flexibility offered by the traditional methods of region-growing, multiscale, multiresolution and active contours respectively. At the crux of this framework is a paradigm shift from evolving contours in the continuous domain to evolving multiple masks in the discrete domain. Thus, the active mask framework is particularly suited to segment digital images. We demonstrate the use of the framework in practice through the segmentation of punctate patterns in fluorescence microscope images. Experiments reveal that statistical modeling helps the multiple masks converge from a random initial configuration to a meaningful one. This obviates the need for an involved initialization procedure germane to most of the traditional methods used to segment fluorescence microscope images. While we provide the mathematical details of the functions used to segment fluorescence microscope images, this is only an instantiation of the active mask framework. We suggest some other instantiations of the framework to segment different types of images.
In recent years, fluorescence microscopy has greatly facilitated the task of understanding complex systems at cellular and molecular levels. We focus on the task of segmenting punctate* patterns of cells to delineate individual cells in multicell images. Unlike traditional applications such as multimedia, fluorescence microscope images lack edges (see Fig. 1(a)). Hence, most of the segmentation algorithms developed by the computer vision and image processing communities cannot be used without modifications to segment these images.
In an earlier work, we demonstrated the utility of combining multiresolution transforms with active contours to segment cells in fluorescence microscope images.18 In this we described two masks—in and out—that were evolved, starting from a random initialization, to segment an image. The limitation of using only one mask to denote multiple cells in the foreground is that it becomes difficult to separate overlapping cells (see Fig. 1(b)). To overcome this drawback, we resorted to evolving multiple masks. Thus, in the present work, we build upon the earlier ideas to use one mask for the background and a mask to represent each cell in the foreground. As this work began as an improvisation of the active contour method and rather than the contour, it is the mask—the boundary and everything in it—that evolves, we call it the active mask algorithm.16
Most of the algorithms in the literature that rely on discernable edges cannot be used as such to delineate cells from punctate patterns. There are a host of region-growing algorithms that may be adapted to segment fluorescence microscope images. This is based on the relative homogeneity of statistics of these patterns. However, these ideas have not directly been used to segment fluorescence microscope images.1, 7, 12, 13 A region-based algorithm that is used to segment punctate patterns is the Vornoi, to derive neighborhood relationships and the seeded watershed to delineate cells.6, 11 Various modifications have been suggested to improve this algorithm over the years.8, 20 The principal limitation of this method is it is not designed to produce tight contours around the objects of interest. Further, the method does not always work effectively with random seeds. Accurate seeds are often not easy to determine automatically in fluorescence microscope images of punctate patterns.
In this work we present the mathematical framework of active masks that combines (a) statistical modeling offered by region-growing methods, (b) the advantage of smoothing offered by multiscale techniques, (c) the advantage of speed offered by multiresolution methods and (d) flexibility of active contour methods to segment punctate patterns of cells. In Section 2, we highlight the need for statistical modeling and a representation based on multiple masks for the example we have considered in this work. The idea of using multiple masks to represent regions in an image is not new to region-growing techniques. Similar ideas have also been seen in PDE literature.2, 10, 14 We then describe the functions used to evolve these multiple masks with the active mask framework. These functions form an instantiation of active mask framework. In Sections 3 and Section 4, we highlight the advantages multiscale and multiresolution bring to the active mask framework. Finally, in Section 5, we briefly discuss changes to the instantiation of the framework to adapt to a different class of images.
The statistical modeling we use to segment fluorescence microscope images is based on a statistical region-based function in combination with a majority voting-based function. This section describes the design details of these functions.
As shown in Fig. 1, statistical properties coarsely distinguish the foreground from the background. We first convolve the input image f with a low-pass filter h to “connect the dots” in the image (see Fig. 2).
Next, we apply a soft-threshold to obtain a coarse separation of the foreground from the background (see Fig. 3). The resulting image is a region-based distributing function,
where α (−1, 0) is the skewing factor, sig a sigmoid function (for example, a Gaussian), β, harshness of the threshold, f h, convolution of image f with a low-pass filter h and γ, the average intensity of the foreground-background border.
After this step, we must distinguish between different regions in the foreground. For this we call upon the representation of multiple masks. We then use a majority voting based distributing function to evolve the mask. As the cells we seek to delineate in the foreground are smooth, the function ensures the multiple masks suitably partition the image with smooth boundaries.
To be able to delineate multiple regions, we first define ψ, a collection of multiple binary masks, μm, to represent the image. Thus, ψ can be thought of as a multihued image of the same size as the input image f. Corresponding to each hue (or value) m in ψ, we have a characteristic function μm defined as
Starting from a large collection of randomly initialized masks, the majority voting based distributing function gives us a way of evolving ψ:
where i is the iteration number and M the largest m in ψi+1 and g is a lowpass filter, whose scale parameter determines the degree of smoothness of the boundararies between masks. For simplicity of notation we refer to the voting based distributing function described by (2) as V. If any μm = 0 for all n, then mask m ceases to exist in ψ and cannot be recovered during the course of evolution. Empirically, we find that the masks merge to partition the image with smooth boundaries between them. Fig. 4 shows the action of V starting with a ψ = rand(256). Eventually, the masks converge to a zero change configuration. In other words, the stopping criterion is ψi+1 − ψi = 0.
Clearly, Fig. 4 does not use any information from the input image. Hence, to obtain a meaningful partition of the image, we must use some property of the image itself. In the case of punctate patterns of fluorescence microscope images, this is region-based distributing function R.
As we noted from Fig. 3, R provides a satisfactory separation of the foreground from the background. Thus, we can skew the voting so one of the masks arbitrarily favors the background. Thus, the modified voting function can be written as
where the is arbitrarily chosen to represent the background region. Since the multiple objects in the foreground are statistically indistinguishable, we do not bias any of the other masks. Consequently, the background region no longer participates in the voting process as before. This ensures that the mask that represents the background does not completely “take over” the foreground. At the same time, no mask from the foreground region eats into the background. Based on the cell geometry, certain pixels at the foreground-background interface might change their membership from one mask to another. Thus, the masks in the region corresponding to the foreground configure themselves based on the geometry of the cells. Fig. 5 shows the combined action of R and V, starting with a random initialization of the multiple masks used to represent the image is shown in Fig. 1(a).
This skewed majority voting procedure or the combination of R and V described above, forms an instantiation of the active mask (AM) framework for the segmentation of punctate patterns of fluorescence microscope images. The skewing ensures the masks achieve a zero change configuration faster than without a bias.
We note from Fig. 1, the segmentation result is not a satisfactory one. This is because of spurious splits: pixels corresponding to a single cell in the foreground have been assigned to multiple masks. As we noted earlier, the scale parameter of the lowpass filter g used in (2) determines the degree of smoothness of the boundaries between the masks. Suppose g is the radially symmetric function,
where a is the scale parameter.
A small value of a results in spurious splits as well as a longer time for the convergence (due to a large number of evolving masks). A larger a implies a higher degree of smoothing. This translates to the masks merging rapidly to ensure smoother boundaries between them. If we set the value of a to be a very large value, then once a mask m is subsumed by another, it can never be recovered (unless ψ is reinitialized). This would be counter productive if pixels corresponding to two distinct cells in the foreground were assigned to the same mask.
There are different ways to vary the scale parameter; one way could be to start with a small value of a and gradually increase it. Another way is to start with an initial value of a based on the size of the cell and resolution of the image. We can then gradually decrease this value. Thus, once the masks achieve a zero-change configuration, we can gradually decrease the scale parameter and allow the masks to evolve again. This allows the details in the boundary to be traced more accurately. Thus, multiscale used in conjunction with AM provides the advantage of appropriate smoothing to enable an accurate tracing of the boundary. Fig. 6 shows the effect of changing the scale parameter and allowing the algorithm to iterate on the segmentation result shown in Fig. 5(c).
This is the Multiscale AM (MsAM) version of the algorithm.
Whereas application of the MsAM algorithm would be slow on a large image, say of size 1024 × 1024 pixels, it is much faster on smaller images, say 128 × 128 pixels. This is because in the former there are far more pixels for which the distributing functions R and V must assign masks. Applying the algorithm at a low resolution to obtain a coarse segmentation and successively refining the result offers the advantage of speed.9, 18 The low resolution version of an image can be created using a filter bank to decompose the image to K levels.19 Since the application is segmentation, we would only work with the coarse approximation of the image. Moreover, since smoothing is a part of the decomposition process, it automatically provides the lowpass filtering action shown in Fig. 2. The scale parameter of V can be suitably adjusted at different resolutions to obtain a quick and meaningful segmentation outcome. This forms the multiresolution multiscale active mask (MrMsAM) algorithm for the segmentation of punctate patterns in fluorescence microscope images. The final result shown in Fig. 6 is after the application of multiresolution and multiscale techniques.
MrMsAM achieves experimental convergence. That is, the evolving masks arrive at a zero change state at which point the procedure comes to a halt. The action of iterative smoothing is well studied as the maximum (or minimum) principle21 (see22 for a discrete version of the maximum principle). The conditions under which a lowpass filter produces a coarse version of the image have been presented in the formulation of the diffusion equation.23 It has been rigorously proved that a version of the maximum principle from the theory of parabolic differential equations is equivalent to the condition of applying such a lowpass filter on an image so as not to produce any new zero-crossings (or edges) in the image.24 Unfortunately, we cannot apply these results directly to prove the convergence of MrMsAM as it is not continuous. Unfortunately, the discrete counterpart is not easy to analyze.14 However, we note that the active mask framework is closely related to threshold growth of cellular automata.3 Attempts have been made to characterize such dynamic systems, although a rigorous proof remains elusive in this setting.
One of the principal advantages of active contour algorithms is their ability to accommodate multiple forces that work in tandem to perform segmentation. In the active mask algorithm we have presented above, the two distributing functions R and V form an instantiation of the framework.
For the region-based function R, a soft-threshold is sufficient to segment punctate patterns of fluorescence microscope images. However, we can easily replace this with in any function—such as one that describes textures—to suit the application on hand. Fig. 7 shows a class of images that is very distinct from fluorescence microscope images. Computation of the variances (rather than average intensity around a given pixel) or an edge-based mask might be more appropriate R’s for this type of image.
Likewise, the voting-based distributing function we use successfully segments blobs (objects with smooth boundaries). We can design similar functions to describe different shapes. For instance, instead of using the majority voting based approach, V could be any other dynamical system that is known to converge. In summary, the AM framework can be adapted to different images through the design of appropriate filters and distributing functions. Thus, the AM framework is a flexible one.
We have presented an active mask framework that combines the advantages of different approaches such as region-growing, multiscale and multiresolution techniques and active contours. We have demonstrated the advantages each component brings to the framework through the example of segmentation of punctate patterns of fluorescence microscope images. While we have only treated this as an example to describe the features of the framework, details of the biological study and extensive experimental analysis of AM’s performance are available for punctate patterns of fluorescence microscope images.15–17 In this work, we have also discussed how the instantiation of the framework to segment images from other modalities through suitable changes to the distributing functions. While there is much work to be done to rigorously characterize the evolutionary behavior of the functions used in the active mask framework, experiments have categorically demonstrated the use of the active mask framework in practice.
This work was supported in part by NSF through awards DMS-0405376, ITR-EF-0331657, by NIH through the award R03-EB008870, and the PA State Tobacco Settlement, Kamlet-Smith Bioinformatics Grant.
*Punctate refers to dotted patterns or patterns with very small holes. This describes a large class of fluorescence microscope cell images such as those of subcellular protein locations.