PMCCPMCCPMCC

Search tips
Search criteria 

Advanced

 
Logo of nihpaAbout Author manuscriptsSubmit a manuscriptHHS Public Access; Author Manuscript; Accepted for publication in peer reviewed journal;
 
Proc SPIE Int Soc Opt Eng. Author manuscript; available in PMC 2010 August 2.
Published in final edited form as:
Proc SPIE Int Soc Opt Eng. 2009; 7446(744603): 744603.
doi:  10.1117/12.825776
PMCID: PMC2913509
NIHMSID: NIHMS131558

Multiresolution Multiscale Active Mask Segmentation of Fluorescence Microscope Images

Abstract

We propose an active mask segmentation framework that combines the advantages of statistical modeling, smoothing, speed and flexibility offered by the traditional methods of region-growing, multiscale, multiresolution and active contours respectively. At the crux of this framework is a paradigm shift from evolving contours in the continuous domain to evolving multiple masks in the discrete domain. Thus, the active mask framework is particularly suited to segment digital images. We demonstrate the use of the framework in practice through the segmentation of punctate patterns in fluorescence microscope images. Experiments reveal that statistical modeling helps the multiple masks converge from a random initial configuration to a meaningful one. This obviates the need for an involved initialization procedure germane to most of the traditional methods used to segment fluorescence microscope images. While we provide the mathematical details of the functions used to segment fluorescence microscope images, this is only an instantiation of the active mask framework. We suggest some other instantiations of the framework to segment different types of images.

Keywords: active contours, active masks, cellular automata, fluorescence microscopy, multiresolution, multiscale, segmentation

1. INTRODUCTION

In recent years, fluorescence microscopy has greatly facilitated the task of understanding complex systems at cellular and molecular levels. We focus on the task of segmenting punctate* patterns of cells to delineate individual cells in multicell images. Unlike traditional applications such as multimedia, fluorescence microscope images lack edges (see Fig. 1(a)). Hence, most of the segmentation algorithms developed by the computer vision and image processing communities cannot be used without modifications to segment these images.

Figure 1
(a) HeLa cells stained with sec13 marker.4 (b) Segmentation result obtained using multiresolution active contour transformation.18

In an earlier work, we demonstrated the utility of combining multiresolution transforms with active contours to segment cells in fluorescence microscope images.18 In this we described two masks—in and out—that were evolved, starting from a random initialization, to segment an image. The limitation of using only one mask to denote multiple cells in the foreground is that it becomes difficult to separate overlapping cells (see Fig. 1(b)). To overcome this drawback, we resorted to evolving multiple masks. Thus, in the present work, we build upon the earlier ideas to use one mask for the background and a mask to represent each cell in the foreground. As this work began as an improvisation of the active contour method and rather than the contour, it is the mask—the boundary and everything in it—that evolves, we call it the active mask algorithm.16

Related work

Most of the algorithms in the literature that rely on discernable edges cannot be used as such to delineate cells from punctate patterns. There are a host of region-growing algorithms that may be adapted to segment fluorescence microscope images. This is based on the relative homogeneity of statistics of these patterns. However, these ideas have not directly been used to segment fluorescence microscope images.1, 7, 12, 13 A region-based algorithm that is used to segment punctate patterns is the Vornoi, to derive neighborhood relationships and the seeded watershed to delineate cells.6, 11 Various modifications have been suggested to improve this algorithm over the years.8, 20 The principal limitation of this method is it is not designed to produce tight contours around the objects of interest. Further, the method does not always work effectively with random seeds. Accurate seeds are often not easy to determine automatically in fluorescence microscope images of punctate patterns.

In this work we present the mathematical framework of active masks that combines (a) statistical modeling offered by region-growing methods, (b) the advantage of smoothing offered by multiscale techniques, (c) the advantage of speed offered by multiresolution methods and (d) flexibility of active contour methods to segment punctate patterns of cells. In Section 2, we highlight the need for statistical modeling and a representation based on multiple masks for the example we have considered in this work. The idea of using multiple masks to represent regions in an image is not new to region-growing techniques. Similar ideas have also been seen in PDE literature.2, 10, 14 We then describe the functions used to evolve these multiple masks with the active mask framework. These functions form an instantiation of active mask framework. In Sections 3 and Section 4, we highlight the advantages multiscale and multiresolution bring to the active mask framework. Finally, in Section 5, we briefly discuss changes to the instantiation of the framework to adapt to a different class of images.

2. STATISTICAL MODELING

The statistical modeling we use to segment fluorescence microscope images is based on a statistical region-based function in combination with a majority voting-based function. This section describes the design details of these functions.

2.1 Region based distributing function

As shown in Fig. 1, statistical properties coarsely distinguish the foreground from the background. We first convolve the input image f with a low-pass filter h to “connect the dots” in the image (see Fig. 2).

Figure 2
Effect of smoothing a fluorescence microscope image containing a punctate pattern. (a) The original image. (b) After convolution with a low-pass filter, the edge of the cell becomes clearly defined.

Next, we apply a soft-threshold to obtain a coarse separation of the foreground from the background (see Fig. 3). The resulting image is a region-based distributing function,

Figure 3
An illustration of the effect of the region-based distributing function R1 applied to image f.17 (a) Smoothed image (f [large star] h)(n); (b) soft-thresholded image as in (1).

R(n):=αsig(β((fh)(n)γ)),
(1)

where α [set membership] (−1, 0) is the skewing factor, sig a sigmoid function (for example, a Gaussian), β, harshness of the threshold, f [large star] h, convolution of image f with a low-pass filter h and γ, the average intensity of the foreground-background border.

After this step, we must distinguish between different regions in the foreground. For this we call upon the representation of multiple masks. We then use a majority voting based distributing function to evolve the mask. As the cells we seek to delineate in the foreground are smooth, the function ensures the multiple masks suitably partition the image with smooth boundaries.

2.2 Majority voting based distributing function

To be able to delineate multiple regions, we first define ψ, a collection of multiple binary masks, μm, to represent the image. Thus, ψ can be thought of as a multihued image of the same size as the input image f. Corresponding to each hue (or value) m in ψ, we have a characteristic function μm defined as

μm(n)={1ifψ(n)=m,0otherwise.

Starting from a large collection of randomly initialized masks, the majority voting based distributing function gives us a way of evolving ψ:

ψi+1=argmaxm{μ1ig,μ2ig,,μMig},
(2)

where i is the iteration number and M the largest m in ψi+1 and g is a lowpass filter, whose scale parameter determines the degree of smoothness of the boundararies between masks. For simplicity of notation we refer to the voting based distributing function described by (2) as V. If any μm = 0 for all n, then mask m ceases to exist in ψ and cannot be recovered during the course of evolution. Empirically, we find that the masks merge to partition the image with smooth boundaries between them. Fig. 4 shows the action of V starting with a ψ = rand(256). Eventually, the masks converge to a zero change configuration. In other words, the stopping criterion is ψi+1ψi = 0.

Figure 4
An illustration of the action of the voting based distributing function V on a collection of multiple masks, ψ. (a) Initial ψ with 256 masks. (b) ψ after one iteration of (2) with 237 masks. (c) ψ at i=179 with M=11. (d) ...

Clearly, Fig. 4 does not use any information from the input image. Hence, to obtain a meaningful partition of the image, we must use some property of the image itself. In the case of punctate patterns of fluorescence microscope images, this is region-based distributing function R.

2.3 Active mask segmentation of punctate patterns

As we noted from Fig. 3, R provides a satisfactory separation of the foreground from the background. Thus, we can skew the voting so one of the masks arbitrarily favors the background. Thus, the modified voting function can be written as

ψi+1=argmaxm{(μ1ig)+R,μ2ig,,μMig},
(3)

where the μ1i is arbitrarily chosen to represent the background region. Since the multiple objects in the foreground are statistically indistinguishable, we do not bias any of the other masks. Consequently, the background region no longer participates in the voting process as before. This ensures that the mask that represents the background does not completely “take over” the foreground. At the same time, no mask from the foreground region eats into the background. Based on the cell geometry, certain pixels at the foreground-background interface might change their membership from one mask to another. Thus, the masks in the region corresponding to the foreground configure themselves based on the geometry of the cells. Fig. 5 shows the combined action of R and V, starting with a random initialization of the multiple masks used to represent the image is shown in Fig. 1(a).

Figure 5
Active mask segmentation applied to the image in Fig. 1(a).17 (a) Random initialization: 256 masks are used to describe the original image. (b) After the first iteration of (3), the background is separated from the foreground by the region-based distributing ...

This skewed majority voting procedure or the combination of R and V described above, forms an instantiation of the active mask (AM) framework for the segmentation of punctate patterns of fluorescence microscope images. The skewing ensures the masks achieve a zero change configuration faster than without a bias.

3. MULTISCALE ACTIVE MASK SEGMENTATION

We note from Fig. 1, the segmentation result is not a satisfactory one. This is because of spurious splits: pixels corresponding to a single cell in the foreground have been assigned to multiple masks. As we noted earlier, the scale parameter of the lowpass filter g used in (2) determines the degree of smoothness of the boundaries between the masks. Suppose g is the radially symmetric function,

g(n)=ena,
(4)

where a is the scale parameter.

A small value of a results in spurious splits as well as a longer time for the convergence (due to a large number of evolving masks). A larger a implies a higher degree of smoothing. This translates to the masks merging rapidly to ensure smoother boundaries between them. If we set the value of a to be a very large value, then once a mask m is subsumed by another, it can never be recovered (unless ψ is reinitialized). This would be counter productive if pixels corresponding to two distinct cells in the foreground were assigned to the same mask.

There are different ways to vary the scale parameter; one way could be to start with a small value of a and gradually increase it. Another way is to start with an initial value of a based on the size of the cell and resolution of the image. We can then gradually decrease this value. Thus, once the masks achieve a zero-change configuration, we can gradually decrease the scale parameter and allow the masks to evolve again. This allows the details in the boundary to be traced more accurately. Thus, multiscale used in conjunction with AM provides the advantage of appropriate smoothing to enable an accurate tracing of the boundary. Fig. 6 shows the effect of changing the scale parameter and allowing the algorithm to iterate on the segmentation result shown in Fig. 5(c).

Figure 6
An illustration of the evolution of the multiresolution multiscale active mask algorithm starting from Fig. 5(d) at resolution level k = 3 and scale of the region-based lowpass filter a = 4 to Fig. 6(c).17 (a) Segmentation outcome at resolution level ...

This is the Multiscale AM (MsAM) version of the algorithm.

4. MULTIRESOLUTION MULTISCALE ACTIVE MASK SEGMENTATION

Whereas application of the MsAM algorithm would be slow on a large image, say of size 1024 × 1024 pixels, it is much faster on smaller images, say 128 × 128 pixels. This is because in the former there are far more pixels for which the distributing functions R and V must assign masks. Applying the algorithm at a low resolution to obtain a coarse segmentation and successively refining the result offers the advantage of speed.9, 18 The low resolution version of an image can be created using a filter bank to decompose the image to K levels.19 Since the application is segmentation, we would only work with the coarse approximation of the image. Moreover, since smoothing is a part of the decomposition process, it automatically provides the lowpass filtering action shown in Fig. 2. The scale parameter of V can be suitably adjusted at different resolutions to obtain a quick and meaningful segmentation outcome. This forms the multiresolution multiscale active mask (MrMsAM) algorithm for the segmentation of punctate patterns in fluorescence microscope images. The final result shown in Fig. 6 is after the application of multiresolution and multiscale techniques.

MrMsAM achieves experimental convergence. That is, the evolving masks arrive at a zero change state at which point the procedure comes to a halt. The action of iterative smoothing is well studied as the maximum (or minimum) principle21 (see22 for a discrete version of the maximum principle). The conditions under which a lowpass filter produces a coarse version of the image have been presented in the formulation of the diffusion equation.23 It has been rigorously proved that a version of the maximum principle from the theory of parabolic differential equations is equivalent to the condition of applying such a lowpass filter on an image so as not to produce any new zero-crossings (or edges) in the image.24 Unfortunately, we cannot apply these results directly to prove the convergence of MrMsAM as it is not continuous. Unfortunately, the discrete counterpart is not easy to analyze.14 However, we note that the active mask framework is closely related to threshold growth of cellular automata.3 Attempts have been made to characterize such dynamic systems, although a rigorous proof remains elusive in this setting.

5. MRMSAM SEGMENTATION: A FLEXIBLE FRAMEWORK

One of the principal advantages of active contour algorithms is their ability to accommodate multiple forces that work in tandem to perform segmentation. In the active mask algorithm we have presented above, the two distributing functions R and V form an instantiation of the framework.

For the region-based function R, a soft-threshold is sufficient to segment punctate patterns of fluorescence microscope images. However, we can easily replace this with in any function—such as one that describes textures—to suit the application on hand. Fig. 7 shows a class of images that is very distinct from fluorescence microscope images. Computation of the variances (rather than average intensity around a given pixel) or an edge-based mask might be more appropriate R’s for this type of image.

Figure 7
A DIC microscope image of yeast cells.5

Likewise, the voting-based distributing function we use successfully segments blobs (objects with smooth boundaries). We can design similar functions to describe different shapes. For instance, instead of using the majority voting based approach, V could be any other dynamical system that is known to converge. In summary, the AM framework can be adapted to different images through the design of appropriate filters and distributing functions. Thus, the AM framework is a flexible one.

Summary

We have presented an active mask framework that combines the advantages of different approaches such as region-growing, multiscale and multiresolution techniques and active contours. We have demonstrated the advantages each component brings to the framework through the example of segmentation of punctate patterns of fluorescence microscope images. While we have only treated this as an example to describe the features of the framework, details of the biological study and extensive experimental analysis of AM’s performance are available for punctate patterns of fluorescence microscope images.1517 In this work, we have also discussed how the instantiation of the framework to segment images from other modalities through suitable changes to the distributing functions. While there is much work to be done to rigorously characterize the evolutionary behavior of the functions used in the active mask framework, experiments have categorically demonstrated the use of the active mask framework in practice.

Acknowledgments

This work was supported in part by NSF through awards DMS-0405376, ITR-EF-0331657, by NIH through the award R03-EB008870, and the PA State Tobacco Settlement, Kamlet-Smith Bioinformatics Grant.

Footnotes

*Punctate refers to dotted patterns or patterns with very small holes. This describes a large class of fluorescence microscope cell images such as those of subcellular protein locations.

References

1. Ballester C, Caselles C, Gonzales M. Affine invariant segmentation by variational method. SIAM Journ Appl Math. 1996;56(1):294–325.
2. Cao F. Geometric curve evolution and image processing, volume 1805 of Lecture Notes on Math. Springer; 2003.
3. Gravner J, Griffeath D. Cellular automaton growth on Z2: Theorems, examples and problems. Adv Appl Math. 1998;21:241–304.
4. Guo Y, Linstedt AD. COPII-Golgi protein interactions regulate COPII coat assembly and Golgi size. Journ Cell Biol. 2006;174(1):53–56. [PMC free article] [PubMed]
5. Howson R, Huh W-K, Ghaemmaghami S, Falvo JV, Bower K, Belle A, Dephoure N, Wykoff DD, Weissman JS, O’Shea EK. Construction, verification and experimental use of two epitope-tagged collections of budding yeast strains: Research papers. Comparative and Functional Genomics. 2005 Feb;6:2–16. [PMC free article] [PubMed]
6. Jones TR, Carpenter AE, Golland P. Voronoi-based segmentation of cells on image manifolds. Lecture Notes in Computer Science, pages. 2005:535–543.
7. Koepfler G, Lopez C, Morel JM. A multiscale algorithm for image segmentation by variational method. SIAM Journ Num Anal. 1994;31(1):282–299.
8. Krtolica A, Solorzano CO, Lockett S, Campisi J. Quantification of epithelial cells in coculture with fibroblasts by fluorescence image analysis. Cytometry. 2002;49:73–82. [PubMed]
9. Leroy B, Herlin IL, Cohen LD. Multi-resolution algorithms for active contour models, volume 219 of Lecture Notes in Control and Inform Sci. Springer; Berlin, Germany: 1996.
10. Merriman B, Bence JK, Osher SJ. Motion of multiple junctions: A level set approach. Journ Comp Phys. 1994;112:334–363.
11. Meyer F. Topographic distance and watershed lines. Signal Proc. 1994;38(1):113–125.
12. Nock R, Nielsen F. Statistical region merging. IEEE Trans Patt Anal and Mach Intelligence. 2004;26(11):1452–1458. [PubMed]
13. Pavlidis T. Segmentation of pictures and maps through functional approximation. Comp Graphics and Image Proc. 1972;1:360–372.
14. Ruuth SJ, Merriman B. Convolution-thresholding methods for interface motion. Journ Comp Phys. 2001;169:678–707.
15. Srinivasa G. PhD thesis. Carnegie Mellon Univ; 2009. Active mask framework for segmentation of fluorescence microscope images.
16. Srinivasa G, Fickus MC, Gonzales-Rivero MN, Hsieh S Yichia, Guo Y, Linstedt AD, Kovačević J. Active mask segmentation for the cell-volume computation and Golgi-body segmentation of HeLa cell images. Proc. IEEE Int. Symp. Biomed. Imaging; Paris, France. May 2008.pp. 348–351.
17. Srinivasa G, Fickus MC, Guo Y, Linstedt AD, Kovačević J. Active mask segmentation of fluorescence microscope images. IEEE Trans Image Proc. 2009 To appear. [PMC free article] [PubMed]
18. Srinivasa G, Fickus MC, Kovačević J. Multiscale active contour transformations for the segmentation of fluorescence microscope images. Proc. SPIE Conf. Wavelet Appl. in Signal and Image Proc; San Diego, CA. Aug. 2007.pp. 18pp. 1–15.
19. Vetterli M, Kovačević J. Signal Processing. Prentice Hall; Englewood Cliffs, NJ: 1995. Wavelets and Subband Coding. http://waveletsandsubbandcoding.org/
20. Wählby C. PhD thesis. Uppsala Univ; 2003. Algorithms for Applied Digital Image Cytometry.
21. Nirenbarg L. A strong maximum principle for parabolic equations. Commun Pure and Appl Math. 1953;6:167–177.
22. Perona P, Malik J. Scale-space and edge detection using anisotropic diffusion. IEEE Trans PAMI. 12(7)
23. Koenderink JJ. The structure of images. Journ Bio Cybern. 1984;50(5):363–370. [PubMed]
24. Hummel RA. Readings in computer vision: issues, problems, principles, and paradigms, chapter Representations based on zero-crossings in scale-space. Morgan Kaufmann Publishers, Inc.; CA, USA: 1987. pp. 753–758. Morgan Kaufmann Readings Series archive.