The fluorescence microscope has developed into a versatile quantitative measuring device in cell biology. Advanced imaging techniques including four-dimensional imaging and fluorescence recovery after photobleaching (FRAP) can be used to visualize and analyze complex biological processes in cells, organelles and subcellular compartments1,2
. However, quantification with statistical significance requires sampling high numbers of events. This poses a challenging problem for the quantification of rare cellular events.
In principle, computational image analysis can be used to detect specific cells in microscopy images automatically. Several open-source packages for postacquisition image processing of cell-based assays already include classification by machine learning for cell recognition. Examples are CellProfiler Analyst/CellClassifier3,4
, which provide workflows for fixed cell images as well as CellCognition6
, a package dedicated to the classification of large time-lapse datasets with a fast image-processing engine. Several commercial tools, for example, Acapella from Perkin-Elmer or ScanR from Olympus, enable analysis of a few generally applicable cell-based assays but lack the extensibility to new algorithms that open-source packages have. In contrast, not only dedicated screening platforms but also state-of-the-art research microscopes have become fully motorized and computer controlled either by microscope vendor or open-source software packages such as µManager. But there has been no software solution to leverage the power of machine learning based cell recognition to automatically control the whole repertoire of fluorescence-based imaging assays.
To fill this gap, we developed the Micropilot software that effectively replaces a human expert at the microscope to identify cells of interest and automate even complex fluorescence microscopy–based imaging assays (Supplementary Software
). Micropilot provides a machine learning-based module that can be rapidly trained by the user to automatically identify desired cell states during an unattended, fast, low-resolution prescanning mode. Upon identification of a cell of interest Micropilot switches to a complex imaging mode that automatically runs desired complex high-resolution and high-information-content imaging assays such as high-resolution three-dimensional (3D) time-lapse imaging or FRAP. After completion of the desired complex imaging procedure, Micropilot returns to the low-resolution prescanning mode to identify additional cells of interest (). We used Lab View for the main Micropilot program and C for the image analysis routines. We tested Micropilot on the Microsoft Windows platform.
Figure 1 Schematic workflow of Micropilot. (a) After autofocussing different positions to find the best focal plane (yellow frame), low-resolution prescan images (optionally maximum z-dimension projections, gray frames) are presented to the automatic classification. (more ...)
Micropilot relies on a robust communication channel to read the low-resolution prescan microscope images, interrupt the scan to analyze them and reconfigure the microscope for the desired complex imaging procedure (). Pseudocode examples are listed in Supplementary Table 1
. The hardware prerequisites for such automatic imaging systems are (i) motorization of the microscope stage, (ii) automatic changing of objectives and/or laser scanner zoom, and (iii) switching of fluorescence filter and/or laser lines. The microscope system should also provide multipositioning or grid scanning along with autofocus capabilities. Micropilot can remotely control any microscope offering interfaces to this specification. We implemented Micropilot for Leica SP5, µManager, Olympus ScanR, PerkinElmer Ultra View ERS and Zeiss 510. Certainly, µManager as an open-source microscope control software allocates most freedom for users in terms of further developments.
To test the power of Micropilot, we applied this platform to three different fluorescence microscopy–based assays, focusing on the biological process of cell division for which low-resolution imaging data from functional genomic studies have recently become available7,8
. We aimed to identify automatically the two most transient mitotic stages, prophase and anaphase, in low-resolution prescan images and wanted to assay three different biological processes: (i) the biogenesis of endoplasmic reticulum exit sites (ERESs) after mitosis using high-resolution immunofluorescence, (ii) the assembly of the microtubule spindle using 3D time-lapse imaging and (iii) the binding of chromobox homolog 1 to condensing chromosomes using FRAP. To test the hardware flexibility of the software, we used two different microscopy systems for these assays.
ERESs disappear during metaphase in mitosis and reassemble at the telophase or cytokinesis transition9
. To understand ERES biogenesis quantitatively, we trained the Micropilot classifier to recognize mitotic cells by their chromosome configuration in the DNA (Hoechst) channel and select those that showed quantifiable expression of YFP-tagged SEC31, a COPII component and ERES marker (). The trained classifier provided 92% sensitivity (Supplementary Fig. 1a
) with a positive predictive value (PPV) of 88% (). For three microscope slides containing HeLa-Kyoto cells, we acquired 2,700 image fields at low resolution and detected 173 high-resolution confocal 3D stacks (each comprising 50 slices) of anaphase and later mitotic stages () during three unattended overnight scans with only 1.5 h combined hands-on time. After manual quality inspection of the image data, we quantified the number of ERESs in 91 of the high-resolution 3D stacks. This revealed that the average number of ERESs exponentially increased with the volume increase of the nuclei () but showed no clear correlation with the increase in inter-nuclei distance (Supplementary Fig. 1b
). As nuclear volume marks the progression of cells from anaphase to cytokinesis, these data are consistent with earlier work9
suggesting ERES biogenesis after mitosis to be most efficient at the telophase or cytokinesis transition. In the future, extending the experiments we performed with additional markers of ERES composition and systematic microscopy-based quantification enabled by Micropilot will shed more light on the mechanism of postmitotic ERES biogenesis.
Figure 2 Assays of SEC31 and H2B-tubulin HeLa cells. (a) Examples for Hoechst-labeled (blue; DNA label) and SEC31-labeled (green) cells representing null or artifact and anaphase or telophase cells (insets, close-up images). Scale bars, 10 µm. (b) Confusion (more ...)
The assembly of the mitotic microtubule spindle is a highly dynamic process, which starts upon mitotic entry in prophase and is a key prerequisite for genome segregation. We aimed to automatically image prophase cells and the first appearance of microtubule asters in four dimensions, which enables dynamic measurement of structural parameters of the forming spindle such as its length. We trained the Micropilot classifier to identify prophase HeLa cells based on the condensing chromatin (histone H2B tagged with mCherry, abbreviated H2B-mCherry), and subsequently recorded fluorescently labeled microtubules (EGFP-tagged α-tubulin, abbreviated EGFP-tubulin) and chromosomes in two-color, 3D time-lapse sequences of live cells at high spatial and temporal resolution. We fed 3,644 fluorescently labeled H2B-mCherry
cells into the Micropilot multiclass support vector machine (SVM) classifier to recognize six major mitotic phases (Supplementary Fig. 2a
). The resulting classifier achieved 97% mean sensitivity and 87% mean PPV (Supplementary Fig. 2b
) in good accordance with other mitotic classifications10,11
and could be used without retraining to control a second confocal microscope from the same manufacturer with very similar prophase PPV of 85% (Supplementary Fig. 2c
). These findings indicate good generalization properties of the classifier that is not dramatically affected by hardware variations of the two similar confocal microscopes or biological sample variation over 6 months. In 15 unattended overnight runs, more than 120 high-resolution, 3D time-lapse mitosis movies were generated, instead of an expected 2 months of full-time manual work by an experienced microscopist ( and Supplementary Video 1
). Quantitative analysis of the movies revealed that the mean pole-to-pole distance of the metaphase spindle of HeLa cells was 15.6 ± 1.6 µm (±s.d.) and increased to 25.3 ± 1.7 µm in telophase (), which is consistent with previous measurements12
. Exploiting this automatic quantitative assay for spindle assembly, we analyzed the cells in which expression of the essential centromere protein CENP-E had been knocked down by siRNA. Automatic sampling of 71 CENPE
-depleted cells arrested in prophase or prometaphase13
, revealed three different subpopulations of spindle lengths, with mean pole-to-pole distances of 15.5 ± 3.2 µm, which correlated with the number of unaligned chromosomes at the poles (). The good statistical sampling of prophase or prometaphase stages enabled by Micropilot thus revealed that depletion of CENP-E activity by RNAi led to different metastable spindle defects rather than a continuous spectrum of abnormal spindle lengths.
Condensation of mitotic chromosomes is a dynamic structural reorganization of the genome in mitotic prophase and is poorly understood molecularly. To define candidate proteins for this process their binding to condensing chromosomes needs to be assayed. As an example candidate we focused on chromobox homolog 1 (CBX1 or heterochromatin protein 1 beta (HP1β)). In interphase, CBX1 binds in heterochromatic areas of chromosomes14
, but it is largely released from chromosomes in metaphase and freely diffuses in the cell15
. To analyze CBX1 binding to condensing chromosomes, we performed automatic FRAP analysis on prophase and interphase nuclei. We trained the Micropilot classifier to identify prophase 3T3 cells based on the condensing chromatin (H2B-mCherry), and subsequently performed automatic FRAP experiments on CBX1-EGFP by photobleaching half of the nucleus and following the recovery of CBX1 fluorescence during 1 min ( and Supplementary Videos 2
). The binary prophase classifier had 92% mean sensitivity and 95% mean PPV on untrained images (Supplementary Fig. 3a
). In four unattended overnight runs we bleached 115 interphase and 117 prophase cells. The manual selection of the rare prophase stages would have required a full month of work by an expert microscopist. The recovery curves of 50 interphase cells exhibited a characteristic halftime of recovery (t1/2
) of 6.7 ± 1.4 s (±s.d.), resulting from the exchange of the heterochromatin-bound pool (). The t1/2
values of 64 prophase cells were distributed in two distinct populations with distinct kinetics (Supplementary Fig. 3b
) suggesting that CBX1-chromatin interactions undergo discrete changes during prophase. By analyzing the chromosome condensation state in the two populations, we found that the two clusters of recovery halftimes corresponded to early and late condensation stages (Supplementary Fig. 3b
). In early prophase CBX1 recovered significantly slower than in interphase with a mean t1/2
of 7.8 ± 0.9 s (P
= 0.0013), indicating that its binding is increased during condensation. By contrast, in late prophase, when chromosome condensation was nearly complete, CBX1 was largely unbound and recovered significantly faster than in interphase with a mean t1/2
of 4.2 ± 1.1 s (P
< 0.0001) (). The good statistical sampling of prophase stages enabled by Micropilot revealed that CBX1 binding to condensing chromosomes is transiently elevated in early prophase before its release at the end of prophase, suggesting a possible function in the condensation process.
Figure 3 Examples and measurements of automatic FRAP on CBX1-EGFP cells. (a) After the automatic selection of an interphase or prophase cell with a trained prophase SVM classifier, a prebleached image was taken, followed by bleaching of the predefined upper half (more ...)
To create quantitative data suitable for systems biology at the cellular level of biology, intelligent automation will be crucial to increase throughput and provide statistical relevance of the measurements of very different cellular processes. Micropilot automation reduces bias in cell selection and liberates cell biologists from the tedious work of repetitive manual data generation. Micropilot is broadly applicable in cell biology to carry out a host of quantitative imaging assays in a systematic fashion as required to characterize a large number of genes or perturbations such as in RNAi studies or drug screens. It can be adapted to virtually any imaging system that allows automation and online control based on the results of image classification by machine vision. It will be straightforward in the future to use Micropilot to automate additional high-resolution assays beyond multicolor high-resolution confocal 3D stacks, time-lapse movies and FRAP, such as fluorescence lifetime imaging microscopy and fluorescence cross-correlation spectroscopy.