PMCCPMCCPMCC

Search tips
Search criteria 

Advanced

 
Logo of nihpaAbout Author manuscriptsSubmit a manuscriptHHS Public Access; Author Manuscript; Accepted for publication in peer reviewed journal;
 
Conf Proc IEEE Eng Med Biol Soc. Author manuscript; available in PMC 2010 June 27.
Published in final edited form as:
PMCID: PMC2892735
NIHMSID: NIHMS208837

Serial Reconstruction and Montaging from Large-Field Electron Microscope Tomograms

Abstract

Electron microscope tomography [1] has been proven as an essential technique for imaging the structure of cells beyond the range of the light microscope down to the molecular level. However, because of the extreme difference in spatial scales, there is a large gap to be bridged between light and electron microscopy. Various techniques have been developed, including increasing size of the sensor arrays, serial sectioning and montaging. Data sets and reconstructions obtained by the latter techniques generate many 3D reconstructions that need to be glued together to provide information at a larger spatial scale. However, during the course of data acquisition, thin slices may become warped in optical and electron microscope preparations. We review some procedures for de-warping sections and reassembling them into larger reconstructions, and present some data from electron microscopy.

I. BACKGROUND

During the past two decades, major effort in the biological sciences has been devoted to understanding the structure and function of proteins and other molecules in living systems. It is now recognized that this is insufficient for the complete understanding of what is happening in living cells. The electron microscope allows us to image cells at the next level of organization. High resolution EM images reveal a complex ultrastructure inside the cells of living organisms. Combined with tomographic techniques [1], electron microscopy, has been an indispensable tool for probing the three-dimensional structure of cells and sub-cellular organelles. Because of these findings, there is a second gap to be bridged, the gap between conventional light microscopes and electron microscopes. For example, the difference in spatial scales under investigation between an electron microscope and a light microscope can be as much as 105. In terms of volume covered this is a factor of 1015. Thus, an atlas of 3D EM tomographic reconstruction covering the same area as a single reconstruction from a light microscope may require as many as 1015 entries. Immense volumes of data must be collected to build a picture coherent with light microscope data. Such a task could be termed “biology at the petascale”. The process of data collection has been somewhat simplified by moving to larger image detectors – up to 8K × 8K pixels or larger. However, to develop even partial coverage of volumes corresponding to light microscope images, many tomographic reconstructions – serial sections or montages –are required. Serial sections, as the name implies, are generally employed to reconstruct thicker sections; for example, a light microscope section may be sliced into many thinner sections for EM study. Montaging techniques are employed to extend in the other directions [2]; two overlapping regions of a section are imaged and reconstructed separately and then glued together after the reconstruction.

II. PROBLEMS WITH SERIAL TOMOGRAPHY

Because of the curvilinearity of electron trajectories [4], larger detector arrays produce additional problems. This is because electrons travel in helicoidal paths under the influence of magnetic fields in the optical components of the EM [3]. The combination of geometric and optical aberrations at higher sample rotations may produce significant deviations from the straight trajectories. Deviations from the linear projection model are not the only effects which may degrade reconstruction quality. Sample warping [1] and other complex optical aberrations [3] may confound the simple helicoidal effects and produce irregular and unpredictable distortion. Digital reassembling is complicated by sample warping for both serial sections and montaging. We describe some of the problems associated with warping in the next section.

The problems associated with curvilinear trajectories are quite subtle both for montaging and with serial sections. Because alignment markers are generally placed on the surfaces of the sample, the electron trajectories are given by the position of the markers on the surface [5]. In the case of montaging, this means that correspondences between trajectories from one tilt series to another, taken at an adjacent region, can be defined on the surfaces of the overlaps. However, the correspondence is unspecified in the interiors of the overlap regions. This is true even if the beam does not change during acquisition of the tilt series, because sample mass loss will cause apparent displacement of features in the interior of the sample. Often, because of random placement of markers on the surfaces, there may be no markers in regions of overlap, so determining a precise correspondence of electron trajectories in overlap regions may not even be possible near the surfaces of the sample. The situation is slightly different with serial sections because there is no common marker nor overlapping areas. We discuss the problems with serial sections in the present paper, but problems with precise alignment of montagined reconstructions are are still open at present.

A further complication, from an engineering perspective is the need to automate the process. With the advent of large field capabilites, increases in sheer numbers of tomograms, and increasing research efforts in bridging the gap between light and electron microscopy, huge volumes of data must pass through computational facilities devoted to tomography. Manual intervention in the reconstruction process at any stage slows down the purely mechanical steps associated with tomography, and puts unnecessary burdens on human resources. With recent reports of automated tracking and alignment [6], automated processing from raw data to finished tomograms appears possible. At present montaging requires detection and alignment of structures within overlapping regions. Doing this automatically is a difficult AI problem [2], and still a largely open field in electron microscope microscopy. An alternative solution, would be to “straighten” trajectories in the overlaps, so that the alignment is reduced to a 2D problem, which is solvable by current techniques. We will discuss this in later sections.

III. HEURISTIC METHODS FOR DE-WARPING SAMPLES

A. Introduction

In practice, two types of sample deformation are apparent in electron tomography. On one hand, volume bending/twisting are out-of-plane deformations that do not preserve specimen boundaries. On the other hand, in-plane (or internal) displacements within the reconstructed volumes prevent a seamless serial stacking, creating discontinuous interfaces. Even though these two processes are without any doubt coupled, one can adopt an heuristic approach that consists in correcting for those deformations one after the other, by first flattening volumes and then rearranging their internal components for a better stacking.

Physical mechanisms leading to the sample deformation are unclear and result probably from a combination of many effects, such as the mechanical stress that occurs during the microtome step or the electron-beam-induced stress during data acquisition. Apparent deformations may also not be real, being artifacts in the reconstructions related, for instance, to the curvilinear nature of the electron trajectories. The complexity of the problem is such that the approach we adopt in the following is mostly motivated by practical considerations.

B. Flattening

In this section, we suppose that the z-axis is roughly perpendicular to the thin slab-like biological sample, and that the external specimen boundaries are described with two algebraic surfaces z = f1(x, y) and z = f2(x, y); volume density is denoted u(x, y, z). Note that outside of the reconstruction for z < f1(x, y) and z > f2(x, y), u(x, y, z) [similar, equals] 0.

Flattening the volume reduces to the problem of finding the appropriate transformation that associates u to another volume density ū delimited by two plane boundaries z = z1 and z = z2.

One can choose to transform the volume so every one of its internal slices remains orthogonal to its displacement during the transformation (see left schematic in figure 2). The corresponding transformation can be evaluated by use of an electrostatic analogy. Consider the capacitor whose plates match the initial and final position of an internal membrane to the volume; displacement of the membrane should occur along the electric field lines. The general solution of this problem involves solving a set of Laplace’s equations in a geometry specified by the initial (warped) and final (flattened) states of the volume. Even though this solution appeals to one’s common sense, there is absolutely no guarantee that the sample underwent such a transformation – or inverse transformation to be more specific.

Fig. 2
During the flattening procedure, each z-section of the specimen is de-warped. The involved transformation can be fully 3D with an approach based on electrostatic analogy (left) or purely 1D with a shear-type approach (right).

A simpler solution is to adopt a shear type de-warping procedure (see right schematic in figure 2). This consists in readjusting the volume density along the z direction to be delimited by [z1, z2]. The flattened density ū is then:

u¯(x,y,z)=u(x,y,z),
(1)

where z is given by

z=zz1z2z1[f2(x,y)f1(x,y)]+f1(x,y)
(2)

Finally, an additional constraint between z1 and z2 needs to be assumed to close this problem. The transformation can be isovolumic, with the same overall volume for the warped and de-warped case. Other possibilities consist in setting the difference z2z1 to match either the minimal, average or maximal width of the original warped volume.

As opposed to the previous 3D approach, this solution is a strictly 1D problem, and thus easy to implement.

C. Two-dimensional Remapping

During the stacking procedure, all the flattened volumes are first assembled along the z-direction; this preliminary step is carried out with translations and rotations. Next, an in-plane structure rearrangement is accomplished by considering corresponding contours between adjacent volumes.

Those contours are provided on both sides of an interface, and their differences generate a set of constraints that will support the in-plane rearrangement. The latter consists in providing, at each z-section of the stack, a deviation map:

(i,j)δi,j,
(3)

which associates for every pixel (i, j) a two-dimensional vector δi,j representing the displacement to apply for a refined stacking.

The problem now reduces to generating the deviation map for the entire stack from a sparse input, the set of “random” 1D constraints defined within a few 2D images.

Let us first consider the case of a simple interfacial z-section. We denote C the set of pixels for which the displacement field is known. For simplicity, we also ensure that C contains the points belonging to the external edges of the image – for which i = 0, j = 0, i = Nx + 1 and j = Ny + 1 by extrapolation. Nx and Ny are the number of image pixels in the x and y direction.

How then do we fill up the deviation map within the entire section? The simplest scheme is an interpolation-type procedure where the displacement field at each pixel of the image results from averaging the ones of its four next-neighbors:

δi,j=δi1,j14+δi1,j+14+δi+1,j14+δi+1,j+14.
(4)

This should be carried out for any pixel (i, j) that does not belong to C. Note also, that the displacement field is only relative to the two volumes on both sides of the interface.

The field δi,j can then be obtained by solving a simple linear problem; the only issue here is that the number of variables can be huge, on the order of NxNy where Nx and Ny can be as big as 8 × 103. A possible approach is to evaluate δi,j with a recursive scheme, using the constraint map as the initial values. However, we find it computationally more efficient to solve the problem by minimizing the following quantity:

F=i,jδi,j22i,j;iNxδi,j·δi+1,j4i,j;jNyδi,j·δi,j+14,
(5)

with respect to every pixel value of the displacement field that does not belong to C.

Stacking constraints impose a 2D displacement field for both outer z-sections of any specimen slice. The final deviation map is then evaluated at every z-section of the volume by interpolating the outer displacement field along the z-direction.

This interpolation-like routine relies on an appropriate set of initial deviation constraints. It can be very effective even when drastic variations of the in-plane deviations are present on a wide range of length scales. This kind of situation seems to be common in practice.

D. Segmentation

Structure segmentation is required as input to both the flattening and the in-plane remapping routines. At present, these segmentation steps are still manual.

For flattening the biological specimen, two sets of points need to be provided, one set for each boundary. The two functions f1 and f2 introduced in section III-B are then approximated with polynomials of x and y, and their coefficients are calculated using a least-squares method.

In the case of the two-dimensional remapping, multiple pair of matching contours must be provided for each interfacial zone. Coordinates of points that belong to corresponding contours are parametrized with an identical natural parameter t (with 0 ≤ t ≤ 1), and described with polynomial functions. The polynomial coefficients are calculated by regression on the segmented points. The initial relative displacement map is then evaluated by comparing the corresponding contours for the same parameter values t.

During the in-plane rearrangement, one of the z-section contiguous to an interface is kept fixed, while the other is fully distorted according to the calculated relative deviation map.

In another version of this 2D re-warping scheme, sets of four (or more) consecutive matching contours are considered in the interfacial zones to evaluate the relative displacement field. This allows for a better account of structural changes along the z direction,

IV. RESULTS AND DISCUSSION

In this work, we considered a series of warped tomographic reconstructions obtained for a study of dendrites. Slices containing these dendrites were taken from the hippocampus area of a mouse brain. At sample preparation, the specimen was sliced into several serial components, and electron tomography techniques were applied to build a 3D view of these thin slab-like components.

An example of one of the warped reconstructions is shown in figure 3. In this case, the apparent bending angle of the specimen is nearly 10°. We applied the shear-type flattening scheme described in section III-B; the result is shown on figure 4. Surface boundaries were described with polynomial functions of order 10 and the difference z2z1 was chosen to match the minimal width in the initial reconstruction. As is made clear by a comparison figures 3 and and4,4, the flattening step is very efficient and a good preamble to the in-plane remapping.

Fig. 3
Example of a warped reconstruction for a dendrite specimen from a mouse hippocampus. The z-section is taken so that part of the reconstruction is warped out of plane.
Fig. 4
Flattened reconstruction. This shows a z-section which is the same a in figure 3. The reconstruction surface is no longer warped out of plane.

To illustrate the 2D remapping procedure, we consider the case of an interface separating two distinct reconstructions. Figure 6 displays the set of contour pairs that were manually tracked and should be used as constraints for the displacement field. Figure 7 displays the whole relative deviation map that needs to be corrected for. Notice the smooth distribution of δi,j. For implementing the 2D remapping, we make use of the cvRemap function in the openCV software package.

Fig. 6
Typical contours segmentation used for an in-plane remapping. These contours follow the intersection of cut membranes with the opposing surfaces of the sections.
Fig. 7
Relative displacement field to correct at an interface. This field was reconstructed from sampling the relative displacements between contours in opposing surfaces.

V. FUTURE WORK

The final problem we will discuss is the problem of stitching reconstructions from large field EM images. Although it is possible to stitch together the reconstructions by matching structures in the object, it is also desirable to “straighten” the trajectories so that they agree on overlap regions. We have given the rationale for this in section II.

We are investigating application of a method developed by S. Patch [7], [8] for the purpose of rebinning the X-ray transform. Because the ray transform is defined in terms of four parameters on a three dimensional space, it is overdetermined. Thus the X-ray transform satisfies a set of ultrahyperbolic equations known as the Fritz John equations. Patch reports rebinning methods based on this method. In essence she was able to obtain synthetic images corresponding to other families of trajectories by treating the consistency conditions as a boundary value problem, and solving this problem via numerical methods.

Similar consistency conditions can be obtained if we cut up the object by a set of parallel planes and integrate along broken paths which connect a point in one plane to a point in the next plane via a straight line path. Any trajectory can be approximated by a series of broken line segments, and the method developed by Patch can be applied to individual segments. By iterating over the set of segments, a broken trajectory can be straightened to a straight-line trajectory. Thus the method developed by Patch can be extended to an iterative code for creating a synthetic X-ray transform. We plan to implement these methods in a computer code.

Fig. 1
Serial stacking of warped volumes includes two steps. The tomographic reconstructions are first flattened in step (a). Straightened volumes are then stacked and rearranged using a contour-based correspondences 2D remapping in step (b).
Fig. 5
Segmentation of the specimen boundary planes. The colored sets correspond to the two surfaces of the warped sample.

Acknowledgments

This work was supported by grants from the NIH National Center for Research Resources (NCRR) under award number P41RR008605, The National Biomedical Computation Resource to Peter Arzberger, and award number P41RR004050, The National Center for Microscopy and Imaging Research to Mark Ellisman.

The authors would like to thank Eric Bushong, Tom Deerinck and Naoko Yamada for the preparation and processing of the dendrite specimen.

References

1. Frank J. Electron Tomography. 2. Plenum Publishing Corporation; New York: 2006.
2. Koshevoy Pavel A, Tasdizen Tolga, Whitaker Ross T. Automatic Assembly of TEM Mosaics and Mosaic Stacks Using Phase Correlation. SCI Institute; 2007. p. 004.
3. Reimer L. Springer Series in Optical Sciences. 4. Vol. 36. 1997. Transmission Electron Microscopy: Physics of Image Formation and Microanalysis.
4. Lawrence Albert, Bouwer James C, Perkins Guy, Ellisman Mark H. Transform-based backprojection for volume reconstruction of large format electron microscope tilt series. Journal of Structural Biology. 2006;154:144–167. [PubMed]
5. Phan Sébastien, Bouwer James, Lanman Jason, Terada Masako, Lawrence Albert. Non-linear bundle adjustment for electron tomography. World Congress on Computer Science and Information Engineering; 2009.
6. Sorzano Sanchez, Oscar Carlos, Messaoudi C, Eibauer M, Bilbao-Castro, Hegerl R, Nickell S, Marco S, Carazo Jm. Marker-free image registration of electron tomography tilt-series. BMC Bioinformatics. 2009;10 [PMC free article] [PubMed]
7. Patch Sarah K. MSRI workshop on Inverse Problems and Applications. 2001. F. John’s ultrahyperbolic equation and 3D computed tomography.
8. Patch Sarah K. Consistency condition upon 3D CT data and the wave equation. Phys Med Biol. 2002;47:2637–2650. [PubMed]