3D representation of biostructures imaged with an

optical microscope

Part A: Digital optical sectioning

A Diaspro, M Sat-tore and C Nicolini

A method for acquiring and processing 30 information from biological specimens imaged with an optical microscope is presented. It is physically based on optical sectioning microscopy, and utilizes a set of digital filters implemented for deblurring optical section images. Rather than the provision of an exact solution for the problem, we have found of more practical use a new approximate method that significantly improves the situation, within reasonable cost parameters.

Keywords: optical sectioning, digital filtering, 30 approximate method, 30 chromatin layout

Digital image processing is one of the most active fields in which research and application efforts are widely performed. In the last 15 years a variety of image analysis techniques have been proposed, in conjunction with the rapid progess which has been made by hardware technology, and the pressing need felt in many areasi*z.

As a part of a continual investigation into the structure and function of chromatin within the nuclei of intact cell+7, we have developed a method to map the spatial organization of intracellular and subnuclear components. Since cells and nuclei are intrinsically three-dimensional, an imaging system was designed and realized to inspect their 3D characteristics.

This is the first part of a two part paper on 3D representation of biostructures imaged within the frame of light optical microscopy. Part A is concerned with the acquisition and restoration of 3D information from optical section images taken from cells and nuclei. Part B is devoted to the display of this information in 3Ds.

Istituto di Biofisica, Universita di Genova, Via Giotto 2-16153, Genova, Sestri Ponente, Italy

Our goal is to follow dense chromatin zones (paths), with the aid of 3D modelling and cytological mapping, to reveal a number of consistent organizational motifs that characterize their spatial organization.

The biophysical basis of imaging cytometry is that functional changes in any cell are always accompanied (or even preceded) by significant changes in the higher order structure of the cell and of its main constituentss.

The acquisition methodology is physically based on a computer controlled optical sectioning of the object in conjunction with an electromechanic light shuttering system9.10. This sectioning allows us to look closely into the sample without disturbing it.

During the last few years there has been a great deal of interest in confocal microscopestl, designed specifi- cally to give optical sections. Although confocal micro- scopes are much better at rejecting information that contaminates section images than is optical sectioning microscopy, confocal microscopes present some dis- advantages for our application (such as those discussed below that made us prefer the classic optical sectioning techniquet*.is). Because the sectioning from a state of images has been attempted befores, we have developed (in a limited number of cases) a different procedure both for acquisition and for digital processing.

The method implemented to remove the out-of-- focus information uses a new approximate method, developed in terms of spatial frequency analysisi4. The whole system has been realized using as main constituents a PC workstation and a modified optical microscope.

3D DATA COLLECTION

The purpose of this comparison between two different techniques suitable for 3D data collection is to clarify our choice of one technique with respect to the other.

130

0262-8856/90/020130-10 0 1990 Butterworth Sr Co (Publishers) Ltd

image and vision computing

Optical sectioning and confoeal scanning microscopy techniques

The 3D reconstruction techniques discussed here belong to that class involved with the acquisition of specimen sections. For this purpose we ignore tech- niques in which the object is physically divided (cut) into sections which are individually imaged. Discussion here is concerned only with methods of obtaining sections through an intact specimen.

This may be done by making a through focal series of pictures. A series of images is made in which the focus of the objective is varied through the specimen. Because of the non-zero depth of focus of the lens, each acquired frame contains not only the ‘true’ image of the section at that plane, but it is also combined with out-of-focus contributions from adjacent sections, and limited by the transfer function of the image formation system r,g,r5.

To date, an elegant and complete example is given by a series of works concerned with the analysis of DNA packing in Drosophila Melanogaster sperm headsr6, the 331) reconstruction of the polythene chromosomesr7, and of the diploid embryonic chromosomes $2 of the Drosophila Melanogaster. Erhardt et al. 18 provide another example, with an application to the nucleus of a cell from the mucosa of the Iarge intestine, and to some other test objects, like the shell of Radiolaria Auionia hexagonia and crossing bristles of an insect wing. The above applications are very specific, and have been performed using very particular specimens.

In confocal laser scanning microscopy, a different and (some might say) more elegant version of the optical sectioning procedure has been realized. (For the interested reader, details of basic confocal principles may be found in Reference 11, and demonstrations of the imaging capabilities with biological samples can be found in References 13, 19 and 20.) A beam of light is focused by the objective lens and scanned across each focal plane of the specimen. The focused point is subsequently imaged onto a detector pinhole. This is a confocal method because the image of the illumination pinhole and the back projection of the detector pinhole have a common focus in the specimen.

Although confocal microscopy is much better in rejecting out-of-focus contamination from adjacent sections. it is not perfect’“. Among the different versions of confocal scanning microscopes, a common characteristic is the time needed to acquire a single frame. As White et al. 13, depending on the architec- ture, the acquisition times are typically of the order of 10 to 60slimage. Some faster systems not yet commercially available may acquire frame images at a time in the order of 0.125 slimager3.

The nature of 3D imaging is such that the acquisition of section images is accomplished with considerably less photobleaching using optical sectioning, especially when in conjunction with our automatic light shuttering system, rather than using confocal scanning methodsr’.

One problem for our application is the reduced choice of available excitations, and the impossibility of having a direct view of the object under study. Moreover, in some commercial inst~ments much arbitrary preprocessing is performed (at least the average on many images to improve the signal/noise

ratio, but at the expense of the acquisition time), making the system something like a ‘black-box’, unsuitable for quantitative studies and for some non- standard applications. Another problem concerned with acquisition arises from the frequent adjustments of the supply voltage of the (generally used) photo- multiplier tube (PMT), performed automatically (or even manually) to achieve maximum sensitivity. It may be difficult to control the behaviour of the PMT gain with respect to the applied voltage when that voltage varies over a wide range. A final, equally important negative factor for this type of microscope is the high cost, which increases very rapidly under special, unavoidable conditions (changing laser excitations, changing image output format, etc.). For all the above reasons it was decided to develop our 3D system on the basis of the optical sectioning method, achieving at a relatively low cost a highly transportable and effective system.

MATERIALS AND METHODS

Biological

Rat epatocytes were obtained from three month old male Sprague-Dawley animals. Cells were stained with DAPI solution (15pM in 0.1 MTris HCl pH 8). Micro- scopic samples were simply prepared smearing 25~~1 of ceils, suspended in st.s., on to a microscope slide: liquid was covered with a coverslip. By staining with DNA-specific fluorescent dye (DAPI: 4, &diamidino- 2-phenolindole), DNA-containing structures can be imagedzr. (Occasionally, the specimen was fixed using 2% gluteraldeide or 3 : 1 V./V etOH solutions.)

This method has also been used with CHO cells (clone Kl), grown in medium to obtain the reverse counterpart. The cells have then been synchronized in metaphase and S phase, in situ hybridization has been performed using a myc probe, and just before observation the cells were stained with either DAPI or with Fuelgen reaction22.

Microscopy

Fluorescence-dependent images have been acquired with a 100x 1.3 NA oil-immersion objective mounted on a Zeiss Axioplan microscope set-up for epifluores- cence (HBO 100 Osram mercury lamp, Zeiss filter set at 487402). The whole system (see Figures 1 and 2) is completely computer controlled, and has three possible operating modes, both in fluorescence and in transmis- sion: automatic; semi-automatic; and manual.

Successive images are acquired by progressive defo- cusing, i.e. by changing the focal plane in either direction within the object by small increments of 0.35~0.7,~m. The progressive defocusing is per- formed by a computer driven stepper motor. During the focus changes, the agitation of the sample is switched off by an automatic, computer-synchronized electromechanic shutterra. The shutter is mechanically realized by an anticorodal cylinder mounted on the principal axis of a stepper motor, and directly plugged into a filter’s lodging on the microscope along the excitation light path. It produces an ON/OFF effect on the excitation light without acting on the mercury lamp, and at a (variable) relatively high speed. This

~018 no 2 may 1990 131

Personal Computer I

I - Z-stage

Tape El unit

Figure 1. Simplified block diagram of the developed imaging station

Figure 2. Imaging station system with (upper arrow) the automatic light shuttering system, and (lower arrow) the hardware cards built to control shutter and z-axis movement

original component, designed and realized in our laboratory, allows us to damage the sample as little as possible during 3D acquisition sessions.

At each z-position, averaged (10 times) and not- averaged images are taken at the standard rate of 30 frames/s. A typical data set consists of 12 to 24 digital images, which are windowed before saving to reduce the occupation of memory space. Images are directly digitized from the microscope using a TV camera attached to a frame grabber.

Hardware

The computer workstation consists of an IBM enhanced AT computer, equipped with 0.64Mbyte of internal memory, 20Mbyte of hard disc memory, 1.2Mbyte plus 0.72Mbyte of floppy disc drivers, and 60Mbyte of removable tape memory. A mathematic coprocessor (80287) is used for floating point calcu-

lations. The image digitization module, a frame grabber manufactured by Imaging Technology (PFG 8350AT PCVISION), is directly plugged into the computer workstation, occupying a single slot. A standard monochrome video monitor and a colour Sony Trinitron video monitor are used as display devices. A DAC/IBM card is used to control the z-axis movements and the shuttering system drivers.

Software

To manage the acquisition, process and display opera- tions, an IMAGO software system was used (a C programming language environmentlo). The system is both machine and frame grabber independent, except during the acquisition and display sessions when it uses the DAC card and an output graphic monitor, respec- tively. Every step, from digital image processing to 3D generation of image files, can be performed indepen- dently using the graphic card or frame grabber, and only the format of the input and output image has to be specified for processing.

Furthermore, many steps may be executed over- night, when a great number of experiments can be processed, using a command file utility and tape unit.

DIGITAL IMAGE RESTORATION

Once a 3D optical section data set has been collected, all of the contaminating effects must be removed plane- by-planei,9.15. We have developed a set of digital filters that work very efficiently on small computer systems, allowing a good resolution to be achieved. These digital filters may be successfully employed for a wide range of problems in cell biology. Although the system was initially developed for fluorescent micro- scopy, we are also working with transmission images following the same acquisition method and restoring procedure22.

General background

To remove distortions from the images, due to both the method itself and the intrinsic resolution limits of the light microscope, it is necessary to understand in detail the image formation process (for a detailed discussion of this subject, readers are referred to References 9,15 and 18).

For optical sectioning microscopy a knowledge of how the image formation system modifies the images as a function of defocus, and of how contaminating effects arise plane-by-plane, is required. Families of 2D contrast transfer functions can be calculated from different approximationsi123.24 to determine the 3D contrast transfer function of the imaging system. Direct analytic calculations 1% is, or experimental deter- minationzs, have been conducted to determine the behaviour of the 3D contrast transfer function. Unfortunately, many ideal assumptions go into the calculation and application of the 3D contrast transfer function when applied to this kind of inverse problem. Furthermore, an exact method would be very time consuming unless specialized, fully-dedicated hardware (i.e. array processors) were used.

132 image and vision computing

Image formation principles

To recover the in-focus information from a state of images, several approaches have been followedi,9.i*. In general, it would be necessary to reverse a convo- lution operation that establishes a connection among the observed image 0(x, y, z), the ‘true’ image Z(X, y, z), and the microscope’s total 3D point spread function S(X, y, 2)‘s.

To simplify the notation we synthesize the problem in terms of Fourier transforms, and explicitly separate the section image contributions. Considering the object as the summation of a stack of N specimen planes located at different positions along the optical axis, and assuming a uniform spacing between acquired planes (k above and below the focal plane), we can write for a generic image plane Z with reasonable approximationi.9:

I+k Oj = Z, ’ Sj f C i Z; ’ S(i_j)

1-k If1

where 0, I and S indicate the relative 2D Fourier transforms of o, i and S, respectively, at a certain position along the z axis, reducing the 3D problem to a finite summation of 2D multiplication in the spatial-frequency domain.

Recasting equation (l), in principle it is possible to recover the ‘true’ image spectrum:

(2) 1’1

and consequently, by reversing the Fourier transforma- tion, the image at the Zj position:

,ij = 3 -‘VA (3)

Unfortunately, because the Zi are unknown, it is impos- sible to directly compute the quantity ZiS,,_j,.

A possible approach, using some approximate S, would be to solve the equation system in equation (2) with some iterative methodi.9.

APPROXIMATE METHOD

Rather than an exact solution, impractical under many points of view’, 15, we have developed a reasonable approximate method that yields good performance.

Having abandoned a simultaneous solution approach such as that which solves the system, reported in equation (2), using an approximate relationship for S 1.9, we assume as correct the statement that the observed image spectrum Oj, at the Zth plane, contains the specimen spectrum Zj, plus an excess of low frequencies coming from the image-formation system and from adjacent plane contamination (in addition to an excess of high frequencies arising from the image detector).

To eliminate low frequencies arising from defocusing we apply a ‘robust’ filter F’ that cuts off very low frequencies from the image spectrum.

Weinstein and Castleman2” suggest that contami- nation from sections above and below could be

approached by considering as relevant contributions from those planes immediately adjacent (Z- 1, Z + 1). Taking into account this observation, we consider that the contribution from the immediately adjacent planes in Oj appears in a certain spatial frequencies range that may be identified by an inspection of the relative Fourier spectra. To the original acquired image we then apply another filter F”, a low-pass filter that cuts off high frequency noise and enhances the contaminat- ing effects arising from the planes above and below the object.

As a final step to this approximate method, we sub- tract this enhanced effect due to the immediately adjacent planes from the F’ filtered image, in the space domain.

For simplicity and ease of calculation, we consider the image spectra contained in the square matrices N x N, with the reference taken at the matrix coordi- nate N/2, N/2 corresponding to the zero frequency.

Operatively, let us define 0; as:

0; = 0; F (4)

where F is one of a number of preprocessing filters used to eliminate different sources of noise or particu- larly strong diffusion effects. To reduce generic elec- tronic noise, the T(r) or the E(r,, rY, (Y) filters are used. A D(p) filter is used in the second case. Plate 1 (see page 140) shows the effect of a D(7) filter: the most out of focus image spectrum, the 7th, has been used to preprocess the original images.

Following preprocessing on the ori$nal spectrum 0, of the observed image, we process Oj by means of the F’ and F” filters, respectively implemented by the HP(r) and L(r) digital filters:

r; =O;.F’

r; = 0; . F” C-9

Then, an inverse Fourier transformation .1;2 -I, allows us to obtain an estimation of i at the Zth plane:

;,=c, .<~-‘{z;}-c2..~-‘{l;r} (6)

The constants C1 and C2 are used to balance out the relative contributions of the Zth plane’s defocusing and the adjacent plane’s effect.

The filter parameters r are heuristically determined by inspecting the image spectra. When l images have been obtained, thresholding is applied as a last step before 3D representation*.

Plates 2 and 3 (see page 140) show the effect of a HP(4) filter on two different nuclei. In Plate 4 (see page 140), on a sequence of original images, the effects of L(33) and HP(4) digital filters are shown. Plate 5 shows some biological features extracted by utilizing the approximate method (equations (5) and (6)), and after a colour threshold has been applied8 (see page 140).

Digital filters

The digital filters used to apply this simple but effective approximate method are:

l T(r): this filter has as its zero frequency reference

~018 no 2 may 1990 133

the 0,O matrix location. It cuts off frequencies within a circle of radius r centred in N/2, N/2.

E(r,, rY, (Y): this filter eliminates spatial frequencies within an ellipse defined by the semi-axes r, and r,,, and an a rotation with the r, axis. D(p): this operation is performed by subtracting from the desired Oj spectrum the 0, spectrum; the

P th image used is usually the most out of focus image up or down the focal plane. HP(r): the first low frequencies inside a circle of radius r, centred at O,O, are cancelled from the image spectrum. As an example we use, in a mono- dimensional case, an HP(l) filter. In this case, the 0th and 1st components of the image spectrum are cancelled. The 0th component cancellation means eliminating the average intensity value of the process, while removal of the 1st component cancellation means cutting off a sinusoid of a period equal to that of the considered frame. L(r): only the frequencies that are inside of radius r, centred at O,O, are preserved.

a circle

Propagative technique

As further development of the approximated method we have implemented27 a ‘propagative technique’, summarized through the algorithm:

/ ‘A =ioA lo

11 .A =;

1 -~3(ioAh-6z

< . (7)

N-1

IN .A =;

N

where A stands for accurate solution, g is the propaga- tion operator of the j th image plane ij of a spatial quantity (N-i)&, and (N -i)Sz is the distance between the plane to be propagated and the actual plane considered. The initial guess is set equal to the processed observed image 2,.

CONCLUSION

A method for acquiring and processing 3D information from biological specimens imaged with an optical microscope has been presented. The methodology, physically based on optical sectioning of the sample, has been improved by using an automatic light shutter- ing system and an approximate procedure for image restoring. The former allows study of the specimens with minimal perturbations; the latter allows the extraction of important features from the optical sections. A brief comparison with the confocal scanning microscopy technique has also been carried out to clarify our choice of the optical sectioning method.

The approximate method allows us to obtain good results at a low cost. It has been extensively and successfully used for fluorescence and transmission- dependent images of different biological samples. In addition, the approximate method has been used to study chromatin distribution in rat epatocytes during different states of the cell cycle, and for studying oncogene distribution in CHO Kl reverse cells.

The whole system has been developed on the basis of two common pieces of laboratory equipment: a PC and an optical microscope. With the addition of some easily available additional cards, this system could easily be implemented in any research laboratory.

Part B: 3D graphic representation

A Diaspro, M Sat-tore and C Nicolini

A 30 graphic display system has been developed for relatively rapid and reasonably accurate imaging of the 30 organization of biostructures imaged in-situ through an optical microscope. This system makes possible 30 study of relevant biostructural assemblies. The 30 representations are achieved from feature points isolated from a sequence of images taken by the digital optical sectioning method, described in Part A. The programs may run in graphic-card independent mode.

Keywords: 30 representation, optical sections, 30 edit- ing, 30 biostructural assemblies, 30 portable system

The 3D representation of objects is a critical and fundamental step in structural analysis. A technique has been developed (and is described in part A above) that allows us to study important biostructures in-situ with a minimum of perturbation.

During recent years there has been an increasing demand for ways of representing the 3D architecture of biological features from available 2D information2g32. In the case of biophysical research, the possibility of inspecting the 3D structure (organization) of bio- structures within their natural environment - the cell - is a key step in the elucidation of their biological function3,5y6. Computer graphic display methods for visualizing biostructures have been applied to different experimental methods according to different pro- cedures, offering different tradeoffs between the complexity of implementation and the level of detail of 3D scene representation28,32-3s. Among the techniques that have been implemented to display biological structures in 3D, we are more in touch with those developed and utilized in the field of electron microscopy30,34,36,39.

Our approach is simple and not very time consuming, allowing the user to inspect, in different levels of detail,

134 image and vision computing

fundamental information within intact cells and nuclei. We have developed a 3D graphic display system on the basis of experimental data obtained from a digital optical sectioning technique, described in Part A. Optical section data, in digital format, were computa- tionally processed using an original approximate method to remove out-of-focus information that con- taminates each plane. The interesting graphical zones were extracted using digital thresholding and contour- following techniques4w4.

The combination of the acquisition method with the capabilities of this 3D graphic display allows the fundamental organization of biostructures within cells and nuclei to be effectively probed and studied for the interpretation of structure, and for the correlation of structure with biological functions.31.

It is possible to examine more closely the link between chromatin structure and nuclear morphology, focusing first on the relationship between changes in nuclear morphology and changes in the intranuclear DNA arrangement at the micron level of resolution during the cell cycle”‘,4’.

MATERIALS AND METHODS

The materials and methods used here have been described in Part A in detail. The package for 3D representation exists in two versions: one version is frame grabber dependent, and another is completely independent from any graphics card. These versions have been written in C and Fortran 77, respectively, and run under the DOS operating system.

INPUT DATA FOR 3D DISPLAY

The digital optical sectioning technique, as described in Part A, has made possible the extraction of the x-y-z features of relevant biostructural assemblies. The filter- ing adopted before 3D representation allows us to recognize and locate partially occluded 2D objects plane-by-plane.

To extract information of interest from the processed optical sections, two different methods are used. In the first, a contour-line method, the boundary of the object is determined using a Roberts filter in conjunction with some preprocessing (smoothing or pill-box fiher- ing). In the second method, each structure is extracted by interactive thresholding.

For the near future we are studying a single-pixel contour extractor from images highly affected with noise44.

Once the interesting features have been isolated we are left with a set of binary or multilevel arrays that are assumed to be parallel planar slices. Each array represents homogeneous zones. Figure 3 shows a set of binary chromatin-DNA layouts as extracted from an optical state of images. Multilevel images contain more than one interesting object feature, e.g. nuclear envelope and chromatin distribution, or differently condensed organized chromatin. Each feature is individuated by a particular grey level. Plate 5 (see page 140) shows three multilevel images containing information on both the nuclear morpho- logy and the chromatin distribution for differently stained rat epatocytes nuclei.

Figure 3. Set of binary chromatin-DNA layouts extracted from an optical state of images processed using the approximate method ,following interactive thresholding

3D DISPLAY

The primary need for 3D representation consists of gaining an insight into the connectivity of an object within a specimen. Arrays generated from the proces- sing of optical sections (assumed virtually independent of each other at this stage) can make accessible the 3D (not all volumetric) structural relationship in the specimen, in the sense that the x-y-z information that has been recorded is visualized together. Different structures can be reproduced (e.g. nuclear and cellular envelopes, internal chromatin distribution at different density levels, cytoskeleton components) using separate representation, or by combining them using transparency effects”” that mask one part with respect to the others.

Our actual implementation produces separated 3D pictures of those interesting specimen parts as stacks

~018 no 2 may 1990 135

Figure 4. The x-y-z coordinate system for 30 repre- sentation

of slices, either with or without tilting and rotation. Any angle of rotation around the X, y, z axes, as shown in Figure 4, can be assigned, to achieve, from a user’s view, the best perspective. It should be said that the view under different perspectives does not imply a volumetric knowledge of the object, as the volumetric information is limited by the numerical aperture of the measuring system, and by the acquisition method usedis.

The 3D object is drawn assigning a thickness for each optical section without postulating any particular connectivity among them. In this way, the specimen is approximated with a stack of object planes separated at equal intervals Sz along the z-axis.

The 3D image is formed by the overlay of successive sections top-down or bottom-up, loaded interactively or automatically: each optical section image, tilted and rotated by the angles r and p (respectively related to a planar rotation and to tilting on the z-axis), is endowed with a thickness proportional to the 6z spacing used for acquisition, and is superimposed onto the preceding optical section image.

The steps followed to obtain 3D representation of the object as shown in Figure 5, are:

l image stack definition: a text file ‘stack.3D’ must be created and filled with the names of the image file to be used;

l image stack loading: the user recalls the ‘stack.3D’ file containing the names of the binary silhouettes of the planes to be drawn;

l single plane thickness definition: the 6z spacing, in pixel units, has to be indicated;

l view angle definition: the perspective of the visual- ized object can be changed by specifying a rotation planar angle /I which acts on the 2D section, and a tilting angle T which defines the inclination of the

Figure 5. 30 editing activities

optical section with respect to the z-axis (/3 = 0” means no rotation on the x-y plane, and T= 45” means that the angle between the x-y and the y-z planes is 180” to 60”; T = 90” and r = 0” generate, respectively, a side view and a top view of the object);

l view: 2D stack visualization in a 3D view.

At this stage, the user has two options for assembling the object in 3D:

l recalling plane-by-plane the interesting 2D zones and interactively drawing the connections and related 3D images;

l using an automatic procedure.

The drawing procedures, both automatic and inter- active, can be stopped at any time, producing a partial image that may be used for library purposes. Furthermore, help can be obtained and standard graphic operations performed (see Figure 5). ‘Save’ and ‘Abort’ sessions can be carried out to exit the procedure.

Interactive 3D drawing

The 3D object can be drawn by analysing the optical section image, and by utilizing complementary data which is always on-line and available in different formats: iconic (alternative images, i.e. transmission or phase contrast microscopic images acquired simul- taneously), text and numeric (containing information on the method of preparation, specimen characteristics, shape and form factors, etc.). Complementary data can be recalled on a graphic screen viewport, or in the computer monitor. Plate 6 (see page 140) shows a trans- mission image recalled during an interactive session and displayed on a predefined viewport on the graphic screen.

The user may select different lookup tables and use the library by recalling or creating library images. A number of constraints are automatically imposed during 3D editing. These are those constraints which arise from the definition of the /?, 7 and 6z parameters.

Plate 7 (see page 140) shows an interactive editing

136 image and vision computing

stage (above) and a partial representation of 3D chromatin layouts (below). Plate 8 shows some 3D editing steps conducted to represent a nuclear envelope structure (see page 140).

Automated 3D drawing

The 3D image is automatically built as follows:

The specified stack image is automatically loaded, image by image, into a predefined window on-screen (a working matrix in the case of the frame grabber independent mode), following the order specified in the ‘stack.3D’ file. Rotation and tilting of the p and r quantities assigned are performed into the initial predefined window (working matrix). ‘The windowed area (matrix) is automatically scanned top-down (columns per row). Each scan gives the feature a thickness, generally set proportional to the real distance 6z between the optical sections. ‘The resulting picture is transferred to another tscreen zone (matrix position) at an altitude (row) proportional to 6z (in pixel unit) multiplied by the number of the current sections (the sections are usually numbered from the bottom to the top of the object). The procedure is iterated until the last stack image, generally the lowest, has been treated.

During the computing time, in the frame grabber- dependent mode the operator visualizes parts of the ob,ject, since the image is drawn plane-by-plane, and can decide to stop the procedure at any instant, to save a partial image, to visualize complementary informa- tion, or to modify the ‘stack.3D’ file or pre-fixed parameters.

An image corresponding to 12 frames (128 x 128 pixels) is drawn in approximately seven minutes. Image drawing may be conducted without using any graphic output, since the image has been completed. In this wa,y, 3D assembly may be conducted independently from the graphic card’s use, or more exactly, on any computer without the need for any graphic tool. The final 3D image is stored directly onto an image file for future display. In this case, the assembly time of seven minutes quoted before is reduced by about 20%.

Plate 9 (see page 141) shows an automatic computer- generated top-view (p = 0) of chromatin layouts: this view is interesting because it may be directly related to the original image taken at the focal plane. The pixel intensity grows from the lower level to the higher level simply by adding the sections, plane-by-plane, starting from an offset grey level value.

Plate 10 (see page 141) shows four 3D views for chromatin layouts extracted from images of two differ- ently stained nuclei. The images are generated with a 0 inclination of 45” and for two different planar rotations 7 = 0” and T = 180”.

In the case of a stack of multilevel images, the 3D automatic assembler produces as many 3D images as there are levels. Because each level is utilized to represent a single sample’s feature, each specimen substructure is separately imaged in 3D.

3D colour range

Because we are representing a 3D object on a 2D devicess, we need to artificially create a sense of perspective. This effect is obtained by acting on the choice of colours34.45. For a predefined lookup table, we simply assign the intensity value as a function of the distance with respect to an observer positioned in front of the video monitor.

CONCLUSION

Three-dimensional representation based on optical sectioning data is a suitable way of obtaining in situ structural knowledge on the organization of biostruc- tures. Arrays generated from optical sectioning make clear the 3D structural relationships in the specimen, in the sense that (x, y, z) information has been recorded and 3D information derived from the path, along with the nuclear features. This can be used effectively in investigating the correlation between chromatin organ- ization and cell functioning.

One of the goals of this work was, in fact, to develop an operating procedure that could be used by the biophysician or biologist, and which did not require a specific knowledge of computer science. Interactive and automated 3D imaging, as provided by this package, presents many advantages:

3D views, from different viewpoints, of different features of a more complex biological object, can be obtained; complementary data may be invoked and used during the 3D display session; the system has a ‘blind operating’ mode which is completely independent from the graphic card, not used until the final display; the software package is highly transportable, and runs on small machines within a reasonable time span and at a reasonable cost.

Although these studies and research efforts have focused on the 3D organization of chromatin in situ, it should be apparent that both the digital filters and the 3D reconstitution method are quite general, and are suitable for many other problems in structural cell biology. The 3D reconstitution package is actually used to image STM topographies, as well as electron microscope parallel sections of biological specimens. This program, well suited to the assembly and display of the 3D organization of cellular and nuclear struc- tures, has proven itself a powerful and flexible analysis tool. It is used successfully by experts and non-experts in computer science in the biophysical laboratory, and is actually used to assemble images taken from em microscopes and scanning tunnel microscope topo- graphies in 3Dz2.

ACKNOWLEDGMENTS

We would like to acknowledge Piero Draghi for having tested the interactive 3D editor, for his suggestions concerning the representation modalities and for his support during the earlier stages of this work; Paolo Milani for technical support; Manolo Pancorbo and Eloy Anguiano of the Istituto de Ciencia de Materiales-

~018 no 2 may 1990 137

CSIC Universidad Autonoma de Madrid, for providing us with image hardcopy facilities on a TEKTRONIX 4693 hardcopier and other imaging tools; Dr Laura Vergani for the preparation of samples; and MAE(Offanengo) for providing stepper motors. This work has been supported by research grants from MPI, NATO and CNR (nr.87.01377.44). For Dr Diaspro, this work has been partially supported by a COMEIT project of the Commission of the European Com- munity.

REFERENCES

1

2

Castleman, K R Digital Image Processing Prentice- Hall, NJ, USA (1979) Amdt-Jovin, D J, Nicoud, R M, Kaufmann, J and Jovin, T M ‘Fluorescence digital-imaging micro- scopy in cell-biology’ Science Vol 230 (1985) pp 247-256

20

3

4

7

8

Nicolini, C Biophysics and Cancer Plenum Publish- ing, NY, USA (1986) Nicolini, C ‘Nuclear morphometry, quinternary chromatin structure and cell growth’ 1. Sub- microsc. Cytol. Vol 12 (1980) pp 475-505 Kendall, F M, Beltrame, F, Zietz, S Belmont, A and Nicolini, C ‘The quinternary cromatin- DNA structure: three-dimensional reconstruc- tion and functional significance’ Cell Biophys. Vol 2 No 4 (1980) pp 373-404 Nicolini, C, Diaspro, A, Vergani, L, Bertolotto, M and Germano, P ‘Nuclear architecture, intra- nuclear DNA distribution and nuclear digestion’ Cell Biophys. Vol 13 No 1 (1988) pp 1-14 Diaspro, A, Adami, M and Nicolini, C ‘Three dimensional representation of biostructures imaged with an optical microscope: Part B: 3D graphic display’ Image & Vision Comput. Vol 8 No 2 (May 1990) pp 134-139 Agard, D A ‘Optical sectioning microscopy: cel- lular architecture in three dimensions’ Ann. Rev. Biophys. & Bioeng. Vol 13 (1984) pp 191-219 Diaspro, A, Adami, M, Sarfore, M and Nicolini, C ‘IMAGO: a complete system for acquisition, pro- cessing, 2D/3D 2nd temporal display of micro- scopic bioimages’ Comput. Methods & Programs in Biomed. (in press 1990) Sheppard, C J R and Choudhury, A ‘Image formation in the scanning microscope’ Opt. Actu Vo124 (1977) pp 1051-1073 Agard, D A, Hiraoka, Y and Sedat, J W ‘Three-dimensional light microscopy of diploid drosophila chromosomes’ Cell Motility & the Cytoskeleton Vol 10 (1988) pp 18-27 White, J G, Amos, W B and Fordham, M ‘An evaluation of confocal versus conventional imaging of biological structures by fluorescence light microscopy’ J. Cell Biol. Vol 105 (1987) pp 41-48 Goodman, J W Introduction to Fourier Optics McGraw Hill, NY, USA (1968) Bianco, B and Diaspro, A ‘Analysis of three dimensional cell imaging obtained with optical microscopy techniques based on defocusing’ Cell Biophys. Vol 15 No 3 (1989) pp 189-200 Agard, D A and Sedat, J W ‘Three-dimensional analysis of biological specimens utilizing image

21

5 22

23

24

25

26

9 27

10 28

11

processing techniques’ Proc. SPZE Vo1264 (1980) pp 110-117 Agard, D A and Sedat, J W ‘Three-dimensional architecture of a polythene nucleus’ Nature Lond. Vol302 (1983) pp 676-681 Erhardt, A, Zinser, G, Komitowski, D and Bille, G ‘Reconstructing 3D light-microscopic images by digital image processing’ Appl. Optics Vo124 No 2 (1985) pp 194-200 Brakenhoff, G J, van der Voort, H T M, van Spronsen, E A, Linnemans, W A M and Nanninga, N ‘Three-dimensional chromatin distribution in neuroblastoma nuclei shown by confocal scanning laa;~~~Fscopy’ Nature Vol 317 (1985) pp

Brakenhoff, G J, van der Voort, H T M, van Spronsen, E A and Nanninga, N ‘Three-dimen- sional imaging of biological structures by high resolution confocal scanning laser microscopy’ Scanning Microsc. Vol 2 No 1 (1988) pp 33-40 Morikawa, K and Yanagida, Y ‘Visualization of individual DNA molecules in solution by light microscopy: DAPI staining method’ Biochem. Vol 89 No 10 (1981) pp 693-696 Institute of Biophysics, University of Genova, Italy (unpublished data) Hopkins, H H ‘The frequency response of a defocused optical system’ Proc. Royal Sot. Lond. A Vol231 (1955) pp 91-103 Stokseth, P A ‘Properties of a defocused optical system’ 1. Optical Sot. Am. Vol 59 No 10 (1969) pp 1314-1321 Hiraoka, Y, Sedat, J W and Agard, D A ‘The use of a charge-coupled device for quantitative optical microscopy of biological structures’ Science Vol 238 (1987) pp 36-41 Weinstein, M and Castleman, K R ‘Reconstructing 3D specimens from 2D section images’ Proc. SPZE Vo126 (1971) pp 131-138 Diaspro, A Computer simulation of an optical centred system using Fourier optics Dissertation, Facolta di Ingegneria, University of Genova, Italy (1983) Huijsmans, D D, Laners, W H, Los, J A and Stackee, J ‘Toward computerized morphometric facilities: an aided review of 58 software packages for computer-aided three-dimensional reconstruc- tion, quantification, and picture generation from parallel serial sections’ Anut. Rec. Vol 216 (1986) 449-470

12 29

13 30

14

15

31

32

16

Baba, N, Nakamura, S, Kino, I and Kanaya, K ‘Three-dimensional reconstruction from serial section images by computer graphics’ J. Electron. Microsc. Tech. Vol 3 (1986) pp 401-406 Carazo, J M, Jiemenez, J, Secilla, J P and Carascosa, J L ‘Computer visualization of bio- logical structures’ Am. Biotechnol. Lab. Vol 1 No 5 (1987) pp 48-55 Moens, P B and Moens, T ‘Computer measure- ments and graphics of three-dimensional cellular ultrastucture’ J. Ultrustruct. Res. Vol 75 (1980) pp 131-141 Marko, M, Leith, A and Parsons, D ‘Three- dimensional reconstruction of cells from serial section and whole-cell mounts using multilevel contouring of stereo micrographs’ J. Electron.

138 image and vision computing

17

18

19

Microsc. Tech. Vol 9 (1988) pp 395-411 33 Newman, W R and Sproull, R F Principles of

Interactive Computer Graphics McGraw Hill, NY, USA (1984)

34 Geraud, G, Soyer, A and Epelboin, K ‘Three dimensional computer reconstruction from serial sections of cell nuclei’ Biol. of the Cell Vol 62 (1988) pp 111-117

35 Menhardt, W, Lockhausen, S and Kristen, V ‘An environment for three-dimensional shaded per- spective display of cell components’ Micron. Micros. Acta Vol 17 (1986) pp 349-357

36 Jimenez, J, Santisteban, A, Carazo, J M and Carrascosa, J L ‘Computer graphic display method for visualizing three-dimensional biological struc- tures’ Science Vol 232 (1986) pp 1113-1115

37 Takahashi, T, Chiba, Y and Yaegashi, H ‘Three dimensional reconstruction of biostructures and its characterization: a possible extension of stereology’ Acta Stereof. Vol 6 No 3 (1987) pp 723-73 1

38 Park, K S and Lee, S N ‘A Three-dimensional Fourier descriptor for human body representation/

reconstruction from serial cross sections’ Comput. Biomed. Res. Vol 20 (1987) pp 125-140

39 Jimenez, E J, Cogollos, M and Bernal, L J. Fluid. Mech. Vol 152 (1985) pp 125

40 Udupa, J K ‘Interactive segmentation and boun- dary surface formation for 3D digital images’ Comput. Vision, Graph. Image & Process. Vol 18 (1982) pp 213-235

41 Wezka, J J, Nagel, R N and Rosenfeld, D ‘A threshold selection technique’ IEEE Trans. Comput. Vol 23 (1987) pp 125-140

42 Rosenfeld, A and Kak, A C Digital Picture Proces- sing Academic Press, London, UK (1988)

43 Kendall, F M, Swenson, R, Borun, Y, Rowinki, J and Nicolini, C ‘Nuclear morphometry during the cell cycle’ Science Vol 196 (1977) pp 1106-1109

44 Adami, M, Diaspro, A, Sartore, M and Nicolini, C ‘A method for extracting single-pixel contours from images highly affected with noise’ (in preparation)

45 Farrell, E S ‘Color display and interactive inter- pretation of 3D data’ IBM J. Res. & Dev. Vol 27 (1983) pp 356-365

~018 no 2 may 1990 139

3D representation of biostructures imaged with A Diaspro et al

Plate 1. Effect of a D(p) filter (below) on a set of original defocused sections (above)

Plate 2. Effect of a HP(4) filter on a nucleus fixed with etOH: (left) original image; (right) HP(4) filtered image. The single horizontal line histogram (above) of the grey level values shows enhancement of the fluorescently marked zones

Plate 3. Effect of a HP(4) filter on an unfixed nucleus: (left) original image; (right) HP(4) filtered image

Plate 4. Effect of the L(33) and HP(4) filters on a set of defocused section images: (above) original sections; (centre) L(33) filtered images; (below) HP(4) filtered images

Plate 5.’ Internal (chromatin) layouts and external (nuclear borders) features extracted from cells marked with DAPI using the approximate method. Rat epatocytes nuclei fixed with: (above left) 270 gluteralde- heide; (above right) 3:l v.Iv of etOH; and (below left) an unfixed sample

an optical microscope

Plate 6. A help image has been recalled and displayed in a pre- defined viewpoint of the graphic screen during the interactive editing phase

Plate 7. An interactive 30 editing stage (above), and a partial result (below)

Plate 8. Building a nuclear envelope structure from a state of nuclear contours via interactive editing stages

140 image and vision computing

3D representation of biostructures imaged with an optical microscope A Diaspro et al

Plate 9. Top view of chromatin layouts. The pixel intensity is direc- tionally proportional to the distance of the observer from the optical section contributing to the overall picture

Plate 10. Four 30 representations of differently viewed chrornatin layouts (see text)

Feature labelling in infrared oceanographic images N Krishnakumar et al

Plate I. Edges overlapped with infrared image

~018 no 2 may 1990

Plate 2. Feature labelling output

141