www.elsevier.com/locate/micron

Micron 38 (2007) 115–120

Pattern projection for subpixel resolved imaging in microscopy

Dror Fixler a, Javier Garcia b, Zeev Zalevsky a,*, Aryeh Weiss a, Mordechai Deutsch c

a School of Engineering, Bar-Ilan University, Ramat-Gan 52900, Israelb Departamento de Optica, Universitat de Valencia, C/Dr. Moliner 50, 46100 Burjassot, Spain

c Department of Physics, Bar-Ilan University, Ramat-Gan 52900, Israel

Abstract

In this paper, we present a new approach providing super resolved images exceeding the geometrical limitation given by the detector pixel size

of the imaging camera. The concept involves the projection of periodic patterns on top of the sample, which are then investigated under a

microscope. Combining spatial scanning together with proper digital post-processing algorithm yields the improved geometrical resolution

enhancement. This new method is especially interesting for microscopic imaging when the resolution of the detector is lower than the resolution

due to diffraction.

# 2006 Elsevier Ltd. All rights reserved.

Keywords: Microscopy; Geometrical limitation

1. Introduction

Fluorescence microscopy is one of the most widely used

tools for localizing proteins in intracellular compartments, with

the advantage of molecular selectivity in imaging, enabling live

observation, and allowing visualization of specific molecules in

living cells. The development of protein microarray detection

has accelerated over the past few years. Protein microarrays can

be widely used in diagnostics, drug screening and testing,

disease monitoring, drug discovery and medical research

(Jocelyn and Leodevico, 2002). Observation of fluorescent-

stained living cells on the microscope stage requires special

consideration of experimental conditions such as temperature

control, pH, nutrition and other growth conditions (Lakowicz,

1999).

The microscopy fluorescence images are affected by several

noise sources and limitations. A main distortion is introduced

by the superposition of out-of-focus images to the focused

image. Two common approaches for improving the SNR under

this situation are the ‘‘iterative de-convolution’’ which is

described in Agard (1984), Wallace et al. (2001) and Neil et al.

(1997) and optical sectioning by means of structured

illumination (Neil et al., 1997). Some of the algorithms are

computer time consuming, especially in large images (a few

* Corresponding author. Tel.: +972 3 531 7055; fax: +972 3 534 0697.

E-mail address: zalevsz@eng.biu.ac.il (Z. Zalevsky).

0968-4328/$ – see front matter # 2006 Elsevier Ltd. All rights reserved.

doi:10.1016/j.micron.2006.07.007

mega bits per image) (Yang et al., 2002), making them hard to

apply in practice.

In addition, the fluorescence image is limited by the spatial

resolution, which is determined by the Rayleigh diffraction limit,

or by the detector pixel size. The diffraction limit is mainly

related to the numerical aperture of the microscope lens and its

absolute limit is about half a wavelength on the sample plane. For

lenses with low numerical apertures it is possible to increase the

resolution by means of structured illumination, that serves as an

encoding method for high-resolution features (Lukosz, 1967;

Gustafsson, 2000; Fixler et al., 2006). With respects to sensor

limitation, the pixel size is given by the specific choice of camera,

which is application dependent, paying attention to a plurality of

factors such as overall sensor size (related to the field of view),

sensitivity, resolution or pricing. One of the crucial aspects is the

light sensitivity. In most of the cases the fluorescence intensity

(FI) that is emitted is very low and hard to detect. Even green

fluorescent protein (GFP) measurements, which enabled

fluorescent labeling of proteins in living cells without protein

purification (Tsien, 1998), suffer in most instances from low FI

and frequently are barely visible. The signal-to-noise ratio (SNR)

of images in a live cell experiment is usually limited by the

reductions in FI and exposure time necessary for a successful

experiment. A common approach to cope with this problem is to

use a sensor with large pixels size, which gathers a higher number

of photodetections and increases at the same time the intensity

dynamic range. As a result, many of the commercially available

high sensitivity cameras are built on large pixels sensors, as

D. Fixler et al. / Micron 38 (2007) 115–120116

compared to general market digital cameras. In these cases the

final resolution is dictated by the sensor especially for low

magnifications at high numerical aperture, when the diffraction

spot size measured in the detector plane is small (typically a few

micrometers).

Several efforts have been made to overcome the detector

pixel size limitation when several displacement frames of the

same sample have been captured (Granrath and Lersch, 1998;

Gillette et al., 1995; Zalevsky and Mendlovic, 2004; Zalevsky

et al., 2000). Most of the methods are based on interlacing or

averaging the frames after a subpixel resolved registration.

Nevertheless, an increase of resolution by a factor of two is

usually considered the maximum realistic improvement, owing

to the loss of information in each frame (Zalevsky and

Mendlovic, 2004). In Zalevsky et al. (2000), a technique using a

mask with subpixel details attached to the detector is proposed,

in order to obtain a higher resolution improvement. The

implementation of this method is very costly because of the

need to prepare a high-resolution mask and to set it in physical

contact with the sensor (or fabricate the mask over it).

In this paper, we propose and experimentally demonstrate a

method to surpass the resolution of the sensor in microscopy

imaging in the cases where it is a limiting factor, mainly at low

magnification. The approach starts by the projection of periodic

patterns on top of the examined sample. By scanning the sample

with subpixel steps and applying proper digital processing, a

geometrical super resolved imaging is obtained, i.e. the image

captured has much smaller pixels, and therefore is more

effective. Unlike previous approaches the only modification

over a conventional microscope system is the use of structured

light in the illumination of the sample, not requiring any

modification on the sensor itself. The resolution improvement

corresponds to the number of the subpixel scanning steps. This

is possible as long as the generated effective pixels are not

smaller than the diffraction limitation of the optics.

Section 2 illustrates the theory of the proposed method. In

Section 3 we present the system and the samples that we use.

Section 4 reports the experimental results and Section 5 the

conclusions.

2. Theory

For the mathematical model we assume an M pixels sensing

device that samples the scene N times. In each sample, the device

is shifted with respect to the image a distance of Dx/N, where Dx

is the pitch of the detector’s pixels. At the same time a periodic

pattern, with period Dx, is used to illuminate the sample. This

pattern is displaced synchronously with the detector. For CCD

sensors each pixel integrates all the light impinging upon it

within the cycle. We will deal with the 1D mathematical

derivations, although the extension to 2D is straightforward. In

our analysis we are about to neglect the effect of diffraction. This

approximation can hold as far as the diffraction spot size in the

detector plane is smaller than the target resolution after the

method is applied. This is, when the resolution limit is given by

the pixel size and not by diffraction. In addition, since we work at

low NA and neglect out of focus effects.

The light distribution just before the detector for a subpixel

displacement nDx/N is:

sðxÞg�

x� nDx

N

�(1)

where n = 0, 1, 2, . . ., N � 1, s(x) is the imaged object and g(x)

is the projected pattern. Denoting by p(x) the spatial shape of

the pixel (a rectangle function for instance), the integrated

signal for pixel m at subpixel displacement n is:

ym;n ¼Z þ1�1

sðxÞg�

x� nDx

N

�p

�x� mDx� n

Dx

N

�dx (2)

with m = 0, 1, 2, . . ., M � 1. Notice that Eq. (2) implies a

synchronous subpixel movement of the pattern and the detector.

This can easily be accomplished if instead of projecting the

pattern on the object, the pattern is set directly in contact with

the detector. Then the movement of the pattern will follow the

movement of the detector. A simpler way to achieve the

scanning, without physical attachments on the detector, is to

displace the object, keeping static the rest of the system.

Owing to the periodicity of the pattern g(x) = g(x � mDx).

Moreover, we can join the two indices m and n in a single index

as k = n + mN, that runs continuously from 1 to NM, covering

all pixels and subpixel displacements. Thus, we can rewrite:

yk ¼Z þ1�1

sðxÞg�

x� kDx

N

�p

�x� k

Dx

N

�dx (3)

For simplicity we can redefine:

cðxÞ ¼ gðxÞ pðxÞ (4)

This gives the unit cell of the pattern modified by the pixel. In the

simplest case, where the pixel is just a rectangle function, c(x)

represents a single period of the pattern. With this definition:

yk ¼Z þ1�1

sðxÞc�

x� kDx

N

�dx (5)

Eq. (5) states a convolution operation between the original

signal and the unit cell of the combination given by the product

of the projected pattern and the sensing array. The convolution

is sampled at the rate given by the subpixel displacements. And,

according to the sampling theorem, these values are sufficient

to fully recover a spatial distribution sampled at rate of Dx/N.

The convolution in Eq. (5) can be expressed in the Fourier

domain as a product. Naming {Sk}{Yk} and {Ck} the discrete

Fourier transforms (DFT) of the sampled input and output and

unit cell distributions:

Yk ¼ SkCk (6)

The sampled original signal can be recovered from the samples

{yk} by means of an inverse filtering that removes the con-

volution effect of c(x). This operation is easily performed in the

Fourier domain; the restored high-resolution image can be

obtained as:

Sk ¼ DFT�1

�Ym

Cm

�(7)

D. Fixler et al. / Micron 38 (2007) 115–120 117

In order to summarize, the full process consists on: (a) the

object is illuminated with a pattern such that on the detector

plane its period coincides with the detector pixel pitch; (b) a set

of images, each for a subpixel object displacement, is recorded

at the detector’s resolution (M pixels); (c) the images are

interlaced to give an image with subpixel pitch sampling

(MN pixels); (d) the influence of the pattern is removed by

inverse filtering, rendering the final reconstructed image.

At this point it is worth considering the gain obtained by

using the pattern projected on top of the object. In principle, the

same method could be use without the pattern, the only

difference being that the output image would be a convolution

with the pixel function [c(x) = p(x)], typically a rectangle. The

main drawback of this approach is that [according to Eq. (6)]

the frequencies are attenuated with the Fourier transform of the

pixel function. Normally, this means a strong attenuation of

high frequencies and the presence of zeros in the spectrum

where the signal cannot be recovered. In general a two-fold

improvement in resolution is considered the maximum

achievable (Zalevsky and Mendlovic, 2004). On the other

hand using a projected pattern with subpixel details will

enhance the high frequencies with respect to the conventional

illumination case making possible the recovery of the signal at

higher resolutions. As an extreme case example, if the pattern

contains a single transparent spot then the obtained resolution

will be the size of this spot. Also, a proper design of the subpixel

structure can avoid the presence of zeros in the Fourier

transform of the unit cell (Zalevsky et al., 2000), avoiding the

need of any regularization when applying Eq. (7).

3. Materials and methods

3.1. Apparatus

The experiments were done on biological samples. Beads

and human cells were imaged using an Epi-Fluorescence

microscope (BX61, Olympus, Japan), with 20� and 4� (NA of

0.6 and 0.13, respectively) LCPlanFl objective (Olympus,

Japan) and 488 nm of Ar Ion laser, (Spectra-Physics CA).

Images were collected via the photometric CoolSNAPHQ

monochrome CCD camera with a 1392 � 1040 imaging array

and 6.45 mm � 6.45 mm pixels (Roper Scientific Inc., Trenton,

NJ). This cooled CCD camera system provides 12-bit

digitalization at both 10 and 20 MHz.

3.2. Biological samples

For the experiments with cells, human Jurkat T-lymphoblast

cell line was grown in a humidified atmosphere containing 5%

CO2, in RPMI 1640 medium (Biological Industries, Israel);

supplemented with 10% (v/v) heat inactivated fetal calf serum

(Biological Industries, Israel), 2 mM L-glutamine, 10 mM

Hepes buffer solution, 1 mM sodium pyruvate, 50 U/ml

penicillin, and 100 mg/ml streptomycin. The cells were washed

twice with incomplete RPMI 1640 medium, without phenol

red, containing 10 mM HEPES buffer solution. An aliquot of

100 ml of cell suspension (5 � 106 cells/ml) was added to 50 ml

of fluorescein diacetate (FDA) staining solution (Sigma, St.

Louis, MO, USA, F7378) dissolved in PBS to a final

concentration of 2 mM in PBS, and incubated at room

temperature for 5 min. At the end of incubation, cells were

loaded onto a microscope slide and measured.

For the experiments with beads, polyscience fluorescent

beads Fluoresbrite (fluorescein) of 10 � 0.1 mm diameter, with

FI values of 2000–50,000 molecules of equivalent soluble

fluorochromes, MESF, were used. These were obtained from

Bangs Laboratories Inc. (IN, USA).

4. Experiments

The principle of the proposed method is demonstrated by

means of two experiments, one in transmission microscopy and

the other in fluorescence microscopy. For the purpose of

comparison, we use the binning capability of the camera to

simulate a large pixels detector. This way a reference high-

resolution image is also available.

4.1. Microscopic super resolution measurements of beads

The periodic pattern projection method was first tested on

10.0 � 0.1 mm diameter beads using the 4� lens (NA of 0.13).

The diffraction resolution limit is thus about 2 mm in the sample

plane, corresponding to 8 mm in the sensor plane. Thus, the

diffraction spot size is slightly larger than the physical pixel

size (6.45 mm). A two-dimensional grating with 200 line pairs/

mm was projected on the beads at such a magnification that

80 line pairs/mm periodicity approximately was generated on

the sample. The grating is formed by square tessellation of

replicas of a unit cell consisting on a transparent square on a

black background. The projection system magnification was

fine tuned until a full grating period covered eight pixels in the

captured image. The profile of the grating is known from

previous measurements and is used for the inverse filtering

needed for the reconstruction. For the proof of concept the

captured images were binned into macro pixels of 8 � 8

original pixels. The scan was two-dimensional with eight steps

in each axis and thus a total of 64 images were taken prior to the

digital processing. Fig. 1 shows the high-resolution images

prior to binning while a long exposure time was applied. Fig. 2

demonstrates the low-resolution image when the 8 by 8 binning

was done and the exposure time was reduced by a factor of 64.

Indeed many spatial features have disappeared. Fig. 3 illustrates

the reconstruction using the suggested approach via illumina-

tion with the periodic pattern. The processing consists on

interlacing the 64 images according to the shift of the sample

for each one of them and then applying the inverse filtering

correction. These operations are very fast to perform and do not

compromise the speed of the final imaging. Note that the

resolution is comparable to that of Fig. 1. The square artifacts

(8 � 8 pixels) in some parts of the image are due to non-

uniform transmittance of the projected grating. Also, the non-

uniformities in the mechanical scanning of the sample may

introduce a periodic pattern in the resulting image. Therefore,

aside from the interlacing and unit cell effect compensation, a

Fig. 1. High-resolution image of polyscience fluorescent beads obtained with

long exposure time.

Fig. 3. Reconstruction obtained with the suggested approach including the

application of illuminating with a periodic pattern.

D. Fixler et al. / Micron 38 (2007) 115–120118

simple digital processing was performed in order to obtain

higher uniformity, consisting on removing the periodic patterns

appearing in the spectrum of the reconstructed image. A more

complete processing can be made by taking an image over a

uniform sample to calibrate the local grating transmittance. For

comparison in Fig. 4 we present the reconstruction obtained

from the set of 64 images as in Fig. 3, but without projecting the

two-dimensional grating. Despite the increased resolution with

respect to Fig. 2, the beads are not separable as in Fig. 3. This

operation is equivalent to ‘‘microscanning’’ (Gustafsson, 2000)

and its resolution enhancement is limited by the fill factor of the

detector’s pixels. When the projected grating is used, this

limitation in the resolution improvement factor no longer

exists, and it is then directly proportional to subpixel scanning

steps.

Fig. 2. Image obtained via an 8 � 8 binning. Exposure time is reduced from

Fig. 1 by a factor of 64, although the resolution is not enough for resolving the

beads.

4.2. Microscopic super resolution measurements of human

cells stained with fluorescein

In this section, we present the experiment performed with

Jurkat T-lymphoblast cell lines (described in Section 3) while

the binning and scanning were done in one dimension only. The

scanning was one-dimensional and thus only eight images were

captured to fit the binning factor of eight. A projection grating

of 600 lines/mm was used, fitting the magnification to match on

the camera one period of the grating to eight camera pixels. The

diffraction resolution limit for the 20� (NA of 0.6) lens is in

this case about 0.5 mm in the sample plane, corresponding to

10 mm in the sensor plane. Fig. 5 shows the high-resolution

image obtained with a long exposure time. This image was

obtained without binning. The inset displays a magnified

Fig. 4. Reconstruction from a set of 64 images, as in Fig. 3, but without the

grating illumination.

Fig. 5. High-resolution image of Jurkat T-lymphoblast cells obtained with long

exposure time. The inset displays a magnified portion of the image with high-

resolution details for comparison.

Fig. 6. Image obtained via binning of 8 � 1. Exposure time is reduced from

Fig. 5 by a factor of eight, although the resolution is not enough for resolving the

details.

Fig. 7. Reconstruction obtained with the suggested approach, illuminated with

a grating. The artefacts (8 � 1 pixels) in some parts of the image are due to non-

uniformities of the grating.

Fig. 8. Reconstruction from a set of eight images as in Fig. 7, but without the

grating illumination.

D. Fixler et al. / Micron 38 (2007) 115–120 119

portion of the image with high-resolution details and will be

used for reconstruction quality comparison. Fig. 6 displays the

image obtained via 8 � 1 binning. Exposure time is reduced by

a factor of eight, although the resolution is not enough for

resolving the details. In Fig. 7 one can see the reconstructed

results obtained with the suggested approach including the

grating projection. Note that the resolution is comparable with

that of Fig. 5. The artefacts (8 � 1 pixels), in some parts of the

image, are due to non-uniformities of the grating. The

reconstruction included simple digital enhancement of the

image in order to improve its spatial uniformity, as described in

the previous subsection. Fig. 8 shows the reconstruction from a

set of eight images as in Fig. 7, but this time without the grating

illumination. Despite the increased resolution with respect to

Fig. 6 due to the ‘‘microscanning’’ operation, the high-

resolution details are not as clearly resolved as in Fig. 7.

5. Conclusion

In this manuscript, we have demonstrated a new approach to

geometrical super resolution generated by projection of

periodic structures on top of a scanned sample, while the

scanning is done with subpixel steps and some digital image

enhancement is applied on the captured images. The approach

was employed in both transmission and fluorescence micro-

scopy, proving a significantly improved spatial resolution when

the limiting factor of resolution is the pixel size of the detector.

Acknowledgements

Javier Garcia acknowledges the support of the Spanish

Ministry of Science and Education and the Ministry of Science

and Technology under the codes PR-2004-0543 and FIS2004-

06947-C02-01.

References

Agard, D.A., 1984. Optical sectioning microscopy: cellular architecture in three

dimensions. Annu. Rev. Biophys. Bioeng. 13, 191–219.

D. Fixler et al. / Micron 38 (2007) 115–120120

Fixler, D., Garcia, J., Zalevsky, Z., Weiss, A., Deutsch, M., 2006. Speckle

random coding for 2D super resolving fluorescent microscopic imaging.

Micron, submitted for publication.

Gillette, Stadtmiller, Hardie, 1995. Aliasing reduction in staring infrared

imagers utilizing subpixel techniques. Opt. Eng. 34, 3130–3137.

Granrath, D., Lersch, J., 1998. Fusion of images on affine sampling grids. J. Opt.

Soc. Am. A 15, 791.

Gustafsson, M.G.L., 2000. Surpassing the lateral resolution limit by a factor of

two using structured illumination microscopy. J. Microsc. 198, 82–87.

Jocelyn, H.N., Leodevico, L.I., 2002. Biomedical applications of protein chips.

J. Cell. Mol. Med. 6 (3), 329–340.

Lakowicz, J.R., 1999. Principles of Fluorescence Spectroscopy, 2nd ed. Plenum

Press, New York.

Lukosz, W., 1967. Optical systems with resolving powers exceeding the

classical limit II. J. Opt. Soc. Am. 57, 932–941.

Neil, M.A.A., Juskaitis, R., Wilson, T., 1997. Method of obtaining optical

sectioning by using structured light in a conventional microscope. Opt. Lett.

22, 1905–1907.

Tsien, R.Y., 1998. The green fluorescent protein. Annu. Rev. Biochem. 67, 509–

544.

Wallace, W., Schaefer, L.H., Swedlow, J.R., 2001. A workingperson’s guide to

deconvolution in light microscopy. Biotechniques 31, 1076–1097.

Yang, Y.H., Buckley, M.J., Dudoit, S., Speed, T.P., 2002. Comparison of

methods for image analysis on cDNA microarray data. J. Comput. Graph.

Stat. 11 (1), 108–136.

Zalevsky, Z., Mendlovic, D., 2004. Optical Super Resolution. Springer, New

York.

Zalevsky, Z., Mendlovic, D., Marom, E., 2000. Special sensor masking

for exceeding system geometrical resolving power. Opt. Eng. 39, 1936–

1942.