1742 J. Opt. Soc. Am. A/Vol. 14, No. 8 /August 1997 Marroquin et al.

Adaptive quadrature filters and the recoveryof phase from fringe pattern images

J. L. Marroquin

Centro de Investigacion en Matematicas, Apartado Postal 402 Guanajuato, Guanajuato 36000, Mexico

M. Servin and R. Rodriguez-Vera

Centro de Investigaciones en Optica, Apartado Postal 1-948 Leon, Guanajuato 37000, Mexico

Received July 19, 1996; revised manuscript received December 12, 1996; accepted February 11, 1997

A principled approach, based on Bayesian estimation theory and complex-valued Markov random-field priormodels, is introduced for the design of a new class of adaptive quadrature filters. These filters are capable ofadapting their tuning frequency to the local dominant spatial frequency of the input image while maintainingan arbitrarily narrow local frequency response; therefore they may be effectively used for the accurate recoveryof the phase of broadband spatial-carrier fringe patterns, even when they are corrupted by a significantamount of noise. Also, by constraining the spatial variation of the adaptive frequency to be smooth, they per-mit the completely automatic recovery of local phase from single closed fringe pattern images, since the spu-rious discontinuities and sign reversals that one obtains from the classical Fourier-based methods are avoidedin this case. Although the applications discussed here come from fringe pattern analysis in optics, these fil-ters may also be useful in the solution of other problems, such as texture characterization and segmentationand the recovery of depth from stereoscopic pairs of images. © 1997 Optical Society of America[S0740-3232(97)00908-3]

1. INTRODUCTIONLinear, shift-invariant quadrature filters (QF’s) are char-acterized by a real frequency response that corresponds toa rapidly decaying window function, such as a Gaussian(in which case one obtains the well-known Gabor filters1)centered on a particular multidimensional tuning fre-quency v0 . In the spatial domain, these filters have acomplex impulse response, whose real and imaginaryparts correspond to a damped cosinusoid w(x)cos(v0 • x)and a damped sinusoid w(x)sin(v0 • x), respectively,where the damping function w(x) is the inverse Fouriertransform of the window function in the frequency do-main (see, for example, Refs. 2–8).Consider applying this QF to a signal of the form

g~x ! 5 b~x !cos@v0 • x 1 a~x !#

5b~x !

2(exp$i@v0 • x 1 a~x !#%

1 exp$2i@v0 • x 1 a~x !#%),

where x is the two-dimensional spatial variable, the op-erator (•) denotes inner product, and b(x) and a(x) arereal functions. If the function b(x) is smooth and themagnitude of the function a(x) is small compared withv0 • x, the power spectrum of g(x) will be concentratednear v0 and 2v0 [we say in this case that g(x) is quasi-monochromatic; see Section 2 and Figs. 3(b) and 3(c)], andtherefore the output of the QF [i.e., the result of convolv-ing g(x) with the corresponding complex kernel] will be ofthe form

0740-3232/97/0801742-12$10.00 ©

f~x ! 5 w~x ! 1 ic~x ! 'b~x !

2exp$i@v0 • x 1 a~x !#%,

so that we may recover b(x) from its local magnitude

M~x ! 5 @w2~x ! 1 c2~x !#1/2

and a(x) from its local phase

f~x ! 5 arctan@c~x !/w~x !#. (1)

[the formulas are b(x) 5 2M(x) and a(x) 5 f(x) 2 v0• x]. This property makes these filters a valuable tool inmany fields; they have been applied, for example, to therecovery of phase from fringe pattern images in optics,9–11

to texture characterization and segmentation,2–5 to therecovery of depth from stereoscopic pairs of images,6,12

and to the recovery of optical flow (extending them to thespatiotemporal domain).7,8

Their usefulness, however, decreases as the underlyingassumptions are violated. For example, if the modulat-ing phase a(x) has a relatively large variation over thespatial domain under consideration (an important case infringe pattern analysis; see Fig. 4 in Subsection 3.A), thespectrum of g(x) will no longer be quasi-monochromaticand may in fact overlap with the spectrum of other un-wanted components, making the phase estimate given byEq. (1) unreliable.One way to overcome this difficulty is to note that in

this case g(x) may be written in the form

g~x ! 5 b~x !cos@v~x ! • x 1 a~x !#, (2)

where v(x) denotes a spatially varying frequency andua(x)u is small compared with uv(x) • (x)u. This meansthat it should be possible to obtain an accurate phase re-

1997 Optical Society of America

Marroquin et al. Vol. 14, No. 8 /August 1997 /J. Opt. Soc. Am. A 1743

covery even in this case by using, instead of a classicalQF, one whose tuning frequency is adapted dynamicallyto the local dominant frequency v(x). The purpose ofthis paper is to present a principled approach for the de-sign of such adaptive quadrature filters (AQF’s). Thisapproach is based on Bayesian estimation theory, withMarkov random fields (MRF’s) as prior models, and ispresented in Section 2. In Section 3 we discuss the ap-plication of these filters to the automatic recovery of localphase from single fringe pattern images and present someexperiments that illustrate their performance. In Sec-tion 4 we present the concrete algorithms that were usedfor the implementation of these filters, and, finally, themain results are summarized and discussed in Section 5.

2. ADAPTIVE QUADRATURE FILTERSThe problem of constructing an AQF may be formulatedin the following way: Consider a subset S of sites takenfrom a regular lattice L, which consists of the set of pixelsof an image where observations are present and the solu-tion is to be computed (for instance, in the example of Fig.9, discussed in Subsection 3.C, the set S consists of thepixels inside the circular pupil where observations areavailable). Given an observed image $g(x), x P S%, theproblem is to construct a complex field $f(x) 5 w(x)1 ic(x), x P S% that best approximates a locally quasi-monochromatic component of g(x); in other words, w(x)may be considered as the in-phase component of a noise-free fringe pattern that best approximates g(x), whilec(x) represents the corresponding quadrature compo-nent.This problem is not well posed in a mathematical

sense, since it is not possible, with this information, to re-cover f from g uniquely. To solve it, it is necessary to in-clude in the reconstruction algorithm additional priorconstraints on the solution. For example, one may re-quire that the field f should be locally quasi-monochromatic and that the dominant frequency v(x)should vary smoothly across L.A powerful and systematic way for constructing such

constrained reconstruction algorithms is to consider f andv to be random fields on L and to define the desired solu-tion as the maximizer of the a posteriori probability dis-tribution, which, from Bayes’s rule,13,14 is of the form

Pf,vug~ f, v! 51KPguf,v~ f, v!Pf,v~ f, v!, (3)

where K is a normalizing constant.The fidelity to the observations is embodied in the con-

ditional distribution Pguf,v( f, v); if one assumes indepen-dent observation noise and notes that the field v is not di-rectly observable, this distribution may be written in theform

Pguf,v~ f ! 51K8

expF2(xPS

F( f ~x !, g~x !)G , (4)

where K8 is a constant and the function F depends on thenoise model [for example, for zero-mean, Gaussian obser-vation noise, F(f(x), g(x)) 5 (1/2s2)u f (x)2g(x)u2].

The prior constraints on f and v are specified in theprior distribution Pf,v( f,v). If one assumes that therandom fields f and v are Markovian (in the sense thatthe probabilistic dependencies are spatially localized),this distribution may be written in the form of a Gibbsianmeasure (see, for example, Refs. 13 and 14 for details):

Pf,v~ f, v! 51Zexp@2Uf,v~ f, v!#. (5)

Here Z is a normalizing constant and the energyUf,v(f,v) is obtained as the sum of a set of potential func-tions $VC%, each of which depends on the configurations ofthe fields f and v on a small subset C of interacting sites(these subsets are called the cliques of an appropriateneighborhood system defined on S):

Uf,v~ f, v! 5 (C

VC~ f, v!.

Substituting Eqs. (4) and (5) into Eq. (3), one gets thatthe optimal (maximum a posteriori) estimator for thefields f, and v is obtained by minimizing a posterior en-ergy function U, which is given by

U~f, v! 5 (xPS

F( f ~x !, g~x !) 1 (C

VC~ f, v!.

The potentials $VC% embody the prior constraints aboutthe behavior of the solution. For example, the constraintthat f be smooth may be enforced locally by potentialsthat force the values of f at neighboring sites to be simi-lar, i.e., the cliques are nearest-neighbor pairs of sites xand y, and Vxy( f ) 5 u f (x)2f (y)u2 (the magnitude istaken because f is, in general, complex-valued). One ob-tains in this case the classical membrane energy function:

U~f ! 5 (xPS

u f ~x ! 2 g~x !u2 1 l (@x, y#

u f ~x ! 2 f ~ y !u2,

where @x, y# indicates that the sum is taken over allnearest-neighbor pairs of sites. The minimizer of thisfunction corresponds to the equilibrium configuration of asystem of interacting particles [represented as opencircles in the one-dimensional example of Fig. 1(a)] thatare constrained to move in copies of the complex planethat pass through the corresponding lattice sites and thatare connected to the observations (represented as closedcircles) and to neighboring particles by linear springs(bold lines in the same figure), where the constant of in-terparticle springs equals l times the constant of theparticle-observation springs. Note that the equilibriumpositions are in this case confined to the real axes.If the copies of the complex plane at neighboring loca-

tions are rotated by an angle v0 with respect to eachother [Fig. 1(b)], the equilibrium positions of the particleswill be, in general, outside the real axes and will corre-spond to the configuration f that minimizes the function

U~f ! 5 (xPS

uf~x ! 2 g~x !u2 1 l (@x, y#

uf~x !

2 f~ y !exp@iv0 • ~x 2 y !#u2.

For large values of l, these equilibrium positions will bealigned, but in the original (unrotated) coordinate system,

1744 J. Opt. Soc. Am. A/Vol. 14, No. 8 /August 1997 Marroquin et al.

they will describe a circular helix, so that the correspond-ing real and imaginary parts will describe sinusoidal pat-terns with frequency v0 and with a phase difference ofp/2. In other words, the potentials given by

Vxy~ f ! 5 u f ~x ! 2 f ~ y !exp@iv0 • ~x 2 y !#u2

5 U f~x !expF2i2

v0 • ~x 2 y !G2 f ~ y !expF i2 v0 • ~x 2 y !GU2 (6)

[where (x, y) is a nearest-neighbor pair of sites] enforcethe constraint that the complex signal f should be a quasi-monochromatic analytic signal, i.e., close to the formf(x) 5 M exp(iv0•x) (see Ref. 15 for details).Thus a regularized solution to the problem of recon-

structing a quasi-monochromatic component f of an ob-served signal g may be obtained by minimizing the func-tion

U~ f ! 5 (xPS

u f ~x ! 2 2g~x !u2

1 l (@x,y#

U f ~x !expF2i2

v0 • ~x 2 y !G2 f ~ y !expF i2 v0 • ~x 2 y !GU2.

The factor of 2 is included in the observation term so thatan input of the form g(x) 5 cos(v0 • x) produces an out-put of unit magnitude, as it should.To get a variable tuning frequency, one should consider

the rotation angle as an additional degree of freedom, i.e.,replacing the constant v0 by a field v(x), so that the po-tentials given by Eq. (6) now take the form

Fig. 1. (a) Physical model for the one-dimensional membraneregularization function extended to the complex case: The esti-mated field is modeled as a system of particles (white circles) al-lowed to move in copies of the complex plane; the particles areconnected to the observations (black circles) and to neighboringparticles by linear springs (bold lines). (b) Model for the AQFregularization function obtained from model (a) by rotating thecopies of the complex plane by an angle equal to the local tuningfrequency (see the text).

Vxy~f ! 5 Uf~x !expF2i2

v~x ! • ~x 2 y !G 2 f~ y !

3 expF i2 v~y ! • ~x 2 y !GU2,subject to the constraint that v(x) should have a smoothvariation across the lattice. This constraint may be en-forced with a second-order (thin-plate) model,16 whosecliques are sets of three adjacent sites $a, b, c% lying on aline and sets of four neighboring sites $p, q, r, s% lying atthe corners of squares with sides equal to the lattice spac-ing (see Fig. 2). The potentials are squares of second dif-ferences:

Vabc~v! 5 u2v~a ! 1 2v~b ! 2 v~c !u2, (7)

Vpqrs~v! 5 2u2v~p ! 1 v~q ! 2 v~r ! 1 v~s !u2 (8)

[note that one should take the magnitude, since in two di-mensions, v(x) is a two-dimensional vector(v1(x), v2(x))].The complete posterior energy takes therefore the form

U~ f, v! 5 (xPS

uf~x ! 2 2g~x !u2

1 l (@x,y#

U f ~x !expF2i2

v~x ! • ~x 2 y !G2 f ~ y !expF i2 v~ y ! • ~x 2 y !GU21 mF (

@a,b,c#Vabc~v! 1 (

@p,q,r,s#Vpqrs~v!G ,

(9)

where @x, y#, @a, b, c#, and @ p, q, r, s# indicate that thesums are taken over all the corresponding cliques in S asindicated above and m is a parameter that controls thesmoothness of field v.The term adaptive quadrature filters (AQF’s) used to

denote the algorithms that minimize function (9) is justi-fied because for a constant tuning frequency these algo-rithms act, in fact, as linear, space-varying filters, whoselocal frequency response (away from the borders of thelattice) closely resembles that of a classical QF (e.g., a Ga-bor filter), whose bandwidth is controlled by the param-eter l (see Ref. 15).For low tuning frequencies, the behavior of these filters

may not be adequate, because components that corre-spond to zero and negative frequencies will be includedin their passband. This may be corrected by modifyingthe data term so that one approximates, instead of the ob-servations themselves, their first differences, since this

Fig. 2. Three- and four-site cliques for the thin-plate two-dimensional model (see the text).

Marroquin et al. Vol. 14, No. 8 /August 1997 /J. Opt. Soc. Am. A 1745

Fig. 3. (a) Phase-modulated signal with two quasi-monochromatic components (see the text), (b) power spectra of components 1 (solidcurve) and 2 (dashed curve), (c) first component extracted by the AQF, (d) phase recovered from the first component (solid curve) and truemodulating phase (dashed curve).

will force a zero in the frequency response at the origin ofthe frequency plane.15 The corrected energy function isthus

U~ f, v! 5 (@x, y#

u f ~x ! 2 f ~ y ! 2 2@g~x ! 2 g~y !#u2

1 l (@x, y#

U f ~x !expF2i

2v~x ! • ~x 2 y !G

2 f ~ y !expF i2 v~ y ! • ~x 2 y !GU21 mF (

@a,b,c#

Vabc~v! 1 (@p,q,r,s#

Vpqrs~v!G .(10)

The fields f and v that result from the minimization offunction (9) or (10) are related in the following way: Thefield f will be of the form

f ~x ! 5 M~x !exp@f~x !#,

where

f~x ! 5 v~x ! • x 1 a~x !, (11)

with ua(x)u small compared with uv(x)•xu [i.e., f(x) is lo-cally quasi-monochromatic], and M(x) is a smooth, real-valued function. Note that the decomposition given byEq. (11) is not unique; different values of m in Eq. (9) or(10) will give different optimal v fields with various de-grees of smoothness; these v fields, however, representonly the local tuning frequency of the AQF and not the de-rivative of the local phase f(x). This phase (computedas f(x) 5 arg@ f (x)#), is, in fact, almost insensitive to the

precise value of m (as long as it remains within certainlimits). One might understand this by considering thatthe variations in the optimal v(x) caused by different val-ues of m are automatically compensated by a(x) (althoughthe a field is never computed explicitly). This meansthat one may use a fixed value for m to recover the correctphase for practically any fringe pattern; we have foundthat it is best to use a high value of m (say m 5 50, withthe observations mapped to the interval @21, 1# as ex-plained in Section 4) to get a smooth v field, particularlyin high-noise situations. However, to get a better conver-gence behavior—particularly for closed fringes—it is bet-ter to start with a lower value (say m 5 5), which may beincreased once the system has converged to the correctglobal attractor (see Section 4).The local bandwidth of the filter, which is controlled by

the parameter l, may be made arbitrarily narrow (e.g., bysetting l 5 100) to get good noise rejection and to controlthe interference from unwanted components, since theadaptive nature of this filter allows it to have a large glo-bal bandwidth even in this case (again, a time-varying lmay be needed to get faster convergence rates for theminimization algorithm as explained in Section 4).Unlike the fixed-frequency case derived in Ref. 15, the

minimization of function (9) or (10) with respect to thevariables $f(x), v(x), x P S% gives rise to a nonlinearproblem, since the equations that define the gradient ofU with respect to these variables are not linear any more(explicit formulas are given in Appendix A). Further, no-tice that if g(x) has a strong component of the formcos(v0 • x 1 a), the values v(x) 5 v0 and f(x) 5 exp(iv0• x 1 a) will correspond to the optimal solution; note,however, that this solution is not unique: For example,v(x) 5 2v0 and f (x) 5 exp(2iv0 • x 2 a) will also be a

1746 J. Opt. Soc. Am. A/Vol. 14, No. 8 /August 1997 Marroquin et al.

stable solution whose phase has the opposite sign. Thismeans that it will be possible to determine the optimal lo-cal phase only up to a global sign change but also that aphase-reconstructing algorithm based on this scheme willbe capable, in principle, of selecting the appropriate signof the local frequency, so that the reconstructed phase hasthe required global smoothness properties and so no spu-rious discontinuities are introduced; we will return to thispoint in Subsection 3.B.The AQF defined by the minimization of Eq. (9) or (10)

is capable of analyzing a signal into its quasi-monochromatic components, even when the correspond-ing spectra have a significant overlap. This is illustratedin Fig. 3: the one-dimensional signal

g~x ! 5 cos@~a1 1 bx !x# 1 cos@~a2 1 bx !x#,

with a1 5 0.2p, a2 5 0.3p, b 5 0.03p, and xP @0, 100#, is shown in plot (a). Plot (b) shows the spec-tra of the two components that were extracted by the AQF[implemented by using the Newtonian descent algorithmto minimize function (9); see Section 4] when the initialstate of the algorithm is set to v(x) [ 0.2p and v(x)[ 0.6p, respectively [i.e., these components correspondto two local minima of Eq. (9)]. Plot (c) shows the firstrecovered component, and plot (d) shows the correspond-ing recovered phase; the results for the other componentare similar. Note that the correct magnitude and phasemay be obtained for each component all through the pat-tern, provided that one knows a priori starting frequen-cies (i.e., 0.2p and 0.6p) where the two components do notoverlap.

3. PHASE RECOVERY FROM SINGLEFRINGE PATTERN IMAGESWe will now present an application of the AQF’s thatwere derived above to an important problem in optics:the recovery of local phase from single fringe pattern im-ages, such as the ones that are obtained in many experi-ments in optical interferometry.17 In most cases the in-tensity of a fringe pattern image is modeled as

g~x ! 5 a~x ! 1 b~x !cos@f~x !# 1 n~x !, (12)

where x is the two-dimensional spatial variable, a(x) is asmooth function that models the variations in irradiance,b(x) models the amplitude modulation and the multipli-cative degradations of the imaging process, n(x) modelsthe measurement noise, and f(x) denotes the phase fieldto be recovered.Depending on whether the field f(x) has isolated criti-

cal points (maxima or minima) within the region of inter-est or not, the fringe patterns may be of two types:

1. Open fringe (spatial-carrier) images [see Fig. 4(a)],in which case f may be written in the form

f~x ! 5 v0 • x 1 a~x !, (13)

where ua(x)u is relatively small.2. Closed fringe (carrier-free) images [see Fig. 6(a) be-

low].

We will now analyze the use of AQF’s for phase recoveryin each of these cases.

A. Open Fringe PatternsIf ua(x)u is small with respect to uv0 • xu and the signal-to-noise ratio is high, ordinary linear QF’s (e.g., Gabor fil-ters) are known to produce good phase recovery, particu-larly away from the boundaries of the fringe pattern.10

They will fail, however, if the global variation of a(x) isrelatively large and if there is a significant amount ofwideband noise present. This is illustrated in the ex-ample of Fig. 4: The fringe pattern of part (a) is a 1283 128 image given by g(x, y) 5 cos@0.25px 1 f(x, y)#1 n(x, y), where the modulating phase f(x, y)5 0.125@(64 2 x)2 1 (64 2 y)2# is shown wrapped inpart (b) and where n(x, y) represents independent obser-vation noise uniformly distributed in the interval@22.85, 2.85#. If one tries to use a linear QF to recoverthe phase and one makes the bandwidth of the filter smallin order to have good noise rejection [e.g., setting l5 100 and keeping v(x, y) [ 0.25p fixed], the recoveredphase will become unreliable as one moves from the re-

Fig. 4. (a) Noisy fringe pattern obtained by modulating a verti-cal grating with the quadratic phase shown (wrapped) in (b) andwith additive uniform observation noise; (c) real part of the out-put of a narrow-band QF applied to (a); (d) reconstructedwrapped phase; (e) and (f ) same as (c) and (d), respectively, for abroadband QF.

Marroquin et al. Vol. 14, No. 8 /August 1997 /J. Opt. Soc. Am. A 1747

gion where the tuning frequency of the filter dominates.This is demonstrated in parts (c) and (d) of Fig. 4, whichshow the real part of the linear filter output (tuned at thedominant frequency at the center of the pattern) and therecovered phase, respectively, of the noisy image of part(a). If, on the other hand, a broadband linear QF is used(e.g., setting l 5 5), the noise will degrade the responseover the whole pattern [see parts (e) and (f) of Fig. 4].With an adaptive filter, one finds a way out of this

dilemma: One may simultaneously have a local narrowband (e.g., with l 5 500), so that the noise is appropri-ately smoothed out, and a global broadband, so that themagnitude of the response—and hence the reliability ofthe phase estimate—remains high in spite of the largespatial variations of a(x). The behavior of the AQF—obtained by minimizing function (9) with use of the New-tonian descent algorithm described in Section 4—appliedto the pattern of Fig. 4(a) is shown in Fig. 5; note that thephase reconstruction remains accurate even close to theborders of the image.

B. Closed Fringe PatternsIn this case the spectrum f(x) is such that one has, infact, frequency components in all directions, and thereforea linear QF cannot be used to recover it. Instead, onemay use a filter whose passband is as wide as possible,but that still eliminates negative frequencies. One thatis widely used10,11 is the Hilbert transform filter (HTF),whose frequency response in one dimension has the form

H~v! 5 H 1 for v > 0

0 otherwise.

When applied to a real signal, this filter produces acomplex output, whose real part is equal to the input sig-nal and whose imaginary part equals its Hilberttransform.18 It may be easily generalized to two dimen-sions by defining a frequency response that vanishes inany half-plane through the origin of the two-dimensionalfrequency domain (for definiteness we assume that it van-ishes in the half-plane v1 , 0). If the input signal is ofthe form

g~x ! 5 b~x !cos@f~x !#

5b~x !

2 $exp@if~x !# 1 exp@2if~x !#% (14)

[we assume that a(x) in Eq. (12) has been eliminated by ahigh-pass filtering operation], the HTF will eliminate oneof the two terms on the right-hand side of Eq. (14), so thatits output will be of the form M(x)exp@if (x)#, from whichf(x) may be recovered by using Eq. (1), provided that thesignal-to-noise ratio is high enough. There is one diffi-culty, however: The desired filtered signal should be lo-cally quasi-monochromatic, so that its phase f(x) may bewritten in the form (11). As one moves along a closedfringe, the local frequency describes a closed curve aroundthe origin in the frequency plane; this curve may be ex-pressed in parametric form as vc(t), with the parametert P @0,T# and vc(0) 5 vc(T). Since the HTF eliminatesall components with v1 , 0, however, it cannot track thefrequency along this curve; instead, it will follow thecurve

vc~t ! 5 H vc~t ! if vc1~t ! . 0

2vc~t ! otherwise,

producing a spurious phase discontinuity along thosecurves in the spatial domain where

]f~x !/]x1 ' v1~x ! 5 0.

Consequently, the phase recovered by the HTF will ex-hibit abrupt sign changes as one moves across these dis-continuities. This artifact is shown in Fig. 6(c), wherethis reconstruction method is applied to the 64 3 64fringe pattern of Fig. 6(a). The true wrapped phase,which is given by

Fig. 5. (a) Real part of the output of the AQF described in thetext applied to the pattern of Fig. 1(a), (b) reconstructed wrappedphase [compare with Fig. 1(b)].

Fig. 6. (a) Fringe pattern generated by the wrapped phaseshown in (b), (c) phase recovered by the HTF (Fourier method),(d) phase recovered by the AQF with l 5 5 and m 5 50.

1748 J. Opt. Soc. Am. A/Vol. 14, No. 8 /August 1997 Marroquin et al.

f~x, y ! 5 30H expF2~x 2 16!2 1 ~y 2 16!2

409.6 G1 expF2

~x 2 48!2 1 ~y 2 48!2

409.6 G J ,is shown in Fig. 6(b).This effect is well known, and although in many cases

it may not be too difficult to mark these spurious discon-tinuities by hand, and thus to correct the phase estimate,this phenomenon has prevented this technique from be-ing used for the fully automatic phase recovery fromsingle closed fringe pattern images.

Fig. 7. (a) First component of the v field that corresponds to thesolution found by the HTF method for the data of Fig. 4(a); darkpixels indicate negative values, and light pixels indicate positiveones. (b) Same as (a) when the energy function is optimizedwith respect to v while maintaining f 5 fHTF fixed. (c) Same as(a), but for the AQF method.

In the formulation that we have presented, on the otherhand, the proper solution is correctly characterized as theglobal minimizer of the energy function (9) or (10), sinceabrupt changes in the local frequency, such as the onesproduced by the HTF, generate creases in the v field,whose high curvature makes significant contributions tothe energy. The phase recovered by an AQF (imple-mented by using the pyramid scheme described in Sub-section 4.B) applied to the pattern of Fig. 6(a) is shown inFig. 6(d). To examine the behavior of the energy func-tion, one may compute the v field associated with theHTF solution, fHTF , as the gradient of the phase fHTF5 arg( fHTF), taken with respect to the spatial variables[the first component, v1(x), of this field is shown in Fig.7(a); the behavior of the second component is similar].The corresponding energy is UHTF 5 39,714.4, while theenergy of the optimal configuration obtained by the AQFis UAQF 5 3373.7. To make this comparison fairer, onemay compute the optimal (smoothed) v field associatedwith fHTF , minimizing the function (9) only with respectto v (i.e., maintaining f 5 fHTF fixed). The energy in thiscase drops to UHTF* 5 5951.7. The first component ofthe resulting v field and of the optimal v field obtained bythe AQF is shown in Figs. 7(b) and 7(c), respectively.If there is noise present in the image, the difference be-

tween the performances of the two methods is even moredramatic; this noise cannot be eliminated by the HTF be-cause of its broad passband, and it makes the spuriousdiscontinuities harder to track by hand or by methods

Fig. 8. (a) 64 3 64 noisy fringe pattern generated by thewrapped phase shown in (b), which is formed by the sum of fiveGaussian functions with centers at (32, 32), (19, 19), (45, 45),(19, 45), and (45, 19), with weights equal to 20, 20, 220, 220,and 220, respectively, plus additive noise uniformly distributedin the interval @21, 1#; (c) phase recovered by the HTF (Fouriermethod); (d) phase recovered by the AQF with l 5 5 and m5 50.

Marroquin et al. Vol. 14, No. 8 /August 1997 /J. Opt. Soc. Am. A 1749

Fig. 9. (a) Interferogram of an early stage of the manufacturing process of a flat mirror obtained with a Wyko interferometer, (b) phaserecovered by the AQF method, (c) phase recovered by the HTF method.

based in the detection of inconsistencies in the wrappedphase, such as the one proposed in Ref. 19. On the otherhand, since the AQF may have locally a very narrow pass-band, it can effectively eliminate noise and produce anartifact-free phase reconstruction in a fully automaticway. This is illustrated in Fig. 8.

C. Performance with Real ImagesTo assess the performance of this scheme in real applica-tions, we now present two examples of real interferomet-ric fringe pattern images.The first one, shown in Fig. 9(a), is an interferogram

taken at an early stage of the manufacturing process of aflat mirror and was obtained by using a Wyko interferom-eter. Note the spatial variations in illumination and con-trast and the fact that observations are available only in-side a circular pupil. Figure 9(b) shows the (wrapped)phase recovered by an AQF implemented with use of theNewtonian descent algorithm (see Section 4) to minimizefunction (10) (the correction for low tuning frequencies isnecessary in this case). The phase recovered by astraightforward application of the HTF is shown in Fig.9(c); note the spurious discontinuities that are introducedby the HTF and also the distortions near the borders ofthe circular pupil. These adverse effects are avoided inthe AQF case, since one can specify the S set where ob-servations are available to be equal to this pupil (see Sec-tion 2) to get a robust behavior, free from edge artifacts(see Ref. 15).The second example comes from electronic speckle pat-

tern interferometry (ESPI)20 with a system that can dis-play real-time fringes at video rates. This system wasdesigned to study fast transient phenomena, so that theuse of multiphase-stepping techniques is precluded. Theinterferometer was configured to measure out-of-planedisplacement components and was used to measure defor-mations in a 14-cm 3 12-cm 3 0.3-cm clamped alumi-num plate when a perpendicular load was applied on itsposterior surface. The ESPI fringes are shown in Fig.10(a). The real part of the field reconstructed by an AQF[implemented by using the pyramid algorithm of Subsec-tion 4.B to minimize function (10)] is shown in Fig. 10(b);

a high value of l (l 5 100) was used at the finest level toensure good noise rejection. The phase recovered by theAQF and by a HTF applied after low-pass filtering thedata (by using a convolution with a Gaussian kernel thatreduces the bandwidth by a factor of 2) are shown in Figs.10(c) and 10(d), respectively. This example was includedto illustrate the robustness of the AQF method, since itsperformance is not degraded, in spite of the fact that theunderlying assumptions (e.g., additive, white Gaussiannoise) are grossly violated in this case. This robustness,for the particular case of ESPI images, may be increasedby using a more realistic model for the speckle noise inthe conditional distribution; this is a subject of our cur-rent research.

Fig. 10. (a) 256 3 256 image of an ESPI fringe pattern, (b) realpart of the field recovered by the AQF method (see the text), (c)phase recovered by the AQF, (d) phase recovered by the HTFmethod after filtering the fringe pattern with a Gaussian filterthat reduces the bandwidth by a factor of 2 (s 5 256/6 in the fre-quency domain).

1750 J. Opt. Soc. Am. A/Vol. 14, No. 8 /August 1997 Marroquin et al.

4. MINIMIZATION ALGORITHMSA. Gradient-Based MethodsIn this section we present the specific algorithms andimplementation details that were used to produce the ex-perimental results presented above. Since the functionU given by Eq. (9) or (10) is multimodal, one should use,in principle, a global optimization algorithm to minimizeit. In the case of open fringe patterns, however, the mul-timodality of these functions poses, in fact, no problem,since starting points for local minimization algorithmsthat conduct to the desired global minimum are easy togenerate (see below).Among these local minimization schemes, the simplest

one is probably gradient descent,21 but a much faster con-vergence behavior may be obtained by considering thevector variables

u~x ! 5 ~ f ~x !, v~x !! 5 ~w~x !, c~x !, v1~x !, v2~x !!

as the positions of a system of interacting particles in di-mension 4 (one particle for each site x P S), where the in-teraction potentials between particles are given by thecorresponding MRF potentials plus an external potentialgiven by the observation term.22 One may now constructa dynamical system for these positions, considering thatthe net force on the particle at site x is given by the nega-tive gradient of the potential energy U with respect tou(x) (plus a friction term that depends linearly on the ve-locity, which is introduced to avoid oscillatory solutions)and that the system obeys Newton’s second law. Assum-ing unit mass for each particle, the equations of motionare

]2u

]t25 2¹uU~u! 2 2k

]u

]t,

where k is the friction coefficient, t denotes time, and¹uU(u) denotes the gradient of U taken with respect to u(explicit formulas are given in Appendix A).Discretizing the time variable, one gets the following

parallel update equations (see Ref. 22 for details):

u~t11 ! 5 Au~t ! 1 Bu~t21 ! 1 C¹uU(u~t !), (15)

with

A 52

kh 1 1, B 5

kh 2 1kh 1 1

, C 52h2

kh 1 1,

(16)

where h is the time step, which must be chosen smallenough (with respect to the parameter l) to guarantee thestability of the algorithm. Note that if k 5 1/h, the B co-efficient vanishes and the system (15) reduces to the well-known gradient descent algorithm. Smaller values ofk, however, allow the system to move at higher speeds,accelerating significantly the convergence rate. In allthe experiments reported below, we have used h 5 0.5/land k 5 0.005/h.As mentioned above, the selection of an appropriate ini-

tial state [which consists of u (0) and u (21)] is critical forthe convergence of the system to a correct solution. Wehave found consistently good solutions by setting v (0)

5 v (21) [ v0 and f (0) 5 f (21) [ 0, where v0 is a localdominant frequency of the pattern (e.g., the carrier fre-

quency, which is usually known, or an estimate of the lo-cal frequency at the center of the image). In general, ifthe pattern has more than one locally quasi-monochromatic component, setting v (0) 5 v (21) [ v0k ,where v0k is a locally dominant frequency of componentk, and f (0) 5 f (21) [ 0 will cause the system to track thiscomponent over the whole field (cf. Fig. 3).Note that, in particular, if g(x) has a strong component

at zero or a very low frequency [i.e., if the spatial averageof g(x) is different from zero because of the illuminationterm a(x) of Eq. (12)] and function (9) is used to imple-ment the AQF, this component will act as a strong attrac-tor for v, which may then converge to a solution wherev(x) [ 0 and w converges to a smoothed version of g.This problem is avoided if function (10) is used in theimplementation, since in this case the zero-frequency (dc)term is automatically eliminated by the modified dataterm. For this reason it is best, in general, to use func-tion (10) to implement AQF’s, since one gets a more ro-bust performance with only a slight increase in computa-tional complexity [see Eqs. (A1)–(A4) in Appendix A).In all cases it is convenient to map the observations lin-

early to the interval @21, 1# prior to the minimizationstep, i.e., set

g~x !: 52@g~x ! 2 minyPS g~y !#

maxyPS g~ y ! 2 minyPS g~ y !2 1,

so that one may use problem-independent values for theparameters.If the filter is required to have a local narrow passband

(e.g., in high-noise situations), the convergence rate ofthis scheme may be accelerated by starting with a broadpassband (i.e., a small value of l, such as l 5 5) to permita fast adaptation to the coarse frequency variation.Upon convergence of system (15), we may set l to its finaldesired value (e.g., l 5 100) and permit system (15) toconverge again.

B. Closed Fringe Patterns and Pyramid SchemesIn the case of closed fringe patterns, the situation is morecomplicated; the energy landscape given by function (9) or(10) may become nearly flat around some configurationswith phase discontinuities and even exhibit undesired lo-cal minima at these configurations, because smoothingout the creases in the v field that correspond to a discon-tinuous phase requires changing the sign of the frequencyin large regions of the image, and this may be difficult todo by using only local changes in the configuration, asthose produced by algorithm (15). Figure 11 shows onesuch local minimum of function (9) for the data of Fig. 6;the corresponding energy is U local 5 3536.8.Although one may obtain the global minimum that cor-

responds to the desired solution by using, for example,stochastic global minimization algorithms, such as simu-lated annealing,23 these schemes are usually too slow tobe of practical use. Here we use a simple scheme thatpermits the fast computation of the correct solution for arestricted class of fringe patterns.The idea of the method is to apply the local minimiza-

tion algorithm defined by system (15) to a version of thedata that is coarsely subsampled, so that local changes (ofa few pixels) in this coarse grid correspond to large-scale

Marroquin et al. Vol. 14, No. 8 /August 1997 /J. Opt. Soc. Am. A 1751

changes in the original image. Proceeding in this way,one may obtain a coarse approximation to the v field thathas the correct sign. This approximation may then be in-terpolated, rescaled, and propagated to finer grids as theinitial state for system (15), until the final refined solu-tion is obtained.Specifically, this pyramid algorithm24 consists of the

following steps:

• Fine-to-coarse phase:

1: set g0 5 g (the observed image);2: for k 5 1 through kmax do:

2.1: set gk21 equal to a smoothed (low-pass) ver-sion of gk21 ;

2.2: obtain gk by subsampling gk21 every otherrow and every other column (note that if g hasan even number of rows and columns, gk con-tains four times fewer sites than gk21);

• Coarse-to-fine phase:

3: set f kmax(0) [ 0 and vkmax

(0) [ v0 (a locally dominantfrequency of the image gkmax);

4: for k 5 kmax through k 5 0 do:

4.1: set f k(21) 5 f k

(0) and vk(21) 5 vk

(0) ;4.2: iterate system (15) by using gk as observa-

tions, until the optimal solution ( fk* , vk* ) isfound;

4.3: if (k . 0)

4.3.1: obtain f k21(0) (on a finer grid) from f k* by

using bilinear interpolation;4.3.2: obtain vk21

(0) from vk* by using bilinear in-terpolation and rescaling (when movingfrom a coarse to a fine scale, the spatial fre-quencies must be divided by 2).

(f0* , v0* ) is the desired solution.

It is usually convenient to adjust the parameters l andm in the coarse-to-fine phase of the algorithm; we havefound the best results by working with small values (e.g.,l 5 m 5 5) at the coarse levels, so that the system con-verges to the correct global attractor, and then adjustingthem (e.g., setting l 5 100 and m 5 50) at the finest levelto refine the results.Any aliasing introduced by the subsampling process

may mislead the frequency tracking of the AQF and thusseverely affect the results. Therefore it is important touse a low-pass filter that effectively limits the bandwidthof the image by 1/2 before subsampling it. This implies,however, that if the original fringe pattern containsmostly high-frequency fringes, this pyramid scheme willnot work, because these fringes will be eliminated by thesmoothing and subsampling operations. Therefore thisscheme is applicable only to images that contain mostlylow-frequency fringes (e.g., those whose magnitude doesnot exceed 0.25p rad/pixel), such as the ones of Figs. 6, 8,and 10. This kind of image is generated in several inter-esting applications, such as ESPI and optical testing25;moreover, it is common practice to adjust the experimen-tal conditions in interferometric experiments to obtainlow- frequency fringes (e.g., by the use of optical compen-sators), since this is known to improve the accuracy of the

reconstructed phase field. Nevertheless, it is importantto develop fast algorithms that overcome this limitation;this is one of the subjects of our current research.

5. CONCLUSIONSWe have presented a principled approach for the con-struction of QF’s whose tuning frequency adapts to the lo-cal dominant frequency of the input image. While otherapproaches have been proposed for the construction ofthis type of filter (e.g., the well-known phase-lockedloop26), they apply only to the one-dimensional case, andit has not been possible to generalize them to two orhigher dimensions.Our formulation is based on Bayesian estimation with

complex-valued MRF prior models and may be considereda direct extension of the one proposed in Ref. 15. Herethe output of the AQF is characterized as the minimizerof an energy function [Eq. (9) or (10)]; since this functionalso depends on the local frequency field, both fields aresimultaneously computed in the minimization process.The resulting nonlinear AQF shares the same robust-

ness properties of the filters proposed in Ref. 15 (robust-ness with respect to edge effects and missing data, appli-cability to irregularly shaped domains, etc.), but it hassome additional remarkable ones: Since it combines alocal narrow passband with a global wide passband, it caneffectively eliminate noise and at the same time recon-struct the phase of broadband spatial-carrier fringe pat-terns; in addition, since it constrains the local frequencyfield to be smooth, it can track frequency fields thatchange sign within the domain of interest. Thus it canrecover the phase from single closed fringe pattern im-ages in a fully automatic way, without introducing spuri-ous sign changes. Also, since these filters can track downlocally quasi-monochromatic components individually,they permit a signal decomposition—in terms of thesecomponents—that in some cases may be more parsimoni-ous and meaningful than the one provided by classicalFourier analysis (in terms of pure monochromatic compo-nents).

Fig. 11. Phase configuration that corresponds to a local mini-mum of Eq. (9) for the data of Fig. 4.

1752 J. Opt. Soc. Am. A/Vol. 14, No. 8 /August 1997 Marroquin et al.

These properties may make these filters a valuabletool, not only for fringe pattern analysis in optics but alsoin other fields; for example, they may be used for con-structing more robust systems for texture characteriza-tion and segmentation and for the recovery of depth fromstereoscopic pairs of images.The main drawback of this approach (but, in some

sense, also its strength) stems from the multimodal na-ture of the energy function that is minimized. Althoughgood solutions are obtained for open fringe (spatial-carrier) patterns with the algorithms proposed here,these algorithms may be relatively slow, particularly forlarge images: For a 64 3 64 image, the Newtonian de-scent algorithm takes approximately 2 min on a 90-MHzPentium personal computer to minimize function (9), andfor an N 3 N image, this time grows as N2. These algo-rithms, however, are fully parallelizable: We haveimplemented them in a four-processor transputer boardand reduced the convergence time by approximately 70%with respect to that obtained with a single processor.For closed fringe patterns, the situation is more criti-

cal, since relatively fast algorithms are available only forthe restricted class of images that consist mostly of low-frequency fringes, and general global minimization algo-rithms, such as simulated annealing, are usually veryslow. We believe, however, that the main contribution ofthis paper, namely, the characterization of the correct so-lution to the adaptive quadrature filtering problem as theminimizer of a proposed energy function, may serve toguide the development of faster and more robust algo-rithms for performing this task.

APPENDIX AIn this appendix we give explicit formulas for the compu-tation of the gradient of the posterior energy U given byEq. (9) or (10) with respect to the variables f(x) 5 w(x)1 ic(x) and v(x) 5 (v1(x), v2(x))

T. First, we considerEq. (9), and introduce the following notation: S denotesthe set of sites where observations are available and thesolution is to be computed; the components of the spatialvariables x are denoted by (x1 , x2). We define the fol-lowing neighborhoods for the spatial variables:

N~x ! 5 $y P S: ux 2 yu 5 1%,

Nh~x ! 5 $y P S: y1 5 x1 and uy2 2 x2u 5 1%,

Nv~x ! 5 $y P S: y2 5 x2 and uy1 2 x1u 5 1%,

Nhv~x ! 5 $~x1 , x2!,~x1 1 1, x2!,~x1 , x2 1 1 !,

~x1 1 1, x2 1 1 !% ù S;

and we define the following quantities:

axy 5 w~x ! 1 w~y !Gxy 1 c~y !Hxy ,

bxy 5 c~x ! 2 w~y !Hxy 1 c~y !Gxy ,

with

Gxy 5 2sxysyx 2 cxycyx ,

Hxy 5 2cxysyx 1 cyxsxy ,

cxy 5 cosFv~x ! • ~x 2 y !

2 G ,sxy 5 sinFv~x ! • ~x 2 y !

2 G .We also define the vector

Uv~x ! 5 S ]U]v1~x !

,]U

]v2~x ! DT

.

With these definitions we have

]U]w~x !

5 w~x ! 2 2g~x ! 1 l (yPN~x !

axy , (A1)

]U]c~x !

5 c~x ! 1 l (yPN~x !

bxy , (A2)

Uv~x ! 5 l (yPN~x !

@axyc~x ! 2 bxyw~x !#~x 2 y ! 1 md~x !,

where the vector d(x) contains the partial derivatives ofthe thin-plate potentials with respect to the components(v1 , v2) of v:

dk~x ! 5 (yPNh~x !

Vh~vk , y ! 2 2Vh~vk ,x !

1 (yPNv~x !

Vv~vk , y ! 2 2Vv~vk , x !

1 (yPNhv~x !

Vhv~vk , y !

for k 5 1, 2, where

Vh~vk , y !

5 H vk~ y1 , y2 2 1 ! 2 2vk~ y1 , y2! 1 vk~ y1 , y2 1 1 !

if Nh~y ! # S

0 otherwise,

Vv~vk , y !

5 H vk~ y1 2 1, y2! 2 2vk~ y1 , y2! 1 vk~ y1 1 1, y2!

if Nv~ y ! # S

0 otherwise,

Vhv~vk , y !

5 H vk~ y1 , y2! 2 vk~ y1 1 1, y2! 2 vk~ y1 , y2 1 1 !

1 vk~y1 1 1, y2 1 1 ! if Nhv~y ! # S

0 otherwise.

In the case of function (10), the formulas are

]U]w~x !

5 (yPN~x !

$w~x ! 2 w~y ! 2 2@g~x ! 2 g~y !#%

1 l (yPN~x !

axy , (A3)

Marroquin et al. Vol. 14, No. 8 /August 1997 /J. Opt. Soc. Am. A 1753

]U]c~x !

5 (yPN~x !

@c~x ! 2 c~y !# 1 l (yPN~x !

bxy .

(A4)

The expression for Uv(x) remains unchanged.

ACKNOWLEDGMENTSThe authors thank the anonymous reviewers for theircomments, which helped to improve the quality of the pre-sentation. The authors were supported in part by grantsfrom the Consejo Nacional de Ciencia y Tecnologıa,Mexico.

Address all correspondence to J. L. Marroquin at the lo-cation on the title page or at the e-mail addressjlm@fractal.cimat.mx.

REFERENCES1. D. Gabor, ‘‘Theory of communication,’’ J. Inst. Electr. Eng.

(London) 93, 429–457 (1946).2. M. Turner, ‘‘Texture discrimination by Gabor functions,’’

Biol. Cybern. 55, 71–82 (1986).3. J. Beck, A. Sutter, and R. Ivry, ‘‘Spatial frequency channels

and perceptual grouping in texture segregation,’’ Comput.Vis. Graph. Image Process. 37, 299–325 (1987).

4. J. Bergen and E. Adelson, ‘‘Early vision and texture percep-tion,’’ Nature (London) 333, 363–364 (1988).

5. I. Fogel and D. Sagi, ‘‘Gabor filters for texture discrimina-tion,’’ Biol. Cybern. 61, 103–113 (1989).

6. T. Sanger, ‘‘Stereo disparity computation using Gabor fil-ters,’’ Biol. Cybern. 59, 405–418 (1988).

7. E. H. Adelson and J. R. Bergen, ‘‘Spatiotemporal energymodels for the perception of motion,’’ J. Opt. Soc. Am. A 2,284–299 (1985).

8. D. J. Heeger, ‘‘A model for the extraction of image flow,’’ J.Opt. Soc. Am. A 4, 1455–1471 (1987).

9. K. Creath, ‘‘Phase-measurement interferometry tech-niques,’’ in Progress in Optics, E. Wolf, ed. (North-Holland,Amsterdam, 1988), Vol. 26, pp. 349–393.

10. M. Takeda, H. Ina, and S. Kobayashi, ‘‘Fourier transformmethods of fringe pattern analysis for computer-based

topography and interferometry,’’ J. Opt. Soc. Am. 72, 156–160 (1982).

11. Th. Kreis, ‘‘Digital holographic interference phase measure-ment using the Fourier transform method,’’ J. Opt. Soc.Am. A 3, 847–855 (1986).

12. D. J. Fleet, A. D. Jepson, and M. R. M. Jenkin, ‘‘Phase-based disparity measurement,’’ Comput. Vis. Graph. ImageProcess. 53, 198–210 (1991).

13. S. Geman and D. Geman, ‘‘Stochastic relaxation, Gibbs dis-tributions and the Bayesian restoration of images,’’ IEEETrans. Pattern. Anal. Mach. Intell. PAMI-6, 721–741(1984).

14. J. Marroquin, S. Mitter, and T. Poggio, ‘‘Probabilistic solu-tion of ill-posed problems in computational vision,’’ J. Am.Stat. Assoc. 82, 76–89 (1987).

15. J. L. Marroquin, M. Servin, and J. E. Figueroa, ‘‘Robustquadrature filters,’’ J. Opt. Soc. Am. A 14, 779–791 (1997).

16. W. E. L. Grimson, ‘‘A computational theory of visual surfaceinterpolation,’’ Philos. Trans. R. Soc. London, Ser. B 298,395–427 (1982).

17. D. W. Robinson and G. T. Reid, eds., Interferogram Analy-sis: Digital Fringe Pattern Measurement Techniques (In-stitute of Physics, Bristol, UK, 1993).

18. R. N. Bracewell, The Fourier Transform and Its Applica-tions (McGraw-Hill, New York, 1978).

19. A. Spik and D. W. Robinson, ‘‘Investigation of the cellularautomata method for phase unwrapping and its implemen-tation on an array processor,’’ Optics Lasers Eng. 14, 25–37(1991).

20. R. Jones and C. Wykes, Holographic and Speckle Interfer-ometry (Cambridge U. Press, Cambridge, 1989).

21. J. M. Ortega and W. C. Rheinholdt, Iterative Solution ofNonlinear Equations in Several Variables (Academic, NewYork, 1970).

22. J. L. Marroquin, ‘‘Deterministic interactive particle modelsfor image processing and computer graphics,’’ Comput. Vis.Graph. Image Process. 55, 408–417 (1993).

23. S. Kirkpatrick, C. D. Gelatt, and M. P. Vecchi, ‘‘Optimiza-tion by simulated annealing,’’ Science 220, 671–680 (1983).

24. P. J. Burt, ‘‘The pyramid as a structure for efficient compu-tation,’’ in Multiresolution Image Processing and Analysis,A. Rosenfeld, ed., Vol. 12 of Springer Series in InformationSciences (Springer, New York, 1984).

25. D. Malacara, ed., Optical Shop Testing (Wiley, New York,1992).

26. C. S. Shanmugan, Digital and Analog Communication Sys-tems (Wiley, New York, 1979).