www.elsevier.com/locate/patrec

Pattern Recognition Letters 27 (2006) 715–725

Consistent multi-modal non-rigid registration basedon a variational approach

Zhijun Zhang *, Yifeng Jiang, Hungtat Tsui

Electronic Engineering Department, Chinese University of Hong Kong, Shatin, New Territories, Hong Kong, China

Received 23 March 2005; received in revised form 12 September 2005Available online 20 December 2005

Communicated by Dr. C.-F. Westin

Abstract

In this paper, a novel variational approach for multi-modal image registration based on consistent non-rigid transforms is proposed.The forward and backward transforms are computed in a variational framework simultaneously. A consistency energy is added into thevariational registration framework and an iterative method assures that the forward and backward transforms are close approximateinverses of each other. This new scheme can provide more constraints on the forward and backward transforms and make themsmoother. This reduces the chance of being stuck into a local minimum. We can incorporate the consistent transforms together withseveral kinds of information based similarity metrics such as mutual information and correlation ratio into the registration framework.An implicit finite difference scheme is used to solve the Euler–Lagrange equation numerically. Our proposed method is evaluated byusing simulated multi-modal magnetic resonance images (MRI) and real medical images. The results of our proposed consistent regis-tration and those using only forward or backward registration are compared. Improvements at the edge area can be easily seen from theregistration results.� 2005 Elsevier B.V. All rights reserved.

Keywords: Multi-modal image registration; Consistent registration; B-spline based deformation; Implicit finite difference method

1. Introduction

Image registration is the process of finding an optimaltransformation from one image to the other. This tech-nique has wide applications in the areas of medical imagefusion, atlas matching, brain mapping, pathological detec-tion, intra-operative image guided surgery and other com-puter vision applications.

Recently, a large amount of medical images are gener-ated to examine the human anatomy from an anatomicalor functional viewpoints. Due to different imaging princi-ple, we often need to fuse the information in images of dif-ferent image modalities. Since many organs and tissues are

0167-8655/$ - see front matter � 2005 Elsevier B.V. All rights reserved.

doi:10.1016/j.patrec.2005.10.018

* Corresponding author. Tel.: +852 26098251; fax: +852 26035558.E-mail addresses: zjzhang@ee.cuhk.edu.hk (Z. Zhang), yfjiang@ee.

cuhk.edu.hk (Y. Jiang), httsui@ee.cuhk.edu.hk (H. Tsui).

deformable, non-rigid registration is an important issue.Many methods on mono-modal registration (Bajcsy andKovacic, 1989; Thirion, 1998; Christensen et al., 1997;Cachier, 1999; Collins et al., 1994; Ashburner, 1997;Woods et al., 1998) had been proposed in the past. Bajcsyand Kovacic (1989) proposed the elastic model based non-rigid registration, where a Navier–Stoke equation is used todescribe the local deformation. In (Thirion’s, 1998) work,he proposed a method called demon’s algorithm. Cachierand Ray (2000) proved it is similar to the gradient descentmethod and improved it. Ferrant et al. (2001) used a finiteelement model to solve the elastic deformation. Christensenet al. (1997) used the velocity field instead of the displace-ment field in the viscous fluid dynamic model to registernon-rigid images. Collins et al. (1994) used a registrationmethod which maximizes the cross correlation of intensityand edge features to perform an image segmentation

716 Z. Zhang et al. / Pattern Recognition Letters 27 (2006) 715–725

guided by atlas. Ashburner (1997) used a 3D trigonometricbasis to represent the deformation and used a probabilisticframework to maximize the posterior probability. Woodset al. (1998) used a polynomial based image registrationmethod to warp an image. In all the above methods, eachcost function was required to be a sum of squared differ-ence (SSD) or a cross correlation (CC). However, thesecost functions will fail as the features are intensities inthe multi-modal registration where the warped sourceimage and the target image are represented differently.

In the past, information theory based similarities such asmutual information (MI), correlation ratio (CR) and nor-malized mutual information (NMI) have been proposed.They work well as similarity metrics for multi-modal rigidregistration (Wells et al., 1996; Maes et al., 1997; Rocheet al., 2001; Studholme et al., 1997). In this kind of similar-ity measurements, knowledge of the intensity relationbetween two images is not necessary. Later, multi-modalnon-rigid registration methods (Gaens et al., 1998; Gui-mond et al., 2001; Hermosillo et al., 2002; Rueckertet al., 1999; Pluim et al., 2004; Rougon et al., 2003) wereproposed. Hermosillo et al. (2002), Pluim et al. (2004)and Rougon et al. (2003) used a variational method whichfinds a dense displacement field by minimizing the multi-modal matching energy function. This method uses themost recent techniques of functional analysis and thetheory of partial differential equation in image processing.The similarity metric functions are evaluated in a con-tinuous form by using a Parzen window technique. Theproblem of image matching is formulated into an optimiza-tion of a continuous energy functional. Intuition whichhelps proposing new models can be obtained by borrowingideas from the well-established field of applied mathe-matics. For non-rigid registration, several researchers(Alvarez et al., 2002; Cachier, 1999; Johnson and Christen-sen, 2002; Christensen and Johnson, 2001) have proposedenergy functions which enforce consistent transformations.This will also make the transforms smoother. The mainconcept of consistent registration is to consider the forwardand backward transformations simultaneously, and let theforward and backward transform functions closely approxi-mate the inverse of each other. Cachier and Ray (2000)optimized the registration energy which is a function ofthe forward transform and its inverse function assumingthat the inverse of transform function exists. Christensenand Johnson (2001) and Alvarez et al. (2002) consideredan energy function of the forward and backward transfor-mations and let them to be the inverse of each other asymp-totically. However, neither of these methods can be used toregister multi-modal images since the similarity measure-ment they used is the simplest SSD similarity metric suit-able for mono-modal image registration only.

In our former work (Zhang et al., 2004), we proposed aconsistent variational approach for multi-modal registra-tion. By explicitly representing the forward and backwarddisplacement field to be the inverse of each other, the opti-mization will try to make their sum close to zero every-

where. However, this assumption will not be reasonablein those cases where the displacement field is not small.In this paper, we propose a more reasonable assumptionthat the forward and backward transformation functionshould be inverses of each other. In the former work, weused an explicit finite difference method to solve theEuler–Lagrange equation. However, such explicit methodis usually unstable and is also easy to converge to a localminimum. In this paper we use an implicit finite differencemethod which is stable and efficient in solving the Euler–Lagrange equation.

2. Methods

2.1. Variational methods for image matching

We first introduce a variational framework of non-rigidregistration based on a multi-modal image similaritymetric. Denote the Gaussian smoothed source image andtarget image as two functions Ir

s : X � Rn ! ½0 Imax�,n = 2, 3, and Ir

t : X � Rn ! ½0 Imax�, n = 2, 3. The imageshere are smoothed with a Gaussian kernel with a varianceof r2 and are in the space of C1(X). Then all of its deriv-atives are also bounded in the domain X. The objectiveof the registration problem is to find an optimal displace-ment field function h(x) : X! Rn with which the warpedsource image Ir

s ðId þ hÞ will be most similar to the targetimage Ir

t .From a mathematical view the registration problem

can be described as a process of searching for a functionin the admissible functional space which minimizes theenergy functional: hopt = arg minh2H1(X)E(h), where hopt isthe optimal displacement field, and H1(X) is the admis-sible displacement field space which is a Hilbert spacewith the Hilbert product ðh; gÞH1ðXÞ ¼

RX hðxÞTgðxÞdxþR

X traceðDh ðxÞTDgðxÞÞdx, with D(Æ) the Jacobian operatorof a vector function at x. The registration energy functionalis defined by the following equation:

EðhÞ ¼ SðIrt ; I

rs ðId þ hÞÞ þ k

ZuðDhðxÞÞdx. ð1Þ

The energy functional usually consists of two parts. Onepart is the matching energy SðIr

t ; Irs ðId þ hÞÞ which mea-

sures the similarity between the deformed source imageand the target image under the current transformation T,with T(x) = h(x) + Id(x). The other is the regularizationenergy whose purpose is to make the transformation assmooth as possible to avoid resulting in a physicallyimplausible transformation, u is a convex function withD(h(x)) as variable. The two energies are weighted by usinga constant k.

For mono-modal image registration, the similarity mea-sure can be SSD or CC. For multi-modal image registra-tion, information theory based similarity measurementsuch as MI, CR, NMI or a generalized information metric(Pluim et al., 2004; Rougon et al., 2003) can be used into

Z. Zhang et al. / Pattern Recognition Letters 27 (2006) 715–725 717

this variational framework. Since MI, CR and NMIsimilarity metrics are maximized when two images areregistered, we should use negative similarity metrics asenergies. The images are considered as an independent ran-dom process (Hermosillo et al., 2002) and the similaritybetween two images are measured by the statistical depen-dence. Each of these similarity metrics is a function of thejoint probability density. The joint probability and themarginal probability density functions are evaluated byusing a Parzen window technique which turn them intocontinuous and differentiable. For example, a negativemutual information similarity SMI can be represented by

SMIðIrt ; I

rs ðId þ hÞÞ ¼ �

ZX

P hðit; isÞ logP hðit; isÞ

ptðitÞpshðisÞ

dis dit

ð2Þwith P hðit; isÞ ¼ 1

jXjR

X GbðIrhðxÞ � iÞdx as the joint proba-

bility density, where i ¼ ðit; isÞT; IhðxÞ ¼ ðIrt ; I

rs ðxþ IdÞÞT;

Gb ¼ 1ð2p det bÞ exp � 1

2iTb�1i

� �with b ¼ bt 0

0 bs

� �, jXj the

number of pixels (or voxels) in the image domain, andps

hðisÞ ¼ 1jXjR

X gbsðIsðId þ xÞ � isÞdx the marginal prob-

ability density of the deformed source image, withgbs¼ 1ffiffiffiffiffiffiffi

2pbs

p expð� i2

2bsÞ, and pt

hðitÞ ¼ 1jXjR

X gbtðI tðxÞ � itÞdx

the marginal probability density of the target image, with

gbt¼ 1ffiffiffiffiffiffi

2pbt

p expð� i2

2btÞ. Other variational forms of multi-

modal similarity metrics can be obtained from Hermosilloet al. (2002), Petijean et al. (2002) and Rougon et al. (2003).

The necessary condition for the existence of an optimumof the function is that the Gateaux variation of the func-tional should be zero. There is a well-established resultfor this kind of problem if the given problem is a well-posed one. The solution of this minimum problem isusually called an Euler–Lagrange equation, which can bewritten in this formula:

rSðIrt ; I

rs ðId þ hÞÞ � kdivðuDhðDhðxÞÞÞ ¼ 0 ð3Þ

with rSðIrt ; I

rs ðId þ hÞÞ as the Gateaux derivative of the

multi-modal similarity metric functions. It has a generalform:

rSðIrt ; I

rs ðT hðxÞÞÞ ¼ �ðGb � LhÞðIr

t ðxÞ; Irs ðxþ hðxÞÞÞ

� rIrs ðxþ hðxÞÞ; ð4Þ

where Lh is a function differs with different similarity met-rics. We have for mutual information metric:

LMIh ðit; isÞ ¼

1

jXjo2P hðit; isÞP hðit; isÞ

� ps0h ðisÞ

pshðisÞ

� �. ð5Þ

Details of other similarity metrics can be referred to Her-mosillo et al. (2002) and Petijean et al. (2002). Eq. (3) isa nonlinear partial differential equation (PDE) and a directsolution is usually impossible. Usually we add an artificialtime t and let the time derivative oh

ot equal to the negativeGeateaux derivative of the registration energy functional.Then the registration energy will be minimized so thatlimt�>þ1

ohot ¼ 0, and Eq. (3) is solved.

By rewriting Eq. (3) into a time-dependent PDE with aNeumann boundary condition, we have

oh

ot¼ �rSðIr

t ; Irs ðId þ hÞÞ þ kdivðuDhðDhÞÞ;

hð0; �Þ ¼ h0ð�Þ;

oh

o~n¼ 0.

8>>>>>><>>>>>>:

ð6Þ

This equation is a parabolic partial differential equation(PDE). It can be solved by using an explicit or implicitfinite difference methods. The implicit finite differencemethod works better since it is numerically stable (Thomas,1995).

2.2. Regularization

The non-rigid registration is an ill-posed problem. Acommon method to solve this problem is to add some priorknowledge of the displacement field. A very intuitive wayto add some constraints is the regularization method. Itassumes a physical plausible deformation should be regularand smooth. Usually we add a term which will make thetransform as smooth as possible. There are three mostuseful regularization methods. The easiest way is to add aTikhonov regularization term u : X 7! 1

2TrðXTXÞ. Since

we have div(uDu(Du)) = div(Du) = Du, this is a diffusionoperator which makes the displacement field smooth.

The second kind of regularization energy is the Nagel-Enkelmann energy uðDhÞ ¼ 1

2TrðDhT Ir

1DhTÞ. This operator

can preserve the discontinuity of the displacement field bysmoothing along an edge of the image but not across theedge.

The third energy is an elastic energy uðDhÞ ¼12ðnTrðDhTDhÞ þ ð1� nÞTrðDh2ÞÞ, with n ð1

2< n 6 1Þ. The

value of parameter n can be chosen to model differentelastic material.

2.3. Consistent transformation

In general, the transform obtained by using forward reg-istration is not equal to the inverse of the transformobtained by using backward registration. This is due tothe correspondence ambiguity of the voxel based similaritymetric and the inconsistency of the forward and backwardregularization (Christensen and Johnson, 2001; Cachierand Ray, 2000).

The consistent or symmetrical transformation basednon-rigid registration has been proposed by Christensenet al. (1997) and Alvarez et al. (2002). In their methodthe two transforms in the registration energy are asymptot-ically symmetrical or consistent. Here we use a similarmethod to define simultaneously forward and backwarddisplacement field functions. We use a 2D case as an exam-ple. We denote the forward and the backward displacementfield function as h(x) = (u1(x), v1(x))T and g(x) = (u2(x),v2(x))T.

718 Z. Zhang et al. / Pattern Recognition Letters 27 (2006) 715–725

The forward transformation and the backward transfor-mation will be represented by two functions Th(x) = x +h(x) and Tg(x) = x + g(x) respectively. Ideally these twofunctions should be the inverse of each other, i.e.Tg � Th(x) = Id(x) and Th � Tg(x) = Id(x). Then we willadd the consistency energies which will punish a pair oftransformation whose composite function Tg � Th orTh � Tg has large discrepancy with the identical transfor-mation Id(x). The consistency energies will be describedby

RðT g � T h � IdðxÞÞ2 dx and

RðT h � T g � IdðxÞÞ2 dx.

Assume the displacement field as h(x). Then each of theforward and backward registration energies will be madeup of three parts, the similarity metric, the regularizationenergy and the consistency energy. Given the backwarddisplacement field g(x), the forward registration energy willbe described as

Ef ¼ SðIrt ; I

rs ðT hÞÞ þ k1

ZX

uðDðhðxÞÞÞdx

þ k2

ZXðT g � T h � IdðxÞÞ2 dx; ð7Þ

where S(Is, It(Th)) is the multi-modal image similarity met-ric, k1 and k2 are the weights of the energies to balance theeffect of these three energies. It can be determined automat-ically by adjusting the three energies so that they have thesame range of value. This will be introduced in the follow-ing section.

Similarly, given the forward displacement field h(x), thebackward registration energy will be described as the sumof three energies:

Eb ¼ SðIrs ; I

rt ðT gÞÞ þ k1

ZX

uðDðgðxÞÞÞdx

þ k2

ZXðT h � T g � IdðxÞÞ2 dx. ð8Þ

Since the regularization energy and the consistency energyof the backward registration are very close to those in theforward registration, so here the weights are still k1 and k2.

Due to the asymmetric characteristics of the forwardand the backward registration,we need to combine the reg-istration energies together, as below

Eðh; gÞ ¼ Ef þ Eb. ð9ÞThen the Euler–Lagrange equation of this variationalframework should be

oEoh¼ rSðIr

t ; Irs ðT hÞÞ � k1 divðuDhDhÞ

þ k2rT g � T h � ðT g � T h � IdÞ ¼ 0; ð10ÞoEog¼ rSðIr

s ; Irt ðT gÞÞ � k1 divðuDgDgÞ

þ k2rT h � T g � ðT h � T g � IdÞ ¼ 0 ð11Þ

with

rSðIrt ; I

rs ðT hðxÞÞÞ

¼ ðGb � LhÞðIrt ðxÞ; Ir

s ðxþ hðxÞÞÞrIrs ðxþ hðxÞÞ; ð12Þ

rSðIrs ; I

rt ðT gðxÞÞÞ

¼ ðGb � LgÞðIrs ðxÞ; Ir

t ðxþ gðxÞÞÞrIrt ðxþ gðxÞÞ. ð13Þ

Similar to Eq. (6), we formulate the equation into a time-dependent problem with initial displacement fields andNeumann boundary conditions. The steady-state solutionswill be those for Eqs. (10) and (11):

oh

ot¼ �rSðIr

t ; Irs ðT hÞÞ þ k1 divðuDhDhÞ

�k2rT g � T h � ðT g � T h � IdÞ;og

ot¼ �rSðIr

s ; Irt ðT gÞÞ þ k1 divðuDgDgÞ

�k2rT h � T g � ðT h � T g � IdÞ;

hð0; �Þ ¼ h0ð�Þ;

gð0; �Þ ¼ g0ð�Þ;oh

o~n¼ 0;

og

o~n¼ 0.

8>>>>>>>>>>>>>>>>>>>>>>>><>>>>>>>>>>>>>>>>>>>>>>>>:

ð14Þ

2.4. Implementation

2.4.1. Numerical methods

A summary of the discretization of the parabolicequation is shown in the Appendix. We need to solvetwo parabolic equations of h and g respectively. Wewill have f hðxÞ ¼ ðfh;1; . . . ; fh;nÞT ¼ �rSðIr

t ; Irs ðT hÞÞ�

k2rT g � T h � ðT g � T h � IdÞ and f gðxÞ ¼ ðfg;1; . . . ; fg;nÞT ¼�rSðIr

s ; Irt ðT gÞÞ � k2rT h � T g � ðT h � T g � IdÞ. Then we

write them from lexicographical form into a vector

form with ~HðkÞ

:¼ ðfh;1ð~X ÞT; . . . ; fh;1ð~X ÞTÞT and ~GðkÞ

:¼ðfg;1ð~X ÞT; . . . ; fg;1ð~X ÞTÞT, for discretization of the regulari-zation operators, the matrix A (see Appendix) obtainedare same as that in the forward or backward registration.The discrete linear iterative form of Eq. (14) will be

~Hðkþ1Þ ¼ ð1� sk1AÞ�1ð~H ðkÞ þ s~F

ðkÞh Þ;

~Gðkþ1Þ ¼ ð1� sk1AÞ�1ð~GðkÞ þ s~F

ðkÞg Þ.

8<: ð15Þ

2.4.2. Multi-resolution

Non-rigid registration usually takes much longer timethan rigid or affine registration since the displacement vec-tor at each of the pixel has to be evaluated. One criticalproblem in implementation is the speed of the algorithm.A multi-resolution strategy is usually used to increase thespeed of the algorithm. The registration is first donebetween the down-sampled source and target images byusing a Gaussian filter. Then the displacement fieldobtained at the lower resolution level can be used as the ini-tial displacement field at the higher resolution. Multi-resolution scheme can reduce the chance of trapping intoa local minimum by using the down-sampled images.

Fig. 1. The simulated images we used for the registration. In the first row from left to right are the original MR-T1, MR-T2, and MR-PD images, in thesecond row from left to right are the forward and backward displacement field, and the deformed MR-T1 image.

Z. Zhang et al. / Pattern Recognition Letters 27 (2006) 715–725 719

2.4.3. Parameters

Changes of the parameters will affect the registrationresult. We use the same parameters as in (Hermosilloet al., 2002) did in each of the forward and backward reg-istration energies. We need to define the value of k2 here.The value will be determined by letting the amplitude ofthe consistency energy and the similarity energy to be atthe same order of magnitude. In our experiment, we choosek2 = 5 · 10�4. Another parameter we used here is the timestep s. As stated in Eq. (15), we need to let the matrixI � sk1A to be non-singular. We make the matrix I � sk1A

to be diagonal dominant. Assume a = max(jAijj), in ourexperiments we set ska 6 0.01 and it works well for allour experiments.

3. Experiment and results

3.1. Simulated data evaluation

We use simulated data to evaluate our method first. Theimages we use are obtained from Brainweb.1 We deform anMR-T1 image by using a known grid transform. The gridtransform we use is a uniform periodic one in order to com-pare the recovery over the area of the image. The transfor-mation we adopt is u(x, y) = M cos(x Æ p/T), v(x, y) =

1 http://www.bic.mni.mcgill.ca/brainweb/.

Mcos(y Æ p/T), with M = 5 and T = 32. The source image,target image and the ground truth fields (both forward andbackward) are in size of 217 · 181. They are shown inFig. 1 together with the MR-PD image.

Then the deformed MR-T1 image is used as the targetimage and the MR-T2 image is used as the source image.We use non-rigid registration to recover the displacementfield and we compare it with the ground truth displacementfield. We use the sum of squared difference of the recovereddisplacement field with the ground truth displacement fieldto evaluate the accuracy of the registration.

The consistency of the forward and backward transfor-mation is evaluated by using the forward consistency errorfunction Tg � Th � Id and the backward consistency errorfunction Th � Tg � Id. We evaluate these two functions inthe grid deformation form, so when the composite trans-form of Th � Tg is close to an identical transform, the gridshould be a uniform one. The results are shown in Fig. 2.We can see in most of the image area, the composite trans-formation is not an identical transform. In most areas thegrid is deformed and the grid lines which should be straightturn into curves, see Fig. 2(a) and (b). This indicates thatthe forward transform and the backward transform arenot inverse of each other. But in the result of the consistentregistration, since the consistency energy will be minimizedwith iterations, this makes the forward and backwardtransformations are asymptotically inverse of each other,

Fig. 2. The registration results of the MR-T2 image with deformed MR-T1 image by using the implicit finite difference method. (a) and (b) are the forwardand backward consistent displacement error by using separate forward and backward registration, (c) and (d) are the forward and backward consistentdisplacement error by using a consistent registration, (e) and (f) are the forward displacement fields computed by using the forward registration andconsistent registration.

720 Z. Zhang et al. / Pattern Recognition Letters 27 (2006) 715–725

see Fig. 2(c) and (d). From these four figures we can seethat the consistency of the forward and backward transfor-mation is much increased in the area of the boundary andedge area. This means the results from the consistent regis-tration is more biologically meaningful. The recovered dis-placement fields by using forward registration and theconsistent registration are shown in Fig. 2(e) and (f), wecan see the result from consistent registration is smootherand there is not so much waves near the boundary oraround the ventricle area.

Then we take another group of experiments on somemore complex and non-periodic deformation fields. Thedeformation fields are generated by using 2D B-splinenon-rigid deformations. We generate a group of 4 displace-ment fields by randomly assigning the value of the B-splinecontrol grids, and deform the MR-T2 image with each ofthese four displacement fields, the acquired fields andimages are shown in Fig. 3, here we show two of them asillustration for save of space. Then we register the MR-PD image (see in Fig. 1) with each of these four imagesby using the forward registration and the consistent regis-tration method. The displacement fields recovered by usingthese two methods are shown in Fig. 4 as comparison.Then we list the forward and backward consistency errorsof separate forward and backward registration with thoseof the consistent registration. The displacement errors with

ground truth and the consistency transformation errors areshown in Table 1. The e0f and e0b are the sum of squared dif-ference of the recovered displacement fields with theground truth displacement fields, e0f and e0b are the corre-sponding ones of the consistent registration. e1, e2 are thesummed squares of x and y components of the forwardconsistent transformation error and e3, e4, are the summedsquares of x and y components of the backward consistenttransformation error. e01; e

02; e03; e04 are those corresponding

terms for consistent registration. From the data in Table1 we can see generally the transforms obtained by usingconsistent registration have smaller errors with the groundtruth than those obtained by using forward or backwardregistration separately. And also the consistent transforma-tion errors are also smaller than those of forward or back-ward registration. Then we compare the intensity errors ofthe registration methods. Assume the MR-PD sourceimage, MR-T2 target image, ground truth transformwarped MR-PD and the original MR-T2 image areIs; I t; I 0t and I 0s respectively, then we will compare the regis-tration methods by using the sum of square differencekI t � I 0sðT hÞk2

2 and kIs � I 0tðT gÞk22. We denote the forward

and backward transformation in the consistent registrationas T cf

h and T cfg , and the transformations in the forward reg-

istration and backward registration as T fh and T f

g , theresults of the intensity errors of the two kinds of transfor-

Fig. 3. From left to right, the three columns are: two of four simulated MR-T2 images, the forward grid transforms and their inverse transforms.

Fig. 4. The displacement fields. From left to right, the first column is the result of the displacement field obtained by using forward registration, the secondcolumn is the forward consistency error, the third column is the displacement field obtained by using the consistent registration, the fourth column is theconsistent registration error obtained by using consistent registration.

Z. Zhang et al. / Pattern Recognition Letters 27 (2006) 715–725 721

mations are shown in Table 2. We can see from Table 2that the forward and backward intensity errors of the con-sistent registration are smaller than the intensity error ofthe forward registration and the intensity error of the back-ward registration, the intensity difference images can beseen from the Fig. 5. We can see there are much errorsbetween the backward transform warped I 0t with Is in the

boundary of the skull, but in the consistent registrationcases, the intensity differences are much smaller.

3.2. Real data evaluation

Finally we use two real medical images to evaluate ourproposed registration method. The images are two cardiac

Table 1The transformation errors, ef and eb are the sum of squared difference of the transformation with the ground transformation by using forward registrationmethod, and e0f and e0b are the corresponding term by using the consistent registration method

Tests ef eb e1 e2 e3 e4 e0f e0b e01 e02 e03 e04

Test 1 425.9 446.9 273.2 141.6 294.4 146.2 412.1 399.7 17.8 5.78 10.0 5.63Test 2 411.5 484.6 246.7 124.0 255.4 138.6 403.9 398.5 6.58 6.48 6.44 5.53Test 3 409.4 503.2 248.8 150.4 267.2 158.9 398.7 385.7 11.0 10.3 12.8 10.7Test 4 474.0 487.1 261.4 132.8 276.7 138.5 453.0 443.6 19.1 5.7 8.56 5.2

e1, e2, e3, e4 are the x and y component of the forward and backward consistent transformation error and e01; e02; e03; e04 are the corresponding ones for

consistent registration.

Table 2The SSD value of It with I 0sðT hÞ and Is with I 0tðT gÞ for separate forward and backward registrations and consistent registration

Tests kI t � I 0sðTfh Þk

22 kI t � I 0sðT

cfh Þk

22 kIs � I 0tðT f

g Þk22 kIs � I 0tðT cf

g Þk22

Test 1 5871 3892 8329 4851Test 2 5904 3833 8847 3740Test 3 4448 3489 8396 3812Test 4 6182 4763 8304 4096

Fig. 5. The difference images. The first two columns are the difference images of I t � I 0sðT hÞ by using forward registration and consistent registration. Thelast two columns are the difference images of Is � I 0tðT gÞ by using backward and consistent registration.

722 Z. Zhang et al. / Pattern Recognition Letters 27 (2006) 715–725

MRI images, they are shown in Fig. 6(a) and (b), they aresections of a heart in diastolic and systolic periods. Weassume these two images are multi-modal images since thereis obvious difference between the intensity of these twoimages, we use them as source and target images. Fig. 6(c)and (d) show the forward transform warped sourceimage in forward registration and consistent registration,Fig. 6(e) and (f) show the warped target image in the back-ward registration and consistent registration, we can see inboth cases, the warped images in consistent registration aremuch smoother than those in forward or backward registra-tion, and we can see the improvement of the registration at

the left bottom corner of the images. Fig. 6(g) and (i) are theforward and backward grid transform recovered from sep-arate registration, Fig. 6(h) and (j) are those obtained fromconsistent registration, we can see the transformationobtained from consistent registration are much smootherthan those obtained from separate registrations. FinallyFig. 6(k) shows the composite grid transformation of theseparate forward and backward transformations. We seeit is far from identity transform in the whole image area,while that for consistent registration shown in Fig. 6(l) thecomposition of the forward and backward transform isnearly identity transform in the whole image area.

Fig. 6. The cardiac MRI images. (a) and (b) are the source and target images, they are sections of diastole and systole periods respectively. (c) and (d) arethe warped source image by forward registration method and by our proposed method. (e) and (f) are the backward transform warped target image bybackward registration method and by our proposed method. (g) and (i) are the forward and backward grid transform recovered from forward andbackward registrations, (h) and (j) are those obtained from consistent registration. (k) and (l) are the composition transform of the forward and backwardtransform by separate forward and backward registrations and that by our proposed method.

Z. Zhang et al. / Pattern Recognition Letters 27 (2006) 715–725 723

3.3. Computing time

We evaluate all of the registrations under Matlab 7.0under a Pentium4 PC with 2.8 GHz computer and 512MB Memory. Computing a typical forward registrationof two Brainweb images with size of 181 · 217 takes about12 min. A consistent registration takes about 30 min.

4. Summary and conclusions

In this paper, we proposed a novel variational approachfor consistent multi-modal non-rigid registration. Weimproved our former consistent variational approach formulti-modal non-rigid registration by adding a consistencyenergy and solving the Euler–Lagrange equation implicitly

724 Z. Zhang et al. / Pattern Recognition Letters 27 (2006) 715–725

using a numerically stable method. We evaluated our pro-posed registration method with simulated and real imagedata. The registration results obtained are compared withthose obtained from separate forward and backward regis-trations. We found the displacement fields obtained byusing our proposed registration method are muchsmoother than those obtained from separate registrations.The consistency errors of the forward and backward trans-formations are much smaller in our consistent registrationmethod. The improvements are very obvious in the bound-aries and the edge areas.

Acknowledgement

The work described in this paper was partially sup-ported by a Central Allocation Grant from the ResearchGrants Council of the Hong Kong Special AdministrativeRegion (Project no. CUHK1/00C).

Appendix. The discretization of the nonlinear parabolic

equation

For a typical nonlinear parabolic equation, we describethe discretization method here. Suppose we have a para-bolic equation as

ou

ot¼ f ðu; xÞ þ kDuðx; tÞ;

uð0; �Þ ¼ u0ð�Þ;ou

o~n¼ 0.

8>>>><>>>>:

ð16Þ

We use a finite difference method to discretize the nonlinearequation, we define the grid matrix as X, to better representour nonlinear equation into a linear matrix form, we define

0 10 20 30 40 50 60 70

0

10

20

30

40

50

60

70

nz = 302

1

2

3

4

5

6

7

Fig. 7. The non-zero type of the A matrix for laplace and linear elastic ope

~X as the vector form of the grid matrix. For the time dis-cretization we introduce a time-step s, then we denote~U

k:¼ ðu1ð~X ; ksÞT; . . . ; unð~X ; ksÞTÞT; ~F k

:¼ ðf1ðU k; ~X ÞT; . . . ;

fnðUk; ~X ÞTÞT as the discretization of the u(x, t) and f(u, x).Then the linear equation we obtained from the nonlinearPDE will be

~Uðkþ1Þ � ~U ðkÞ

s� kA~U

ðkþ1Þ ¼ ~F ðkÞ ð17Þ

with A the matrix generated from the finite differenceapproximation of the regularization operator, it dependson the regularization operator together with the boundarycondition (Modersitzki, 2004). Usually, the boundary con-dition can be incorporated into the matrix by zero padding(Dirichlet), mirroring (Neumann) or adequate copying(periodic). For 2D image registration problem (n = 2), itis a square matrix of n2N1N2 with N1 and N2 are the widthand height of the images. It is a highly structured matrix,usually a symmetrical sparse matrix. We show two non-zero type of matrices in Fig. 7, the left one shows thenon-zero matrix pattern of the laplace operator with Neu-mann boundary condition for Eq. (16) and the right oneshows the non-zero matrix pattern of the linear elasticoperator with a periodic boundary condition, in both cases,the image width and height are 5 and 7 respectively.

We can rearrange this finite difference into an iterativeformula:

~Uðkþ1Þ ¼ ð1� skAÞ�1ð~U ðkÞ þ s~F

ðkÞÞ. ð18Þ

Instead of calculating the inverse of the matrix, we usuallymove the inverse matrix part to the left of the equation andsolve a linear equation to obtain the intermediate iterativeresult.

0 10 20 30 40 50 60 70

0

0

0

0

0

0

0

0

nz = 630

rators. In both of these two examples, the image size (N1, N2) = (5, 7).

Z. Zhang et al. / Pattern Recognition Letters 27 (2006) 715–725 725

References

Alvarez, L., Deriche, R., Papadopoulo, T., Snchez, J., 2002. Symmetricaldense optical flow estimation with occlussions detection. Proc.ECCV’02 2350 (Part I), 721–736.

Ashburner, J., 1997. Computational neuroanatomy. Ph.D. Thesis.Bajcsy, R., Kovacic, S., 1989. Multiresolution elastic matching. Comput.

Vision Graphics Image Process. 46 (1), 1–21.Cachier, P., Ray, D., 2000. Symmetrization of the non-rigid registration

problem using inversion-invariant energies: Application to multiplesclerosis. Proc. MICCAI’00, pp. 472–481.

Cachier, P., 1999. Fast nonrigid matching by gradient descent: Study andimprovements of the demons algorithm. INRIA Technical Report,June 1999.

Christensen, G.E., Johnson, H.J., 2001. Consistent image registration.IEEE Trans. Med. Imag. 20 (7), 568–582.

Christensen, G.E., Joshi, S.C., Miller, M.I., 1997. Volumetric transfor-mation of brain anatomy. IEEE Trans. Med. Imag. 16 (6), 864–877.

Collins, D.L., Neelin, P., Peters, T.M., Evans, A.C., 1994. Automatic 3Dinter-subject registration of MR volumetric data in standardizedtalairach space. J. Comput. Assist. Tomography 18 (March/April),192–205.

Ferrant, M., Nabavi, A., Macq, B., Jolesz, F.A., Kikinis, R., Warfield,S.K., 2001. Registration of 3-d intraoperative MR images of the brainusing a finite-element biomechanical model. IEEE Trans. Med. Imag.20 (12), 1384–1397.

Gaens, T., Maes, F., Vandermeulen, D., Suetens, P., 1998. Non-rigidmulti-modal image registration using mutual information. MICCAI’981496, 1099–1106.

Guimond, A., Roche, A., Ayache, N., Meunier, J., 2001. Multimodalbrain warping using the demons algorithm and adaptive intensitycorrections. IEEE Trans. Med. Imag. 20 (1), 58–69.

Hermosillo, G., Chefd’hotel, C., Faugeras, O., 2002. Variational methodsfor multimodal image matching. Int. J. Comput. Vision 50 (3), 329–343.

Johnson, H.J., Christensen, G.E., 2002. Consistent landmark and inten-sity-based image registration. IEEE Trans. Med. Imag. 21 (5), 450–461.

Maes, F., Collignon, A., Delaere, D., Vandermeulen, D., Machal, G.,Suetens, P., 1997. Multimodality image registration using mutualinformation. IEEE Trans. Med. Imag. 16 (4), 187–198.

Modersitzki, J., 2004. Numerical methods for image registration. Numer-ical Mathematics and Scientific Computation. Oxford UniversityPress.

Petijean, C., Rougon, N., Preteux, F., 2002. Non-rigid image registrationusing generalized information measures. Technical Reports, March2002.

Pluim, J.P.W., Maintz, J.B.A., Viergever, M.A., 2004. f-Informationmeasures in medical image registration. IEEE Trans. Med. Imag. 23(12), 1508–1516.

Roche, A., Pennec, X., Malandain, G., Ayache, N., 2001. Rigidregistration of 3-D ultrasound with MR images: A new approachcombining intensity and gradient information. IEEE Trans. Med.Imag. 20 (10), 1038–1049.

Rougon, N., Petijean, C., Preteux, F., 2003. Variational non-rigid imageregistration using exclusive f-information. In: Internat. Conf. on ImageProcessing, pp. 703–706.

Rueckert, D., Sonoda, L.I., Hayes, C., Hill, D.L.G., Leach, M.O.,Hawkes, D.J., 1999. Non-rigid registration using free-form deforma-tions: Application to breast MR images. IEEE Trans. Med. Imag. 18(8), 712–721.

Studholme, C., Hill, D.L.G., Hawkes, D.J., 1997. Automated three-dimensional registration of magnetic resonance and positron emissiontomography brain images by multiresolution optimization of voxelsimilarity measures. Med. Phys. 24 (1), 25–35.

Thirion, J.P., 1998. Image matching as a diffusion process: An analogywith Maxwell’s demons. Med. Image Anal. 2 (3), 243–260.

Thomas, J.W., 1995. Numerical Partial Differential Equations—FiniteDifference Methods. Texts in Applied Mathematics. Springer.

Wells, W.M., Viola, P., Atsumi, H., Nakajima, S., 1996. Multi-modalvolume registration by maximization of mutual information. Med.Image Anal. 1 (1), 35–51.

Woods, R.P., Grafton, S.T., Holmes, C.J., Cherry, S.R., Mazziotta, J.C.,1998. Automated image registration: I. General methods and intra-subject, intramodality validation. J. Comput. Assisted Tomography22, 141–154.

Woods, R.P., Grafton, S.T., Watson, J.D.G., Sicotte, N.L., Mazziotta,J.C., 1998. Automated image registration: II. Intersubject validation oflinear and nonlinear models. J. Comput. Assisted Tomography 22,155–165.

Zhang, Z., Tsui, H., Xie, H., 2004. Variational approach for consistentmulti-modal non-rigid registration. Proc. ACCV’04, 1014–1020.