Non-rigid Image Registration Using Geometric Features and Local Salient Region Features

Non-rigid Image Registration Using Geometric Features and Local SalientRegion Features

Jinzhong Yang∗, James P. Williams†, Yiyong Sun†, Rick S. Blum∗, and Chenyang Xu†∗ECE Department, Lehigh University, Bethlehem, PA 18015, USA

{jiy3, rblum}@lehigh.edu†Imaging and Visualization Department, Siemens Corporate Research, Princeton, NJ 08540, USA

{jimwilliams, yiyong.sun, chenyang.xu}@siemens.com

Abstract

We present a novel feature-based non-rigid image reg-istration algorithm using a small number of automaticallyextracted points and their associated local salient regionfeatures. Our automatic registration is a hybrid approachco-optimizing point-based and image-based terms. Moti-vated by the paradigm of the TPS-RPM algorithm [6], wedevelop the RHDM (Robust Hybrid Deformable Matching)algorithm by alternatively optimizing correspondences andtransformations for registration. The local salient regionfeatures and the geometric features, together with the sof-tassign and deterministic annealing techniques, are usedfor solving correspondences. Thin-plate splines are usedfor generating a smooth non-rigid spatial transformation.Our algorithm is built to be extremely robust to feature ex-traction errors. A new dynamic outlier rejection mecha-nism is described for rejecting outliers and generating ac-curate spatial mappings. A local refinement technique isused for correcting non-exactly matched correspondencesarising from image noise and irregular deformations. Incontrast with the TPS-RPM algorithm, which can handleonly outliers in one point set, our algorithm is able to han-dle a considerable number of outliers in both point sets. Theexperimental results demonstrate the robustness and accu-racy of our algorithm.

1. Introduction

In the current literature of image registration, feature-based methods are often used due to their speed [21]. Thesemethods use sparse features extracted from images, suchas points, curves, and surface patches, and the registrationtask is to find their correspondences and compute an op-timal transformation. The key for feature-based methods isto find true correspondences between two feature sets. Mostof them use Euclidean distance-based geometric features forsolving correspondences, for example, the iterative closest

point algorithm [2,19], the softassign method [6,12], shapecontext-based methods [1,20], and kernel correlation-basedmethods [9, 14]. Some of them use intensity-based localsimilarity measures, such as cross correlation, mutual in-formation etc., to determine correspondences [4, 7]. Somehybrid methods integrate geometric features and intensity-based local similarity measures for computing correspon-dences [5, 13].

Feature-based non-rigid image registration methods areoften subject to the impact of image noise, feature outliersand deformations. The presence of noise makes it difficultfor the extracted feature points to be exactly matched. Theoutliers are those feature points detected in one image with-out correspondences in the other. These outliers need tobe rejected during the matching process. Non-rigid reg-istration often involves irregular deformations, which maylead to additional non-exactly matches and outliers. A goodfeature-based non-rigid registration method should be ableto find exactly corresponding points, reject outliers and de-termine an accurate transformation describing the irregu-lar deformation. The TPS-RPM (Thin-Plate Splines RobustPoint Matching) algorithm [6], which is proposed for pointmatching, shows advantages in handling outliers. The basicidea of this algorithm is to use the softassign technique [12]allowing fuzzy and partial matches between two point sets.The deterministic annealing technique [18] is used to di-rectly control the fuzziness and gradually optimize the cor-respondences. Thin-plate splines (TPS) are used to generatea smooth geometric transformation. The inherent structureof the TPS-RPM algorithm allows outliers existing only inthe alignment point set, while it is unable to handle outliersin the reference point set. In automatically extracted featurepoints, many of them may have no correspondence (see Fig.1). To register images using these feature points, an algo-rithm must be able to detect true correspondences subject tooutliers in both point sets.

In this paper, we develop the RHDM (Robust HybridDeformable Matching) algorithm using automatically ex-

Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06) 0-7695-2597-0/06 $20.00 © 2006 IEEE

Figure 1. Two retinal images1 with automati-cally extracted feature points and salient re-gion features.

tracted feature points and local salient region features toregister images. The salient region feature detector initiallyproposed by Kadir and Brady [10] aims to select regionswith highest local saliency in both spatial and scale spaces(Examples of salient region features are shown in Fig. 1).Huang et al. [8] first used these salient region features forrobust rigid registration. Recently, they are extended forregistering non-rigid deformable MR brain images [17]. Inour hybrid approach, the correspondences are optimized notonly using the Euclidean distance-based geometric features,but also using the intensity-based local salient region fea-tures. Since the TPS mapping has a single closed-form rep-resentation for the entire space, and is able to be decom-posed into affine and non-affine subspaces [3, 15], TPS areused for generating a non-rigid spatial transformation. Ouralgorithm is built to be extremely robust to feature extrac-tion errors. We demonstrate a new outlier rejection mech-anism, which can dynamically update point sets during theoptimization process. Once a point is detected as an out-lier, it will be removed from the point sets immediately.Only those corresponding points are used for computing thetransformation. This is very important since for non-rigidregistration each point does not function individually. Ifoutliers are used for computing the transformation, they willbring undesired local deformations and thus negatively af-fect the optimization process. We also introduce a local re-finement technique for correcting non-exactly matched cor-respondences, which arise from noise and irregular defor-mations. The simulated experiments show the great advan-tage of our algorithm in the case that both point sets containa considerable number of outliers. The experimental resultsalso demonstrate the robustness and accuracy of our algo-rithm.

The remainder of this paper is organized as follows. InSection 2 we present our objective function for registration.We develop our RHDM algorithm in Section 3. The exper-imental results are demonstrated in Section 4. Finally weconclude with discussion in Section 5.

1Images courtesy of Prof. Chuck Stewart, Rensselaer Polytechnic In-stitute.

2. Problem formulation and methodology

Suppose we have a reference image I and an alignmentimage J . Two point sets X = {xi : i = 1, ..., L} andY = {yj : j = 1, ..., N} (X,Y ∈ R2) are extracted from Iand J respectively. We will use X and Y to register imageJ to image I .

2.1. Hybrid energy function formulation

Let f denote a non-rigid spatial mapping. The point setX is mapped to X = {xi = f(xi) : i = 1, ..., L}. Wedefine a binary correspondence matrix P with dimension(L + 1) × (N + 1),

P =

p11 · · · p1N p1,N+1

.... . .

......

pL1 · · · pLN pL,N+1

pL+1,1 · · · pL+1,N 0

. (1)

The matrix P consists of two parts. The L × N inner sub-matrix defines the correspondences of X and Y . If xi ismapped to yj , then pij = 1, otherwise pij = 0. The(N+1)th column and the (L+1)th row define the outliers inX and Y respectively. If xi is an outlier, then pi,N+1 = 1.Similarly, if yj is an outlier, then pL+1,j = 1. P satisfies therow and column normalization conditions,

∑N+1j=1 pij = 1,

for i = 1, ..., L, and∑L+1

i=1 pij = 1, for j = 1, ..., N .Our registration task is to find an optimal correspondence

matrix P and an optimal spatial transformation f , whichmatches these two point sets X and Y as closely as possible.Here “close” means not only in the Euclidean sense, butalso in the sense of salient region feature similarity. Wepropose to minimize the following hybrid energy functionfor registration,

[P , f ] = arg minP,f

E(P, f), (2)

E(P, f) = Eg(P, f)+σEd(P, f)+λEs(f)+Ea(P ), (3)

subject to the following constraints,0 ≤ pij ≤ 1, for i = 1, ..., L + 1; j = 1, ..., N + 1.∑N+1

j=1 pij = 1, for i = 1, ..., L. (4)∑L+1i=1 pij = 1, for j = 1, ..., N.

Here Eg(P, f) is the geometric feature-based energy termdefined by Euclidean distance as

Eg(P, f) =N∑

j=1

L∑i=1

pij‖yj − f(xi)‖2. (5)

Ed(P, f) is the salient region feature dissimilarity energyterm that will be addressed in Section 2.2. σ is the prior ofthe weight for Ed(P, f), and λ is the regularization parame-ter for Es(f), the smoothness constraint for the transforma-tion f . In non-rigid registration, smoothness is necessary torestrict the mappings not being too arbitrary. We use TPSto generate a smooth spatial mapping since TPS mapping is


parameter free and has a single closed-form representationfor the entire space. The TPS smoothness energy term is

Es(f) =

�� ∂2f

∂u2

�2

+ 2

�∂2f

∂u∂v

�2

+

�∂2f

∂v2

�2�

dudv.

(6)

We use softassign technique [12] allowing P to takevalues from interval [0, 1] in hybrid energy function (3).The continuous property of P acknowledges ambiguousmatches between X and Y . The energy term Ea(P ) is usedto direct P converging to binary. It is comprised of twoterms,

Ea(P ) = TN∑

j=1

L∑i=1

pij log pij − ζN∑

j=1

L∑i=1

pij . (7)

The first term is an entropy term comes from the determinis-tic annealing technique [18], which is used to directly con-trol the fuzziness of P . T is called the temperature parame-ter. The second term will be described in Section 2.3.

2.2. Salient region features and similarity measures

Each point in X and Y is associated with a local salientregion feature. We introduce a region operator R(·). LetR(c, n) denote a local salient region that centers at point cwith total n points. For each point ri in region R(c, n), ithas a fixed distance and direction to the center c. To sim-plify notation, we omit the number of points for the regionby using R(c) instead of R(c, n). Let I(·) denote the in-tensity values. Then I(R(xi)) denotes a vector of intensityvalues for salient region R(xi). We call I(R(xi)) the lo-cal salient region feature for xi. The translation of salientregion R(c), T (R(c), t), is defined as a new local salientregion R(c + t), which has the same geometric structurewith R(c) and centers at c + t. t is the translation vector.

We now define the salient region feature dissimilarityenergy Ed(P, f) in (3). We use the salient region featuredetector described in [8, 10] to extract local salient regions{R(xi)} from reference image I . Let rk, k = 1, ..., n, de-note the points in region R(xi). We define a transformedsalient region of R(xi), R(f(xi)), in which each pointrk = f(rk), for k = 1, ..., n. The new region center isf(xi). Let tij = yj − f(xi) be the shift vector between yj

and f(xi). The salient region feature dissimilarity energy isdefined as

Ed(P, f) =N∑

j=1

L∑i=1

pij ·

(1 − LCC [I(R(xi)), J(R(yj))]

2

)2

,

(8)

where R(yj) = T (R(f(xi)), tij) is the translation of re-gion R(f(xi)). I(·) denotes the intensity values of im-age I , and J(·) denotes the intensity values of image J .LCC[·, ·] is the local cross-correlation coefficient of salient

region features I(R(xi)) and J(R(yj)), which describestheir similarity.

2.3. Outlier rejection mechanism

The correspondence matrix P explicitly includes the de-scription of outliers. Each entry of the continuous P canbe viewed as the probability of one point in X mapped toone point in Y or being an outlier. For the ith row of P ,the first N entries denote the probabilities of xi mapped toeach point in Y , and pi,N+1 is the probability of xi beingan outlier. If pi,N+1 is greater than a threshold, 1 − ε, xi

can be rejected as an outlier. Similarly, yj is rejected as anoutlier if pL+1,j is greater than 1 − ε. The outlier rejectionis performed throughout the optimization process. X and Yare updated dynamically.

Posed in the dynamic update of feature point sets, thereare two issues to be addressed. First, there is no clear wayto estimate the probability of a point being an outlier. Chuiand Rangarajan [6] proposed to define a cluster center witha very large variance. All outliers are placed in this clus-ter. The cluster center is put at the center of mass of eachpoint set. However, it is difficult to choose a proper clus-ter variance, and the center of mass of a point set may notbe good for the outlier cluster center. For simplification,we assume each point has a small prior ρ to be an outlier.Since we are able to estimate the L×N inner submatrix ofP , we may use these estimated values together with ρ fornormalization to satisfy constraints (4). Then the dynamicoutlier rejection can be applied after normalization. Sec-ond, since dynamically deleting points from feature pointsets can always minimize the energy function (3), we needa mechanism to prevent too many points from being rejectedas outliers. Thus we use the term −ζ

∑Nj=1

∑Li=1 pij in (7)

to balance the outlier rejection. ζ is a weighted parameterto control the effect of this term.

3. The RHDM algorithm

The minimization problem in (2) includes solving for anoptimal P and an optimal f . A natural solution is to alterna-tively optimize one while having the other one fixed. Thuswe suggest to iteratively perform the alternating optimiza-tion so that P and f converge to local optima.

3.1. Alternating optimization

First, we fix f in (3) to optimize P . The optima of P ={pij : i = 1, ..., L; j = 1, ..., N} can be obtained by solving∂E(P, f)/∂pij = 0. It consists of three terms, pij = µ ·mij · sij . The first term, µ = exp(ζ/T ), is the protectionterm for outlier rejection. Especially when T is small andP is becoming binary, this term works to avoid too manyzeros in the L × N inner submatrix of P . The second termis the geometric matching probability term,

mij = exp[− (yj − f(xi))T (yj − f(xi))

T

]. (9)


This is the probability of point xi matched to yj in Eu-clidean sense. We call M = {mij} the geometric matchingprobability matrix. The third term is the salient feature sim-ilarity term,

sij = exp

�−σ (1 − LCC [I(R(xi)), J(R(yj))])

2

4T

�. (10)

This is the probability of point xi matched to yj in the senseof salient region feature similarity. We call S = {sij} thesalient region feature similarity matrix. To satisfy the con-straints of P in (4), the alternating row and column normal-ization [6, 20] is applied:

pij =pij∑N+1

l=1 pil

, i = 1, ..., L; j = 1, ..., N + 1. (11)

pij =pij∑L+1

l=1 plj

, i = 1, ..., L + 1; j = 1, ..., N. (12)

Second, we fix P in (3) to optimize f . To simplify,we use correspondence matrix P estimating a new pointset {zi =

∑Nj=1 pijyj : i = 1, ..., L} that corresponds to

{xi : i = 1, ..., L}. Thus the TPS finds an optimal transfor-mation f by minimizing the following TPS bending energyfunction, which is a simplification of (3),

ETPS(f) =L�

i=1

‖zi − f(xi)‖2+

λ

�� ∂2f

∂u2

�2

+ 2

�∂2f

∂u∂v

�2

+

�∂2f

∂v2

�2�

dudv.

(13)

The homogeneous coordinates are used for easy descrip-tion. Each point xi is represented as a vector (1, xiu, xiv),and X is an L × 3 matrix by stacking all xi together. Ma-trix Z is built in the same way. A unique solution of f thatminimizes (13) consists of two matrices a and w as [6, 15]

f(xi) = xi · a + K(xi) · w, (14)where a is a 3 × 3 matrix representing the affine transfor-mation and w is an L×3 matrix representing the non-affinewarping transformation. K(xi) is a vector with each en-try Kj(xi) = ‖xj − xi‖2 log ‖xj − xi‖, for j = 1, ..., L.All K(xi), i = 1, ..., L, stack together to comprise the TPSkernel K. Applying QR decomposition on X ,

X = (Q1 Q2)(

R0

), (15)

we obtain the optimal solution for a and w as

w = Q2

(QT

2 KQ2 + λI)−1

QT2 Z, (16)

a = R−1QT1 (Z − Kw) . (17)

We also adopt the penalization on affine transformation pro-posed in [6] to avoid undesired nonphysical reflection map-pings.

3.2. Implementation remarks

To implement the RHDM algorithm, the parameters needto be set properly and some practical techniques are utilized.To help for easily choosing annealing parameters, we scalecoordinates of X and Y into a uniform box. During an-nealing procedure, T is first set to an initial temperature T0

and then gradually decreases according to a linear schedule,Tnew = Toldγ, where γ is the annealing rate. The anneal-ing is stopped when a final temperature Tfinal is reached.In order to flexibly control the smoothness for transforma-tion, we opt to anneal the regularization parameter λ by set-ting λ = λ0T , where λ0 is an initial constant. The salientregion feature similarity term sij is not effective for iden-tifying true correspondences when T is at a high temper-ature. Thus we adjust the functionality of sij by settingσ = σ0 exp(−T/δ2). σ0 is the prior weight for sij andδ controls the increasing rate of σ when T is decreasing.Most parameters used in our algorithm are not sensitive tothe input data except ζ, the parameter controlling the outlierrejection. However, automatic data-based determination ofζ seems feasible and is a topic of our future work.

3.3. Summary of the RHDM algorithm

The RHDM algorithm is briefly summarized in Algo-rithm 1. The feature extraction is non-trivial work forfeature-based image registration. We do not elaborate thesalient region feature detector here but refer the readersto [8, 10] for details.

Algorithm 1 The RHDM Algorithm Pseudo-codeInput: Reference point-set X and alignment point-set YOutput: Correspondence P and transformation f = {a, w}

Initialize: T ⇐ T0, λ ⇐ λ0T and σ ⇐ σ0 exp(−T/δ2)Initialize: P , a and wwhile T > Tfinal do {Begin deterministic annealing}

repeat {Begin alternating update}{Step I: Update correspondence}Compute M and S using (9) and (10)Update correspondence P ⇐ µM ◦ S 2

Normalize P using (11) and (12) iteratively{Step II: Outlier rejection}Remove xi from X if pi,N+1 > 1 − ε for i = 1, ..., LRemove yj from Y if pL+1,j > 1 − ε for j = 1, ..., N{Step III: Update transformation}Update w and a using (16) and (17) respectively

until P , a and w convergedT ⇐ Tγ, update λ ⇐ λ0T and σ ⇐ σ0 exp(−T/δ2)

end while

Due to noise and irregular deformations, there may ex-ist non-exactly matched correspondences, in which the ex-tracted points are only close to, but not the exactly matchedpoints (see examples in Fig. 4). We introduce a local refine-ment technique to correct these non-exactly matched cor-

2The operator “◦” denotes element-wise product of two matrices.


(a) Brain Image (b) Retinal Image

(c) Thorax Image (d) Aerial Image

Figure 2. Four images for simulation tests.

respondences. This technique is applied when computingS in Algorithm 1. Assume in the previous iteration xi ismapped to yj with the greatest probability. Let W(yj) de-note a small h×h window centering at yj . We compute theLCCs between R(xi) and R(y′

j), for all y′j ∈ W(yj), and

choose y′j that maximizes the LCC as the new feature point

yj . The maximal LCC is then used for computing sij . Thusthe feature point yj is locally “refined” to an improved onethat is possibly the exactly matched point of xi.

4. Experiments

4.1. Simulation tests on robustness to outliers

The prominent feature of the RHDM algorithm is its ca-pability of detecting true correspondences subject to a con-siderable number of outliers in both point sets. Extensiveexperiments have been carried out to demonstrate the ro-bustness of the RHDM algorithm. For a given image I , apoint set X = {xi : i = 1, ..., N} is extracted and a ran-dom TPS transformation of median degree of deformationis applied to obtain point set Y = {yj : j = 1, ..., N} anddeformed image J . Two cases of the tests are performed.(I) Randomly add l outliers to X for testing outliers in ref-erence point set. (II) Randomly add l outliers to Y for test-ing outliers in alignment point set. We carry out 20 randomtrials on each l for each case. Four images in Fig. 2 are usedfor these tests. We record the normalized mean square error(NMSE) between registered image J ′ and original imageI for each test. For each l outliers, we compute the meanand standard deviation of NMSE over 20 trials. They areplotted versus outlier to data ratio shown in Fig. 3. Forcomparison we also carried out the same tests on the TPS-RPM algorithm. From Fig. 3 we can see that the RHDM isvery robust to outliers in both reference point set and align-ment point set, even if both point sets contain outliers with

0 0.5 1 1.5 2−0.01

0

0.01

0.02

0.03

0.04

Outlier to Data Ratio

Nor

mal

ized

MS

E (

NM

SE

)

Outliers in Reference Point Set (Brain image, N=30)

0 0.5 1 1.5 2−0.005

0

0.005

0.01

0.015

0.02


Nor

mal

ized

MS

E (

NM

SE

)

Outliers in Alignment Point Set (Brain image, N=30)

0 0.5 1 1.5 20

0.005

0.01

0.015

0.02

0.025

0.03


Nor

mal

ized

MS

E (

NM

SE

)

Outliers in Reference Point Set (Retinal image, N=37)

0 0.5 1 1.5 2−0.01

0

0.01

0.02

0.03

0.04

0.05


Nor

mal

ized

MS

E (

NM

SE

)

Outliers in Alignment Point Set (Retinal image, N=37)

0 0.5 1 1.5 2−0.02

0

0.02

0.04

0.06

0.08

0.1

0.12


Nor

mal

ized

MS

E (

NM

SE

)

Outliers in Reference Point Set (Thorax image, N=40)

0 0.5 1 1.5 2−0.02

−0.01

0

0.01

0.02

0.03

0.04


Nor

mal

ized

MS

E (

NM

SE

)

Outliers in Alignment Point Set (Thorax image, N=40)

0 0.5 1 1.5 2−0.01

0

0.01

0.02

0.03

0.04

0.05

0.06


Nor

mal

ized

MS

E (

NM

SE

)

Outliers in Reference Point Set (Aerial image, N=30)

0 0.5 1 1.5 2−0.02

0

0.02

0.04

0.06

0.08


Nor

mal

ized

MS

E (

NM

SE

)

Outliers in Alignment Point Set (Aerial image, N=30)

Figure 3. Comparison of RHDM algorithm (•)and TPS-RPM algorithm (�) in robustness tooutliers. The error bars indicate the standarddeviation of NMSE over 20 random trials.

the outlier to data ratio at 1.0. Column 1 in Fig. 3 shows thegreat advantage of the RHDM algorithm over the TPS-RPMalgorithm when outliers are present in reference point set.The TPS-RPM algorithm cannot handle any outliers in thereference point set. Column 2 in Fig. 3 shows a similar per-formance between the RHDM algorithm and the TPS-RPMalgorithm when outliers are present in alignment point set.

4.2. Simulated non-rigid image registration

In order to validate the robustness and accuracy of theRHDM algorithm, we conducted a series of experiments us-ing simulated non-rigidly deformed image pairs. The simu-lation is built upon the four images in Fig. 2. For each im-age I , a random TPS transformation is applied to obtaineda deformed image J . We examine 5 different degrees of de-formation (0.02, 0.04, 0.06, 0.08 and 0.10) for each image.The degree of deformation (DEG) is the standard deviationof a zero mean Gaussian distribution, which stipulates thefreedom of movement for feature points. A big standard de-


Brain Image Retinal Image Thorax Image Aerial ImageDEG L N C E GCC L N C E GCC L N C E GCC L N C E GCC0.02 30 28 19 1 0.9644 30 34 16 0 0.9791 30 30 25 1 0.9907 37 40 15 0 0.96790.04 30 31 18 1 0.9364 30 34 19 2 0.9428 30 30 22 1 0.9802 37 40 19 0 0.94340.06 30 30 22 2 0.9578 25 26 17 1 0.9593 30 40 23 0 0.9798 45 50 19 0 0.91990.08 31 30 17 3 0.8864 59 60 17 2 0.9530 30 40 24 0 0.9875 37 37 19 0 0.93490.10 30 27 16 3 0.9172 30 30 17 1 0.9494 30 30 25 1 0.9749 50 60 23 0 0.8784

Table 1. Simulated registration results for images in Fig. 2.



Figure 4. Correspondence illustration on de-formed image with DEG at 0.06. Circles (◦) de-note the ground truth of deformed points. Pluses (+) de-note the points extracted from the deformed image. A circleoverlapping a plus means one existing correspondence. Tri-angles (�) denote the detected correspondences by RHDMalgorithm. A triangle overlapping a circle means a goodcorrespondence detection.

viation means a large deformation. Table 1 shows the sim-ulation results for the four images. We recorded the num-ber of reference points (L) extracted from I , the number ofalignment points (N ) extracted from J , the number of cor-respondences (C) detected by RHDM algorithm, the num-ber of error detected correspondences (E), and the globalcross-correlation coefficient (GCC) of registered image J ′

and reference image I . Table 1 shows that the RHDM al-gorithm can obtain a good registration for different degreesof deformation, even if only half of the extracted points aretrue correspondences. We illustrate the correspondences inFig. 4 for each data set with DEG at 0.06. The deformationgrids acquired by RHDM algorithm are compared with theground truth in Fig. 5.

The registration error normally comes from two parts.One is the false matches that are detected as correspon-dences. The other is that detected correspondences arenot enough to characterize the deformations. Most falsematches come from non-exactly matched correspondences.



Figure 5. Deformation grids. Dotted lines, solidlines, and dashed lines denote the original grids, the groundtruth deformation grids, and the deformation grids obtainedby RHDM algorithm, respectively. Circles (◦) and pluses(+) denote the reference points on original grids and align-ment points on ground truth deformation grids, respectively.

Fig. 4 shows the existing non-exactly matched correspon-dences, especially in the retinal and aerial images. Thesenon-exactly matched correspondences often decrease theaccuracy of registration by introducing small local defor-mations (Fig. 5). The local refinement technique is usedto alleviate this effect. Fig. 4 and Fig. 5 show that mostnon-exactly matched correspondences are corrected. Gen-erally speaking, the existing correspondences are enough tocharacterize the deformations. In case that there exist in-sufficient correspondences, one can increase the number ofextracted feature points for additional matches, e.g. retinalimage with DEG at 0.08 and aerial image with DEG at 0.10in Table 1. In some cases there may be undetected existingcorrespondences (Fig. 4a). Our experiments show that asmall number of undetected correspondences generally donot affect the registration results.

4.3. Real non-rigid image registration applications

Real images often contain significant levels of noise andintensity inhomogeneity. This would yield difficulties infeature extraction and produce additional outliers. To vali-


(I)

(II)

(III)(a) (b) (c) (d) (e) (f)

Figure 6. Registration of six pairs of real world images. (I.a-f): reference images. (II.a-f): alignmentimages. (III.a-f): edges from the registered images superimposed on the reference images.

date the robustness and accuracy of our RHDM algorithm,we applied it to a variety of real cases.

The RHDM algorithm is used to register six pairs ofreal images shown in Fig. 6(I-II). Fig. 6(a) is a pair offace images 3 obtained from the MIT-CBCL face recogni-tion database [16]. These two images are one face in dif-ferent poses. Fig. 6(b-c) are stereo images obtained fromCMU/VASC image database 4. The stereo images containmany structures appearing/disappearing. Fig. 6(b) is a pairof stereo aerial images of white house, and Fig. 6(c) is astereo pair of partial view of a Rubik’s cube occluding awooden block 5. Fig. 6(d) is a pair of curved human retinalimages with partial matching and rigid transformation. Theextracted feature points and salient region features for reg-istering these two images are shown in Fig. 1. Fig. 6(e) is apair of brain images 6 with one of which contains a tumor.The tumor in the alignment image significantly changes itsappearance. Fig. 6(f) is a pair of CT thorax images that ispreregistered rigidly. To see how effective and robust theregistration is, we extracted the edges of the registered im-ages and superimpose the edges on the reference images,which are shown in Fig. 6(III). The edge maps show goodregistrations for all of these images. The registered result inFig. 6(III.c) contains some local errors. These errors comefrom false matches that are detected as correspondences dueto a big structure change at the conjunction of the cube and

3 c©Copyright 2003 - 2005 Massachusetts Institute of Technology.4http://vasc.ri.cmu.edu/idb/html/stereo/index.html5Images courtesy of Dr. Steven Cochran, USC IRIS.6Images courtesy of Prof. J. M. Fitzpatrick at Vanderbilt University.

the wooden block.To further demonstrate the efficiency of our RHDM algo-

rithm, We applied it to register a series of MR lung imagesthat are taken at different time. Fig. 7(I.a-d) shows fourlung images in a breathing cycle. We registered images inFig. 7(I.b-d) to Fig. 7(I.a) using the RHDM algorithm. Thesubtraction maps of alignment images in Fig. 7(II.b-d) andthe subtraction maps of registered images in Fig. 7(III.b-d)are placed together for comparison. The difference betweenregistered images and the reference image is much smallerthan that between alignment images and the reference im-age, which shows the good registration results.

5. Discussions and conclusions

It is worth to mention that the similarity measure forsalient region features plays an important role in our RHDMalgorithm. The computational complexity largely comesfrom the calculations of the similarity. For a typical ex-ample of two 256 × 256 images with 2 sets of 30 extractedsalient region features, it takes about 1 minute to run theRHDM algorithm, which is implemented using MATLABon a desktop PC with Pentium 4 1.8GHz and 640MB RAM.Although we use the salient region feature in this paper,other types of local features are possible with our algorithmas well. In multi-modal registration (e.g. CT-PET registra-tion) where large deformations are present, existing salientregion feature detectors may not capture enough local in-formation to yield an accurate registration. Recently a newrobust feature detector [11] that combines SIFT (Scale In-variant Feature Transform) and global context seems favor-


(I)

(II)

(III)(a) (b) (c) (d)

Figure 7. Registration of a series of lung im-ages at different time points. (I.a): referenceimage. (I.b-d): alignment images. (II.b-d):alignment images subtracted from the refer-ence image. (III.b-d): registered images sub-tracted from the reference image.

able in this case. We may incorporate this feature detectorto our algorithm in the future.

To conclude, we have presented a novel feature-basednon-rigid image registration algorithm using geometric fea-tures and local salient region features. Our algorithm isa hybrid approach co-optimizing point-based and image-based terms, and is built to be extremely robust to featureextraction errors. We introduced a new dynamic outlierrejection mechanism and a local refinement technique forhelping find true correspondences, reject outliers and deter-mine an accurate transformation. The experimental resultsdemonstrate the robustness and accuracy of our algorithmand its capability of handling a considerable number of out-liers in both point sets.

Acknowledgements

We would like to thank C. Florin, C. Guetter, A. Azar, andDr. F. Sauer at Siemens Corporate Research for inspiringdiscussions. We are grateful to X. Huang for her initial workon salient region feature-based registration, and Dr. H. Chuifor helpful discussions on the TPS-RPM algorithm. We ac-knowledge G. Wolz and C. Chefd’Hotel for providing tho-rax images and lung time series images respectively. Thiswork is funded by Siemens Corporate Research, Inc.

References

[1] S. Belongie, J. Malik, and J. Puzicha. Shape matching andobject recognition using shape contexts. IEEE Trans. Pat-tern Anal. Mach. Intell., 24(4):509–522, 2002.

[2] P. J. Besl and N. D. McKay. A method for registration of 3-Dshapes. IEEE Trans. Pattern Anal. Mach. Intell., 14(2):239–256, 1992.

[3] F. L. Bookstein. Principal warps: Thin-plate splines and thedecomposition of deformations. IEEE Trans. Pattern Anal.Mach. Intell., 11(6):567–585, 1989.

[4] P. Cachier, E. Bardinet, D. Dormont, X. Pennec, and N. Ay-ache. Iconic feature based nonrigid registration: the PASHAalgorithm. Comput. Vis. Image Understand., 89(2-3):272–298, 2003.

[5] P. Cachier, J.-F. Mangin, X. Pennec, D. Riviere,D. Papadopoulos-Orfanos, J. Regis, and N. Ayache. Mul-tisubject non-rigid registration of brain MRI using intensityand geometric features. In MICCAI, pages 734–742, 2001.

[6] H. Chui and A. Rangarajah. A new point matching algorithmfor non-rigid registration. Comput. Vis. Image Understand.,89(2-3):114–141, 2003.

[7] A. Guimond, A. Roche, N. Ayache, and J. Meunier. Three-dimensional multimodal brain warping using the demonsalgorithm and adaptive intensity corrections. IEEE Trans.Med. Imag., 20(1):58–69, 2001.

[8] X. Huang, Y. Sun, D. Metaxas, F. Sauer, and C. Xu. Hybridimage registration based on configural matching of scale-invariant salient region features. In IEEE Workshop on Im-age and Video Registration, 2004.

[9] B. Jian and B. C. Vemuri. A robust algorithm for pointset registration using mixture of Gaussians. In Proc. ICCV,pages 1246–1251, 2005.

[10] T. Kadir and M.Brady. Saliency, scale and image descrip-tion. Int. J. Comput. Vis., 45(2):83–105, 2001.

[11] E. Mortensen, H. Deng, and L. Shapiro. A SIFT descriptorwith global context. In Proc. CVPR, pages 184–190, 2005.

[12] A. Rangarajan, H. Chui, and F. Bookstein. The softassignprocrustes matching algorithm. In Proc. IPMI, pages 29–42, 1997.

[13] D. Shen and C. Davatzikos. HAMMER: hierarchical at-tribute matching mechanism for elastic registration. IEEETrans. Med. Imag., 21(11):1421–1439, 2002.

[14] Y. Tsin and T. Kanade. A correlation-based approach to ro-bust point set registration. In Proc. ECCV(3), pages 558–569, 2004.

[15] G. Wahba. Spline models for observational data. SIAM,Philadelphia, PA, 1990.

[16] B. Weyrauch, J. Huang, B. Heisele, and V. Blanz.Component-based face recognition with 3D morphablemodels. In IEEE Workshop FPiV’04, 2004.

[17] G. Wu, F. Qi, and D. Shen. Learning best features for de-formable registration of MR brains. In Proc. MICCAI, pages179–187, 2005.

[18] A. L. Yuille and J. J. Kosowsky. Statistical physics algo-rithms that converge. Neural Comput., 6(3):341–356, 1994.

[19] Z. Zhang. Iterative point matching for registration of free-form curves and surfaces. Int. J. Comput. Vis., 13(2):119–152, 1994.

[20] Y. Zheng and D. Doermann. Robust point matching for non-rigid shapes: a relaxation labeling based approach. LAMP-TR-117, University of Maryland, College Park, MD, 2004.

[21] B. Zitova and J. Flusser. Image registration methods: a sur-vey. Image Vis. Comput., 21(11):977–1000, 2003.


Non-rigid Image Registration Using Geometric Features and Local Salient Region Features

Documents

Transcript of Non-rigid Image Registration Using Geometric Features and Local Salient Region Features