A Generalization of Free-Form Deformation ImageRegistration Within the ITK Finite Element Framework

Nicholas J. Tustison, Brian B. Avants, Tessa A. Sundaram,Jeffrey T. Duda, and James C. Gee

Penn Image Computing and Science LaboratoryUniversity of Pennsylvania

Philadelphia, PA 19104-6389

Abstract. Since the 1970’s B-splines1 have evolved to become the de facto stan-dard for use in curve and surface representation. This resulted in a relatively re-cent proliferation of nonrigid image registration techniques based on B-splines.These techniques fall under the general Free-Form Deformation (FFD) approachin which the object to be registered is embedded within a B-spline object. Thedeformation of the B-spline object represents the transformation space of the reg-istered object. In this paper, we describe the implementation of our finite elementmethodological (FEM) approach using B-splines. This registration frameworksubsumes essential components of currently popular FFD image registration al-gorithms, while providing a more principled and generalized control mechanismfor nonrigid deformation. Our implementation constitutes an extension of the ex-isting FEM library of the Insight Toolkit (ITK). We discuss the theoretical impli-cations and provide experimental results of our proposed methodology.

1 Introduction

FFD approaches to modeling are characterized by embedding the object of interestwithin an encompassing geometric primitive. The resulting deformation of the objectoccurs via the transformation of the surrounding geometric space. Due, in large part,to their “local support” property and computational efficiency, FFD image registrationbased on B-splines has received substantial attention in the research literature. In thispaper, we discuss our FEM-based B-spline approach to image registration, which is ageneralization of current FFD B-spline image registration algorithms. In this sense, weare not proposing a new FFD algorithm which distinguishes itself from existing algo-rithms by one of the features discussed below, but rather we are claiming that currentapproaches can be subsumed within our proposed FEM image registration framework.

The constitutive framework of the general registration algorithm is best elucidatedwithin the registration schema outlined in [2] which follows closely that of [3] and [4].According to [2], the three major components comprising the general image registrationalgorithm are:

1 Contrary to popular usage in the CAGD community, we follow de Boor’s distinction discussedin [1] of using the term ‘B-spline’ to denote the shape function of limited support and not theB-spline object (e.g curve, surface, volume).

J.P.W. Pluim, B. Likar, and F.A. Gerritsen (Eds.): WBIR 2006, LNCS 4057, pp. 238–246, 2006.c© Springer-Verlag Berlin Heidelberg 2006

A Generalization of Free-Form Deformation Image Registration 239

– the similarity metric,– the optimization strategy, and– the transformation model.

While the transformation model addressed in this paper is B-spline-based, it is impor-tant, in distinguishing previous work, to discern the absence or presence of an explicitregularization component of the transformation. In addition, discussion of the first twocomponents will prove helpful in differentiating previous work.

The explicit regularization term is best understood by interpreting the image registra-tion problem in a Bayesian decision-theoretic context [5]. Potential solutions are viewedas posterior probabilities defined in terms of both the data (likelihoods) and the regular-ization (prior probabilities) components. The often-used variational formulation followsby employing an assumption of Gaussianity for the probability distribution functions.We write this general variational energy, Π , as

Π(I, J, T ) =∫

Ω

(Π∼(I, J, T )(x) + ΠR(T )(x)) dΩ, (1)

where I and J are the fixed and moving images, respectively, and T is the transforma-tion which maps between I and J . Π∼ is the similarity metric and ΠR is the explicitregularization term. Some of the algorithms discussed below rely solely on the B-splinetransform, T , to provide the desired continuity, at the exclusion of the ΠR contribution.

2 Previous Work

Both Szeliski et al. [6] and Thevenaz et al. [7] use the Levenberg-Marquardt optimiza-tion strategy to formulate their respective B-spline image registration algorithms. Thelatter work was extended in [8] to include an explicit regularization term designed tominimize the weighted divergence and curl of the resulting deformation field. Unlikethe aforementioned algorithms, which seek to minimize a mean squares image intensitysimilarity metric (MSQ), Rueckert et al. [9] and Mattes et al. [10] use normalized mu-tual information (NMI). The former method uses a gradient descent optimization strat-egy whereas the latter is optimized via a limited-memory, quasi-Newton minimizationpackage (LBFGS). Also, the algorithm in [9] employs an explicit regularization term inthe form of the well-known thin-plate energy to prevent topological folding of the de-formation field for high-resolutions of the control point grid. This work was extended in[11] to include an isochoric regularization term, i.e. a regularization term which penal-izes deviations of the Jacobian from unity. These algorithms are summarized in Table1. By taking advantage of the modular nature of the ITK image registration framework[3], the basic elements of each of these algorithms can be reimplemented using ITKwith relative ease.

While these FFD methods are geometrically-based, finite element methods are inher-ently physics-based. Physics-based approaches to image registration were pioneered byBroit and Bajcsy [12] in which image registration is modeled via the deformation of aphysically-defined elastic object. Regularization occurs via the Cauchy-Navier operatorderived from continuum mechanics, which is given by

L = µ∇2 + (λ + µ)∇(∇·), (2)

240 N.J. Tustison et al.

Table 1. Summary of FFD B-Spline Registration Algorithms

Method Similarity Metric Optimization Strategy Explicit Regularization

Szeliski [6] MSQ Levenberg-Marquardt noThevenaz [7] MSQ Levenberg-Marquardt noSorzano [8] MSQ Levenberg-Marquardt Divergence and CurlRueckert [9] NMI Gradient Descent Thin-PlateMattes [10] NMI LBFGS no

Rohlfing [11] NMI Gradient Descent Isochoric

where the Lame constants, λ and µ, specify the material properties. This approach waslater recast in variational form in [5] (see Equation (1)). This reformulation suggests theuse of well-established finite element solution methods. Currently, this strategy is freelyavailable for use and development as part of the ITK FEM software library. The imple-mentation of B-spline finite elements within the ITK library allows for a generalizedapproach to B-spline image registration where the similarity metrics and regularizationterms are easily tailored, in a modular fashion, to the problem of interest. This also al-lows for the essential components of the algorithms listed in Table 1 to be duplicatedwithin our framework.

3 FEM Subsumption of FFD B-Spline Image Registration

The outline given of previous work provides the necessary context for understandingthe subsumption of the major components of existing FFD methods into our FEM-basedmethod.

3.1 Similarity Metric

Numerous similarity metrics are available for image registration in addition to the met-rics used by the algorithms given in Table 1. Measures such as normalized cross corre-lation, Mattes’ mutual information, and pattern intensity, as well as others, are all avail-able within the ITK library. Instead of creating a new FFD algorithm for each metric,our FEM-based image registration framework allows for modular exchange of metricsdictated by the user.

3.2 Optimization Strategy

Although not explicit in the previous work listed, except for [6], it is assumed that finitedifference approximations were used to estimate the value of the cost function and itsderivatives for nonlinear optimization. Although justification is typically derived froma Taylor expansion perspective, finite differences can be thought of in terms of gen-eralized FEM. From this vantage, finite difference solutions are simply FEM approxi-mations based on localized, discontinuous shape functions. Although such methods areeasy to implement, theoretically speaking, they are less accurate than correspondingFEM approaches [13].

A Generalization of Free-Form Deformation Image Registration 241

3.3 Explicit Regularization

Of the six algorithms surveyed in Table 1, three use no explicit regularization. Thesealgorithms depend on the inherent continuity of higher order B-splines to satisfy smooth-ness constraints. Of the other three, the work in [9] uses only an extremely tiny con-tribution of the thin-plate energy regularization. The weighting value, λ = 0.01, wasdetermined empirically and is only used when the control point density is sufficientlylarge (although the determination of “sufficiently large” is left unexplained). In [11] itwas noted that the thin-plate energy has no inherent volume-preserving capabilitieswhich motivated the introduction of a Jacobian-based regularization term. Unfortu-nately, there are scenarios in which this assumption is invalid, e.g. intersubject brainregistration. In [8] the capability of regularizing in terms of both the divergence andcurl of the displacement field is proposed. For many of the results given in the paper,however, there was no contribution from the curl regularization term whereas the di-vergence term had a weighting of 4 (assumed to be set empirically). In contrast to thealgorithm-specific regularization of the listed FFD methods, FEM-based image regis-tration allows for simple, modular exchange of regularization types (e.g. membrane orthin-plate regularization).

4 FEM Image Registration Using B-Splines

4.1 Shape Functions and Nodal Placement

At its conceptual level, FEM solutions discretize the continuum of the problem domaininto elements. Each of these elements has a governing set of both nodes (or control

−1 −0.5 0 0.5 1−0.2

0

0.2

0.4

0.6

0.8

1

1.2

−1 −0.5 0 0.5 1−0.4

−0.2

0

0.2

0.4

0.6

0.8

1

1.2

−1 −0.5 0 0.5 1−8

−6

−4

−2

0

2

4

6

8

(a) (b) (c)

−1 −0.5 0 0.5 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

−1 −0.5 0 0.5 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

−1 −0.5 0 0.5 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

(d) (e) (f)

Fig. 1. Shape functions: (a) C1 Lagrangian, (b) C2 Lagrangian, (c) C9 Lagrangian, (d) C1

B-spline, (e) C2 B-spline, and (f) C9 B-spline

242 N.J. Tustison et al.

Fig. 2. Placement of nodes for Lagrangian (left) versus B-spline elements (right)

points) and shape functions. In calculating the value of the FEM solution at a specificpoint within a given element, the shape functions are used to average, in a weighted andsmooth fashion, the contribution of the corresponding nodal values. Finite elementsusing B-splines, and a comparison with more traditional elements, have been investi-gated previously [14] in a general FEM context. We compare B-spline elements withLagrangian elements, due to their historical ubiquity in the FEM literature [13], in orderto provide a point of reference in discussing B-spline elements.

For simplicity in comparing shape functions, we plot the univariate shape functionsof both the Lagrangian and B-spline varieties. Multivariate extensions are simply thetensor product of the respective univariate set. These shape functions, parameterizedover the interval [−1, 1], are illustrated in Figure 1. Note that, when specified, the FFDB-spline image registration algorithms surveyed use cubic B-splines (see Figure 1(e)).

The Lagrangian shape functions are interpolatory in that the function values are iden-tical with the nodal values at the nodal locations. This is, in general, not the case withB-spline elements (the linear case is the sole exception in which Lagrangian and B-spline shape functions are identical). Also, notice that the B-spline shape functions arestrictly nonnegative over the interval in contrast to their Lagrangian counterparts. TheLagrangian shape functions also exhibit undulations meaning that the nodal influencein calculating the elemental function value does not necessarily decrease monotonicallywith distance as it does with B-splines. This undulatory phenomenon is exacerbated inhigher order shape functions, e.g. Figure 1(c).

In addition to the distinct shape functions, nodal location differs between Lagrangianand B-spline elements. For a given mesh size, B-spline elements span a subspace of theLagrangian elements so the theoretical accuracy will necessarily be less [14]. However,where computational costs prohibit fine resolution of Lagrangian element meshes (asin the case with certain image registration problems), B-spline elements provide a lesscostly substitute while guaranteeing a specified order of continuity. Shown in Figure 2are two diagrammatic representations of the nodal placement for four C2 Lagrangianelements (left) and four C2 B-spline elements (right). Note that neighboring Lagrangian

A Generalization of Free-Form Deformation Image Registration 243

elements only share bordering nodes whereas neighboring C2 B-spline elements share12 common nodes.

4.2 Multiresolution Strategy

The current ITK FEM registration methodology allows for a mulitresolution approachin which more global deformation can be inferred using a low resolution grid which canthen be propagated to higher levels for more localized deformation. We generalize thehierarchical approach for doubling the resolution of cubic B-spline surfaces discussedin [15] to bivariate and trivariate B-spline objects of arbitrary order. This allows forpropagating the FEM B-spline registration solution at a lower level to the next higherlevel.

5 Experimental Results

We experimentally demonstrate results from our image registration framework usingtwo 2-D brain slice images (size = 256×256) from two different subjects. These imagesare shown in Figures 3(a) and 3(b) which comprise the moving and fixed images, re-spectively. Preprocessing of the images included histogram matching. The mean squareimage intensity similarity metric was used in each of the examples as was a single levelresolution. Two different grid sizes were used (64×64 and 128×128 elements). Thesegrids were composed of quadratic (C1) B-spline elements using both membrane andlinear elastic regularization models [13].

The image registration results are illustrated in Figure 4. The top row consists ofthe results obtained from the membrane regularization model whereas the bottom row

(a) (b)

Fig. 3. 2-D Brain slice images used to demonstrate the FEM-based B-spline image registrationmethodology. (a) Original moving image and (b) fixed image.

244 N.J. Tustison et al.

(a) (b) (c)

(d) (e) (f)

Fig. 4. Registered images using different B-spline elements and Lame constants: (a) membraneelement, µ = 25000, λ = 0, 64 × 64 nodal mesh; (b) membrane element, µ = 25000, λ = 0,128 × 128 nodal mesh; (c) membrane element, µ = 50000, λ = 0, 128 × 128 nodal mesh;(d) linear elastic element, µ = 25000, λ = 0, 64 × 64 nodal mesh; (e) linear elastic element,µ = 25000, λ = 0, 128 × 128 nodal mesh; (f) linear elastic element, µ = 50000, λ = 0,128 × 128 nodal mesh;

contains the results from the linear elastic model. The first column of image results inFigure 4 were derived from grids of size 64×64 elements whereas the last two columnsof images were derived from grids of size 128 × 128 elements. Experimentation led tothe selection of the Lame constants used for the membrane results. Lower values forµ in the membrane model caused folding of the deformation field which dictated ourselection of the given parameters. We employed the same µ and λ values used in themembrane model for the linear elastic model for comparative purposes.

To assess the quantitative difference between the resulting images, we list variousmeasures in Table 2. These measures include the mean squares intensity differencebetween the resulting image and the fixed image as well as the minimum and max-imum Jacobian of the resulting transformation. These numbers seem to accord withintuition. The results derived from a higher resolution grid have a lower mean squareserror value. Also, we see that the membrane regularization produces deformation fieldswith extreme Jacobian values closer to unity and lower mean squares error values thanthe linear elastic regularization.

A Generalization of Free-Form Deformation Image Registration 245

Table 2. Quantitative measures for the results in Figure 4

Membrane Membrane Membrane Linear El. Linear El. Linear El.64 × 64 128 × 128 128 × 128 64 × 64 128 × 128 128 × 128λ = 0 λ = 0 λ = 0 λ = 0 λ = 0 λ = 0

µ = 25000 µ = 25000 µ = 50000 µ = 25000 µ = 25000 µ = 50000(a) (b) (c) (d) (e) (f)

Mean squares 237.7 190.0 193.0 221.7 175.1 206.1Min. Jacobian 0.72 0.52 0.42 0.65 0.28 0.53Max. Jacobian 1.36 1.73 1.76 1.48 1.76 1.61

6 Conclusion

We provided a discussion of our open-source ITK finite element library for nonrigid im-age registration based on B-splines and how it is related to current FFD B-spline imageregistration approaches. Experimental results were also illustrated using componentsunique to the FEM framework. Although specific FFD implementations are perhapseasier to implement, the FEM approach provides for more principled control over thenonrigid deformation. While we find the discussion presented in this paper meritorious,we recognize that the scope of our experimental results was minimal and that otherareas of investigation would certainly be welcomed by the research community. Suchareas of investigation would include, but not be limited to, the consequences of usingother regularization models (e.g. thin plate splines), the affects of the multiresolutionstrategy, differences in the results produced by different orders of B-splines, or the af-fects of different similarity metrics. Such comprehensive discussion will certainly beincluded in future work.

References

1. de Boor, C.: B-spline basics. In: Fundamental Developments of Computer-Aided GeometricModeling. American Press (1993) 27–49

2. Crum, W.R., Hartkens, T., Hill, D.L.G.: Non-rigid image registration: theory and practice.The British Journal of Radiology 77 (2004) 140–153

3. Ibanez, L., Ng, L., Gee, J.C., Aylward, S.: Registration patterns: The generic framework forimage registration of the Insight Toolkit. In: IEEE International Symposium on BiomedicalImaging. (2002) 345–348

4. Brown, L.G.: A survey of image registration techniques. ACM Computing Surveys 24 (1992)325–376

5. Gee, J.C., Bajcsy, R.K.: Elastic Matching: Continuum Mechanical and Probabilistic Analy-sis. In: Brain Warping. Academic Press (1999) 183–198

6. Szeliski, R., Coughlan, J.: Spline-based image registration. International Journal of Com-puter Vision 22 (1997) 199–218

7. Thevenaz, P., Ruttimann, U.E., Unser, M.: A pyramid approach to subpixel registration basedon intensity. IEEE Transactions on Image Processing 7 (1998) 27–41

246 N.J. Tustison et al.

8. Sorzano, C.O.S., Thevenaz, P., Unser, M.: Elastic registration of biological images usingvector-spline regularization. IEEE Transactions on Biomedical Engineering 52 (2005) 652–663

9. Rueckert, D., Sonoda, L.I., Hayes, C., Hill, D.L.G., Leach, M.O., Hawkes, D.J.: Nonrigidregistration using free-form deformations: Application to breast MR images. IEEE Transac-tions on Medical Imaging 18 (1999) 712–721

10. Mattes, D., Haynor, D.R., Vesselle, H., Lewellen, T.K., Eubank, W.: PET-CT image registra-tion in the chest using free-form deformations. IEEE Transactions on Medical Imaging 22(2003) 120–128

11. Rohlfing, T., Jr., C.R.M., Bluemke, D.A., Jacobs, M.A.: Volume-preserving nonrigid regis-tration of mr breast images using free-form deformation with an incompressibility constraint.IEEE Transactions on Medical Imaging 22 (2003) 730–741

12. Bajcsy, R., Broit, C.: Matching of deformed images. In: International Conference on PatternRecognition. (1982)

13. Zienkiewicz, O.C., Taylor, R.L.: The Finite Element Method, Volume 1: The Basis. 5th edn.Butterworth Heinemann (2000)

14. Sabin, M.A.: Spline Finite Elements. PhD thesis, Leeds University (1997)15. Lee, S., Wolberg, G., Shin, S.Y.: Scattered data interpolation with multilevel b-splines. IEEE

Transactions on Visualization and Computer Graphics 3 (1997) 228–244