Multi-modal non-rigid registration of medical images based on mutualinformation maximization

Edoardo Ardizzone, Orazio Gambino, Marco La Cascia, Liliana Lo Presti, and Roberto PirroneUniversita di Palermo - Dipartimento di Ingegneria Informatica

Viale delle Scienze, 90128, Palermo, Italylopresti@csai.unipa.it

Abstract

In this paper, a new multi-modal non-rigid registrationtechnique for medical images is presented. Firstly, the reg-istration problem is outlined and some of the most com-mon approaches reported, then, the proposed algorithm ispresented. The proposed technique is based on mutual in-formation maximization and computes a deformation fieldthrough a suitable globally smoothed affine piecewise trans-formation. The algorithm has been conceived with particu-lar attention to computational load and accuracy of results.Experimental results involving intra-patient, inter-patientsand atlas images on brain CT and MR (T1, T2 and PDmodalities) are reported.

1. Introduction

Given two images, the floating image f and the referenceimage g, image registration is the process aiming at findingthe geometric transformation T that applied to f producesa very similar image to g that is usually called registeredimage r. Image registration is an important tool in medicalimaging. Registering a couple of medical images for exam-ple can help physician to monitor the course of disease inpatient (for example in Alzheimer and in multiple sclero-sis cases), to compare the anatomical structures of differentpatients or to study anomalies in groups of individuals com-paring their structures with those of a medical image atlas.To measure how similar are the registered and the referenceimages, a similarity metric is needed. The metric is thenused to guide the registration process in determining theoptimal transformation. Bi-dimensional geometric transfor-mation, similarity criteria and optimization algorithm arethe main components of a registration technique [1]. Thesuccess of the registration process arises from the chosencomponents, that are domain problem dependent and de-terminant for execution time. Strategies are also needed to

speed up the convergence to the transformation realizing theoptimal alignment. The base registration algorithm usuallyconsists of iterative optimization of transformation param-eters based on some similarity measure and on some exitcondition. In the following, we describe the main issues no-ticeable in registration problem and several useful strategiesto face with them. Then we describe the components andthe main characteristics of the proposed registration algo-rithm. In particular, we propose a non-rigid registration inwhich the deformation field is computed to improve globalsimilarity by maximizing sub-images local similarities; be-sides, our implementation combines properly several com-mon methods in order to perform the registration process.

2. Related works

As the registration process aims at aligning anatomicalstructures, many algorithms take advantage of prior infor-mation about their position [1]. For example it is possibleto define a set of points or surfaces correspondences be-tween the two images and use this information to extendthe correspondences to all other pixels in the floating im-age by interpolation [5]. These methods are usually namedlandmark and surface based registration. It is necessaryto point out that identifying the surfaces in an image is noteasy in all modalities. For example in functional exams likePET [11] manual recognition is often required. To deter-mine the displacement of each point in the deformed image,many methods make use of forces fields [13]; others assumesome physical model for the registration problem and solvethe correspondent partial differential equation to find thegeometric transformation. For example, in viscous fluid ap-proach image is modeled like a fluid and deformation comesfrom the solution of the Navier-Stokes equation [4]. As re-gard to the optimization method, many numeric algorithmssuch as Nelder-Mead, Levenberg-Marquardt or Powell canbe used to find transformation parameters. Moreover, be-cause of computational complexity of the problem, regis-

14th International Conference on Image Analysis and Processing (ICIAP 2007)0-7695-2877-5/07 $25.00 © 2007

tration is often realized using a multi-resolution approach.A Gaussian pyramid of the initial image is constructed andthe estimation of transformation parameters is done for eachpyramidal level in a coarse to fine way. In [11], severalmulti-resolution optimization strategies for multimodal reg-istration are compared. Other registration methods find cor-respondences based on the grey levels in the two images[1]. In these cases, there are no assumption about the ge-ometric forms and spatial positions of the structures in thetwo images but it is only assumed that there is some rela-tion among their intensity distribution functions. In thesecases it makes sense to distinguish between mono-modaland multi-modal registration according to the type of im-ages to which the method is applied to. The former refersto those algorithms realized for images acquired with thesame type of exam (for example MR-PD); the latter refersto those realized for images acquired with different modal-ities (for example MR-PD versus MR-T1). In mono-modalregistration the distance between grey levels of each point inthe two images is usually minimized. In multi-modal reg-istration it is preferable to use criteria based on statisticalproperties of the grey levels distribution function. Mutualinformation (MI) and its variants are maybe the most pop-ular similarity criteria used in multimodal non-rigid regis-tration. Wells et al. [16] were the first to introduce MI torigidly register multi-modal volumes of the same patient. In[10], Maes et al. describe another multimodal rigid registra-tion to align intra-patient volumes. Their implementation isbased on MI maximization too but grey levels probabili-ties are estimated like relative frequencies. Multi-resolutionstrategy is also adopted to speed up performance. There areonly a few implementation in which MI is used in a fluid-viscous approach. For example in [4] the authors proposean hybrid MI-based fluid algorithm that combines the effi-cient smoothed deformation model offered by the viscousapproach with the advantage offered by MI to align inten-sity patterns without no other morphological information.In [15], to handle computational complexity, Thevenaz et al.implements a geometric multi-resolution strategy to realizea MI maximization based registration. In [12], an initialrigid transformation roughly aligns the two images throughPowell’s optimization of MI. Evaluation of MI is done bytri-linear partial volume interpolation. Then, an elastic reg-istration is carried out by subdividing the rigidly registeredfloating image in sub-images and valuing the translationfactor that maximizes their local MI resulting in a vectorfield of local displacements. In [4], Fookes et al. reportan algorithm in which displacements in a few points are es-timated by a block-matching strategy and then are spreadto the whole image by Gaussian convolution. The mostimportant consequence is that the resulting deformation issmoothed. Similarly in [13], Pekar et al. use a set of controlpoints distributed on an irregular grid on the image. At these

points, Gaussian-shaped forces are applied and the image istreated like an elastic medium that reacts to them. Finally,in [9], a segmentation guided registration algorithm is pro-posed. A level set method is used to find the contours of themain anatomical structures in an image and these informa-tion are then used to guide the whole deformation processof the floating image.

3. Geometric transformation

It is very common to distinguish registration algorithmsbased on the geometric transformation adopted and in par-ticular between rigid and non-rigid registration [1]. Rigidregistration uses a rigid body deformation and consists ofapplying rotational and translational transformations to thefloating image. As a consequence all distances and pro-portions between structures of the floating image remainunchanged in the registered one [3, 6, 17]. This type ofregistration is very useful when it is necessary to compareimages from the same patient taken in different moments,with different modalities or positions. Figure 1 shows anexample of rigid registration.

Non-rigid registration techniques use also scaling andshearing transformations so proportions and distances be-tween structures usually changes and collinearity is notguaranteed. Non-rigid registration is done through affine orelastic transformations. In particular, an elastic transforma-tion produces local changes on pixels of the floating image.This type of registration is more adequate when it is nec-essary to compare images from different patients or to anatlas.

According to the type of geometric transformation, it isalso possible to distinguish between parametric and non-parametric transformation. Parametric transformations arefully defined by a finite number of parameters while non-parametric transformations are defined by a deformationfield. Parametric transformations can be global or local. Asglobally defined transformations are usually inadequate torecover local deformations of the whole image affine piece-wise transformation [2, 7, 17] realizing a mapping betweencorrespondent triangles of two point grids defined on thefloating image and on the registered one are often used.

Figure 1. Rigid registration: floating image,reference image and rigidly registered image.

14th International Conference on Image Analysis and Processing (ICIAP 2007)0-7695-2877-5/07 $25.00 © 2007

3.1. Affine transformation

Let (u, v) a pixel of the deformed image after the trans-formation while (x, y) is the corresponding pixel in thefloating image. If the applied transformation is affine, thenthe mapping is expressed by the following equation: x

y1

=

a b cd e f0 0 1

·

uv1

3.2. Deformation field

In non-parametric methods, a displacement vector isfound for each point of the floating image [13]. Allthese displacements are iteratively evaluated and memo-rized through a so called deformation field [4, 7, 12]. Inmany implementations, it comes from solving a partial dif-ferential equation representing a physical model for the reg-istration process [4, 12]. When deformation field is used,transformation is applied with the following equations:

x = u + δu

y = v + δv

Displacements δu and δv are iteratively obtained and atthe k-th step, displacements value:

δuk = δuk−1 + ∆u

δvk = δvk−1 + ∆v

where ∆u and ∆v are computed in such a way that the reg-istered and the reference images are more and more similar.

4. Similarity Criteria

Similarity criteria is critical in the image registration pro-cess because it measures the quality of the process itself.Metrics based on statistical information assume that imageintensity is a random variable and assign a probability to ev-ery possible value. Probabilities can be assigned defining aParzen window estimator [15] or by relative frequencies ofintensity in the image. To limit computational complexity,we have chosen to use the latter method: we evaluate thejoint histogram of the deformed and the reference imagesrepresenting the joint probability that a couple of grey levelsis in the two images. Prior probabilities for each image greylevels can be obtained summing joint histogram per columnand per row. To align intensity patterns, we use the criteriaof mutual information [16]. It states that MI is maximumwhen the two images are correctly aligned. MI uses statis-tical information directly obtained from the images during

the registration process and its application does not requireany information about position of anatomical structures. Weonly assume that there is some relation between grey levelsof the two images. MI comes from information theory andmeasures the quantity of information that an image containsof another. It can be expressed in terms of entropies of theregistered and the reference images. Let p and q the randomvariables of the grey levels in the two images r and g. Theentropies of the floating and the reference images, H(r) andH(g) respectively, result by the following equations:

H(r) = −L−1∑p=0

Pr(p) · log(Pr(p))

H(g) = −L−1∑q=0

Pg(q) · log(Pg(q))

In these equations Pr(p) and Pg(q) are the probabilitiesassigned to grey levels p and q of the registered and thereference images with p and q in the range [0 . . . L − 1]where L is the maximum grey level. Joint entropy resultsfrom:

H(r, g) = −L−1∑p=0

L−1∑q=0

Prg(p, q) · log(Prg(p, q))

where Prg(p, q) is the probability that the grey level q inthe reference image g correspond to grey level p in the reg-istered image r. Mutual information MI is defined as:

MI(r, g) = H(r) + H(g)−H(r, g)

5. Proposed algorithm

Our approach realizes registration through a sequence ofsteps. Figure 2 shows a diagram of the whole image regis-tration process.

Figure 2. Block diagram to represent our reg-istration method.

Experiments show MI is very sensitive to noise; to re-duce this drawback and improve the method, we apply aninitial pre-processing to eliminate background by maskingthe two images. In figure 3, there is an example of noisyimage before our pre-processing was applied and the maskused to cut background.

14th International Conference on Image Analysis and Processing (ICIAP 2007)0-7695-2877-5/07 $25.00 © 2007

Figure 3. Noisy MR-PD image from brain vol-ume and its correspondent mask.

To speed up convergence, first we apply a global affineregistration, then a non rigid registration. This strategy, ap-plied in [8] too, reduces initial large misalignments. De-formation field is then initialized with displacements result-ing from the estimated affine global transformation. Thechoice of applying a deformation field arises from its sim-plicity, its high degree of freedom in deforming image andin its easyness in inverse transforming. We have applied aglobally smoothed affine piecewise transformation. Unlikethe common piecewise transformation, we do not use gridpoints or triangulation but, rather, we iteratively transformthe floating image by a sequence of local optimum affinetransformation. In particular, deformation field is evaluatedin a coarse to fine way i.e. effects and intensity of eachcomputed transformation reduce throughout the executionbecoming more and more local to align fine details. To dothis, we divide the floating image in several sub-images par-tially overlapping. For each sub-image, the local optimumaffine transformation aligning it to the reference image iscomputed by maximizing local measure of MI. Then wesuitably propagate the transformation to the whole imageupdating the deformation field in the following manner:

∆u = [T (u + δuk−1)− (u + δuk−1)] ·Ψ(d)∆v = [T (v + δvk−1)− (v + δvk−1)] ·Ψ(d)

i.e. the local transformation is applied to all points of theimage; the resulting displacements are weighted with a suit-able function Ψ(d) to smooth the transformation and toavoid blocking artifact; the resulting ∆u and ∆v are thenused to update the deformation field. Figure 4 shows a com-parison among global affine transformation, affine piece-wise transformation and globally smoothed affine piecewisetransformation.

Ψ(d) is a 2nd-order Butterworth weight function (seefig. 5) centered on the block-center, with cutoff frequencyd0 equal to half of the block side length. Each local transfor-mation results from an optimization process in which localMI is maximized through a Nelder-Mead optimizer. Param-eters are valued with respect to those of the identity trans-formation. Local affine transformation is applied to the nor-malized coordinates of each point.

(a) (b)

(c) (d)

Figure 4. Globally smoothed affine piecewisetransformation a) the floating image. b) theimage after the affine transformation T. c,d)the deformed image when T is applied only tothe second quadrant and when weight func-tion is also applied.

Figure 5. Weight function: 2nd order Butter-worth function centered in the first quadrant

6. Implementation details

In developing the algorithm, we have taken particularcare to the execution time. To speed up convergence, wehave adopted a spatial multi-resolution strategy. In particu-lar, a three levels pyramid has been constructed from float-ing and reference images. Sub-sampling has been done withsteps 2, 4, and 8. For each pyramidal level, deformationfield has been evaluated. Moreover, sub-sampling reducesfine details in the images and favors coarse registration atpyramidal bottom level. A drawback of multi-resolution isthe need to suitably propagate deformation to each pyrami-dal level. We have constructed a deformation field pyramidwhose levels are the deformation field for the image at thecorresponding level in the image pyramid. When a localtransformation has been found, it is used to appropriatelyupdate each level of the deformation field pyramid. A prob-lem arises from the way in which updating must correctlybe done; indeed, the weight function is influenced by thescale factor due to sub-sampling. Propagation of the de-formation obtained at each pyramidal level to all the otherlevels requires particular care [14]. So, the weight function

14th International Conference on Image Analysis and Processing (ICIAP 2007)0-7695-2877-5/07 $25.00 © 2007

Ψ(d) must be adjusted to take into account this scale factorin the following manner:

Ψ(d) =1

1 +(

dλ·d0

)4

where λ = 2k−1 and k is the pyramidal level numberwith respect to the current one. At each pyramidal levelthe number of blocks in which the image has been dividedchanges; in particular, it increases with the image spatialresolution to make more and more local the transformation.Blocks are partially overlapped i.e. each one overlaps for ahalf the nearest four blocks. Blocks subdivision and multi-resolution decrease the number of samples used to evaluateMI reducing the accuracy of the similarity measure. Par-tial overlap has the effect to increase the number of sam-ples and the coherence of the transformation that, in thisway, take into account the transformation produced in theprevious iteration. To speed execution and to improve ouralgorithm, we also use grey levels quantization. It resultsin a better statistic of grey levels and a more careful MImeasure. Moreover, quantization conveniently controls thedetail level reducing details less meaningful at bottom pyra-midal levels. Quantization degree is selected on the basisof the pyramidal level and of the blocking resolution in acoarse to fine way.

7. Experimental results

Different experiments have been conducted to evaluatethe performance of the proposed algorithm both on syn-thetic and real data. Preliminarily experiments have beenconducted on synthetic images representing simple forms(circles, quadrilaterals . . . ) in different grey levels to simu-late multi-modal images. They have shown the robustnessof the method and the coherence of our geometric trans-formation. For test on real data we used 8-bit images with256 × 256 spatial resolution acquired by different sources.Many experiments on mono-modal and multi-modal intra-patient images were carried out. In these cases, imagescame from some volumes of brain MR (T1, T2 and PDmodalities). These volumes have been acquired by Gen-eral Electric Signa 1.5 Tesla, scanning sequence was SpinE-cho, flip angle 90◦, slice thickness of 4 mm, while repeti-tion time was of 400 ms and 1800 ms for T1 and T2/PDmodalities respectively; Echo time was of 90 ms and 15 msfor PD and T1/T2 modalities respectively. Experiments ontwo consecutive slices showed the ability of the algorithmto recover little misalignments in the two images. Inter-patient and atlas registrations have also been tested. At-las come from the Simulated Brain Database (SBD), de-fault parameters where used to generate these volumes. Totest robustness and reliability of our method, some images

have been extracted from Alzheimer and multiple sclero-sis diseased volumes available on the whole brain atlas athttp://www.med.harvard.edu/AANLIB/home.html. Theseexperiments showed that our algorithm produces accurateresults when severe diseases are evident too. Table 1 showssome results obtained with our algorithm.

Our implementation has been realized in Matlab envi-ronment; execution time varies according to misalignments.The average execution time is of 15 minutes on a PentiumM 760, 2Ghz, 1GB of RAM.

8. Conclusions and future works

A new registration algorithm has been proposed in whichdeformation is iteratively computed maximizing local mu-tual information. Testing has been conducted on multi-modal brain images. Good results have also been obtainedwith inter-patient volumes. Experiments have shown a limitto our similarity measure: in spite of all the techniques im-plemented to try to improve its robustness, MI occasionallyassumes a value that does not represent the really similaritybetween the two images. Likely, this arises from the lackof samples and from the presence of some local minimumin MI local measure. In these cases, coherent deformationfield is not guaranteed. To deal with this problem, a suitablecontrol on the transformation is done before updating defor-mation field. In particular, local transformation is acceptedand globally applied only whether global MI increases. Thisimproves our method limiting only partially the deforma-tion freedom. In future works we will investigate the pos-sibility of improving our method modifying the similaritymeasure. In particular, we could incorporate anatomical in-formation in the metric to obtain an hybrid method aligningintensity pattern guided by anatomical structures informa-tion.

9. Acknowledgements

For the volumes used to test our algorithm, we acknowl-edge Dr. T. Piccoli of Neurology Institute of University ofPalermo.

References

[1] W. R. Crum, T. Hartkens, and D. L. G. Hill. Non-rigid im-age registration: theory and practice. The British Journal ofRadiology, 77:140–153, 2004.

[2] D. N. Fogel and L. R. Tinney. Image registration usingmultiquadric functions, the finite element method, bivariatemapping polynomials and thin plate spline. NCGIA 96-1,National Center for Geographic Information and Analysis,1996.

14th International Conference on Image Analysis and Processing (ICIAP 2007)0-7695-2877-5/07 $25.00 © 2007

Floating image Reference image Registered image Note

Inter-patientregistration

MR-PD versusMR-T1

Inter-patientregistration

MR-PD versusMR-T1

Inter-patientregistration

MR-T1 versusMR-PD

Atlas basedregistration

MR-T2 versusMR-T1

Atlas basedregistration

MR-T1 versusMR-T2

Inter-patientregistration CTversus MR-T2

Inter-patientregistration

MR-T2 versusMR-PD

Inter-patientregistration

MR-PD versusMR-T2

Inter-patientregistration CT

versus CT

Table 1. Experimental results.

[3] J. D. Foley, A. van Dam, S. K. Feiner, and J. F. Hughes.Computer Graphics: principles and practice. Addison-Wesley, 2001.

[4] C. Fookes and A. Maeder. Comparison of popular non-rigid image registration techniques and a new, hybrid mu-tual information-based fluid algorithm. In Proc. of APRSWorkshop on Digital Image Computing, Brisbane, Australia,2003.

[5] M. Fornefett, K. Rohr, and H. S. Stiehl. Elastic registrationof medical images using radial basis function with compactsupport. In Proc. of IEEE Conference on Computer Visionand Pattern recognition, pages 402–407, Fort Collins, CO,1999.

[6] C. A. Glasbey and K. V. Mardia. A review of image warpingmethods. Journal of Applied Statistics, 25:155–171, 1998.

[7] R. C. Gonzalez and R. E. Woods. Digital image processing- Second Edition. Prentice Hall Inc., 2002.

[8] J. Kohlrausch, K. Rohr, and H. S. Stiehl. A new class of elas-tic body splines for nonrigid registration of medical images.Journal of Mathematical Imaging and Vision, 23(3):253–280, 2005.

[9] J. Liu. Deformable model-based image registration. Tech-nical report, School of Electrical Engineering and ComputerScience, Ohio University, 2006.

[10] F. Maes, A. Collignon, D. Vandermeulen, and et al. Multi-modality image registration by maximization of mutual in-formation. IEEE transactions on medical imaging, 16(2),1997.

[11] F. Maes, D. Vandermeulen, and P. Suetens. Compara-tive evaluation of multiresolution optimization strategies formultimodality image registration by maximization of mutualinformation. Medical Image analysis, 3(4):373–386, 1999.

[12] J. B. A. Maintz, E. H. W. Meijering, and M. A. Viergever.General multimodal image registration based on mutual in-formation. Technical report, Image Sciences Institute, Uni-versity Medical Center Utrecht, Holland, 1998.

[13] V. Pekar, E. Gladilin, and K. Rohr. An adaptive irregulargrid approach for 3d deformable image registration. Physicsin Medicine and Biology, 51:361–377, 2006.

[14] P. Thevenaz, U. E. Ruttimann, and M. Unser. A pyramidapproach to subpixel registration based on intensity. IEEETransactions on Image Processing, 7(1), 1998.

[15] P. Thevenaz and M. Unser. Optimization of mutual informa-tion for multiresolution image registration. IEEE Transac-tions on Image Processing, 9(19), 2000.

[16] W. M. Wells, P. Viola, and et al. Multi-modal volume reg-istration by maximization of mutual information. MedicalImage analysis, 1(1):35–51, 1996.

[17] L. Zagorchev and A. Goshtasby. A comparative study oftransformation functions for nonrigid image registration.IEEE Transactions on Image Processing, 15(3):529–538,2006.

14th International Conference on Image Analysis and Processing (ICIAP 2007)0-7695-2877-5/07 $25.00 © 2007