AdvancedNonrigidRegistrationAlgorithmsfor

ImageFusion

SimonK. Warfield AlexandreGuimond Alexis Roche

Aditya Bharatha Alida Tei Florin Talos JanRexilius

JuanRuiz-Alzola Carl-FredrikWestin StevenHaker

SigurdAngenent Allen Tannenbaum FerencA. Jolesz

RonKikinis

1

Intr oduction

Medical imagesarebroughtinto spatialcorrespondence,or aligned, by theuseof registration

algorithms. Nonrigid registrationrefersto the setof techniquesthat allow the alignmentof

datasetsthat aremismatchedin a nonrigid, or nonuniformmanner. Suchmisalignmentscan

resultfrom physicaldeformationprocesses,or canbea resultof morphologicalvariability. For

example,physicaldeformationin thebraincanoccurduring neurosurgeryasa resultof such

factorsasswelling,cerebrospinalfluid (CSF)loss,hemorrhageandtheinterventionitself. Non-

rigid deformationis alsocharacteristicof theorgansandsoft tissuesof theabdomenandpelvis.1This chapterappearedin Brain Mapping: The Methods,SecondEdition, aschapter24 on pages661–690,

publishedby AcademicPressof SanDiego,USA in 2002.

In addition,nonrigidmorphologicaldifferencescanarisewhencomparisonsaremadeamong

imagedatasetsacquiredfrom different individuals. Thesechangescanbe a resultof normal

anatomicalvariability or theproductof pathologicalprocesses.Becausethegrossstructureof

the brain is essentiallysimilar amonghumans(andeven amongrelatedspecies),the factors

describedabovetendto producelocal nonrigidshapedifferences.

Nonrigidbrainregistrationtechniqueshavenumerousapplications.They havebeenusedto

alignscansof differentbrains,permittingthecharacterizationof normalandpathologicalmor-

phologicalvariation(brainmapping).They have alsobeenusedto align anatomicaltemplates

with specificdatasets,thusfacilitatingsegmentation(i.e. segmentationby registration).More

recently, thesetechniqueshavebeenusedto capturechangeswhichoccurduringneurosurgery.

With theongoingdevelopmentof robustalgorithmsandadvancedhardwareplatforms,further

applicationsin surgical visualizationandenhancedfunctionalimageanalysisareinevitable.

Oneexciting applicationof nonrigidregistrationalgorithmsis in theautomaticregistration

of multimodalimagedata.Rigid registrationof multimodaldatahasbeengreatlyfacilitatedby

the framework provided by mutual information(MI). However, MI-basedstrategiesto effec-

tively capturelargenonrigidshapedifferencesarestill beingexplored. An alternateapproach

is to normalizemultimodality imagesandthusreducetheproblemto a monomodalitymatch.

In thefirst section,wepresentanonrigidregistrationmethodwhichusesanintensitytransform

which allows a singleintensityin onemodalityto bemappedonto(up to) two intensities.The

methodis iterative, combiningin eachiterationan intensitycorrectionanda geometrictrans-

form using intensity-similaritycriteria. The methodis appliedin two caseswith promising

results.

In thenext section,we turn our attentionto theissueof imageregistrationandfusiondur-

ing neurosurgery. It is commonto desireto align preoperative datawith imagesof thepatient

acquiredduringneurosurgery. It is now widely acknowledgedthatduringneurosurgicalopera-

2

tions,nonrigidchangesin theshapeof thebrainoccurasa resultof theinterventionitself and

dueto reactive physiologicalchanges.Thesedeformations(“brain shift”) make it difficult to

relatepreoperative imagedatato theintraoperativeanatomyof thepatient.Sincepreoperative

imagingis not subjectto thesametime constraintsandlimitations in tissuecontrastselection

methodsasintraoperative imaging,a majorgoalhasbeento developrobustnonrigidregistra-

tion algorithmsfor matchingof preoperative imagedataonto intraoperative imagedata. We

presentourbiomechanicalmodelingalgorithmwhichcancapturenonrigiddeformationsbased

on surfacechangesandinfer volumetricdeformationusinga finite elementdiscretization.We

alsodescribeour early prospective experienceusing the methodduring neurosurgical cases,

andprovideexamplesof theenhancedvisualizationswhichareproduced.

In thethird section,webuild uponthethemeof physics-basedmodelsby presentinganovel

inhomogeneouselasticitymodelwhichusesalocalsimilarity measureto obtainaninitial sparse

estimateof thedeformationfield. Themethodincludesautomaticfeaturepoint extractionus-

ing a nonlineardiffusion filter. Correspondencedetectionis achieved by maximizinga local

normalizedcross-correlation.The sparseestimatesof the deformationfield calculatedat the

featurepointsarethenintroducedasexternalforces,restrictingtheregistrationprocesssothat

thedeformationfield is fixedat thosepoints.An advantageof themethodis thatfeaturepoints

andcorrespondencesareestablishedautomatically. Thusneithersegmentationnor themanual

identificationof correspondencesis required.

In thefourth sectionwe discussregistrationof Dif fusionTensorMRI dataandintroducea

framework for nonrigid registrationof tensordata(including thespecialcaseof vectordata).

The approachis basedon a multiresolutiontemplatematchingschemefollowed by interpo-

lation of the sparsedisplacementfield usinga Kriging interpolator. After warping the data,

the tensorsare locally realignedbasedon information from the deformationgradientof the

displacement.

3

In the fifth section,we presenta novel methodfor producingareapreservingsurfacede-

formations,andmore generalmasspreservingareaandvolumedeformations,basedon the

minimizationof a functionalof Monge–Kantorovich type. Thetheoryis basedaroundtheop-

timal masstransportproblemof minimizing thecostof redistributingacertainamountof mass

betweentwo distributionsgivena priori. Herethecostis a functionof thedistanceeachbit of

materialis moved,weightedby its mass.Theproblemof optimaltransportis classicalandhas

appearedin econometrics,fluid dynamics,automaticcontrol,transportation,statisticalphysics,

shapeoptimization,expert systems,andmeteorology. We show how the resultinglow-order

differentialequationsmaybeusedfor imageregistration.

The challengeof nonrigid registrationremainsoneof the outstandingopenproblemsin

medicalimageanalysis.New algorithmdevelopments,often targetedtowardspecificclinical

applications,havehelpedto identify furtherunsolvedissues.Thischapterprovidesanoverview

of thenonrigidregistrationalgorithmsbeingpursuedtodayattheSurgicalPlanningLaboratory.

4

1 Inter -Modality and Multi-Contrast Images

1.1 Intr oduction

Automatic registrationtechniquesof brain imageshave beendevelopedfollowing two main

trends: 1) registrationof multimodal imagesusing low to intermediatedegreetransforma-

tions (lessthan a few hundredparameters),and2) registrationof monomodalimagesusing

high-dimensionalvolumetricmaps(elasticor fluid deformationswith hundredsof thousands

of parameters).Thefirst category mainly addressesthe fusionof complementaryinformation

obtainedfrom different imaging modalities. The secondcategory’s predominantpurposeis

the evaluationof either the anatomicalevolution processpresentin a particularsubjector of

anatomicalvariationsbetweendifferentsubjects.Despitepromisingearlywork suchas(Hata,

1998),densetransformationfield multimodalregistrationhas,sofar, remainedrelatively unex-

plored.

Researchonmultimodalregistrationculminatedwith theconceptof mutualinformation(MI)

(Viola andWells, 1995;Wells et al., 1996b;Hataet al., 1996;Viola andWells, 1997;Maes

et al., 1997),leadingto a new classof rigid/affine registrationalgorithms.In this framework,

the registrationof two imagesis performedby maximizingtheir MI with respectto thetrans-

formationspace.A significantreasonfor thesuccessof MI asasimilarity measureresidesin its

generality, asit doesnotuseany prior informationabouttherelationshipbetweentheintensities

of the images.For instance,MI doesnot assumea linear relationshipasis typically thecase

in standardopticalflow techniques.Also, unlikesomeearlierapproaches,MI doesnot require

theidentificationof correspondingfeaturesin theimagesto beregistered.

Significantwork hasbeendonein establishingthe applicability of MI for nonrigid regis-

tration (Gaenset al., 1998;Maintz et al., 1998;Meyer et al., 1999;Likar andPernus,2000;

Hellier andBarillot, 2000;Rueckert et al., 2000;Hermosilloet al., 2001). Someauthorshave

5

further improvedtherobustnessof theapproachby modifying theoriginal MI measure,either

by including someprior information on the joint intensitydistribution (Maintz et al., 1998;

Likar andPernus,2000),or by usinghigher-orderdefinitionsof MI which incorporatespatial

information(Rueckert et al., 2000).

Ourapproachdescribedherestemsfrom theobservationthatanumberof multimodalrigid

registrationproblemscanbesolvedin practiceusingothersimilarity measuresthanMI, oneof

which is thecorrelationratio(CR)(Rocheetal.,1998).TheCRis muchmoreconstrainedthan

MI asit assumesa functional,thoughnon-linear, relationshipbetweentheimageintensities.In

otherwords,it assumesthatoneimagecouldbemadesimilar to theotherby asimpleintensity

remapping.Thus,theCR methodamountsto anadaptiveestimationstrategy whereoneimage

is alternatelycorrectedin intensityandin geometryto progressively matchtheother.

For mostcombinationsof medicalimages,thefunctionaldependenceassumptionis gener-

ally valid for amajorityof anatomicalstructures,but not for all of them.Althoughthisproblem

doesnot turn out to be critical in a rigid/affine registrationcontext, we observe that it may

seriouslyhampertheestimationof ahigh-dimensionaltransformation.Weproposehereanex-

tensionof thefunctionaldependencemodel,which we call thebifunctionalmodel,to achieve

betterintensitycorrections.While thebifunctionalmodelis morerealisticthanthefunctional

one,it remainsstronglyconstrainedandthusenablesa goodconditioningof the multimodal

non-rigidregistrationproblem.

1.2 Method

The registrationalgorithmdescribedhereis iterative andeachiterationconsistsof two parts.

The first part transformsthe intensitiesof anatomicalstructuresof a sourceimageS so that

they matchtheintensitiesof thecorrespondingstructuresof a target imageT. Thesecondpart

is concernedwith theregistrationof S (after intensitytransformation)with T usinganoptical

6

flow algorithm.

1.2.1 Intensity Transformation

Theintensitycorrectionprocessstartsby definingthesetC of intensitypairsfrom correspond-

ing voxelsof T andS. Hence,thesetC is definedas

C Sx T x ;1 x N (1)

whereN is thenumberof voxelsin theimages.Sx andT

x correspondto theintensityvalue

of thexth voxel of SandT, respectively, whenadoptingthecustomaryconventionof consider-

ing imagesasone-dimensionalarrays.Weshallnow show how to performintensitycorrection

if wecanassumethatasingleintensityvaluein Shaseither1) exactlyonecorrespondinginten-

sity valuein T (monofunctionaldependence)or 2) at leastoneandat mosttwo corresponding

intensityvaluesin T (bifunctionaldependence).

Monofunctional DependenceAssumption Our goal is to characterizethe mappingfrom

voxel intensitiesin S to thosein T, knowing that someelementsof C areerroneous,i.e. that

would not be presentin C if S andT were perfectlymatched. Let us assumeherethat the

intensityin T is a functionof theintensityin Scorruptedwith anadditivestationaryGaussian

whitenoiseη,

Tx f

Sx η

x (2)

where f is an unknown function to be estimated. This is exactly the model employed by

Rocheet al. (Rocheet al., 2000) which leadsto the correlationratio as the measureto be

maximizedfor registration.In thatapproach,for agiventransformation,oneseeksthefunction

thatbestdescribesT in termsof S. In a maximumlikelihoodcontext, this function is actually

a leastsquares(LS) fit of T in termsof S.

7

Themajordifferencebetweenour respective problemsis thatwe seeka high-dimensional

geometricaltransformation.As opposedto affine registrationwherethetransformationis gov-

ernedby the majority of goodmatches,elasticdeformationsmay be computedusingmainly

local information(i.e. gradients,local averages,etc.).Hence,wecannotexpectgooddisplace-

mentsin onestructureto correctfor badonesin another;wehaveto makecertaineachvoxel is

movedproperlyat eachiteration.For this,sincethegeometricaltransformationis foundusing

intensitysimilarity, themostpreciseintensitytransformationis required.Consequently, instead

of performinga standardLS regression,we have optedfor a robust linearregressionestimator

whichwill removeoutlyingelementsof C duringtheestimationof theintensitytransformation.

To estimatef weusetheleasttrimmedsquares(LTS)methodfollowedby abinaryreweighted

leastsquares(RLS) estimation(RousseeuwandLeroy, 1987). The combinationof thesetwo

methodsprovidesaveryrobustregressiontechniquewith outlierdetection,while ensuringthat

amaximumof pertinentvoxel pairsaretakeninto account.

Dif ferenttypesof functionscanbeusedto model f . In (Guimondetal.,2001)wemadeuse

of polynomialfunctions.TheintensitycorrespondencesbetweenT andS is thendefinedas:

Tx θ0 θ1S

x θ2S

x 2 θpS

x p (3)

whereθ θ0 θp needsto be estimatedand p is the degreeof the polynomial function.

This model is adequateto register imagesthat have a vastrangeof intensities;the restricted

polynomialdegreeimposesintensityspaceconstraintson thecorrespondences,mappingsimi-

lar intensitiesin S to similar intensitiesin T.

In the casethat S is a labeledimage,neighboringintensitiesin S will usuallycorrespond

to differentstructures.Hencethe intensityspaceconstraintis no longer required. f is then

modeledasa piecewiseconstantfunction,suchthateachlabelof S is mappedto theLTS/RLS

estimateof intensitiescorrespondingto thatlabelin T.

8

Bifunctional DependenceAssumption Functionaldependenceasexpressedin (2) andin (3)

impliesthattwostructureshaving similarintensityrangesin Sshouldalsohavesimilar intensity

rangesin T. With somecombinationsof images,this is a crudeapproximation.For example,

CSFandbonesgenerallygivesimilarintensityvaluesin T1-weightedimages,while they appear

with very distinct valuesin PD-weightedscans. Conversely, CSF and gray matterare well

contrastedin T1-weightedimages,while they correspondto similar intensitiesin PD weighted

scans.

To circumvent this difficulty, we have developeda strategy thatenablesthemappingof an

intensityvaluein S to not only one,but two possibleintensityvaluesin T. This methodis a

naturalextensionof theprevioussection.Insteadof computinga singlefunctionthatmapsthe

intensitiesof Sto thoseof T, two functionsareestimatedandthemappingbecomesaweighted

sumof thesetwo functions.

We startwith theassumptionthat if a point hasanintensitys in S, thecorrespondingpoint

in T hasanintensityt that is normallydistributedaroundtwo possiblevaluesdependingon s,

f1s and f2

s . In statisticalterms,this meansthat given s, t is drawn from a mixture of

Gaussiandistribution,

Pt s π1

s N

f1s σ2 π2

s N

f2s σ2 (4)

whereπ1

s andπ2

s 1 π1

s aremixing proportionsthatdependon the intensityin the

sourceimage,andσ2 representsthevarianceof thenoisein thetarget image.Consistentwith

the functional relationship,we will restrict ourselves to polynomial intensity functions, i.e.

f1s θ0 θ1s θ2s2 θpsp, and f2

s ψ0 ψ1s ψ2s2 ψpsp.

An intuitive way to interpretthis modelingis to statethat for any voxel, thereis a binary

“selector”variableε 1 2 thatwould tell us,if it wasobserved,which of thetwo functions

f1 or f2 actuallyservesto maps to t. Without knowledgeof ε, thebestintensitycorrectionto

9

applyto S(in theminimumvariancesense)is aweightedsumof thetwo functions,

fs t P

ε 1 s t f1

s P

ε 2 s t f2

s (5)

in which theweightscorrespondto theprobabilitythatthepointbemappedaccordingto either

thefirst or thesecondfunction.To estimatethefunctions,weemploy asequentialstrategy that

performstwo successive LTS/RLSregressionsasin themonofunctionalcase.Detailson how

theotherparametersaredeterminedcanbefoundin (Guimondet al., 2001).

1.2.2 GeometricalTransformation

Having completedthe intensity transformationstage,we endup with an intensitycorrected

versionof the sourceimage,which will be denotedS . In the monofunctionalcaseS x fSx andin thebifunctionalcaseS x f

Sx T x . We mayassumethatS is roughly

of thesamemodality asT in thesensethat correspondinganatomicalstructureshave similar

intensitiesin S and T. The geometricaltransformationproblemmay then be treatedin a

monomodalregistrationcontext.

Many algorithmshave beendevelopedthatdeformonebrain so its shapematchesthatof

another(Maintz andViergever, 1998; Toga,1999). The procedureusedherewasinfluenced

by a varietyof opticalflow methods,primarily thedemonsalgorithm(Thirion, 1995;Thirion,

1998).At agiveniterationn, eachvoxel x of T is displacedaccordingto avectorvn

x soasto

matchits correspondinganatomicallocationin Sn. Weusethefollowing scheme:

vn 1

x Gσ vn Sn hn

x T

x!

∇Sn hn

x ! 2 Sn hn

x" T

x 2∇Sn hn

x#$ (6)

whereGσ is a 3D Gaussianfilter with isotropic varianceσ2, denotesthe convolution, denotesthecomposition,∇ is thegradientoperatorandthe transformationhn

x is relatedto

10

thedisplacementby hn

x x vn

x . As is commonwith registrationmethods,wealsomake

useof multilevel techniquesto accelerateconvergence.Detailsaboutthenumberof levelsand

iterationsaswell asfilter implementationissuesareaddressedin Section1.3. We show here

how ourmethodcanberelatedto threeotherregistrationmethods:theminimizationof thesum

of squareddifference(SSD)criterion;opticalflow; and,thedemonsalgorithm.

1.2.3 Relation to SSDMinimization

In theSSDminimizationframework, onesearchesfor thetransformationh thatminimizesthe

sumof squareddifferencesbetweenthe transformedsourceimageandthe target image. The

SSDis thendefinedas:

SSDh 1

2

N

∑x% 1

S hx T

x 2 (7)

Theminimizationof (7) maybeperformedusinga gradientdescentalgorithm. By differ-

entiatingtheabove equation,we get for a givenx: ∇SSDh S h

x& T

x ∇S h

x .

Thus,thegradientdescentconsistsof aniterativeschemeof theform:

vn 1 vn α Sn hn

x" T

x ∇Sn hn

x (8)

whereα is thesteplength. If we setα to a constantvalue,this methodcorrespondsto a first

ordergradientdescentalgorithm.Comparing(8) to (6), weseethatourmethodsets

α 1!∇S hn x ! 2 T

x" S hn

x 2 (9)

andappliesa Gaussianfilter to provide a smoothdisplacementfield. Cachieret al. (Cachier

et al., 1999;Pennecet al., 1999)have shown that using(9) closelyrelates(6) with a second

ordergradientdescentof theSSDcriterion,in whicheachiterationn setshn 1 to theminimum

of the SSD quadraticapproximationat hn. We refer the readerto thesearticlesfor a more

11

technicaldiscussionon this subject.

1.2.4 Relation to Optical Flow

T andSareconsideredassuccessivetimesamplesof animagesequencerepresentedby Ix t ,

wherex x1 x2 x3 is a voxel position in the imageand t is time. The displacementsare

computedby constrainingthebrightnessof brainstructuresto beconstantin timesothat

dIx t

dt 0 (10)

It is well known that(10) is not sufficient to provide a uniquedisplacementfor eachvoxel. In

fact,this constraintleadsto

fx ∂I

x t ' ∂t!

∇xIx t ! 2∇xI

x t (11)

which is the componentof thedisplacementin thedirectionof the brightnessgradient(Horn

andSchunck,1981).

Otherconstraintsneedto beaddedto (10) to obtainthedisplacementcomponentsin other

directions. Many methodshave beenproposedto fulfill this purposeandthusregularizethe

resultingvector field (Barron et al., 1994). One that can be computedvery efficiently was

proposedby Thirion (Thirion,1998)in hisdescriptionof thedemonsregistrationmethod,using

a completegrid of demons.It consistsof smoothingeachdimensionof the vectorfield with

a Gaussianfilter Gσ. He also proposedto add ∂Ix t ' ∂t 2 to the denominatorof (11) for

numericalstability when∇xIx t is closeto zero,a term which servesthe samepurposeas

α2 in the original optical flow formulationof Horn andSchunck(Horn andSchunck,1981).

As is presentedby Bro-NielsenandGramkow (Bro-NielsenandGramkow, 1996),this kind of

regularizationapproximatesa linearelasticitytransformationmodel.

12

With this in mind, thedisplacementthatmapsa voxel positionin T to its positionin S is

foundusinganiterativemethod,

vn 1

x Gσ vn ∂I

x t ' ∂t!

∇xIx t ! 2 ∂I

x t ' ∂t 2∇xI

x t # (12)

Spatialderivativesmay be computedin several ways (Horn andSchunck,1981; Brandt,

1997; Simoncelli,1994). We have observed from practicalexperiencethat our methodper-

formsbestwhenthey arecomputedfrom theresampledsourceimageof thecurrentiteration.

As shown in Section1.2.3,this is in agreementwith theSSDminimization. Temporalderiva-

tivesare obtainedby subtractingthe target imagesfrom the resampledsourceimageof the

currentiteration.Theseconsiderationsrelate(12) to (6). Thereadershouldnotethatthemajor

differencebetweenthis methodandotheroptical flow strategiesis that regularizationis per-

formedafter thecalculationof thedisplacementsin thegradientdirectioninsteadof usingan

explicit regularizationtermin aminimizationframework.

1.2.5 Relation to the DemonsAlgorithm

Our algorithm is actually a small variation of the demonsmethod(Thirion, 1995; Thirion,

1998)usingacompletegrid of demons,itself closelyrelatedto opticalflow asdescribedin the

previoussection.Thedemonsalgorithmfindsthedisplacementsusingthefollowing formula:

vn 1

x Gσ vn S hn

x T

x!

∇Tx ! 2 S hn

x" T

x 2∇T

x # (13)

In comparing(13) and(6), it is apparentthat theonly differencebetweenour formulationand

thedemonmethodis thatderivativesarecomputedontheresampledsourceimageof thecurrent

iteration. This modificationwasperformedfollowing theobservationson theminimizationof

theSSDcriterion.

13

1.3 Resultsand Discussion

In the following sectionwe presentregistrationresultsinvolving imagesobtainedfrom mul-

tiple modalities.First, we show a typical examplewheremonofunctionaldependencecanbe

assumed:theregistrationof anatlas(Collinset al., 1998b)with aT1-weightedMR image.We

next presentanexamplewherebifunctionaldependencemaybeassumed:theregistrationof a

PD-weightedimagewith thesameT1-weightedimage.

All of the imagesusedin this sectionhave a resolutionof 1 ( 1 ( 1mm3 andrespectthe

neurologicalconvention,i.e. oncoronalslices,thepatient’s left is on theleft sideof theimage.

Before registration,imagesareaffinely registeredusing the correlationratio method(Roche

et al., 1998).

The multilevel processwasperformedat threeresolutionlevels, namely4mm, 2mm and

1mm per voxel. Displacementfields at one level are initialized from the result of the pre-

vious level. The initial displacementfield v0 is set to zero. 128 iterationsareperformedat

4mm/voxel, 32 at 2mm/voxel and8 at 1mm/voxel. Thesearetwice the numberof iterations

usedfor registrationof monomodalimagesusingtheconventionaldemonsalgorithm. We be-

lieve that makinguseof a betterstoppingcriterion, suchasthe differenceof the SSDvalues

betweeniterations,would probablyimprove theresultsshown below. This aspectis presently

underinvestigation.TheGaussianfilter Gσ usedto smooththedisplacementfield hasa stan-

darddeviation of 1 voxel regardlessof theresolution.This modelsstrongerconstraintson the

deformationfield at thebeginningof theregistrationprocessto correctfor grossdisplacements,

andweaker constraintsneartheendwhenfine displacementsaresought.Theresamplingpro-

cessmakesuseof trilinear interpolation,exceptin thecaseof theatlaswherenearest-neighbor

interpolationis used.

Computationtime to obtainthe following resultsis around60 minuteson a 450MHz PC

with 500MB of RAM (10 minutesat 4mm,20minutesat 2mmand30 minutesat 1mm).Most

14

of this time ( ) 85%)is devotedto theintensitycorrectionpart,which hasnot beenoptimized

in this first versionof our program.Theother15%is takenby thestandardregistrationcode

which is stableandwell optimized.

1.3.1 Monofunctional Dependence

We presentheretheresultof registeringtheatlaswith a T1-weightedimage.This is a typical

exampleof monofunctionaldependencebetweentheintensitiesof theimagesto register:since

theatlascanbeusedto generaterealisticMR images,it is safeto assumea functionaldepen-

dencebetweenthe intensityof theatlasandthoseof theT1-weightedimage. Also, sincethe

sourceimageS is a labeledimage,thefunction f is modeledasa piecewiseconstantfunction.

In this case,eachintensitylevel (10 in all) correspondsto a region from which to estimatethe

constantfunction.

The resultof registrationis presentedin Figure1. The first image(Figure1a) shows one

slice of the atlas. The secondone(Figure1b) is the correspondingslice of the T1-weighted

image. The third andfourth images(Figures1c and1d) presentthe resultof registeringthe

atlaswith theT1-weightedimageusingour algorithm. Figure1c shows theresultwithout the

intensity transformation;we have simply appliedto the atlasthe geometricaltransformation

resultingfrom the registrationprocedure.Figure1d shows the imageresultingfrom the reg-

istrationprocess.It hasthesameshapeastheT1-weightedimage(Figure1b) andintensities

havebeentransformedusingtheintensitycorrection.To facilitatethevisualassessmentof reg-

istrationaccuracy, contoursobtainedusingaCanny-Dericheedgedetector(ontheT1-weighted

image)havebeenoverlaidovereachimagein Figure1.

Figure1eshowsthejoint histogramof intensitiesafterregistration.Valueshavebeencom-

pressedlogarithmically and normalizedas is depictedin the color scale. The histogramis

color-codedandrangesfrom darkredrepresentinghigh point densitiesto light yellow depict-

15

ing low point densities.Thevaluesof thepiecewiseconstantfunction f areoverlaidaswhite

dots.

1.3.2 Bifunctional Dependence

Whenregisteringimagesfrom differentmodalities,monofunctionaldependencemaynot nec-

essarilybe assumed.We presentedin Section1.2.1suchan example: the registrationof PD

andT1-weightedimages.The mainproblemin this caseis that theCSF/GMintensityof the

PD imageneedsto bemappedto two differentintensitiesin theT1-weightedscan.

To solve this problem,we appliedthe methoddescribedin Section1.2.1 to register PD

andT1-weightedimageswheretwo polynomial functionsof degree12 areestimated.This

polynomialdegreewassetarbitrarily to a relatively high valueto allow significantintensity

transformations.

As shown in Figure1f-j, the CSF, which is white in the PD-weightedimage(Figure1f)

andblack in theT1-weightedimage(Figure1g), is well registered.Also, the intensitytrans-

formationis adequate(Figure1i). Thesamecommentsapplyto theGM, which is white in the

PD-weightedimage(Figure1f) andgrayin theT1-weightedimage(Figure1g).

Figure1j presentsthejoint histogramof thetwo imagesafterregistration.Thefunctions f1

and f2 foundduringtheregistrationprocessaresuperimposed,theblueline correspondsto f1

andthegreenoneto f2. Theline width for agivenintensitys is proportionalto thevalueof the

correspondingπε

s .

As canbeobservedin Figure1j, thepolynomialfunctions f1 and f2 fit well with thehigh

densityclustersof thejoint histogram.In particular, we seethat theCSF/GMintensityvalues

from the PD-weightedimage(with valuesaround220) get mappedto two different intensity

valuesin theT1-weightedscan:75and45. Themappingto 75 representstheGM (redpolyno-

mial) while themappingto 45 representsCSF(bluepolynomial).

16

Note that in theregistrationof theatlaswith theT1-weightedimageandthePD- with the

T1- weightedimage,we selectedas the sourceimagethe one which hasthe bestconstrast

betweenstructures.This is simplybecauseouralgorithmpermitsmany structuresof thesource

imageto be mappedto a single intensity. But a single intensity in the sourceimagecanbe

mappedto at most two intensitiesin the target image. Hence,it is alwaysbetterto usethe

imagewith thegreaternumberof visiblestructuresasthesourceimage.

1.4 Conclusion

Wehavepresentedanoriginalmethodto performnon-rigidregistrationof multimodalimages.

This iterativealgorithmis composedof two steps:theintensitytransformationandthegeomet-

rical transformation.Two intensitytransformationmodelsweredescribedwhichassumeeither

monofunctionalor bifunctionaldependencebetweenthe intensityvaluesin the imagesbeing

matched.Both of thesemodelsarebuilt usingrobustestimatorsto enablepreciseandaccurate

transformationsolutions.

17

0

0.5

1

50

100

150

200

250

Bac

kgro

und

CS

FG

ray

Mat

ter

Whi

te M

atte

rFa

t

Mus

cle

/ Ski

nS

kin

Sku

llG

lial M

atte

rC

onne

ctiv

e

0

0.5

1

0 75 150 225

50

100

150

200

250

T1

Inte

nsiti

esAtlas

PD Intensities

T1

Inte

nsiti

es

a

c

g

b

d

i

j

f

h

e

Figure1: Resultsof 3D registration.Registrationof Atlas with T1-weightedimage: (a)Atlas(Source).(b)T1-weightedimage(Target).(c) Atlaswithout intensitycorrection,afterregistrationwith T1. (d) Atlaswith intensitycorrection,afterregistrationwith T1. (e)Thejoint histogramof theatlasandT1-weightedimageafterregistration;valuesrangefrom dark red representinghigh point densitiesto light yellow depictinglow point densities;thewhite dotscorrespondto the intensitytransformationfoundby registeringtheatlaswith theT1-weightedimageand assumingmonofunctionaldependence(piecewise constantfunction). Registration of PD-weighted withT1-weightedimage: (f) PD-weightedimage(Source).(g) T1-weightedimage(Target). (h) PD-weightedimagewithout intensity correction,after registrationwith T1-weightedimage. (i) PD-weightedimagewith intensitycorrectionafter registrationwith T1-weightedimage. (j) The joint histogramof PD-weightedimageand T1-weightedimagem,after registration;theblue line correspondsto f1 andthegreenoneto f2; the line width for agivenintensityvalues in T1 correspondsto thevalueof thecorrespondingπε * s+ . ContourswereobtainedusingaCanny-Dericheedgedetectoron thetargets(b andg) andoverlaidon theotherimagesto betterassessthequalityof registration.Thejoint histogramsvalueshave beencompressedlogarithmicallyandnormalizedasis depictedin thecolorscale. 18

2 Image Fusion During Neurosurgery with a Biomechanical

Model of Brain Deformation

Intr oduction

A critical goalof neurosurgery is to accuratelylocate,accessandremove intracraniallesions

withoutdamaginghealthybraintissue.Theoverridinggoalis to preserveneurologicalfunction.

This requirestheprecisedelineationof thefunctionalanatomyandmorphologyof thepatient’s

brain,aswell aslesionmargins. Thesimilar visualappearanceof healthyanddiseasedbrain

tissue(e.g.aswith infiltrating tumors)andtheinability of thesurgeonto seecritical structures

beneaththe brain surfacecanposedifficultiesduring the operation.Somecritical structures,

suchaswhitematterfibertracts,maynotbevisibleatall. Moreover, thedifficulty in perceiving

lesion(e.g.tumor)boundariesmakescompleteresectionextremelydifficult (Jolesz,1997).

Over thelastdecade,advancesin image-guidedneurosurgery(IGNS) techniqueshavecon-

tributedto thegrowth of minimally-invasive neurosurgery. Theseproceduresmustbecarried

out in operatingroomswhicharespecially-equippedwith imagingsystems.Thesesystemsare

usedto acquireimagesintraoperatively, asnecessitatedby the procedure.The improved vi-

sualizationof deepstructuresandtheimprovedcontrastbetweenthelesionandhealthytissue

(dependingon themodality)allow thesurgeonto planandexecutetheprocedurewith greater

precision.

IGNS haslargely beena visualization-driventask. In thepast,it hadnot beenpossibleto

makequantitativeassessmentsof intraoperative imagingdata,andinsteadphysiciansreliedon

qualitative judgments.In orderto createa rich visualizationenvironmentwhichmaximizesthe

informationavailableto thesurgeon,previouswork hasbeenconcernedwith imageacquisition,

registrationanddisplay. Algorithm developmentfor computer-aidedIGNS hasfocussedon

improving imagingquality andspeed.Anothermajor focushasbeento developsophisticated

19

multimodality imageregistrationandfusion techniques,to enablefusion of preoperative and

intraoperative images.However, clinical experiencewith IGNSinvolving deepbrainstructures

hasrevealedthelimitationsof existing rigid registrationapproaches.Thismotivatesthesearch

for nonrigid techniquesthatcanrapidly andfaithfully capturethemorphologicalchangesthat

occurduringsurgery. In thefuture,theuseof computer-aidedsurgicalplanningwill includenot

only threedimensional(3D), modelsbut alsoinformationfrom multiple imagingmodalities,

registeredinto thepatient’sreferenceframe.Intraoperativeimagingandnavigationwill thusbe

fully integrated.

Variousimagingmodalitieshavebeenusedfor imageguidance.Theseinclude,amongoth-

ers,digital subtractionangiography(DSA), computedtomography(CT), ultrasound(US),and

intraoperative magneticresonanceimaging (IMRI). IMRI representsa significantadvantage

over othermodalitiesbecauseof its high spatialresolutionandsuperiorsoft tissuecontrast.

However, even the most advancedintraoperative imaging systemscannotprovide the same

imageresolutionor tissuecontrastselectionfeaturesaspreoperative imagingsystems.More-

over, intraoperative imagingsystemsareby necessitylimited in theamountof time available

for imaging. Multimodality registrationcanallow preoperative datathat cannotbe acquired

intraoperatively [suchasnuclearmedicinescans(SPECT/PET),functionalMRI (fMRI), MR

angiography(MRA), etc.] to bevisualizedalongsideintraoperativedata.

2.1 Nonrigid Registration for IGNS

During neurosurgical operations,changesoccurin theanatomicalpositionof brainstructures

andadjacentlesions.The leakageof cerebrospinalfluid (CSF)afteropeningthedura,hyper-

ventilation,theadministrationof anestheticandosmoticagents,andretractionandresectionof

tissueall contribute to shifting of thebrainparenchyma.This makesinformationbasedupon

preoperatively acquiredimagesunreliable. The lossof correspondencebetweenpre- andin-

20

traoperative imagesincreasessubstantiallyasthe procedurecontinues.Thesechangesin the

shapeof the brainhave beenwidely recognizedasnonrigiddeformationscalled“brain shift”

(see(Nabavi et al., 2001)).

Suitableapproachesto capturetheseshapechangesandto createintegratedvisualizationsof

preoperativedatain theconfigurationof thedeformedbrainarecurrentlyin activedevelopment.

Previouswork aimedatcapturingbraindeformationsfor neurosurgerycanbegroupedinto two

categories. In the first categorey arethoseapproachesthat usesomeform of biomechanical

model (recentexamplesinclude (Hagemannet al., 1999; Skrinjar andDuncan,1999; Miga

et al., 1999; Skrinjar et al., 2001; Ferrantet al., 2000b)). In the secondcategory are those

approachesthat usephenomenologicalmethods,relying upon imagerelatedcriteria (recent

examplesinclude(Hill et al., 1998;Hata,1998;Ferrantet al., 1999b;Hataet al., 2000)).

Purelyimage-basedmatchingmaybe ableto achieve a visually pleasingalignment,once

issuesof noiseandintensityartifactareaccountedfor. However, in our work on intraoperative

matchingwe favor physics-basedmodelswhich ultimately may be expandedto incorporate

importantmaterialproperties(suchasinhomogeneity, anisotropy) of thebrain,oncetheseare

determined.However, evenamongphysics-basedmodels,thereexist aspectrumof approaches,

usuallyinvolving a trade-off betweenphysicalplausibilityandspeed.

A fastsurgery simulationmethodhasbeendescribedin (Bro-Nielsen,1996). Here,high

computationalspeedswere obtainedby converting a volumetric finite elementmodel into a

modelwith only surfacenodes.Thegoalof thiswork wasto achieveveryhighgraphicsspeeds

consistentwith interactive computationanddisplay. This is achievedat thecostof simulation

accuracy. This typeof modelis bestsuitedto computergraphics-orienteddisplay, wherehigh

frameratesareneeded.

A sophisticatedfinite elementbasedbiomechanicalmodel for two-dimensionalbrain de-

formationsimulationhasbeenproposedin (Hagemannet al., 1999). In this work, correspon-

21

denceswereestablishedby manualinteraction.Theelementsof thefinite elementmodelwere

the pixels of the two dimensionalimage. The manualdeterminationof correspondencescan

betimeconsuming,andis subjectto humanerror. Moreover, whenmethodsaregeneralizedto

threedimensions,thenumberof pointswhich mustbeidentifiedcanbevery large. Dueto the

realitiesof clinical practice,two-dimensionalresultsarenot practical. A (threedimensional)

voxelwisediscretizationapproach,while theoreticallypossible,is extremelyexpensive from a

computationalstandpoint(evenconsideringa parallelimplementation)becausethenumberof

voxels in a typical intraoperative MRI datasetleadingto a largenumberof equationsto solve

(256x256x60= 3,932,160voxels, which correspondsto 11,796,480displacementsto deter-

mine).Downsamplingcanleadto fewervoxels,but leadsto a lossof detail.

Edwardset al. (Edwardset al., 1997)presenteda two dimensionalthreecomponentmodel

for tracking intraoperative deformation. This work useda simplified materialmodel. How-

ever, the initial 2D multigrid implementationrequired120–180minuteswhenrun on a Sun

MicrosystemsSparc20,whichmaylimit its feasibility for routineuse.

Skrinjar et al. (Skrinjar andDuncan,1999)have presenteda very interestingsystemfor

capturingreal-timeintraoperative brain shift during epilepsysurgery. In this context, brain

shift occursslowly. A very simplifiedhomogeneousbrain tissuematerialmodelwasadopted.

Following the descriptionof surfacebasedtrackingfrom intraoperative MRI driving a linear

elasticbiomechanicalmodel in (Ferrantet al., 2000b),Skrinjar et al. presenteda new imple-

mentation(Skrinjaret al., 2001)of their systemusinga linearelasticmodelandsurfacebased

trackingfrom IMRI with thegoalof eventuallyusingstereoscopiccamerasto obtainintraoper-

ativesurfacedataandhenceto captureintraoperativebraindeformation.

Paulsenet al. (Paulsenet al., 1999)andMiga et al. (Miga et al., 1999;Miga et al., 2001)

havedevelopedasophisticatedfinite elementmodelto simulatebraindeformation.Theirmodel

is interestingbecauseit incorporatessimulationsof forcesassociatedwith tumortissue,aswell

22

asthoseresultingfrom retractionandresection.A limitation of the existing approachis that

thepreoperativesegmentationandtetrahedralfinite elementmeshgenerationcurrentlyrequire

aroundfive hoursof operatortime. Nevertheless,this approachholdspromisein actuallypre-

dicting braindeformation.

Thereal-timeneedsof surgerydictatethatany algorithmusedfor prospectiveimagematch-

ing must rapidly, reliably andaccuratelycapturenonrigid shapechangesin the brain which

occurduringsurgery. Ourapproachis to constructanunstructuredgrid representingthegeom-

etry of thekey structuresin theimagedataset.This techniqueallowsusto usea finite element

discretizationthatfaithfully modelskey characteristicsin importantregionswhile reducingthe

numberof equationsto solveby usingmeshelementsthatspanmultiplevoxelsin otherregions.

Thealgorithmallowstheprojectionof preoperative imagesontointraoperativeimages,thereby

allowing thefusionof imagesfrom multiplemodalitiesandspanningdifferentcontrastmecha-

nisms.Wehaveusedparallelhardware,parallelalgorithmdesignandefficient implementations

to achieve rapidexecutiontimescompatiblewith neurosurgery.

2.2 Method

Figure2 is anoverview, illustratingtheimageanalysisstepsusedduring intraoperative image

registration. The critical imageprocessingtasksinclude segmentation(the identificationof

anatomicalstructures)andregistration.Segmentationdataareusedbothfor preoperativeplan-

ning,andto createintraoperativesegmentations.Thesegmentationdataareusedto calculatean

initial affine transformation(rotation,translation,scaling)which rigidly registersthe images,

thusinitializing thedatafor nonrigidmatchingusingour biomechanicalsimulation.Usingthe

biomechanicalmodel,thevolumetricdeformationis inferredthrougha mechanicalsimulation

with boundaryconditionsestablishedvia surfacematching. This sophisticateddeformation

modelcanbesolvedduringneurosurgery, providing enhancedintraoperativevisualization.

23

Image Analysis During Image Guided Neurosurgery:

Tissue Segmentation

InitialRegistration

BiomechanicalSimulation

Preoperative data

Intraoperative MRI

SurfaceMatching

Visualization

Neurosurgeryprogresses

Figure2: Schematicillustratingimageanalysistaskscarriedoutduringneurosurgery.

2.2.1 Preoperative Data Acquisition and Processing

Thetime availablefor imageprocessingduringsurgery is extremelylimited comparedto that

availablepreoperatively. Consequently, preoperativedataacquisitioncanbemorecomprehen-

sive,andmoreextensiveimageanalysis(for examplesegmentation)canbeperformed.

A variety of manual(Gering et al., 1999), semi-automated(Kikinis et al., 1992; Yezzi

et al., 2000)andautomated(Warfield et al., 2000a;Kauset al., 1999;Warfieldet al., 2000b)

segmentationapproachesareavailable. We selectthe mostaccurate,robust approachbased

uponthepreoperative dataandtheparticularcritical structures.For thematchingexperiments

which will bedescribedbelow, we have usedananatomicalatlas,althoughotherpreoperative

datasuchasmagneticresonanceangiographyor diffusiontensorimagescouldultimatelyalso

be used. The atlaswasconstructedfrom a high resolutionscanof a singlepatient,in which

over 200 structuresweresegmented(Kikinis et al., 1996)usinga combinationof automated

and interactive techniques.During surgery, we are especiallyinterestedin the corticospinal

tract, a region of white matterwhich canbe difficult or impossibleto directly observe with

24

conventionalMRI, andwhichmustbepreserved.Wehavepreviouslyshownthatwecanproject

thecorticospinaltractfrom theatlasontopatientscansfor preoperativesurgicalplanning(Kaus

et al., 2000).

2.2.2 Intraoperati ve ImageProcessing

Intraoperative imageprocessingconsistsof: 1.) acquiringoneor moreintraoperativevolumet-

ric datasets;2.) constructinga segmentationof the intraoperative acquisition;3.) computing

anaffine registrationof thepreoperativedataontothenew acquisition;4.) identifying thecor-

respondencesbetweenkey surfacesof the preoperative andintraoperative data;5.) solving a

biomechanicalmodelto infer a volumetricdeformationfield; 6.) applyingthedeformationto

thepreoperativedataandconstructinga new visualizationmerging critical structuresfrom the

preoperativedatawith theintraoperativedata.

Segmentationof Intraoperati ve Volumetric Images

In theexperimentsconductedbelow, a rapidsegmentationof thebrainandventricleswasob-

tainedusing a binary curvaturedriven evolution algorithm (Yezzi et al., 2000). The region

identifiedasbrain or ventriclewastheninteractively correctedto remove misclassifiedtissue

usingthesoftwaredescribedby Geringet al. (Geringet al., 2001). This approachallows the

surgeonto inspectandinteractively edit thesegmentationdata,increasingits accuracy.

Alternatively, wehaveexperimentedwith automatedintraoperativesegmentation(Warfield

et al., 1998b;Warfield et al., 2000a)utilizing tissueclassificationin a multi-channelfeature

spaceusinga modelof expectedanatomyasan initial template.Automatedapproacheswill

likely bepreferableoncethey havebeenvalidated.

25

Unstructured MeshGenerationand SurfaceRepresentation

Wehave implementedameshgeneratorwhich is optimizedfor usewith biomedicalstructures,

building uponpreviously describedtechniques(Schroederet al., 1996;Geiger, 1993). During

meshgeneration,weextractanexplicit representationof thesurfaceof thebrainandventricles

basedon thepreoperative segmentation.We alsocreatea volumetricunstructuredmeshusing

a multiresolutionversionof themarchingtetrahedraalgorithm. Themesherbeginsby subdi-

viding theimageinto cubes,which arethendividedinto 5 tetrahedrausinganalternatingsplit

patternwhich preventsdiagonalcrossingson thesharedfaces.Themeshis iteratively refined

in theregion of complex boundaries,andthena marchingtetrahedra-like approachis applied

to this multiresolutionmesh.For eachcell of thefinal mesh,the labelvalueof eachvertex is

checked,andif different,thetetrahedronis dividedalongtheedgehaving differentnodelabels.

A detaileddescriptioncanbefoundin (Ferrantet al., 1999b;Ferrantet al., 2000a).

The meshingprocessis extremely robust, allowing us to generatetetrahedralmeshesof

thebrainandventriclesfrom rapidsegmentationsof eachvolumetricintraoperativeacquisition

carriedout during the surgery. This facilitatesintraoperative matchingfrom onevolumetric

acquisitionto thenext.

Affine Registration of Preoperative to Intraoperati ve Image Datasets

For affine registration(rotation, translation,scaling),we usea fast parallel implementation

of a robust algorithm which is baseduponaligning segmentedimagedata,usinga rapidly-

converging multiresolutionsearchstrategy (Warfield et al., 1998a). Applying the resulting

transform,segmentationsandgreyscaledatafrom thepreoperativeandintraoperativescansare

rigidly registered.

26

Volumetric BiomechanicalSimulation of Brain Deformation

During the procedure,the brain undergoesnonrigid shapechangesfor the reasonsdescribed

above. During IGNS thesurgeonis ableto acquirea new volumetricMRI whenhewishesto

review the currentconfigurationof the entirebrain. A volumetricdeformationfield relating

earlieracquisitionsto this new scanis computedby first matchingsurfacesfrom the earlier

acquisitionto the currentacquisition,and then calculatingthe volumetric displacementsby

using the surfacedisplacementsas boundaryconditions. The critical conceptis that forces

areappliedto thevolumetricmodelthatwill producethesamesurfacedisplacementsaswere

obtainedby the surfacematching. The biomechanicalmodelcanthenbe usedto computea

volumetricdeformationmap.

Establishing SurfaceCorr espondences Thesurfacesof thebrainandlateralventriclesare

iteratively deformedusinga dual active surfacealgorithm. Image-derived forcesareapplied

iteratively to anelasticmembranesurfacemodelof theearlyscan,therebydeformingit soasto

matchtheboundaryof thecurrentacquisition.Thederivedforcesarea decreasingfunctionof

theimageintensitygradients,soasto beminimizedat theedgesof objectsin thevolume.We

have includedprior knowledgeabouttheexpectedgraylevel andgradientsof theobjectsbeing

matchedto increasethe convergencerateof theprocess.This algorithmis fully describedin

(Ferrantet al., 1999a).

BiomechanicalSimulation of Volumetric Brain Deformation Wetreatthebrainasahomo-

geneouslinearlyelasticmaterial.Thedeformationenergy of anelasticbodyΩ, underno initial

stressesor strains,andsubjectto externallyappliedforces,canbedescribedby the following

model(Zienkiewicz andTaylor, 1994):

Eu 1

2 , Ωσ - ε dΩ , Ω

u - F dΩ (14)

27

wherethe variablesaregiven in termsof the stressvector, σ, the strainvector, ε, the forces

F Fx y z appliedto theelasticbody(forcesperunit volume,surfaceforcesor forcescon-

centratedat the nodesof the mesh)andu ux y z v x y z w x y z - , the displacement

vectorfield we wish to compute.Sincewe areusinga linearelasticityframework, we assume

smalldeformations.Hencethestrainvectorε is givenby

ε ∂u∂x

∂v∂y

∂w∂z

∂u∂y

∂v∂x

∂v∂z

∂w∂y

∂w∂x

∂u∂z# - (15)

which canbe written as ε L u whereL is a linear operator. The elastomechanicalrelation

betweenstressesandstrainscanbeexpressedby thegeneralizedHooke’s law as

σ σx σy σz τxy τyz τzx - D ε (16)

Assumingisotropicmaterialpropertiesfor eachpoint,we obtaina symmetricelasticitymatrix

D in theform

D E1 ν 1 2ν

.//////////////01 ν ν ν 0 0 0

ν 1 ν ν 0 0 0

ν ν 1 ν 0 0 0

0 0 0 1 1 2ν2 0 0

0 0 0 0 1 1 2ν2 0

0 0 0 0 0 1 1 2ν2

24333333333333335(17)

with physicalparametersE (Young’s modulus)andν (Poisson’s ratio). See(Zienkiewicz and

Taylor, 1994)for thefull details.

For thediscretization,we usethefinite elementmethodappliedover thevolumetricimage

domainsothatthetotal potentialenergy canbewritten asa sumof potentialenergiesfor each

28

element:Eu ∑Nnodes

e% 1 Eeue . Themeshis composedof tetrahedralelementsandthuseach

elementis definedby four meshnodes.Thecontinuousdisplacementfield u everywherewithin

elemente of the meshis definedasa function of the displacementat the element’s nodesuei

weightedby theelement’s interpolatingfunctionsNei

x ,

ux Nnodes

∑i % 1

I Nei

x ue

i (18)

Linearinterpolatingfunctionsareusedto definethedisplacementfield insideeachelement.

Theinterpolatingfunctionof nodei of tetrahedralelemente is definedas

Nei

x 1

6Ve 6 aei be

i x cei y de

i z7 (19)

Thecomputationof thevolumeof theelementVe andtheinterpolationcoefficientsaredetailed

in (Zienkiewicz andTaylor, 1994,pages91–92).

The volumetric deformationof the brain is found by solving for the displacementfield

that minimizesthe deformationenergy describedby Equation(14). For our finitite element

approachthis is describedby

δEu M

∑e% 1

δEeue 0 (20)

where

δEeue Nnodes

∑i % 1

∂∂ue

iEe

ue δui

e Nnodes

∑i % 1

∂∂ve

iEe

ue δvi

e Nnodes

∑i % 1

∂∂we

iEe

ue δwi

e (21)

Sinceδuie δvi

e andδwie areindependent,definingmatrix Be

Bei Nnodes

i % 1 with Bei LNe

i for

29

everynodei of eachelemente, yieldsin thefollowing equation:

0 , ΩBe - DBeuedΩ , Ω

Ne - FedΩ (22)

with the elementstiffnessmatrix Ke 98Ω Be - DBe dΩ. An assemblyof the equationsfor

all elementsfinally leadsto a global linear systemof equations,which canbe solved for the

displacementsresultingfrom theforcesappliedto thebody:

K u F (23)

The displacementsat the boundarysurfacenodesare fixed to matchthosegeneratedby

theactive surfacemodel. Let :u be thevectorrepresentingthe displacementto be imposedat

theboundarynodes.Theelementsof the rows of thestiffnessmatrix K correspondingto the

nodesfor which a displacementis to be imposedaresetto zeroandthediagonalelementsof

theserows areset to one. The force vectorF is set to equalthe displacementvector for the

boundarynodes:F :u (Zienkiewicz andTaylor, 1994). In this way solvingEquation(23) for

theunknown displacementswill producea deformationfield over theentirevolumetricmesh

thatmatchestheprescribeddisplacementsat theboundarysurfaces.

Hardwareand Implementation

Thevolumetricdeformationof thebrainis computedby solvingfor thedisplacementfield that

minimizestheenergy describedby Equation(14),afterfixing thedisplacementsat thesurface

to matchthosegeneratedby theactivesurfacemodel.

Threevariables,representingthe x, y andz displacements,mustbe determinedfor each

elementof thefinite elementmesh.Eachvariablegivesriseto onerow andonecolumnin the

global K matrix. The rows of thematrix aredividedequallyamongsttheCPUsavailablefor

30

computationandtheglobalmatrix is assembledin parallel.EachCPUassemblesthelocal Ke

matrix for eachelementin its subdomain.AlthougheachCPUhasanequalnumberof rows to

process,becausetheconnectivity of themeshis irregular, someCPUsmaydo morework than

others.

Following matrix assembly, the boundaryconditionsdeterminedby the surfacematching

areapplied.TheglobalK matrix is adjustedsuchthat rows associatedwith variablesthatare

determinedconsistof asinglenon-zeroentryof unit magnitudeon thediagonal.

The volumetric biomechanicalbrain model systemof equations(and the active surface

membranemodel equations)are solved using the Portable,ExtensibleToolkit for Scientific

Computation(PETSc)package(Balayet al., 1997;Balayet al., 2000a)usingtheGeneralized

Minimal Residual(GMRES)solver with block Jacobipreconditioning.During neurosurgery,

the systemof equationswassolved on a SunMicrosystemsSunFire6800symmetricmulti-

processormachinewith 12750MHzUltraSPARC-III (8MB Ecache)CPUsand12GB of RAM.

This architecturegivesussufficient computecapacityto executetheintraoperative imagepro-

cessingprospectively duringneurosurgery.

Intraoperati ve Visualization

Oncethevolumetricdeformationfield hasbeencomputed,it canbeappliedto earlierdatato

warp it into the currentconfigurationof the patientanatomy. The imagingdatacanthenbe

displayedby texture mappingonto flat planesto facilitatecomparisonswith currentintraop-

erative dataaswell asprior scans. Triangle modelsof segmentedstructures(i.e. basedon

registeredvolumetric data)can be usedto display surfacerenderingsof critical anatomical

structures,overlaid on intraoperative imagedata. This allows readyappreciationof the 3D

anatomyof thesesegmentedstructurestogetherwith the imagingdatain the form of planes

passingthroughor over the3D trianglemodels(Geringet al., 2001). This augmentsthesur-

31

Figure3: Open-configurationmagneticresonancescannerduringneurosurgery.

geon’sability to seecritical structureswhichmustbepreserved(suchasthecorticospinaltract)

andto betterappreciatethelesionandits relationshipto otherstructures.

Figure3 shows theopen-configurationmagneticresonancescanneroptimizedfor imaging

duringsurgical procedures(Jolesz,1997;Black et al., 1997). The imagewe constructedwas

presentedon theLCD andincreasedthe informationavailableto thesurgeonastheoperation

progressed.

2.3 Resultsand Discussion

The imageregistrationstrategy describedherehasbeenappliedprospectively during several

neurosurgical cases.We presenthereillustrative resultswhich demonstratethe ability of our

algorithmto captureintraoperativebraindeformations.

Theenhancementprovidedby intraoperativenonrigidregistrationto thesurgical visualiza-

tion environmentis shown by ourmatchingthecorticospinaltractof apreoperatively prepared

anatomicalatlasto the initial andsubsequentintraoperative scansof a subject.This matching

32

Figure4: This figureshows thecorticospinaltractfrom our anatomicalatlasin blue,projectedinto theshapeof thebrainof thesubjectshown in Figure3.

wascarriedoutprospectively duringtheneurosurgery, demonstratingthepracticalvalueof the

approachandits ability to meetthe real-timeconstraintsof surgery. We have alsoconducted

parallelscalingexperimentswhich have yieldedvery encouragingresults. The entire image

analysisprocesscanbe completedin lessthan10 minutes,which hasbeenadequateto dis-

play theinformationto thesurgeon.Interestingly, themostcomputationallyintensive task(the

biomechanicalsimulation)hasalsobeenoptimizedthe mostandis now the fasteststep. We

anticipatethatsegmentationtechniquesrequiringlessuserinteractionwill resultin significant

improvementsin speed.

BiomechanicalSimulation of Volumetric Brain Deformation

Figure4 shows thecorticospinaltractfrom ouranatomicalatlasprojectedinto theshapeof the

brainof thesubject.Thisvisualizationhelpsthesurgeonto betterappreciatethe3D relationship

of this essentialstructureto the lesionandotherregionsof the brain. The corticospinaltract

cannotbereadilyobservedin IMRI acquisitions.

33

(a) A single slice from anearly3D IMRI scan.

(b) Thecorrespondingsliceina later 3D IMRI scan,show-ing significantbrain shift hasoccurred.

(c) The matchedslice of thefirst volume after simulationof thebraindeformation.

(d) Visualizationof the mag-nitude of the deformationfield computedin matchingimage(a) to image(b).

Figure5: Two dimensionalslicesthroughthree-dimensionaldata,showing the matchof thesimulateddeformationof the initial brain onto the actualdeformedbrain. The quality of thematchis significantlybetterthancanbeobtainedthroughrigid registrationalone.

34

Figure 5 is a typical caseillustrating the amountof brain deformationwhich can occur

duringneurosurgery, aswell astheeffectivenessof our algorithmin capturingthis shift during

neurosurgery. As shown, the quality of the matchis significantlybetterthancanbe obtained

throughrigid registrationalone.

Our earlyexperiencehasshown thatour intraoperative biomechanicalsimulationof brain

deformationis a robustandreliablemethodfor capturingthechangesin brainshapethatoccur

during neurosurgery. The registrationalgorithmrequiresno userinteractionandthe parallel

implementationis sufficiently fastto beusedintraoperatively. Weintendto incorporatepatient-

specificpreoperative datain placeof theanatomicalatlasto increasethesurgical valueof the

intraoperativeupdates.

As we refineour approach,we expectto appreciateperformancegainsbasedon moreau-

tomatedsegmentationmethods,andfurtheroptimizedparallelimplementationswhich address

loadimbalances.Improvementsin theaccuracy of thematchcouldresultfrom a moresophis-

ticatedmodelof the materialpropertiesof the brain (suchasmoreaccuratemodelingof the

cerebralfalx andthe lateralventricles).SophisticatedMR imagingmethodssuchasdiffusion

tensorMRI now enablethe preoperative imagingof inhomogeneousanistropicwhite matter

structure,which could be incorporatedinto the materialmodel. Ultimately, the predictionof

braindeformation,asopposedto thecaptureof observeddeformationdescribedhere,will most

likely requirea nonlinearmaterialmodeltogetherwith extensive monitoringof physiological

data.Suchpredictioncouldbeusedto indicatewhennew intraoperative imagingis necessary

to appropriatelyupdateboththesimulationmodelandthesurgeon’sunderstandingof thebrain

shape.

35

2.4 Conclusion

Nonrigid changesin brainmorphologyoccurduringneurosurgeryandlimit theusefulnessof

preoperative imagingfor intra-treatmentplanningandsurgical navigation. Intraoperativenon-

rigid registrationcanaddsignificantlyto the valueof intraoperative imaging. It providesfor

quantitativemonitoringof therapy application,includingtheability to make quantitativecom-

parisonswith a preoperatively-definedtreatmentplanandenablespreoperative imagedatato

bealignedwith thecurrentconfigurationof thebrainof thepatient.We have shown thateven

a relatively complex biomechanicalmodelcanbe initialized andsolvedduring neurosurgery,

providing enhancedsurgical visualization. Ultimately, suchapproachesmay provide a truly

integrated,multimodalityenvironmentfor surgicalnavigationandplanning.

36

3 Physics-BasedRegularization with an Empirical Model of

Anatomical Variability

An importantissuein nonrigidregistrationfor computer-assistedneurologyandneurosurgery

is thegenerationof deformationfieldsthatreflectthetransformationof animagein a realistic

waywith respectto thegivenanatomy. Dueto lackof imagestructure,noise,intensityartifacts,

computationalcomplexity anda restrictedtime frame(e.g. during surgery), it is not feasible

to measuredirectly the deformationoccuringat eachvoxel. This leadsto estimatesof the

deformationfield only at sparselocationswhich have to beinterpolatedthroughouttheimage.

Recently, physics-basedelasticandviscousfluid modelsfor nonrigidregistrationhave be-

comepopular(BajcsyandKovacic,1989),sincethey have thepotentialto constraintheunder-

lying deformationin a plausiblemanner. However viscousfluid models(Lesteret al., 1999;

WangandStaib,2000)have to bechosencarefully, sincethey allow largedeformations.This

is notalwayssuitablefor medicalapplicationsconcerningthebrain.Furthermore,viscousfluid

modelsdrivenby alignmentof similar grayvaluesmayallow anatomicallyincorrectmatches

of differentbut adjacentstructuresthroughthe samemechanismby which large-deformation

matchesarepermitted.For example,onegyrusmayflow from thesourcebrain to matchtwo

or moredifferentgyri in a targetbrain,producingananatomicallyincorrectmatch.

In termsof physics-basedelasticmodels,recentwork has(Davatzikos,1997;Ferrantetal.,

2000b)proposedanactivesurfacealgorithmcomputedat theboundaryof a regardedstructure

as an initial estimateof the deformationfield which was then introducedinto a volumetric

elasticmodelto infer thedeformationinsideandoutsidethesurface.A drawbackof thismethod

is thatalthoughit hasbeenshown to beaccuratecloseto theobject’s boundary, away from the

boundariesthesolutioncouldbelessaccurate.Thework by (WangandStaib,2000)represents

animprovementin thatstatisticalshapeinformation(basedon a setof imageswith manually-

37

identified boundarypoints) was includedas an additionalmatchingcriterion. Even though

suchmethodsarepromisingfor specificbrainstructures,a robust3D shaperepresentationof

thewholebrainstill remainsdifficult to achieve.

In (Collins,1994)anothernonrigidregistrationalgorithmwasproposed,basedon anitera-

tiverefinementof a local similarity measureusingasimplex optimization.In thisapproachthe

deformationfield wasconstrainedonly by smoothingaftercorrespondenceestimation,andthus

canonly beaccuratefor specificregionsof thebrain.To achievebetterresults,themethodwas

improvedby introducingvariousgyri andsulci of the brain asgeometriclandmarks(Collins

et al., 1998a).

In order to obtainrealisticdeformations,we proposeherea physics-basedelasticmodel.

Themethoddoesnot requirea segmentationanddoesnot have thedrawbackthat initial esti-

matesof thedeformationareonly generatedfor theboundaryof aconsideredstructure.Instead,

theseestimatesarecalculatedbasedon a templatematchingapproachwith a local similarity

measure.Furthermorewe have incorporateda modelfor inhomogeneouselasticitiesinto our

algorithm.Thediscretizationof theunderlyingequationis doneby a finite elementtechnique,

which hasbecomea popularmethodfor medicalimagingapplications(e.g. see(Bro-Nielsen,

1998)and(Ferrantet al., 2000b)).

3.1 Method

The processof registrationcanbe describedasan optimizationproblemthat minimizesthe

deformationenergy betweena templateanda referenceimage. Assumingthat both images

representthe samephysicalobject, the deformationthat aligns them is thereforerelatedto

the theoremof minimum potentialenergy. The ideaof our registrationprocesscannow be

describedasfollows: basedonasetof pointsextractedoutof animageasdescribedin (3.2),an

initial sparseestimateof thedeformationfield is foundby a local normalizedcross-correlation

38

(3.3). In a next stepnonrigid registrationis performedusingan elasticmodel(3.4) which is

constrainedby thesparseestimatescomputedin thepreviousstep.

3.2 FeaturePoint extraction

Let Ω denotethedomainof a volumeS: Ω ;=< with voxel positionsx x y z - x > Ω. In

a first stepa setof featurepointsis extractedout of thereferenceimage.For thatpurposewe

calculatethe gradientmagnitudeout of blurredimageintensities. In orderto obtainsuitable

featurepointsfor aninitial sparseestimateof thedeformationfield, only voxel higherthantwo

standarddeviationsabovethemeanof themagnitudeof thegradientareusedfor thecorrespon-

dencedetection(3.3).Figure6 showsthisprocessfor onesliceof aMagneticResonance(MR)

scanof thebrain.

To overcomethepooredge-preservingpropertiesof linearlow-passfilters,weuseanonlin-

eardiffusionfilter. A filteredversionp of volumeScanbedescribedasasolutionof thepartial

differentialequation(PDE):

∂t p div ? g ∇pσ 2 ∇p@ (24)

with Neumannboundaryconditionsandthe original imageas initial state(Weickert, 1997).

The diffusion function g : <A;B< is usedto reducethe noisesensivity andthusdependson

themagnitudeof thegradientof smoothedimageintensities,computedby convolving p with

a Gaussiankernelof standarddeviation σ. The ideaof the diffusion function is to stop the

filtering processat regionswith high gradients,(i.e. at edgesin an image),andto provide a

valuecloseto zerothere.In our method,we usea diffusionfunctionproposedby Weickert in

(Weickert,1997):

gx2 CDE DF 1 for x2 0

1 exp 1 CG

xH λ I 8 for x2 J 0 (25)

The parameterλ separatesregions of low contrastfrom thoseof high contrast. For values

39

greaterthanλ, thefiltering is reduced,while for valueslessthanor equalto λ strongersmooth-

ing is applied.For theconstantC, Weickert proposesC 3 31448which givesvisually good

resultsandgivestheflux fx x K g x2 theexpectedbehavior (i.e. f is increasingfor values x L λ anddecreasingfor values x J λ). As ana priori determinationof λ is very difficult,

the contrastparameterwassetinteractively for eachvolumein our approach.Furthermorea

paralleladditive operatorsplitting (AOS) schemewasusedfor computationalefficiency. See

(Weickert et al., 1998)for details.

(a) (b) (c)

Figure6: Illustrationof featurepointextraction.For abettervisualimpressiononly adetailof the imageis shown. (a) Sliceof anMR scan;(b) Sliceafterapplyinga nonlineardiffusionfilter; (c) Magnitudeof thegradientof theblurredimageafterthresholding.

3.3 Corr espondencedetection

After extractingfeaturepoints,thecorrespondencesbetweenthe referenceimageR andtem-

plateimageT is computedfor thesepoints.A commonway to minimizethedistancebetween

regionsof two volumesconsistsof finding theglobaloptimumof a function which measures

their similarity. This canbe expressedasa costfunction M : < n ;N< which is optimizedfor

a voxel x betweentwo regionsof R andT in termsof a giventransformationO ϑ. Thesearch

spaceis restrictedby asetof parametervectorsϑ >P< n.

40

Our approachusesthelocal normalizedcross-correlation(NCC)

M ϑ ∑k QSR G x I fR k "K f

T TO 1 1

ϑ

k U

∑k QSR G x I f 2R k VK ∑k QSR G x I f 2

T TO 1 1

ϑ

k XW x > Ω (26)

which is maximizedat a givenvoxel by a bruteforcesearch.Thereforewe assumea window

of sizew ( w ( w aroundavoxel x in thereferenceimage,andcomputethemaximalNCCby

shifting a window of similar sizein the templateimage. In Equation(26), this window is de-

scribedby alocalneighborhoodof avoxelx definedas Y x Z x w y w z w - x

w y w z w - . The searchspacein our methodis restrictedto translationsbecauseother

transformationslike rotationsor scalingwould be of highercomputationalcomplexity. Fur-

thermoretheNCCis only computedfor voxelswith highgradientmagnitudescalculatedoutof

blurredimageintensities,asdescribedin section(3.2). For a betterperformancefor largedata

setstheoptimizationis solvedin parallel.

3.4 Inter polation fr om sparsedisplacementestimates

Thesparsedeformationestimatesobtainedatthefeaturepointscomputedby alocalnormalized

cross-correlation,arenow introducedasexternalforcesinto anelasticmodel.Weuseasimilar

energy termasdescribedin Section2.2.2usingthefinite elementmethodfor thediscretization.

Henceweseekthedeformationu thatminimizesEquation(14)– repeatedherefor convenience

Eu 1

2 , Ωσ - ε dΩ , Ω

u - F dΩ Theunderlyingideais againto restricttheregistrationprocesssothattheresultingdeformation

field is a priori fixedby theestimatesat thesepoints.

For a volume of 256 ( 256 ( 124 voxels, the linear systemof equationswe obtain has

approximately532000unknowns,which is alsosolvedin parallelwith thePortableExtensible

41

Toolkit for ScientificComputation(PETSc)package(Balayet al., 2000a;Balayet al., 2000b;

Balayet al., 1997).Theexecutiontime for thewholeregistrationprocessis usuallyaboutfive

minuteson aclusterwith 12CPUs(seeSection2.2.2for details).

3.4.1 Inferring empirically observed anatomical variability

In orderto describethe mechanicalbehavior of tissueundergoinga deformation,the relation

betweenstressandstrain is expressedby an elasticitymatrix D, generatedduring the matrix

assembly. For isotropicmaterialtwo parametersareneeded:Young’smodulusE asa measure

of stiffnessandPoisson’s ratio ν asameasureof incompressibility.

Typically elasticityparametershave beensetarbitrarily andhomogeneously(Bajcsyand

Kovacic,1989;Ferrantet al., 2000b)which is only a roughapproximationof the underlying

tissue. RecentlyLesteret al. (Lesteret al., 1999)appliedan inhomogeneousviscousfluid

modelto brainandneckregistration.Manualsegmentationsof thebonewereusedasaregionof

high stiffness.Davatzikoset al. (Davatzikos,1997)appliedinhomogeneitiesto brainwarping

settingtheelasticityparametersof thebrainfour timeshigherthantheir valuein theventricles.

Our approachdiffers in that inhomogeneouselasticityparametersarederivedfrom anem-

pirical estimateof anatomicalvariability, sothateachdiscreteelementcanobtainits own ma-

terial propertiesduringthematrix assembly. We useda setof 154MR scansof thebrain,first

segmentedinto white matter, grey matter, cerebrospinalfluid (CSF)andbackgroundusingan

EM-basedstatisticalclassificationalgorithm(Wells et al., 1996a).In thenext step,theheadof

eachscanwasalignedto an arbitrarily selectedscanout of this database,usingglobal affine

transformations(Warfieldet al., 1998a)andour nonrigidregistrationmethod.Figure7 shows

theresultfor thetissueclassesafternonrigidregistration,averagedover all scans.In orderto

generatea model for inhomogeneouselasticities,we usean entropy measurefor eachvoxel.

42

(a) (b) (c) (d)

Figure 7: Imagefrom theaveragedvolumefor differenttissueclassesafternonrigid regis-tration.Dark regionsimply a slightoverlapping.(a)Background;(b) CSF;(c) graymatter;(d)whitematter.

Thereforewedefinethejoint voxelwiseentropy as

hs1 s2 s3 s4 4

∑i % 1

psi ln p si (27)

whereeachsi representsthesumover all scansfor oneof the four differentsegmentedtissue

classesat a certainvoxel. Accordingto theseresults,the elasticityparametersarecomputed

for every voxel. We choosea linearmappingfor thecomputedjoint voxelwiseentropy of the

identifiedbraintissueswherethePoissonratio ν wasscaledin therangeof ν > 0 1 0 4 while

Young’s elasticitymodulusE hada rangeof E > 2kPa 10kPa . Thebackgroundwassetto a

low stiffness(E 1kPa) andincompressibilityparameter(ν 0 05). Figure8 showsasliceof

thecomputedmodelandtheassociatedvaluesfor ν.

(a) (b)

Figure 8: Model of empiricallyobservedvariability. (a) Sliceout of themodelaftervoxel-wiseentropy computation.Darkregionsimply a low entropy value;(b) Computedincompress-ibility parameter(Poissonratio ν) for eachvoxel of thesameslice. Dark regionsimply a lowvaluefor ν.

43

3.5 Illustration of nonrigid registration with homogeneousand inhomo-

geneouselasticities

In orderto demonstratethe behavior of our deformationmodelwith homogeneousandinho-

mogeneouselasticities,the algorithm was appliedto register 159 MR scansof the brain of

youngadults. Eachscanwasfirst globally registeredto an arbitrarily-chosendatasetby an

affine transformation(Warfieldetal., 1998a).Thenonrigidregistrationwith homogeneousand

inhomogeneouselasticitieswasthenappliedto the aligneddata. Figure9 shows the results

of the matchingprocessafter averagingover all scans.Becausewe areperformingregistra-

tion amongdifferentsubjects,a globalaffine transformationnormallywill not beableto align

referenceandtemplateimageproperly. This leadsto a blurredaverageimage(Figure9 (b)).

Thealignmentfor theelasticmodelis shown in Figure9 (c) for homogeneousandin Figure9

(d) for inhomogeneouselasticities.In thecaseof homogeneouselasticitieswe useE 3kPa

for theYoung’s elasticitymodulus,andν 0 4 for thePoissonratio, asusedby Ferrantet al.

(Ferrantet al., 2000b).

An analysisof the summedsquareddifferencesshowed an improvementof 2% usingin-

homogeneouselasticities.This rathersmalleffect is dueto thesettingof featurepointsin our

experiments.As canbeseenin Figure8, largeregionsof whitematteronly havea smallrange

of anatomicalvariability. In other words, the large numberof fixed deformationestimates

constrainsthe interpolationdoneby the elasticmodel. Furtherresearchwill investigatenew

approximationschemesto addressthis.

44

(a) (b)

(c) (d)

Figure9: Resultsof globalaffineandnonrigidregistrationappliedto 159subjects.(a)Sliceof referencevolume; (b)-(d) Resultafter registrationandaveragingover all scansusing: (b)globalaffine registration;(c) nonrigidregistrationwith homogeneouselasticities;(d) nonrigidregistrationwith inhomogeneouselasticities.

45

4 Registration of Diffusion TensorImages

4.1 Intr oduction

A largeamountof researchhasbeendoneover thelasttwo decadeson theregistrationof med-

ical imagesprovidedby differentimagingmodalities,resultingin aproliferationof algorithms

with asolid theoreticalbackground.Non-scalarimagingmodalitiesareemergingin Radiology.

For examplePhaseContrastAngiographyMRI (PCA-MRI) (Dumoulinet al., 1989)provides

adescriptionof speedanddirectionof bloodflow, andDiffusionTensorMRI (DT-MRI) (LeBi-

hanetal.,1986;Basseretal.,1994;Pierpaolietal.,1996)providesdiffusiontensorsdescribing

local mobility of watermoleculesin tissue. The increasingclinical relevanceof suchimage

modalitieshaspromptedresearchfocusedon registrationmethodssupportingthem.

Althoughthetheorythatwill bepresentedin this chapteris generalandvalid for any data

dimensionsandarbitrarytensordata(including thespecialcaseof vectors)thedriving exam-

ple throughoutthis sectionwill beregistrationof DT-MRI data.DT-MRI is a relatively recent

MR imagingmodalityusedfor relatingimageintensitiesto therelativemobility of endogenous

tissuewatermolecules.In DT-MRI, a tensordescribinglocal waterdiffusionis calculatedfor

eachvoxel from measurementsof diffusion in several directions. To measurediffusion, the

Stejskal-Tannerimagingsequenceis used(StejskalandTanner, 1965).Thissequenceusestwo

stronggradientpulses,symmetricallypositionedarounda 180[ refocusingpulse,allowing for

controlleddiffusion weighting. DT-MRI hasshown its clinical value in early assessmentof

brain ischemiaandstroke by showing the decreasedability of the affectedtissuesto diffuse

water (Hajnal andBydder, 1997; Provenzaleand Sorensen,1999). SinceMRI methods,in

general,obtainamacroscopicmeasureof amicroscopicquantity(whichnecessarilyentailsin-

travoxel averaging),thevoxel dimensionsinfluencethemeasureddiffusiontensorat any given

locationin thebrain. Factorsaffectingtheshapeof theapparentdiffusiontensor(shapeof the

46

diffusionellipsoid)in thewhite matterincludethedensityof fibers,thedegreeof myelination,

theaveragefiberdiameterandthedirectionalsimilarity of thefibersin thevoxel. Thedirection

of maximumdiffusivity is describedby theeigenvectorcorrespondingto thelargesteigenvalue.

This is descriptive of theorientationof white matterfiber tractsin thecentralnervoussystem.

This is duetherestricteddiffusioncausedby thepresenceof a tightly packedsheathof myelin

surroundingthe axons(Basseret al., 1994; Peledet al., 1998). Somepostprocessingalgo-

rithms suitedto DT-MRI have arisenover the last years. For example,Westinet al. (Westin

et al., 2001)describesanisotropy analysisandfiltering of DT-MRI data,andRuiz et al. (Ruiz-

Alzola et al., 2001a)describesan approachto point landmarkdetectionin tensordata. The

ability of visualizingandautomaticallytracingwhite mattertractsis expectedto play a major

role in basicneurosciences,in theunderstandingof neurologicaldisorders(especiallythoseas-

sociatedwith white matterdemyelination),agingandbraindevelopment(Pouponet al., 1998;

Pouponet al., 1999;Weinsteinetal., 1999;Westinet al., 2001).

Theapproachpresentedherestemsfrom our work presentedin (Ruiz-Alzolaet al., 2000;

Ruiz-Alzolaet al., 2001a;Ruiz-Alzolaet al., 2001b)andit is basedon templatematchingby

locally optimizingasimilarity function(Sect.4.3).A localstructuredetectorfor generictensor

fields (Sect.4.4) allows us to constrainthe matchingto highly structuredareas. In order to

obtaina densedeformationfield, thesparseestimatesfrom thetemplatematchingareinterpo-

lated.Thewholeapproachis embeddedin amultiresolutionschemeusingaGaussianpyramid

in orderto dealwith moderatedeformationsanddecreasethe influenceof falseoptima. We

alsopresent(Sect.4.5)someillustrativeresultscarriedouton syntheticandclinical data.

4.2 Registration of DT-MRI Data

In addition to our own work (Ruiz-Alzola et al., 2000; Ruiz-Alzola et al., 2001a),previous

work in diffusiontensorregistrationincludestheeffortsof Alexanderandcoworkers (Alexan-

47

deret al., 1999;AlexanderandGee,2000). They extendthemultiresolutionelasticmatching

paradigmin (BajcsyandKovacic,1989;GeeandBajcsy, 1999)to tensordata. Tensorreori-

entationis not includedin the regularizationterm,but tensorsarereorientedin eachiteration

accordingto theestimateddisplacementfield. Severalstrategiesto estimatethetensorreorien-

tationfrom thedisplacementfield arealsoinvestigated.

We statetheproblemof registrationasa mappingof a referenceanatomy, depictedby the

signalSr

x , to a deformedone, representedby the signalSd

x . Equation(28) describesa

modelto characterizetherelationshipbetweenbothsignals,whereD modelsthedeformation

appliedto thereferencesignal,andbothH andthenoisev modeltheinter-scandifferences.

Sd

x H D Sr

x ;x v

x (28)

ThedeformationD representsa space-variantshift systemand,hence,its responseto a signal

Sx is D S x S

x d

x , whered

x is adisplacementfield. With regardto thedifferences

betweenthesystemsgeneratingtheimages(signals), weconsiderH to beanon-memory, pos-

sibly space-variant,systemdependingonasethx

h1

x hp

x t of unknown parameters

andthenoiseto bespatiallywhiteandwith zeromean.With thesesimplificationsanddefining

SHr

x H Sr

x ;x , themodel(28) reducesto:

Sd

x SH

r

x d

x v

x (29)

Thegoalof registrationis to find thedisplacementfield dx thatmakesthebestmatchbetween

Sr

x andSd

x accordingto (29).

48

4.3 TemplateMatching

Severalschemescanbeusedto estimatethedisplacementfield in (29). Whenthereis noa pri-

ori probabilisticinformationaboutthesignalandnoisecharacterization,aLeast-Squares(Moon

andStirling, 2000)approachis a naturalchoice.For this, all that is requiredis a suitabledefi-

nition of aninnerproductand,thereafter, aninducednorm. Notethatscalar, vectorandtensor

fieldsareapplicationsof arealdomainontoEuclideanvectorspacesandthisallowsusto define

the innerproductbetweenfieldsby meansof the integral over thewholedomainof the inner

productsbetweentheir values. Let us considerthe functionalset \ f : D "; V where

D is a real domainandV is an Euclideanspace.Thenan innerproductcanbedefinedon \as ] f1 f2 J A8

D wx ] f1

x f2 x J dx, wherew

x is a weightingfunction for the inner

product. Note that the innerproductin the left-handsideis definedbetweenfieldsandin the

right-handside,insidetheintegral, is definedbetweenvaluesof thefield.

Theleastsquaresestimatoris obtainedby minimizing a costfunction(30) thatconsistsof

thesquarednormof theestimationerror.

C ^ d x ;h x !Sd

x SH

r

x d

x ! 2 (30)

The dependency on the unknown parametershx can be removed by estimatingthem us-

ing constrainedleast-squaresschemes. For example, if the parametersare assumedto be

constantall over the spatialdomain,a least-squaresestimationcan be obtained,hdx

hSd

x Sr

x d

x , andsubstitutedin C ^ to obtaina new costfunction (31) thatonly de-

pendson d (see(Ruiz-Alzolaetal., 2001b)for furtherdetails)

Cdx C ^ d x ; h d x (31)

Theoptimizationof Cdx in orderto obtainthedisplacementfieldd

x is adauntingtaskthat

49

requiresadditionalconstraintsto make it feasible.TemplateMatching tradesoff accuracy and

computationalburdento approximatea solutionfor this optimizationproblem. It essentially

consistsof defininga templatefrom theneighborhoodof every point of thedeformeddataset.

Eachof thesetemplatesis thencomparedor matchedagainsttheneighborhoodsof tentatively

correspondentpoints in the referencedatasetand a similarity measureis obtainedfor each

of them. The tentative point whoseneighborhoodprovidesthe biggestsimilarity is selected

ascorrespondingto the currentpoint in the deformeddatasetandthe displacementbetween

both points is obtained. Thereis a fundamentaltrade-off to be consideredin the designof

the neighborhoods:they must be non-local,and hencelarge in size, in termsof the Sd

x

space-frequenciesto avoid the ill-posednessarising from the lack of discriminantstructure

(apertureproblem(Poggioet al., 1985)),andthey mustbe local, andhencesmall in size, in

termsof the unknown displacementfield spatial-frequenciesto guaranteethe validity of the

local deformationmodel.Adaptive templateswith differentsizesandweightscanhelpto deal

with this problem.

Let Tx x0 bea window functioncenteredin a genericpoint x0 in thedeformeddataset

anddesignedfollowing thepreviousremarks.Thetemplatematchingassumptionstransform(29)

into (32), thatholdsfor everypoint x0 in thedeformeddataset.

Tx x0 Sd

x T

x x0 SH

r

x d

x v

x (32)

Equation(32)hasanintuitiveinterpretation:any neighborhoodin thedeformeddatasetaround

a point x0, definedby the window function Tx x0 , correspondsto a neighborhoodin the

referencedatasetdefinedby thewindow functionTx x0 d

x which hasbeenwarpedby

thedeformationfield. Templatematchingassumesthatamodelis chosenfor thedisplacement

field andfor theparametersof thetransformationhx in a neighborhoodof thepoint x0 to be

registered.For examplethe deformationfield modelmay constrainthe templatejust to shift

50

alongthecoordinateaxes(translation),or to undergo rigid motionshenceallowing alsorota-

tionsor mayevenallow stretchandtwist. In any casethemodelfor thelocaldeformationmust

besuchthat it dependsonly on a few parameters,in orderto make thesearchcomputationally

feasible.With respectto theparameters,thecommonchoiceis to assumethemconstantin the

neighborhood.

The templatematchingsplitsa complex globaloptimizationproblem,i.e. coupledsearch-

ing for all thedisplacements,into many simplelocal ones,i.e. searchingindependentlyfor the

displacementof eachpoint usingtemplatematchingin eachcase.For example,for thecom-

moncasewherethedisplacementfield is assumedto beconstantinsidethetemplate,thecost

function(30) reducesto a setof costfunctions

C ^ d x0 ;h x !Tx x0 _? Sd

x SH

r

x d

x0 @ ! 2 (33)

wherex0 refersto every point in the deformeddataset. One of the main characteristicsof

templatematchingis theabsenceof any globalregularizationthatconstrainsthelocalvariability

of the estimateddeformationfield. While this preventsgetting trappedin falseoptima that

are far from the absoluteoptimum, as global optimizationmethodsare proneto, noisecan

producehigh frequency artifactson theestimateddeformation.Hencea further refinementof

thesolutionmaybe advisabledependingon theapplication,eitherpostfilteringtheestimated

deformationor usingit asaninitial solutionfor aglobaloptimizationscheme.

4.3.1 Similarity Functions

A Similarity Function is a monotonicfunction of the cost (30), SFdx F C

d , which

leavesthelocationsof theoptimaunchangedandremainsinvariantwith respectto theunknown

parameters.The local natureof the templatematchingmethodmakesit necessaryto definea

similarity functionSFdx0 for every point in thedeformeddatasetwhich is to bematched

51

ontothereferenceone,i.e., themonotonicfunctionis appliedto (33). In this sectiontheleast-

squaresmethodreferredto above is usedto obtainsuitablelocal similarity functionsfor the

templatematchingof generictensorfields.

Let usfirst considerthatH is theidentitymappingandthatthedisplacementfield is constant

insidethetemplate.Direct useof (33) leadsto

SFSSD

dx0 !

Tx x0 Sd

x Sr

x d

x0 ! 2 (34)

thatcorrespondsto thewell-known Sumof SquaredDifferencessimilarity function.Extending

it by usinginnerproductsandassumingthat!Tx x0 Sr

x d

x0 ! 2 is almostconstantfor

all possibledx0 leadsto analternativesimilarity functionthatcorrespondsto theCorrelation

measure.

SFC

dx0 ] T2

x x0 Sd

x Sr

x d

x0 J (35)

Let us now considerthat H is a space-invariantaffine transformationof the intensity. In this

case

Tx x0 Sd

x aT

x x0 Sr

x d

x0 bT

x x0 1 x v

x (36)

where1x refersto theonetensorfunction(all thecomponentsareequalto oneeverywhere).

Thecost(33) turnsout to be

C ^ d x0 ;a b !Tx x0 Sd

x " aSr

x d

x0 " b1

x ! 2 (37)

A similarity function invariantto a andb canbeobtainedby replacingthesecoefficientswith

their least-squaresestimationandminimizing theresultingcost.Detailscanbefoundin (Ruiz-

52

Alzola et al., 2001b). The resultingsimilarity function is the absolutevalueof a generalized

versionof thewell-known correlationcoefficient

SFρ ` d ` x0 aba cedddddf

s g 1hth2 i sj t k tl

s g 1hth2 i sj t k t

l j p ` d ` x0 aba g 1hth2 i p ` d ` x0 aba j t k tl

p ` d ` x0 aba g 1hth2 i p ` d ` x0 aba j t k t

lm ddddd (38)

where

s Tx x0 Sd

x (39)

p Tx x0 Sr

x d

x0 (40)

t Tx x0 1 x (41)

Theapplicationof theequationsaboverequiresaproperdefinitionof theinnerproduct

] S1

Kn S2

Kn J , DS1i1 o o o in x S2i1 o o o in x dx (42)

andits inducednorm !S Kp ! 2 , D

Si1 qrqrq in x Si1 qrqsq in x dx (43)

Weassumethatthetensorsarecartesian(definedwith respectto anorthonormalbasis)andwe

areusingtheEinsteinnotationfor sums(any repetitionof anindex entailsasummingover this

index). Notethatany implementationrelieson sampleddataandthereforetheintegralsabove

becomesums.

4.3.2 WarpedVectorsand Tensors

Vectorand tensordataare linked to the body underinspectionand, thereafter, any warping

of thesupportingtissuewill leadto a consequentwarpingor reorientationof thesedata. The

53

warpingof thedomaincanbeexpressedby thetransformation

x Tx tu x t d

x tu (44)

wherex standsfor pointsin thereferencedatasetandx t for pointsin thedeformedone.More-

over, the transformationis assumedto be differentiableandhencethe neighborhoodsof the

correspondentpointsx andx t arerelatedthrough

dx v ∇ Tx tuTw dx tx (45)

wherethe deformationgradient ∇ Tx t canbe easily recognizedasthe Jacobianmatrix

Jx t of thetransformationT

x t v ∇ T

x t w y J

x t δT

x t

δx t (46)

Equation(45)simplystatesthat,asfarasthetransformationis differentiable,a linearmapping

relatesthe local neighborhoodsof bothpoints. For finite sizeneighborhoods,thedeformation

gradientcorrespondsto a linearapproximation,asaTaylor’sexpansionclearlyshows

x ∆x z Tx t ∆x t T

x t 1

1!δT

x t

δx t ∆x t (47)

∆x z δTx t

δx t ∆x t (48)

In this work it will be assumedthat the linear approximationis valid sincethe function data,

vectorsor tensors,arerelatedto infinitesimalpropertiesof thetissue.Consequently, twovectors

v andv t arelocally relatedas

v Jx t| v t (49)

54

(50)

Sincetwo secondorderdiffusiontensorsP andPt canbeconsideredasassociatedto quadratic

forms,they arerelatedby

Pt Jtx tu PJ

x tp (51)

Equation(51) providesa theoreticalway to estimatethealterationof diffusiontensorsdue

to a deformationfield. Neverthelessit is not clear that DT-MRI dataactually are modified

accordingto this modelspeciallyin areasof high anisotropy, i.e. thewhite matterfiber tracts,

wherethesedataaremostrelevant.Theideahereis thattheshapeof thediffusiontensorshould

be preserved throughthe transformationandhenceit mustonly be reorientedasan effect of

local rotation and shear. This essentiallymeansthat the deformationfield only affects the

directionalpropertiesof diffusionbut not its strengthalongits principalaxes.For example,in

a referenceframeintrinsic to a fiber tractdiffusionshouldremaininvariantwith respectto the

deformation.Thishasmotivatedasearchfor tensortransformationsthatmaintaintheshapeand

includeboththeeffect of local rotationandshear. Early experimentson this topic have begun

(Sierra,2001). An ad-hocsolutionto this problemis to scaletheresultingtensorafter (51) is

appliedsoas,for example,preserve theellipsoidvolumeor normalizethe largesteigenvalue.

Anotherpossibilityis to modify thedeformationgradientsoasto avoid undesirableeffectssuch

asthescaling(Alexanderetal., 1999).Neverthelessmuchresearchis still neededto clarify the

appropriatetensortransformationto beused.

A mathematicaltool to dealwith this problemis the Polar DecompositionTheorem (see

for example(Segel, 1987) from the theoryof non-linearelasticity). It allows us to dealnot

only with infinitesimalbut alsowith finite deformations.Thetheoremstatesthat for any non-

singularsquarematrix, suchasthe DeformationGradientJx t ), thereareuniquesymmetric

55

positive definitematricesUx t andV

x t andalsoa uniqueorthonormalmatrix R

x t such

that

Jx t R

x t U

x t Vx t R

x t (52)

This leadsto importantgeometricinterpretationsof the geometricmapping. For example,

noticethat a sphereis first stretchedby the mappingin the directionsof the eigenvectorsof

Ux t andthenrotatedby R

x t . Thereafter, a transformationsuchthatlocally R

x t I is said

to beaPureStrain atx t while if Ux t V

x t I it is saidto beaRigidRotationat thatpoint.

As mentionedabove(Sect.4.3),thematchingapproachto dataregistrationreliesonamodel

for the local displacementfield inside the template. In order to perform the matching,the

vectorsor tensorsinsidethetemplatemustbereorientedaccordingto thehypothesizedmodel.

Note that if a simple model suchas just shifting the templatealong the coordinateaxes is

adopted,i.e. assuminga constantdisplacementfield for all the points inside, the vectorsor

tensorswould not be reoriented.Similarly, if the model is rigid motion no stretchingof the

vectorsor tensorsshouldbe considered.From a practicalpoint of view, no reorientationis

performedduringmatchingandthereforea constantdisplacementfield is assumedinsidethe

template.This is not a limitation aslong asthelocal rotationis smallandin fact it is accepted

in conventionaltemplatematchingof scalardata.Thereorientationis thencalculatedoncethe

displacementfield –andits gradient–hasbeenestimated.

4.4 Structure Measures

Matchingmustbeconstrainedto areaswith localhighdiscriminantstructure.Dependingonthe

dataset,this approachwill leadto very sparsecontrolpointsto estimatethedeformationfield.

The applicability of the methodultimately dependson the characteristicsof the deformation

field – it beingobvious that if the deformationfield hasa large spatialvariability, the sparse

56

displacementfield estimatedfrom thedetectedpointswill suffer from spectralaliasing.Incre-

mentingthe samplingdensityby acceptinglow structurepoints is possibledependingon the

noisecharacteristicsof thedata,sinceit is unacceptableto allow noiseto provide thediscrimi-

nantinformationthatdrivesthetemplatematching.Whenit is not possibleto provide enough

pointsaccordingto thefrequency propertiesof thedeformationfield, it might benecessaryto

resortto regularizedschemes,suchaselasticmodels,thatusethewholedataset.Alternatively,

in someapplications,anadditionalchannelof datais customarilyprovided.Thisis thecase,for

example,in DT-MRI usingEPI sequenceswhereadditionalT2-weightedimagesareprovided.

Thereforeit is possibleto estimatedifferentsparsedisplacementfields from T2 andDT-MRI

andcombinethemin orderto estimatethewholedisplacementfield, providing constraintsboth

from structuralT2 imagesandfrom thediffusiontensors(thewhitematterfiber tracts).

In orderto identify theareasof highstructure,weuseameasureof cornerness(Ruiz-Alzola

et al., 2000;Ruiz-Alzolaetal., 2001a;Ruiz-Alzolaet al., 2001b)which generalizesthelocally

averagedouterproduct(i.e. thecorrelationmatrix)of thegradientfield (Rohr, 1997)frequently

usedin thescalarcase.

4.5 Results

In this sectionresultson both syntheticand real dataare presented.The real imaging data

were acquiredon a GE Signa1.5 TeslaHorizon Echospeed5.6 scanner, using a Line Scan

Diffusiontechnique,with thefollowing parameters:no averaging,effective TR=2.4s, TE=65

ms,bhigh 750s' mm2, fov=22cm,voxel size4 8 ( 1 6 ( 1 5 mm3, 6 kHz readoutbandwidth,

acquisitionmatrix128 ( 128.

Figure (10.a) shows the clustersof points that have beendetectedas highly structured

in a DT-MRI sagittal imageoverlayedon the correspondingT2-weightedMRI imageand

Fig. (10.b) shows theseclustersin the portion correspondingto the highlightedsquare.Re-

57

call thatthematchingis performedusingtheseclusters,not theisolatedlandmarks.Noticethat

thediagonalcomponentsof thetensorfield provide strongerandmorestructuredsignalsthan

theoff-diagonalonesandhow thestructuredetectorfindsthethin structuresin theseimages.It

mustberecalledthatthecooperationof all thecomponentsis whatprovidesthisresult.In order

to obtainthe clusters,we have normalizedthe tensorfield componentsto fit into the interval 1 1 (weaker componentsdo not reachtheextremavalues).Theestimationsof thegradient

andthegeneralizedcorrelationmatriceshave beenmadeusing3 ( 3 neighborhoods.Thedif-

ficulty of presentingillustrative resultsfrom volumedatausing2D figureshasmotivatedusto

reportthis experimentusingasingleDT-MRI slice(thetensorsin it are3D). Nevertheless,the

methodis essentiallyN-dimensionalandit canbedirectlyappliedto volumesof datausingthe

sameparameters,justaddingonemoredimensionin thedefinitionof theneighborhoods.

a) b)

Figure10: a)High structureclustersoverlayedonT2W MRI b) Detailof theclustersinsidethesquare

In orderto assesstheoverall performanceof our nonrigid registrationmethod,Fig. (11.a)

shows a sagittalMRI slice of the corpuscallosumthat is deformedby a syntheticGaussian

58

field as depictedin Fig. (11.b). In order to estimatethe deformation,a Gaussianpyramid

decompositionis obtained,performingthetemplatematchingon structuredareasin eachlevel

andinterpolatingusingKriging (Ruiz-Alzolaetal.,2000;Ruiz-Alzolaetal.,2001b).Fig. (11.c)

shows theresultof reconstructingtheoriginal imagewith thedeformationfield estimatedwith

our approach,usingtheKriging estimatorwith anexponentialvariogram.

a) b) c)

Figure 11: Syntheticdeformation(a) original MRI (b) deformed(c) reconstructedoriginalusingestimateddeformationfield

Figure(12.a)shows a sliceof a DT-MRI datasetof thecorpuscallosumwheretheprinci-

pal eigenvectordirectionshave beenrepresentedusinga color codingrangingfrom blue (in-

planeprojection)to red (orthogonalto plane)(Peledet al., 1998). The whole approachhas

beenappliedto warp this datasetinto anothercorrespondingto a different individual, shown

in Fig. (12.b),usingthreelevelsof a Gaussianpyramid,andanexponentialvariogramfor the

Kriging interpolatorthat is limited to take into accountthe 8 closestsamples.Figure(12.c)

showsa T2W zoomedversionof theright handsideof theformer, correspondingto theposte-

rior corpuscallosumandtheestimateddeformationfield.

59

a) b) c)

Figure12: DT-MRI interpatientwarping. a, b) DT-MRI of different individuals. c) zoomedT2W of theposteriorcorpusof a)andestimateddeformation

4.6 Conclusions

We have describeda framework for non-rigidregistrationof scalar, vectorandtensormedical

data.Theapproachis local,sinceit is basedontemplate-matching,andresortsto amultiresolu-

tion implementationusinga Gaussianpyramidin orderto provide a coarse-to-fineapproxima-

tion to thesolution.This allows themethodto handlemoderatedeformationsandavoidsfalse

local solutions.Themethoddoesnot assumeany globala priori regularization,andtherefore

avoidsthecomputationalburdenassociatedwith thoseapproaches.

60

5 The Monge–Kantorovich Problemand ImageRegistration

In this section,we presenta methodfor producingareapreservingsurfacedeformations,and

moregeneralmasspreservingareaandvolumedeformations,basedon theminimizationof a

functionalof Monge–Kantorovich type.Thetheoryis basedon theproblemof minimizing the

costof redistributing a certainamountof massbetweentwo distributionsgivena priori. Here

thecostis a functionof thedistanceeachbit of materialis moved,weightedby its mass.We

show how theresultinglow-orderdifferentialequationsmaybeusedfor registration.

5.1 The Monge–Kantorovich Problem

Here we presenta methodfor imagewarping andelasticregistrationbasedon the classical

problemof optimalmasstransport.Themasstransportproblemwasfirst formulatedby Gaspar

Mongein 1781,andconcernedfinding theoptimalway, in thesenseof minimal transportation

cost,of moving a pile of soil from onesite to another. This problemwasgivena modernfor-

mulationin thework of Kantorovich (Kantorovich, 1948),andsois now known astheMonge–

Kantorovich problem. This type of problemhasappearedin econometrics,fluid dynamics,

automaticcontrol, transportation,statisticalphysics,shapeoptimization,expert systems,and

meteorology(Rachev andRuschendorf,1998).

Themethodwe introducein this sectionis designedfor elasticregistration,andis basedon

anoptimizationproblembuilt aroundtheL2 Monge–Kantorovich distancetakenasasimilarity

measure.Theconstraintthatweputon thetransformationsconsideredis thatthey obey amass

preservationproperty. Wewill assumethatarigid (non-elastic)registrationprocesshasalready

beenappliedbeforeapplyingour scheme.

Our methodis basedon differentialequationsandin this sensemaybethoughtof asbeing

in the sameclasswith optical flow andelasticdeformationmodelapproachesto registration.

See(Hataetal.,2000),(Davatzikos,1997)and(Lesteretal.,1999)for representativeexamples

61

of thesemethods.Ourmethodalsohasastrongconnectionto computationalfluid dynamics,an

areawhichhaspreviouslybeensuccessfullyappliedto brainregistrationproblems(Christensen

et al., 1996).

Our method,however, hasa numberof distinguishingcharacteristics.It is parameterfree.

It utilizesall of thegrayscaledatain bothimages,andplacesthetwo imageson equalfooting.

It is thussymmetrical,theoptimalmappingfrom imageA to imageB beingthe inverseof the

optimalmappingfrom B to A It doesnotrequirethatlandmarksbespecified.Theminimizerof

thedistancefunctionalinvolvedis unique;thereareno otherlocal minimizers.Thefunctional

at the heartof the methodis suchthat the correspondingdifferentialequationsgoverningits

minimizationareof low order. Finally, it is specificallydesignedto take into accountchanges

in densitythatresultfrom changesin areaor volume.

We believe that this type of elasticwarpingmethodologyis quite naturalin the medical

context wheredensitycanbea key measureof similarity. It alsooccursin functionalimaging,

whereonemaywantto comparethedegreeof activity in variousfeaturesdeformingover time,

andobtaina correspondingelasticregistrationmap. A specialcaseof this problemoccursin

any applicationwherevolumeor areapreservingmappingsareconsidered.

5.1.1 Formulation of the Problem

We now give a modernformulationof the Monge–Kantorovich problem. We assumewe are

given, a priori, two subdomainsΩ0 and Ω1 of Rd, with smoothboundaries,and a pair of

positivedensityfunctions,µ0 andµ1, definedon Ω0 andΩ1 respectively. We assume8 Ω0µ0

8Ω1

µ1 sothatthesametotalmassis associatedwith Ω0 andΩ1 We considerdiffeomorphisms

u from Ω0 to Ω1 whichmaponedensityto theotherin thesensethat

µ0 Du µ1 u (53)

62

which we will call the masspreservation(MP) property, andwrite u > MP Equation(53) is

calledtheJacobianequation.Here Du denotesthedeterminantof theJacobianmapDu, and denotescompositionof functions.In particular, Equation(53) impliesthatif asmallregionin

Ω0 is mappedto a largerregion in Ω1 thentheremustbea correspondingdecreasein density

in orderfor themassto bepreserved.

Theremaybemany suchmappings,andwewantto pick outanoptimalonein somesense.

Accordingly, wedefinethesquaredL2 Monge–Kantorovich distanceasfollows:

d22

µ0 µ1 inf

u > MP , !ux x

! 2µ0

x dx (54)

An optimalMP mapis a mapwhich minimizesthis integral while satisfyingtheconstraint

(53). The Monge–Kantorovich functional(54) is seento placea penaltyon the distancethe

mapu moveseachbit of material,weightedby thematerial’s mass.A fundamentaltheoretical

result (Brenier, 1991;GangboandMcCann,1996), is that thereis a uniqueoptimal u > MP

transportingµ0 to µ1, andthat this u is characterizedasthe gradientof a convex function wi.e., u ∇w This theorytranslatesinto a practicaladvantage,sinceit meansthat thereareno

non-globalminimato stall our solutionprocess.

5.1.2 Computing The Transport Map

Therehave beena numberof algorithmsconsideredfor computinganoptimal transportmap.

Forexample,methodshavebeenproposedbasedonlinearprogramming(Rachev andRuschendorf,

1998),andon Lagrangianmechanicscloselyrelatedto ideasfrom the studyof fluid dynam-

ics (BenamouandBrenier, 2000). An interestinggeometricmethodhasbeenformulatedby

CullenandPurser(CullenandPurser, 1984).Here,we follow closelythework in (Haker and

Tannenbaum,2001a;HakerandTannenbaum,2001b).

Let u : Ω0 ; Ω1 beaninitial mappingwith themasspreserving(MP) property. Inspiredby

63

(Brenier, 1991;Gangbo,1994),weconsiderthefamily of MP mappingsof theform u u s1 1

ass variesoverMP mappingsfrom Ω0 to itself, andtry find ans which yieldsa u without any

curl, thatis, suchthat u ∇w. Oncesuchans is found,we will have theMonge–Kantorovich

mappingu We will alsohave u u s ∇w s known asthepolar factorizationof u with

respectto µ0 (Brenier, 1991).

5.1.3 Removing the Curl

Ourmethodassumesthatwehavefoundandinitial MP mappingu Thiscanbedonefor general

domainsusinga methodof Moser(Moser, 1965;DacorognaandMoser, 1990),or for simpler

domainsusinga type of histogramspecification.Oncean initial MP u is found, we needto

apply the processwhich will remove its curl. It is easyto show that the compositionof two

masspreserving(MP) mappingsis an MP mapping,andthe inverseof an MP mappingis an

MP mapping.Thus,sinceu is anMP mapping,we have thatu u s1 1 is anMP mappingif

µ0 Ds µ0 s (55)

In particular, whenµ0 is constant,thisequationrequiresthatsbeareaor volumepreserving.

Next, we will assumethat s is a function of time, and determinewhat st shouldbe to

decreasetheL2 Monge–Kantorovich functional. This will give usanevolution equationfor s

andin turn anequationfor ut aswell, the latterbeingthemostimportantfor implementation.

By differentiatingu s u with respectto time,we find

ut Dust s1 1 (56)

Dif ferentiating(55)with respectto timeyields

divµ0st s1 1 0 (57)

64

from whichweseethatst andut shouldhave thefollowing forms:

st 1

µ0ζ # s ut

1µ0

Duζ (58)

for somevectorfield ζ onΩ0 with divζ 0 and ζ ~n 0 on∂Ω0 Here ~n denotesthenormal

to theboundaryof Ω0. This lastconditionensuresthats remainsa mappingfrom Ω0 to itself,

by preventingtheflow of s givenby st 6 1

µ0ζ 7 s from crossingtheboundaryof Ω0 This

alsomeansthattherangeof u u s1 1 is alwaysuΩ0 Ω1

Considernow theproblemof minimizing theMonge–Kantorovich functional:

M , u x x 2µ0

x dx (59)

Takingthederivativewith respectto time,andusingtheHelmholtzdecompositionu ∇w χ

with divχ 0 wefind from (58) that

12

Mt , u ζ , χ ζ (60)

Thus,in orderto decreaseM, we cantake ζ χ with correspondingformulas(58) for st and

ut providedthatwe have divχ 0 and χ ~n 0 on ∂Ω0 Thusit remainsto show thatwe

candecomposeu asu ∇w χ for sucha χ 5.1.4 Gradient Descent:Rd

Welet w beasolutionof theNeumann-typeboundaryproblem

∆w divu ∇wT~n u ~n on∂Ω0 (61)

andsetχ u ∇w It is theneasilyseenthatχ satisfiesthenecessaryrequirements.

65

Thus,by (58),wehave thefollowing evolutionequationfor u:

ut 1

µ0Du ? u ∇∆ 1 1div

u@_ (62)

Thisis afirstordernon-localschemefor ut if wecount∆ 1 1 asminus2 derivatives.Notethat

thisflow is consistentwith respectto theMonge–Kantorovich theoryin thefollowingsense.If u

is optimal,thenit is givenasu ∇w in whichcaseu ∇∆ 1 1divu ∇w ∇∆ 1 1div

∇w 0

sothatby (62), ut 0

5.1.5 Gradient Descent:R2

Thesituationis somewhatsimplerin theR2 case,dueto thefact thata divergencefreevector

field χ canin generalbe written asχ ∇ h for somescalarfunction h where represents

rotationby 90deg, so that ∇ h hy hx In this case,we solve Laplace’s equationwith a

Dirichlet boundarycondition,andderive theevolutionequation

ut 1

µ0Du∇ ∆ 1 1div

u (63)

5.1.6 Generalizations

WenotethatwecandefineageneralizedMonge–Kantorovich functionalas

M , Φ u i µ0 (64)

whereΦ : Rd ; R is a positive strictly convex C1 cost function, and i is the identity map

ix x In particular, the L2 Monge–Kantorovich problemdescribedabove, correspondsto

66

thecostfunctionΦx u x u 2 If wedefine

Ψ : ∇Φ u i (65)

thenanalysissimilar to thataboveshowsthatMt mustbeof theform

Mt , Ψ ζ (66)

whereasbefore

ζ µ0st s1 1 (67)

is adivergence-freevectorfield. Thisanalysisyieldsanevolutionequationof theform

ut 1

µ0Du

Ψ ∇∆ 1 1divΨ (68)

whereit is understoodthattheLaplacianis invertedwith respectto appropriateboundarycon-

ditions.

Further, a purelylocal flow equationfor theminimizationof theMonge-Kantorovich func-

tionalmaybeobtainedby setting

ζ ∇divΨ ∆Ψ (69)

It is straightforwardto checkthatin this casedivζ 0 and

Mt 1

2 , curlΨ 2 0 (70)

67

Thecorrespondingsecondorderlocal evolutionequationfor u is

ut 1

µ0Du

∇divΨ ∆Ψ (71)

and(70)showsthatat optimality wemusthavecurlΨ 0 soΨ ∇w for somefunctionw5.1.7 Defining the Warping Map

Typically in elasticregistration,onewantsto seeanexplicit warpingwhich smoothlydeforms

oneimageinto theother. Thiscaneasilybedoneusingthesolutionof theMonge–Kantorovich

problem.Thus,weassumenow thatwehaveappliedourgradientdescentprocessasdescribed

aboveandthatit hasconvergedto theMonge–Kantorovich optimalmappingu It is shown in

(BenamouandBrenier, 2000)thattheflow Xx t definedby

Xx t x t

ux" x (72)

is thesolutionto acloselyrelatedminimizationproblemin fluid mechanicsandprovidesappro-

priatejustificationfor using(72) to defineourcontinuouswarpingmapX betweenthedensities

µ0 andµ1. See(McCann,1997)for applicationsandadetailedanalysisof thepropertiesof this

displacementinterpolation.

5.2 Implementation and Examples

We notethateventhoughour non-localmethodrequiresthat theLaplacianbeinvertedduring

eachiteration, the problemhasbeensetup specificallyto allow for the useof standardfast

numericalsolverswhich useFFT-typemethodsandoperateon rectangulargrids (Presset al.,

1992).

We illustrate our methodswith a pair of examples. In Figure 13 we show a brain de-

68

formation sequence.One slice eachfrom two MR datasets,acquiredat the Brigham and

Women’s hospital,wereused. The first datasetwaspre-operative, the secondwasacquired

duringsurgery, aftercraniotomyandopeningof thedura.Both werepre-processedto remove

theskull. TheMonge–Kantorvichmappingwasfoundusingtheevolution equation(62) with

intensityvaluesasdensities,scaledslightly sothat thesumof theintensitieswasthesamefor

bothimages.This processtook roughly10 minuteson a singleprocessorSunUltra 10. A full

3D volumetricdatasetcantake severalhoursto process.Thedisplacementinterpolation(72)

togetherwith (53) for theintensitieswasthenusedto find thecontinuousdeformationthrough

time. The first image,in the upperleft, shows a planaraxial sliceat time t 0 00. Thebot-

tom right is an axial slice at time t 1 00 Together, theseimagesrepresentthe input to our

algorithm.Theupperright andlower left imagesrepresentthecomputedinterpolationat time

t 0 33andt 0 66respectively.

The secondexampleshows an applicationof our methodto surfacewarping. Figure14

shows a portion of the white mattersurfaceobtainedby segmentingan MRI scan. We cut

the surfaceend to end andflattenedit into the planeusing a conformalmappingtechnique

(Angenentet al., 1999a;Angenentet al., 1999b),asshown in the left of Figure15. It is well

known thatasurfaceof non-zeroGaussiancurvaturecannotbeflattenedby any meanswithout

somedistortion.Theconformalmappingis anattemptto preservetheappearanceof thesurface

throughthepreservationof angles.Theconformalmappingis a “similarity in thesmall” and

sofeatureson thesurfaceappearsimilar in theflattenedrepresentation,up to a scalingfactor.

Further, the conformalflatteningmapscanbe calculatedsimply by solving systemsof linear

equations.For a triangulatedsurfaceof a few hundredthousandtriangles,this takesonly a few

minuteson a singleprocessorcomputer. Parallelizationcanbeachievedusingfreely available

numericallinearalgebrasoftware.

However, in someapplicationsit is desirableto beableto preserve areasinsteadof angles,

69

so that the sizesof surfacestructuresare accuratelyrepresentedin the plane. The Monge–

Kantorovich approachallowsusto find suchanarea-correctflattening.Theideahereis thatthe

conformalflatteningshouldbealteredby moving pointsaroundaslittle aspossible.Oncewe

haveconformallyflattenedthesurface,wedefineadensityµ0 tobetheJacobianof theinverseof

theflatteningmap,andsetµ1 to a constant.TheMonge–Kantorovich optimalmappingis then

area-correctingby (53). Theresultingmaptook just a few minutesto calculate.Detail of the

conformalsurfaceflatteningandtheareacorrectedflatteningareshown in Figure15. Although

correctedfor area,surfacestructuresarestill clearly discernible. The curl-freenatureof the

Monge–Kantorovich mappingavoids distortioneffectsoften associatedwith areapreserving

maps.

5.3 Conclusions

In thissection,wepresentedanaturalmethodfor imageregistrationandsurfacewarpingbased

on the classicalproblemof optimal masstransportation. Although appliedhere to the L2

Monge–Kantorovich problem,themethodusedto enforcethemasspreservationconstraintis

general,asshown in Section5.1.6,andwill haveotherapplications.For example,any weighted

linearcombinationof theMonge–Kantorovich functionalanda standardL2 energy functional

or othermatchingfunctionalcanbeused.Theseideasareacurrentareaof research.

70

Figure 13: A brain deformationsequence.The upper left (time t 0 00) and lower right(t 1 00)aretheinput into thealgorithm.Theupperright (t 0 33) andlower left (t 0 67)representtheinterpolationusingtheMonge–Kantorovich displacementmapandenforcingthepreservationof mass.

71

Figure14: White mattersurface

Figure15: Conformal(left) andareapreserving(right) mappings

72

Acknowledgments The authorsthankWilliam M. Wells III for helpful commentsanddis-

cussionduringthepreparationof this chapter.

This work wasfundedin partby theSpanishGovernment(Ministerio deEducacion y Cul-

ture)with a visiting researchfellowship (FPUPRI1999-0175)(JRA), theEuropeanCommis-

sionandtheSpanishGovernment(CICYT), with thejoint researchgrant1FD97-0881-C02-01,

aNew ConceptAwardfrom theCenterfor Integrationof MedicineandInnovativeTechnology

(SKW), andNIH grantsP41RR13218,P01CA67165,R01CA86879andR01RR11747.

References

Alexander, D. andGee,J. (2000). ElasticMatchingof Dif fusion TensorImages. Computer

VisionandImageUnderstanding, 77(2):233–250.

Alexander, D., Gee,J., andBajcsy, R. (1999). Strategies for datareorientationduring non-

rigid transformationsof diffusiontensorimages.In Taylor, C. andColchester, A., editors,

LectureNotesin ComputerScience. MedicalImageComputingandComputerAssistedIn-

tervention- MICCAI ’98, volume1496,pages463–472,Cambridge,UK. Springer-Verlag.

Angenent,S.,Haker, S.,Tannenbaum,A., andKikinis, R. (1999a).Laplace-beltramioperator

andbrainsurfaceflattening.IEEE Trans.on MedicalImaging, 18:700–711.

Angenent,S.,Haker, S.,Tannenbaum,A., andKikinis, R. (1999b). On areapreservingmaps

of minimal distortion. In Djaferis,T. andSchick,I., editors,SystemTheory: Modeling,

Analysis,andControl, pages275–287.Kluwer, Holland.

Bajcsy, R. andKovacic,S.(1989).Multiresolutionelasticmatching.ComputerVision,Graph-

ics,andImageProcessing, 46(1):1–21.

73

Balay, S., Gropp,W. D., McInnes,L. C., andSmith, B. F. (1997). Efficient managementof

parallelismin objectorientednumericalsoftwarelibraries. In Arge,E., Bruaset,A. M.,

and Langtangen,H. P., editors,Modern Software Tools in ScientificComputing, pages

163–202.BirkhauserPress.

Balay, S.,Gropp,W. D., McInnes,L. C., andSmith,B. F. (2000a).PETSc2.0 usersmanual.

TechnicalReportANL-95/11 - Revision2.0.28,ArgonneNationalLaboratory.

Balay, S., Gropp, W. D., McInnes, L. C., and Smith, B. F. (2000b). PETSchomepage.

http://www.mcs.anl.gov/petsc.

Barron,J. L., Fleet,D. J., andBeauchemin,S. S. (1994). Performanceof optical flow tech-

niques.InternationalJournalof ComputerVision, 12(1):43–77.

Basser, P., Mattiello, J., and LeBihan,D. (1994). Estimationof the effective self-diffusion

tensorfrom theNMR spin-echo.Journalof MagneticResonanceB, 103:247–254.

Benamou,J.-D. and Brenier, Y. (2000). A computationalfluid mechanicssolution to the

monge–kantorovich masstransferproblem.NumerischeMathematik, 84:375–393.

Black, P. M., Moriarty, T., Alexander, E., Stieg, P., Woodard,E. J., Gleason,P. L., Martin,

C. H., Kikinis, R., Schwartz, R. B., and Jolesz,F. A. (1997). The Developmentand

Implementationof IntraoperativeMRI andits Neurosurgical Applications.Neurosurgery,

41(4):831–842.

Brandt,J. W. (1997). Improvedaccuracy in gradient-basedoptical flow estimation. Interna-

tional Journalof ComputerVision, 25(1):5–22.

Brenier, Y. (1991).Polarfactorizationandmonotonerearrangementof vector-valuedfunctions.

Com.PureAppl.Math., 64:375–417.

74

Bro-Nielsen,M. (1996). Surgery SimulationUsing FastFinite Elements.In Visualizationin

BiomedicalComputing, pages529–534.

Bro-Nielsen,M. (1998). Finite elementmodelingin medicalVR. Proceedingsof the IEEE

SpecialIssueonVirtual & AugmentedRealityin Medicine, 86(3):490–503.

Bro-Nielsen,M. andGramkow, C. (1996).Fastfluid registrationof medicalimages.In Hohne,

K. H. andKikinis, R., editors,Proceedingsof the 4th InternationalConferenceVisuali-

sationin BiomedicalComputing(VBC’96,heldin Hamburg, Germany, September22-25,

1996), volume 1131 of Lecture Notesin ComputerScience, pages267–276.Springer-

Verlag.

Cachier, P., Pennec,X., andAyache,N. (1999). Fastnonrigid matchingby gradientdescent:

Study and improvementsof the “demons” algorithm. TechnicalReport3706, Institut

National de Rechercheen Informatiqueet en Automatique. http://www.inria.fr/rrrt/rr-

3706.html.

Christensen,G. E., Rabbit,R. D., andMiller, M. I. (1996). Deformabletemplatesusinglarge

deformationkinematics.IEEE Trans.on MedicalImag., 5:1435–1447.

Collins, D. (1994). 3D Model-basedsegmentationof individual brain structuresfor magnetic

resonanceimagingdata. PhDthesis,MontrealNeurologicalInstitute.

Collins, D., Goualher, G. L., andEvans,A. (1998a). Non-linearCerebralRegistrationwith

SulcalConstraints.In MICCAI 1998, pages974–984,Cambridge,MA, USA.

Collins, D. L., Zijdenbos,A. P., Kollokian, V., Sled,J. G., Kabani,N. J., Holmes,C. J., and

Evans,A. C. (1998b).Designandconstructionof a realisticdigital brainphantom.IEEE

Transactionsin MedicalImaging, 17(3):463–468.

75

Cullen,M. andPurser, R. (1984).An extendedlagrangiantheoryof semigeostrophicfrontoge-

nesis.J. Atmos.Sci., 41:1477–1497.

Dacorogna,B. andMoser, J. (1990). On a partialdifferentialequationinvolving the jacobian

determinant.Ann.Inst.H. PoincareAnal.NonLineaire, 7:1–26.

Davatzikos,C. (1997). SpatialTransformationandRegistrationof Brain ImagesUsingElasti-

cally DeformableModels.Comp.Vis.andImageUnderstanding, SpecialIssueonMedical

Imaging, 66(2):207–222.

Dumoulin,C.,Souza,S.,Walker, M., andWagle,W. (1989).Three-dimensionalphasecontrast

angiography. MagneticResonancein Medicine, 9:139–149.

Edwards,P. J.,Hill, D. L. G., Little, J. A., andHawkes,D. J. (1997). Deformationfor Image

GuidedInterventionsUsinga ThreeComponentTissueModel. In IPMI’97, pages218–

231.

Ferrant,M., Cuisenaire,O.,andMacq,B. (1999a).Multi-ObjectSegmentationof BrainStruc-

turesin 3D MRI Using a ComputerizedAtlas. In SPIE Medical Imaging ’99, volume

3661-2,pages986–995.

Ferrant,M., Nabavi, A., Macq, B., andWarfield, S. K. (2000a). DeformableModeling for

CharacterizingBiomedicalShapeChanges.In Borgefors,G., Nystrom, I., andSannitidi

Baja,G., editors,DGCI2000: DiscreteGeometryfor ComputerImagery; 2000Dec 13–

15; Uppsala,Sweden, volume1953of LectureNotesin ComputerScience, pages235–248,

Heidelberg, Germany. Springer.

Ferrant,M., Warfield,S.K., Guttmann,C. R. G., Mulkern,R. V., Jolesz,F. A., andKikinis, R.

(1999b). 3D ImageMatchingUsinga Finite ElementBasedElasticDeformationModel.

In Taylor, C.andColchester, A., editors,MICCAI 99: SecondInternationalConferenceon

76

Medical Image Computingand Computer-AssistedIntervention;1999Sep19–22;Cam-

bridge, England, pages202–209,Heidelberg, Germany. Springer-Verlag.

Ferrant,M., Warfield,S.K., Nabavi, A., Macq,B., andKikinis, R. (2000b).Registrationof 3D

Intraoperative MR Imagesof theBrain Usinga Finite ElementBiomechanicalModel. In

DiGioia, A. M. andDelp, S., editors,MICCAI 2000: Third InternationalConferenceon

MedicalRobotics,ImagingAndComputerAssistedSurgery; 2000Oct11–14;Pittsburgh,

USA, pages19–28,Heidelberg, Germany. Springer-Verlag.

Gaens,T., Maes,F., Vandermeulen,D., andSuetens,P. (1998). Non-rigid Multimodal Image

RegistrationUsing Mutual Information. In Proceedingsof the First InternationalCon-

ferenceon Medical Image ComputingandComputer-AssistedIntervention(MICCAI’98,

Conferenceheldin Cambridge(MA),USA,October11–13,1998), volume1496of Lecture

Notesin ComputerScience, pages1099–1106.SpringerVerlag.

Gangbo,W. (1994).An elementaryproof of thepolarfactorizationof vector-valuedfunctions.

Arch. RationalMechanicsAnal., 128:381–399.

Gangbo,W. andMcCann,R. (1996). The geometryof optimal transportation.Acta Math.,

177:113–161.

Gee,J.andBajcsy, R. (1999).Brain Warping, chapterElasticMatching:ContinuumMechani-

calandProbabilisticAnalysis,pages193–198.AcademicPress.

Geiger, B. (1993).Threedimensionalmodelingof humanorgansandits applicationto diagno-

sisandsurgical planning.TechnicalReport2105,INRIA.

Gering,D., Nabavi, A., Kikinis, R., Grimson,W., Hata,N., Everett,P., Jolesz,F., andWells,

W. (1999).An IntegratedVisualizationSystemfor Surgical PlanningandGuidanceusing

ImageFusionand InterventionalImaging. In MICCAI 99: Proceedingsof the Second

77

InternationalConferenceon Medical Image ComputingandComputerAssistedInterven-

tion, pages809–819.SpringerVerlag.

Gering,D., Nabavi, A., Kikinis, R., Hata,N., O’Donnell, L., Grimson,W., Jolesz,F., Black,

P., andWells III, W. (2001). An integratedvisualizationsystemfor surgical planningand

guidanceusingimagefusionandanopenMR. J MagnResonImaging, 13(6):967–975.

Guimond,A., Roche,A., Ayache,N., andMeunier, J. (2001). Three-dimensionalmultimodal

brainwarpingusingthedemonsalgorithmandadaptiveintensitycorrections.IEEETrans-

actionsin MedicalImaging, 20(1):58–69.

Hagemann,A., Rohr, K., Stiel, H., Spetzger, U., and Gilsbach,J. (1999). Biomechanical

modelingof the humanheadfor physically based,non-rigid imageregistration. IEEE

TransactionsOn MedicalImaging, 18(10):875–884.

Hajnal,J. andBydder, G. (1997). AdvancedMR imaging techniques, chapterClinical usesof

diffusionweightedimaging.Martin Dunitz, ltd.

Haker, S. andTannenbaum,A. (2001a). Optimal transportandimageregistration. submitted

for publication,IEEE Trans.ImageProcessing.

Haker, S. andTannenbaum,A. (2001b). Optimal transportandimageregistration. In IEEE

workshopon variational andlevel setmethodsin computervision,ICCV2001.

Hata,N. (1998). Rigid anddeformablemedicalimage registration for image-guidedsurgery.

PhDthesis,Universityof Tokyo.

Hata,N., Nabavi, A., Wells,W. M., Warfield,S.K., Kikinis, R., Black,P. M., andJolesz,F. A.

(2000). Three-DimensionalOptical Flow Methodfor Measurementof VolumetricBrain

Deformationfrom IntraoperativeMR Images.J ComputAssistTomogr, 24(4):531–538.

78

Hata, N., Wells, W. M., Halle, M., Nakajima,S., Viola, P., Kikinis, R., and Jolesz,F. A.

(1996). ImageGuidedMicroscopicSurgery SystemUsing Mutual-InformationBased

Registration.In Visualizationin BiomedicalComputing, pages307–316.

Hellier, P. andBarillot, C. (2000). Multimodal non-rigidwarpingfor correctionof distortions

in functionalMRI. In Third InternationalConferenceon Medical Image ComputingAnd

Computer-AssistedIntervention(MICCAI’00), volume1935of LectureNotesin Computer

Science, pages512–520,Pittsburgh.SpringerVerlag.

Hermosillo,G., Chefd’Hotel,C., andFaugeras,O. (2001). A VariationalApproachto Multi-

Modal ImageMatching. TechnicalReport4117,Institut NationaldeRechercheenInfor-

matiqueet enAutomatique.

Hill, D., Maurer, C., Maciunas,R., Barwise,J., Fitzpatrick,J., andWang,M. (1998). Mea-

surementof intraoperative brainsurfacedeformationundera craniotomy. Neurosurgery,

43:514–526.

Horn, B. K. P. andSchunck,B. G. (1981). Determiningoptical flow. Artificial Intelligence,

17:185–203.

Jolesz,F. (1997).Image-guidedProceduresandtheOperatingRoomof theFuture.Radiology,

204:601–612.

Kantorovich, L. V. (1948).On aproblemof monge.UspekhiMat. Nauk., 3:225–226.

Kaus,M. R.,Nabavi, A., Mamisch,C.T., Wells,W. M., Jolesz,F. A., Kikinis, R.,andWarfield,

S. K. (2000). Simulationof CorticospinalTractDisplacementin Patientswith Brain Tu-

mors. In DiGioia, A. M. andDelp, S., editors,MICCAI 2000: Third InternationalCon-

ferenceon MedicalRobotics,Imaging AndComputerAssistedSurgery; 2000Oct 11–14;

Pittsburgh,USA, pages9–18,Heidelberg, Germany. Springer-Verlag.

79

Kaus,M. R.,Warfield,S.K., Nabavi, A., Chatzidakis,E.,Black,P. M., Jolesz,F. A., andKiki-

nis, R. (1999). Segmentationof MRI of meningiomasandlow gradegliomas. In Taylor,

C. andColchester, A., editors,MICCAI 99: SecondInternationalConferenceon Medi-

cal Image ComputingandComputer-AssistedIntervention;1999Sep19–22;Cambridge,

England, pages1–10,Heidelberg, Germany. Springer-Verlag.

Kikinis, R., Shenton,M. E., Gerig, G., Martin, J., Anderson,M., Metcalf, D., Guttmann,C.

R. G., McCarley, R. W., Lorenson,W. E., Cline,H., andJolesz,F. (1992).RoutineQuan-

titative Analysisof Brain andCerebrospinalFluid Spaceswith MR Imaging. Journal of

MagneticResonanceImaging, 2:619–629.

Kikinis, R., Shenton,M. E., Iosifescu,D. V., McCarley, R. W., Saiviroonporn,P., Hokama,

H. H., Robatino,A., Metcalf,D., Wible, C. G.,Portas,C. M., Donnino,R. M., andJolesz,

F. A. (1996).A Digital BrainAtlasfor SurgicalPlanning,ModelDrivenSegmentation,and

Teaching.IEEETransactionsonVisualizationandComputerGraphics, 2(3):232–241.

LeBihan,D., Breton,E.,Lallemand,D., Grenier, P., Cabanis,E.,andLaval-Jeantet,M. (1986).

MR imagingof intravoxel incoherentmotions:applicationsto diffusionandperfusionin

neurologicdisorders.Radiology, 161:401–407.

Lester, H., Arridge, S., Jansons,K., Lemieux,L., Hajnal, J., andOatridge,A. (1999). Non-

linear Registrationwith the Variable Viscosity fluid Algorithm. In IPMI 1999, pages

238–251.

Likar, B. andPernus,F. (2000). A HierarchicalApproachto ElasticRegistrationBasedon

Mutual Information. ImageandVisionComputing, 19(1-2):33–44.

Maes,F., Collignon,A., Vandermeulen,D., Marchal,G.,andSuetens,P. (1997).Multimodality

80

imageregistrationby maximizationof mutualinformation.IEEETransactionsin Medical

Imaging, 16(2):187–198.

Maintz,J.B. A., Meijering,E. H. W., andViergever, M. A. (1998).Generalmultimodalelastic

registrationbasedon mutual information. In Hanson,K. M., editor, Medical Imaging

1998: Image Processing(MI’98, Conferenceheld in SanDiego (CA),USA,Febuary 23–

26 1998), volume 3338 of SPIE Proceedings, pages144–154,Bellingham,WA, USA.

InternationSocietyfor OpticalEngineering.

Maintz,J.B. A. andViergever, M. A. (1998).A survey of medicalimageregistration.Medical

ImageAnalysis, 2(1):1–36.

McCann,R. J. (1997).A convexity principlefor interactinggases.Adv. Math., 128:153–179.

Meyer, C. R.,Boes,J.L., Kim, B., andBland,P. H. (1999).ProbabilisticBrainAtlasConstruc-

tion: Thin-PlateSplineWarpingvia Maximizationof Mutual Information. In Proceed-

ingsof theSecondInternationalConferenceonMedicalImageComputingandComputer-

AssistedIntervention(MICCAI’99, Conferenceheld in Cambridge, England,September

19–22,1999), volume1679of Lecture Notesin ComputerScience, pages631–637,Cam-

bridge,UK. SpringerVerlag.

Miga, M., Paulsen,K., Lemery, J.,Hartov, A., andRoberts,D. (1999).In vivo quantificationof

ahomogeneousbraindeformationmodelfor updatingpreoperativeimagesduringsurgery.

IEEE TransactionsOn MedicalImaging, 47(2):266–273.

Miga, M. I., Roberts,D. W., Kennedy, F. E., Platenik,L. A., Hartov, A., Lunn, K. E., and

Paulsen,K. D. (2001).Modelingof RetractionandResectionfor IntraoperativeUpdating

of Images.Neurosurgery, 49:75–85.

81

Moon, T. K. and Stirling, W. C. (2000). Mathematicalmethodsand algorithmsfor signal

processing. Prentice-Hall,NJ,USA.

Moser, J. (1965). On thevolumeelementson a manifold. Trans.Amer. Math.Soc., 120:286–

294.

Nabavi, A., Black, P., Gering,D., Westin,C., Mehta,V., Jr, R. P., Ferrant,M., Warfield, S.,

Hata,N., Schwartz,R.,3rd,W. W., Kikinis, R.,andJolesz,F. (2001).Serialintraoperative

magneticresonanceimagingof brainshift. Neurosurgery, 48(4):787–97.

Paulsen,K., Miga, M., Kennedy, F., Hoopes,P., Hartov, A., andRoberts,D. (1999). A Com-

putationalModel for TrackingSubsurfaceTissueDeformationDuring StereotacticNeu-

rosurgery. IEEE TransactionsOn MedicalImaging, 47(2):213–225.

Peled,S., Gudbjartsson,H., Westin, C.-F., Kikinis, R., andJolesz,F. A. (1998). Magnetic

resonanceimagingshows orientationandasymmetryof white matterfiber tracts. Brain

Research, 780:27–33.

Pennec,X., Cachier, P., andAyache,N. (1999). Understandingthe“demon’s algorithm”: 3D

non-rigid registrationby gradientdescent.In ?, editor, Proceedingsof theSecondInter-

national Conferenceon Medical Image Computingand Computer-AssistedIntervention

(MICCAI’99,Conferenceheldin Cambridge, England,September19–22,1999), volume?

of Lecture Notesin ComputerScience, page? Springer-Verlag.

Pierpaoli,C., Jezzard,P., Basser, P., Barnett,A., andChiro,G. D. (1996).Dif fusiontensorMR

imagingof thehumanbrain. Radiology, 201(3):637–648.

Poggio,T., Torre, V., andKoch, C. (1985). Computationalvision andregularizationtheory.

Nature, 317:314–319.

82

Poupon,C., Clark, C., Frouin,V., LeBihan,D., Bloch, I., andMangin,J.-F. (1999). Inferring

the brain connectivity from MR diffusion tensordata. In Wells, W. M., Colchester, A.,

andDelp, S.,editors,Lecture Notesin ComputerScience. Medical image computingand

computer-assistedintervention- MICCAI ’98, volume1679,pages453–462,Cambridge,

UK. Springer-Verlag.

Poupon,C., Mangin, J.-F., Frouin, V., Regis, J., Poupon,F., Pachot-Clouard,M., LeBihan,

D., andBloch, I. (1998). Regularizationof MR diffusion tensormapsfor trackingbrain

white matterbundles. In Wells, W. M., Colchester, A., and Delp, S., editors,Lecture

Notesin ComputerScience. Medicalimagecomputingandcomputer-assistedintervention

- MICCAI ’98, volume1496,pages489–498,Cambridge,MA, USA. Springer-Verlag.

Press,W., Teukolsky, S.,Vetterling,W., andFlannery, B. (1992). Numericalrecipesin C: The

art of scientificcomputing. CambridgeUniversityPress.

Provenzale,J.andSorensen,A. (1999). Dif fusion-weightedMR imagingin acutestroke: the-

oretical considerationsand clinical applications. AmericanJournal of Roentgenology,

173:1459–1467.

Rachev, S.andRuschendorf,L. (1998).MassTransportationProblems,VolsI andII . Springer,

New York.

Roche,A., Malandain,G., andAyache,N. (2000). Unifying maximumlikelihoodapproaches

in medicalimageregistration. InternationalJournal of Imaging SystemsandTechnology,

11:71–80.

Roche,A., Malandain,G., Pennec,X., and Ayache,N. (1998). The correlationratio as a

new similarity measurefor multimodal imageregistration. In Wells, W. M., Colchester,

A., andDelp, S., editors,Proceedingsof the First InternationalConferenceon Medical

83

Image ComputingandComputer-AssistedIntervention(MICCAI’98, Conferenceheld in

Cambridge(MA),USA,October11–13,1998), volume1496of LectureNotesin Computer

Science, pages1115–1124.Springer-Verlag.http://www.inria.fr/rrrt/rr-3378.html.

Rohr, K. (1997).Dif ferentialoperatorsfor detectingpoint landmarks.ImageandVision Com-

puting, 15:219–233.

Rousseeuw, P. J. andLeroy, A. M. (1987). Robust RegressionandOutlier Detection. Wiley

seriesin probabilityandmathematicalstatistics.JohnWiley & Sons.

Rueckert, D., Clarkson,M., Hill, D., and Hawkes, D. (2000). Non-rigid registrationusing

higher-ordermutualinformation.In MedicalImaging, volume3979of SPIEProceedings,

pages438–447,SanDiego. InternationSocietyfor OpticalEngineering.

Ruiz-Alzola,J.,Kikinis, R., andWestin,C.-F. (2001a).Detectionof point landmarksin multi-

dimensionaltensordata.SignalProcessing, 81(10):InPress.

Ruiz-Alzola,J.,Westin,C.-F., Warfield,S.K., Alberola,C.,Maier, S.,andKikinis, R. (2001b).

Nonrigid Registrationof 3D TensorMedicalData.MedicalImageAnalysis. In Press.

Ruiz-Alzola,J., Westin,C.-F., Warfield,S. K., Nabavi, A., andKikinis, R. (2000). Nonrigid

registrationof 3d scalar, vectorandtensormedicaldata. In Delp, S. L., DiGioia, A. M.,

andJaramaz,B., editors,Lecturenotesin computerscience. Medicalimagecomputingand

computer-assistedintervention.MICCAI ’00, volume1935,pages541–550,Pittsburgh,

PA, USA. Springer-Verlag.

Schroeder, W., Martin, K., andLorensen,B. (1996). TheVisualizationToolkit: An Object-

OrientedApproach to 3D Graphics. PrenticeHall PTR,New Jersey.

Segel,L. (1987).Mathematicsappliedto continuummechanics. Dover, New York, USA.

84

Sierra,R. (2001). Nonrigid Registrationof Dif fusionTensorImages.Master’s thesis,Zurich,

Switzerland:SwissFederalInstituteof Technology.

Simoncelli,E. P. (1994). Designof multi-dimensionalderivativefilters. In InternationalCon-

ferenceon ImageProcessing, Austin,USA. IEEE.

Skrinjar, O. andDuncan,J. S. (1999). Real time 3D brain shift compensation.In IPMI’99,

pages641–649.

Skrinjar, O.,Studholme,C.,Nabavi, A., andDuncan,J.(2001).StepsTowardaStereo-Camera-

GuidedBiomechanicalModel for Brain Shift Compensation.In Proceedingsof Interna-

tional Conferenceof InformationProcessingin MedicalImaging, pages183–189.

Stejskal,E. andTanner, J. (1965).SpinDiffusionMeasurements:SpinEchoesin thePresence

of aTime-dependentField Gradient.Journalof ChemicalPhysics, 42:288–292.

Thirion, J.-P. (1995). Fastnon-rigidmatchingof 3D medicalimages.TechnicalReport2547,

Institut National de Rechercheen Informatiqueet en Automatique,Sophia-Antipolis.

http://www.inria.fr/rrrt/rr-2547.html.

Thirion, J.-P. (1998). Imagematchingas a diffusion process:an analogywith Maxwell’s

demons.MedicalImageAnalysis, 2(3):243–260.

Toga,A. W. (1999).Brain Warping. AcademicPress.

Viola, P. andWells,W. M. (1995).Alignmentby maximizationof mutualinformation.In Fifth

Intl. Conf. ComputerVision (ICCV), pages16–23.IEEEPress.

Viola, P. andWells, W. M. (1997). Alignmentby maximizationof mutualinformation. Inter-

nationalJournal of ComputerVision, 24(2):137–154.

85

Wang,Y. andStaib,L. (2000). Physicalmodel-basednon-rigidregistrationincorporatingsta-

tistical shapeinformation.MedicalImageAnalysis, 4:7–20.

Warfield,S.K., Jolesz,F., andKikinis, R. (1998a).A High PerformanceComputingApproach

to theRegistrationof MedicalImagingData.Parallel Computing, 24(9–10):1345–1368.

Warfield, S. K., Jolesz,F. A., andKikinis, R. (1998b). Real-TimeImageSegmentationfor

Image-GuidedSurgery. In SC1998: High PerformanceNetworkingandComputingCon-

ference;1998Nov 7–13;Orlando,USA, pages1114:1–14.IEEE.

Warfield,S. K., Kaus,M., Jolesz,F. A., andKikinis, R. (2000a).Adaptive,TemplateModer-

ated,SpatiallyVaryingStatisticalClassification.MedImageAnal, 4(1):43–55.

Warfield, S. K., Mulkern, R. V., Winalski, C. S., Jolesz,F. A., andKikinis, R. (2000b). An

ImageProcessingStrategy for the QuantificationandVisualizationof ExerciseInduced

MuscleMRI SignalEnhancement.J MagnResonImaging, 11(5):525–531.

Weickert,J. (1997).Anisotropicdiffusionin imageprocessing. TeubnerVerlag,Stuttgart.

Weickert,J.,terHaarRomeny, B., andViergever, M. (1998).Efficientandreliableschemesfor

nonlineardiffusionfiltering. IEEETrans.on ImageProcessing, 7:398–410.

Weinstein,D. M., Kindlmann,G. L., andLundberg, E. C. (1999). Tensorlines:Advection-

diffusionbasedpropagationthroughdiffusion tensorfields. In Ebert,D., Gross,M., and

Hamann,B., editors,IEEEVisualization’99, pages249–254,SanFrancisco.

Wells, W., Kikinis, R., Grimson,W., andJolesz,F. (1996a). Adaptive segmentationof MRI

data.IEEE TransactionsonMedicalImaging, 15:429–442.

Wells,W. M., Viola,P., Atsumi,H., Nakajima,S.,andKikinis, R. (1996b).Multi-modalvolume

registrationby maximizationof mutualinformation.MedicalImageAnalysis, 1(1):35–51.

86

Westin, C.-F., Maier, S. E., Mamata,H., Nabavi, A., Jolesz,F. A., andKikinis, R. (2001).

ProcessingandVisualizationof Dif fusionTensorMRI. MedicalImageAnalysis. In Press.

Yezzi,A., Tsai,A., andWillsky, A. (2000).Medicalimagesegmentationviacoupledcurveevo-

lution equationswith globalconstraints.In MathematicalMethodsin BiomedicalImage

Analysis, pages12–19.

Zienkiewicz, O. C. andTaylor, R. L. (1994). TheFinite ElementMethod: BasicFormulation

andLinear Problems. McGraw Hill BookCo.,New York, 4th edition.

87