GeometricFusionfor a Hand-held3D Sensor

AdrianHilton andJohnIllingworth

Centrefor Vision,SpeechandSignalProcessing

Universityof Surrey, GuildfordGU25XH,UK

a.hilton@surrey.ac.uk

Abstract

This articlepresentsa geometricfusionalgorithmdevelopedfor thereconstruc-

tion of 3D surfacemodelsfrom hand-heldsensordata.Hand-heldsystemsallow full

3D movementof the sensorto capturethe shapeof complex objects. Techniques

previously developedfor reconstructionfrom conventional2.5D rangeimagedata

cannotbe appliedto hand-heldsensordata. A geometricfusionalgorithmis intro-

ducedto integratethemeasured3D pointsfrom ahand-heldsensorinto asinglecon-

tinuoussurface.Thenew geometricfusionalgorithmis basedon thenormal-volume

representationof a trianglewhichenablesincrementaltransformationof anarbitrary

meshinto an implicit volumetricfield function.Thissystemis demonstratedfor re-

constructionof surfacemodelsfrom both hand-heldsensordataandconventional

2.5Drangeimages.

Key-words: Object Modelling; 3D Reconstruction; Geometric Fusion; Range Image

Integration

1

1 Capturing 3D models of real objects

Realisticobjectrepresentationis aprimarygoalof computergraphicsresearch

[CurlessandLevoy, 1996]. Model constructionis a major bottleneckin the application

of computergeneratedimagery. Applicationsdemandingrealisticmodelsof realobjects

includefilm animation,virtualmuseums,tele-shopping,tele-conferencing,immersivevir-

tual reality, military simulationandmultimediaeducation/entertainment.

Currentlymanualtechniquesareusedto build modelsof real objectsandhave been

usedto build commercial3D model-banks[Infografica,1996, Datalabs,1996]. However,

this is an expensive processrequiringan expert modelleror animatorandtakesseveral

monthsto obtain a single instanceof a complex object. Reconstructioncostsprevent

realisticmodellingof objectswith a high level-of-detail. In addition,eachnew instance

of an object must be reconstructedfrom scratch. This prohibits realistic modellingof

individualinstancesof organicobjectswhichexhibit largeshapevariationssuchaspeople

or animals.

Image-basedrepresentationshavebeenintroduced[Gortler et al., 1996, Levoy andHanrahan,1996]

whichachievephoto-realisticimagesynthesisof objects.Thisapproachencodesthepro-

jectionof all light raysin asceneasa four dimensionalfunction.Thisapproachachieves

highly realisticvisualisationbut is currentlylimited to staticobjectsandsceneswith fixed

lighting.

1.1 3D Surface Measurement

Automaticreconstructionof modelsof 3D objectshasreceived considerableinterestin

computervisionresearchusingbothactive[Hoppeetal., 1992, CurlessandLevoy, 1996,

Hilton et al., 1998, Pito,1996a,Soucy andLaurendeau,1995b, Turk andLevoy, 1994] and

passive [Niem andWingebermuhle,1997, Fitzgibbonet al., 1998] surfacemeasurement

technologies.Active surfacemeasurementtechniquesprojecta structuredlight pattern

suchasa laserstripe onto the object surfaceandby a processof optical triangulation

measurethedistanceto pointson thestripe.This approachenableshigh-accuracy dense

3D surfacemeasurement.Conventionalactive ‘2.5D range’imagesensoracquiresurface

measurementson a grid correspondingto eithera planeor cylinder. Captureof the full

surfaceshapefor 3D objectsrequirestheacquisitionof multiple2.5Dimages.Automatic

acquisitionof full surfaceshaperequirestechniquesfor viewpointplanning.For complex

2

objectsthis mayrequirethesensorto entertheobjectspaceandrisk collisionswith the

unknown objectsurface. A generalsolution to this problemremainsan openresearch

issue,see[Pito, 1996b] for a review of thiswork.

Hand-heldactive measurementdeviceshave recentlybeendevelopedto enablefull

surfaceshapemeasurementfor complex objects. The basicprinciple is to allow full

six degree-of-freedommovementof an active surfacemeasurementdevice in 3D space

aroundthe object. The freedomof movementenablesan operatorto selectviewpoints

to capturethe full objectsurface.This utilisesour expertknowledgein selectingsensor

positionswithin theobjectspacethatavoid collisions.Theusercanalsoselectandmain-

tain the sensorviewpoint to be approximatelynormal to the object surfaceto achieve

accurateand repeatablemeasurement.Providing a fast solution to acquiringsurface

shapefor complex real objects.Several hand-heldsensorsystemshave beendeveloped

[Fisheret al., 1996] andarecommerciallyavailable including systemsby 3D Scanners

(www.3dscanners.com),Polhemus(www.polhemus.com)andEOIS(www.eois.com).The

sensorpositionandorientationis measuredusinganarticulatedarm,electromagneticsen-

soror opticaldevice. Thechoiceof positionsensordeterminesthesystemcost,measure-

mentvolume,accuracy of thesurfacemeasurementsandany restrictionson movement.

An initial calibrationof thesystemregistersthesensorcoordinatesystemwith theposi-

tion sensorcoordinatesystem.Thisenablesmeasurementsof pointson theobjectsurface

with respectto a singleglobalcoordinatesystem.Theaccuracy of thesurfacemeasure-

mentsfor a hand-heldsensoraredependenton both theaccuracy of the positionsensor

andtheaccuracy of therangemeasurement.Throughoutthis work it is assumedthatthe

positionsensoris repeatablewith a zeromeanerrorover theentireobjectsurface.In this

paperwefocusprimarily onthe3D ScannersModelMakersystem(Figure1(a))for which

thetechniquespresentedwereoriginally developed.

A primary differencebetweenhand-heldsensordataand conventional2.5D range

imagedatais thatthemeasurementsarenotstructuredonagrid. With ahand-heldsensor

suchasthe ModelMaker systemthe operatormovesthe projectedlight stripebackand

forth acrossthe surfacein an actionsimilar to paint spraying. This resultsin a series

of stripesof 3D point measurementson the objectsurfacewhich arenot structuredon

a regular grid anddo not have a commonviewpoint. The raw point measurementsare

illustratedin Figure1(b). Algorithmsfor surfacereconstructionfrom conventionalrange

imagedatacannotbe appliedto handheld sensordata. This paperintroducesa new

algorithmto enablereconstructionof surfacemodelsfrom hand-heldsensordata.

3

1.2 Surface Reconstruction

Rangeimageintegration algorithmsdevelopedfor conventionalsensorscan be classi-

fied into two categories: mesh-based which integratedirectly overlappingmeshregions

into a single mesh[Pito, 1996a, Soucy andLaurendeau,1995b, Turk andLevoy, 1994,

Rutishauseret al., 1994]; andvolume-based which constructan intermediatevolumetric

implicit surfacerepresentationfor fusionof overlappingsurfacemeasurements

[CurlessandLevoy, 1996, Hilton et al., 1996, RothandWibowo, 1995].

It hasbeenshown[CurlessandLevoy, 1996, Hilton et al., 1996] thatvolumetricapproaches

achieve improved reconstructionof complex geometryand greatercomputationaleffi-

ciency. Algorithms for reconstructionof objectmodelsfrom conventionalrangeimage

sensordatahaveassumedthatthedataarestructuredonaregular2Dgrid [CurlessandLevoy, 1996,

Hilton et al., 1998, Pito,1996a,Soucy andLaurendeau,1995b, Turk andLevoy, 1994, Rutishauseret al., 1994].

Thisassumptionallowsreliableestimationof thelocalsurfacetopologybasedonthedis-

tancebetweenadjacentmeasurementonthegrid [Soucy andLaurendeau,1995b, Turk andLevoy, 1994].

In addition,many algorithmsassumethattherangeimagemeasurementsaretakenfrom a

commonview direction[CurlessandLevoy, 1996, Soucy andLaurendeau,1995b] in or-

der to evaluatemeasurementoverlap.Both of theseassumptionsprohibit theuseof pre-

viousrangeimageintegrationalgorithmsfor fusionof hand-heldsensordata.

Exceptionsto this are algorithms,which addressthe more generalproblemof re-

constructionfrom unstructureddata[Boissonnat,1984, Hoppeet al., 1992,Mencl,1995].

However, algorithmsfor unstructureddatado not achieve reliable reconstructionfrom

measurementdataof complex objectsas the Euclideandistancebetweenmeasurement

pointsis usedto estimatelocal surfacetopology. This distancemetric fails wherediffer-

entpartsof theobjectsurfacearein closeproximity suchascreaseedgesandthin sections

[Hilton et al., 1998].

In this paperwe introducea new geometricfusionalgorithmfor reliablereconstruc-

tion of objectmodelsfrom hand-heldor conventional3D sensordata. Thealgorithmis

basedon the constructionof an intermediatevolumetric implicit surfacerepresentation

for integrationof overlappingmeasurements.Thenormal-volumerepresentationof a tri-

angleis usedto incrementallytransforman arbitrarymeshinto an implicit volumetric

representation.For hand-heldsensordatawe take advantageof therelationshipbetween

adjacentmeasurementstripesto reliably estimatelocal surfacetopology. This approach

avoids prior assumptionsof a grid measurementstructureor singleviewpoint enabling

4

reconstructionfrom hand-heldsensordata.Thereconstructedobjectsurfacefor a typical

objectis illustratedin Figure1(d).

A digital colour camerain the hand-heldsensoralso enablesacquisitionof set of

multi-view colourfrom known locations.Texturemappingtechniques

[Niem andWingebermuhle,1997] areusedto projectthe imagesonto the reconstructed

3D surfaceand integrateoverlappingimageregions. This resultsin a highly realistic

colour3D objectmodelsuitablefor computergeneratedimagery.

2 Geometric fusion to reconstruct a 3d model

The goal of geometricfusion is to obtaina surfacemodel from the captured3D point

measurements.This sectionpresentsa generalalgorithmfor reconstructinga surfaceby

fusionof arbitrarysetsof surfacemeasurementsinto a singlevolumetricrepresentation.

A single triangulatedmeshmodel of the surfacecan then be obtainedfrom the fused

volumetricrepresentation.

Hoppe[Hoppeet al., 1992] introducedtheuseof anintermediateimplicit volumetric

representationfor fusion of unstructured3D point measurements.This approachwas

extended[CurlessandLevoy, 1996,Hilton etal., 1996] to achievereliablereconstruction

of detailedsurfacesfrom conventionalrangeimagesby assumingthatthemeasurements

areon a regulargrid in orderto estimatethelocal surfacetopology.

Thenew geometricfusionalgorithmpresentedhereenablesreliableandefficient fu-

sion of surfacemeasurementsfrom both conventionalrangeimagesandhand-held3D

sensordata.This is achievedby introducinganincrementalprocedurefor transformation

of an arbitrarymeshto a volumetricfield function. This approachdoesnot requirethe

measurementdatato beeitherstructuredonagrid or haveacommonview direction.The

geometricfusionalgorithmfor generatinga singleintegratedsurfacemodelproceedsin

four stages:

1. Surface Topology Estimation: Estimatethe local surfacetopologyto triangulate

themeasurements.

2. Volumetric Representation: Transformthe surfacetriangulationto an implicit

volumetricrepresentation.

3. Geometric Fusion: Integrateoverlappingvolumetricrepresentationsinto a single

implicit surface.

5

(a)3DScannersModelMaker system

(b) Raw point stripes (c) Triangulatedpatches(partial)

(d) Fusedmodeltriangulation (e)Fusedmodelsurface

Figure1: Model reconstructionfrom hand-heldsensordata

6

4. Triangulation: Triangulatetheimplicit surfaceto generateasinglemeshmodel.

2.1 Surface Topology Estimation

Conventionalrangesensordatais structuredin a2.5Dgrid enablingestimationof thelocal

surfacetopologybasedonthedistancebetweenadjacentpoints.A step discontinuity con-

strained triangulation[Hilton et al., 1998] haspreviouslybeenused[CurlessandLevoy, 1996,

Pito,1996a, Rutishauseretal., 1994, Soucy andLaurendeau,1995a,Turk andLevoy, 1994]

asa initial stepto generatea meshrepresentationwhich approximatesthelocal topology

of themeasuredsurface.

For hand-heldsensordatameasurementsaretakenin stripesacrosstheobjectsurface

asthesensoris moved. We performa stepdiscontinuitytriangulationbetweenpointsin

adjacentstripesto estimatethe local surfacetopology. As in previouswork a threshold

distance,���������

, is usedto testif the surfaceis continuousbetweenadjacentpoints

on consecutive stripes: � ����� ��� ��� ��� where�

is the known samplingresolutionof

thesensorsystemfor a givensurfacedistance.If adjacentpointssatisfythis criteriathen

they areconnectedin a localsurfacetriangulation.Discontinuitiesbetweenpatchesoccur

dueto eithera stepon the objectsurface,the sensorbeingmovedrapidly or the sensor

changingdirection. For a single patchtheremay not be a commonviewpoint as the

sensororientationis changingcontinuouslyduringacquisition.This triangulationresults

in a seriesof overlappingsurfacepatcheswhich approximatethe surfacegeometryand

topology. Theresultingtriangulationfor hand-heldsensordatais illustratedin Figure1(c)

for a subsetof thetotal patchesgeneratedfrom theraw stripedata.Theobjective is then

to integratethesesurfacepatchesinto a singlesurfacerepresentation.Dueto thelack of

a grid structureor singleviewpoint for surfacepatchesthey cannot be integratedusing

previousapproachesfor conventional2.5Drangeimagedata.

2.2 Volumetric Representation

In this sectionwe introducea generalalgorithmfor convertinganarbitrarytriangulated

mesh � to a volumetricrepresentation.An arbitrarytopologyclosedmanifold surface�canberepresentedin implicit form asan iso-surfaceof a spatialfield function, ������� ,

where ��� � "!$#%!'&)( is any point in Euclideanspace,*,+ . Thuswe canrepresenta surface

by definingthefield function ������� asthesigneddistancefrom a point, � , to thenearest

pointonthesurface.Thisgivestheiso-surface���-���.�0/ for all pointson�

and �������21�3/

7

elsewhere. A discretevolumetricrepresentationcanbe implementedby uniform spatial

subdivision into voxelscellsandlocal planarapproximationfor voxelsnearthe implicit

surface, �������4�5/ . In practice,we requirea discretevolumetricrepresentationfor com-

putationalefficiency of thefusionalgorithm[CurlessandLevoy, 1996].

Let usdefinea voxel grid basedon a uniform spatialsubdivision with voxel centres

�6 i¯

�7� ��8!$#9��!'&;:<(givenby:

=>i¯

?A@CB,DFEHGJILKMON IPD�Q �SRJT B2UVEXW�ILKMON IPU;Q �SR�T BZY�E\[]ILKMON I^Y_Q �SRa` (1)

The discreterepresentationis definedin the range� �6 Q �SRV! �6 Qcbed ( with uniform voxel

resolutionin eachspatialdimension� 6 �5�f�7�g#h�5�i& . Thenumberof voxel cells

in eachdimensionis givenby jlk �m�on d_pJq8rtsud pVvxw�yz|{ !�n~} pJq8r9s } pVvxw9yzF{ !�n�� pJq�rts � pVvxw�yzF{ (. Thevolume

occupiedby the voxel cell correspondingto centre �6 i¯

denoted���<�6 i¯

��� � ��"� zF{� !$#9���zF{� !'&;:�� z|{� ( . For agivenpoint � wecanevaluatethecorrespondingvoxel index i¯by:

i¯?A@ E�D��hD�Q �SR NB > T EHU,�hU;Q �SR NB > T EHY4�hY_Q �SR NB > `

(2)

Volumetric surfacerepresentationis achieved by local planarapproximationof the

surfacein thediscretevoxel structure.Givenaninput triangulatedmesh� composedof

a setof vertices � ��� ��a� !;������! �� ��!;������! ��)�O� sJ� ( andtriangles� ��� � � !;������!�����!;������!�� ��� sJ� ( where������� ��a� ! ��)� ! ��-� ( . We assumethat � is a simply connectedmanifold triangulationwith

no self-intersections.Thuswe canestimatethe local surfacenormalfor eachtriangleas

����x  � n¢¡£�¤ s ¡£8¥ y�¦ n¢¡£8§ s ¡£8¥ y¨C¨ n¢¡£�¤ s ¡£8¥ y�¦ n¢¡£8§ s ¡£8¥ y ¨C¨ . Vertex normalscanbe estimatedby a weightedaverageof the

adjacenttrianglenormals[Taubin,1995]:

=© £8¥ ?«ª �_¬ �V­ � © �x ª �'¬ � ­ � (3)

A volumetric envelope is definedaroundthe mesh � which enablesus to convert

to an implicit volumetricrepresentation.An offset surface �¯® for mesh � is givenby

displacingeachmeshvertex by a distance° � in the vertex normal direction suchthat�²±� � �-�]³ ° � ���� . If we let ° � be a constantoffset distance° Q�b�d then the distanceof

all pointson theoffset surface � ® is lessthanor equalto theoffsetdistance° Q�b�d from

the original mesh � . The offset surfaceis a continuousmeshbut may not be a simple

manifold due to self-intersection.Figure2(b) illustratesthe offset surfacefor a cross-

sectionthrougha mesh. We cannow definea volumetric envelopearound � by two

offset meshes�0´ and � s suchthat eachvertex is displacedby a distance° Qcbed and

8

M M

M’M

M-

M+

(a)Meshandvertex normals (b) Offsetsurface (c) Volumetricenvelope

Figure2: Volumetricsurfacerepresentation

� ° Qcbed in the normaldirection respectively. The spaceenclosedby �0´ and � s is a

closedvolumetricenvelopesuchthateverypoint insidethis region is lessthan ° Q�b�d from

themesh� . Thevolumetricenvelopeenclosedby theoffsetsurfacesis illustratedfor a

crosssectionthroughameshin Figure2(c).

For eachtriangle�����µ� ��-� ! ��)� ! ��a� ( in mesh � the offset meshes� ´ and � s define

a closednormal volume, � � � � � � , between� ´� �m� �� ´� ! �� ´� ! �� ´� ( and

� s� �¶� �� s� ! �� s� ! �� s� ( . The

normalvolumefor atriangularelementis illustratedin Figure3(a).Theconceptof avolu-

metric envelope andnormal volume or fundamental prism haspreviouslybeenintroduced

[Cohenet al., 1996] to definea representationfor meshsimplificationwith boundedap-

proximationerror.

Eachside of the normal volume is a surface� �8� constrainedby threevectors: the

triangleedge �·9��� � ��¸� � ��-� ; andthecorrespondingvertex normals ��%� and ��|� . If thevertex

normalsareequal ���� � ��F� thenthesurface� ��� is a plane. In generalif thenormalsare

notequalwecansatisfytheconstraintsby definingthesurface� ��� asabilinearor Bezier

patch[Cohenet al., 1996].

Thevolumetricenvelopefor mesh � betweenoffsetmeshes� ´ and � s is equiv-

alent to the union of the normal volumesfor all triangles ¹ ��ºV��» � � � � � � � � . Therefore,

transformingthemesh� to anapproximatevolumetricrepresentationcanbereducedto

anincrementalprocessof transformingthenormalvolumefor eachtriangle,� � asfollows:

1. Evaluatethenormalvolume ¼ � E�½ � N .2. Find thesetof voxel centresinsidethenormalvolume, ¾-¿X¿X¿ T => i

¯

T ¿X¿X¿SÀ,Á�¼ � E�½ � N .3. For all voxel centresinsidethenormalvolume ¾-¿X¿X¿ T => i

¯

T ¿X¿X¿SÀ constructa local planarsurface

approximationfrom thenearestpoint on ½ � andthecorrespondingnormal, @ =Â i¯

T =©i¯

`.

To ensurethat the normalvolume, � � � � � � , enclosesall points lessthan ° Q�b�d from� � we canevaluatethevertex offsetdistancefor the à �ÅÄ vertex, ° � , as: ° �Æ� � pJq8rn ¡R ¥�Ç ¡R   y . It is

9

n

nv

+ +

+

-

-

-

v v

v

v

v

v

v

v

t

t

t

s

s

s

r

r

r

r

s

nt

(a)Trianglenormal-volume

n

nr

s

rs

rs

-

s-

s

r

s

r+

+

rs

v v

v

v

v

v

rv

p

dn(p )

x

(b) Bilinearsideof normal-volume

Figure3: Normal-volumetrianglerepresentation

assumedthatfor any trianglein mesh� theanglebetweenatrianglenormalandadjacent

vertex normalis ÈÊÉ /¸Ë to avoid thedegeneratecasewherethenormalvolumereducesto

zero.To obtaina closedvolumetricenvelopefor avoxel size� 6 it is necessaryto setthe

offsetdistance° Qcbed�Ì0Í ��� 6 . Thisgivesadiscretefield function ������� representationfor

all pointswith offsetdistancelessthan Î +� � 6 of the implicit surface ���-���Æ�Ï/ or mesh

� .

Efficientencodingof a trianglein avolumetricrepresentationcanbeimplementedby

first evaluatingthe boundingbox for the normalvolumeon the voxel grid� �РQ �SRJ! �РQcb�d ( .

Thentestingeachvoxel insidetheboundingbox if thevoxel center �6 ¡ � is insidethenormal

volume. Testingif a voxel centeris inside the normalvolumecanbe implementedby

evaluatingif thepoint is insideeachof thenormal-volumesides.Not all thevoxelsin the

boundingbox needto be testedaseachgrid row of voxelscanonly enterandleave the

volumeonce.

2.3 Geometric Fusion

Fusionof multipleoverlappingmeshes,� � !;������! � � , canbeachievedusingthevolumetric

surfacerepresentationintroducedin theprevioussection.For eachmesh� � wecandefine

a closedoffsetenvelopebetween� ´� and � s� with offsetdistance° Q�b�d . Thusfor each

10

triangle� � we candefinea normalvolume � � � � � � andincrementallytransformthemesh

� � to a volumetricrepresentation.In overlappingregionsof surfacemeasurementsfrom

differentmeshesthenormalvolumeswill intersectprovidedthemaximummeasurement

error · Q�b�d È0° Q�b�d . If this conditionis satisfiedwe cancombinethefield functionsfrom

differentmeshesinto a singlevolumetricrepresentation.In practicethis conditioncan

always be satisfiedfor a particularsensorby settingthe voxel size� 6 approximately

equalto themaximummeasurementerror.

Fusionof multiple meshesrequiresan‘overlap’ testto determineif surfacemeasure-

mentsfrom differentmeshesin closespatialproximity correspondto thesameor different

regionsof themeasuredobjectsurface. Definition of a robustoverlaptestis critical for

reliablesurfacereconstructionasdiscussedin previouswork [Hilton et al., 1996]. As in

previous work geometricconstraintsareusedto estimateif overlappingmeasurements

correspondto thesamesurfaceregionbasedon thefollowing criteria:

1. Spatialproximity: distancebetweenoverlappingmeasurementsis lessthanthemaximum

distanceÑ Q�b�d .2. Surfaceorientation:overlappingsurfacenormalswith thesameorientation© � ¿ © �"ÒfÓ .3. Measurementuncertainty:likelihoodof measurementoverlapbasedon estimatesof mea-

surementuncertainty.

Spatialproximity providesacoarsetestof measurementoverlapwhichhasbeenused

in previous work [CurlessandLevoy, 1996]. However, this test is unreliablefor sharp

edgesand for surfacesin closeproximity. Surfaceorientationenablesreliable recon-

structionof creaseedgesandthin surfacesectionsfor continuousimplicit surfacerep-

resentation[Hilton et al., 1996]. Someprevious discretevolumetric representationsfor

fusion of rangeimagedata[CurlessandLevoy, 1996] storedonly position information

which doesnot allow a surfaceorientationtest to be performed. The useof an ori-

entationtest increasesrobustnessfor reconstructionof complex surfaces. Surfaceori-

entationwas previously usedwith a discretevolumetric representationby Roth et al.

[RothandWibowo, 1995].

Measurementuncertaintyfor surfacemeasurementscanbeestimatedfrom therelative

orientationof thesurfaceandsensorviewpoint[Soucy andLaurendeau,1995b, Hilton et al., 1998].

If overlappingmeasurementsaredeterminedtocorrespondto thesamesurfaceregionthey

11

maybecombinedaccordingto a weightedaverage[Turk andLevoy, 1994] or maximum

confidence[Pito, 1996a]. Theweightedaveragefor the i¯

�ÅÄvoxel is evaluatedas:

�Ô i¯

� ª�

i¯� º �JÕ

i �̄ ���t� �Ô i �̄ ���t�ª�u � º �VÕ � ���<� �� i

¯

� ª�

i¯� º �VÕ

i �̄ ���t� �� i �̄ ���t�ª�u � º �JÕ

i �̄ ���<� (4)

where,j i¯

is thenumberof measurementsand Õ i �̄ ���t� is theweightbasedonthemeasure-

mentconfidencefor triangle���

.

Fusionof overlappingsurfacemeasurementsis illustratedin Figure 4. The cross-

sectionthroughoverlappingvolumetricenvelopesfor two meshesareillustratedin Figure

4(a).Theresultingvolumetricenvelopeandintegratediso-surfaceaftergeometricfusion

is illustratedin Figure 4(b). Overlappingregions of the volumetric representationare

combinedto obtainasingleiso-surfaceaccordingto thealgorithmpresentedabove.

Thecomputationalandmemorycostof thevolumetricrepresentationdependon the

numberof voxelsinsidethevolumetricenvelope.For a free-formsurfacethis is propor-

tional to thesurfaceareaor thesquareof thevoxel size.Explicit storageof all voxelshasÖ ��j +£ � memorycostwhich is prohibitively expensive. A run-lengthencodedvoxel struc-

ture [CurlessandLevoy, 1996] is usedto achieve a storagecostwhich is proportionalto

thenumberof voxels insidethevolumetricenvelope.This is proportionalto thesurface

arearesultingin a relatively efficient storagecostofÖ ��j �£ � . Therepresentationaccuracy

is inverselyproportionalto voxel size[Hilton et al., 1998].

2.4 Triangulation

Having, constructedafusedvolumetricimplicit surfacerepresentationfrom themeasure-

ment dataan explicit triangulatedmeshrepresentationcan be extractedusing an iso-

surfacepolygonizationalgorithm. Marching Cubesis appliedto obtain a fusedmesh

representationof theobjectsurface[Bloomenthal,1994, LorensenandCline,1987].

Figure5 shows reconstructedtriangulatedmodelsof complex objectusingthe geo-

metricfusionalgorithm.Thetop two objectswerecapturedusingtheModelMakerhand-

heldsensorsystem.Both datasetsconsistof approximatelyonemillion raw datapoints

which aretriangulatedto form severalhundredoverlappingsurfacepatches.Geometric

fusionof thehand-heldsensordataona200MHzPentiumPCplatformis undertwo min-

utesfor bothobjects.Theresultingmodelsconsistof theorderof onehundredthousand

polygons.Thebottomobjectswereacquiredusingconventionallaserrangeimagesensors

12

M

M1

2

M

(a)Overlappingfield-functions (b) Integratedfield-function

Figure4: Fusionof field-functionsfor cross-sectionsthroughoverlappingvolumetricen-

velopes

from multiple views usinga Cyberwarescanner(bunny) andanNRCCscanner(knight)

[Rioux, 1984]. Thedatasetsconsistof eightandtenrangeimagesrespectively andtake

of the orderof oneminuteon a SUN Sparc10platform to reconstructa fusedmodelof

severalhundredthousandpolygons.

Theseresultsdemonstratethat the geometricfusion algorithmpresentedin this pa-

per canbe usedfor both conventionalandhand-heldrangesensors.The datacaptured

usingthehand-heldrangesensorhasapproximately0.2mmrmserror resultingfrom the

accuracy of theFaroBronzearticulatedarmused.In additionthereareoffseterrorsfor

overlappingsurfacemeasurementstaken on differentsweepsof the sensorof the order

of 0.2mm. The resultingmodelswerereconstructedusinga 0.2mmvoxel resolutionto

integrateoverlappingsurfacemeasurements.A smallervoxel resolutionresultsin visible

ridgeson themodelwhereoverlappingmeasurementsfall into differentvoxels. For the

conventionalrangesensordatathe rms error is × /¸Ø|Ù or lessdueto the electro-optical

noiseof thesensor. Themodelswerereconstructedwith a 0.1mmvoxel sizeresultingin

asmoothsurfacemodelwithoutvisible artifactsdueto thediscretevoxel resolution.The

volumetricapproachintroducedin thispaperreconstructscorrectsurfacemodelsfor both

sourcesof data.

2.5 Geometric Fusion for Large Objects

Hand-held3D sensorscan be mountedon instrumentedplatformsor large articulated

armsto enablecaptureof largeobjects[Levoy, 1999]. Theproblemis thento reconstruct

asinglesurfacemodelto therequiredaccuracy.

Reconstructionof large objectsat high-resolutionis prohibitively expensive usinga

singleresolutiondiscretevolumetricrepresentation.For exampleto reconstructanobject

13

Figure5: Reconstructedmodelsfor ahand-heldsensor(top)andconventionalrangeimage

sensor(bottom)

14

the size of a car requiresa volume of approximatelyÚ � × Ù + . However, reconstruction

accuracy mustbeof theorderof 0.2mmto accuratelyrepresentfinesurfacedetailssuchas

edges.Thenumberof voxelswouldthereforebe j +£ �ÜÛ Ú¸× /�/ + . With run-lengthencoding

this givesapproximatelyj �£ �ÝÛ ×¸Þ²� occupiedvoxels with a memoryrequirementofÛ�� ׸ßÆà . This is prohibitivelyexpensive,thereforewerequiretechniquesfor reconstruction

whichdonot requirehigh-resolutionreconstructionof objectsindependentof size.

Theproblemof reconstructingmodelsof largeobjectshasbeenaddressedby subdi-

viding theobjectspaceinto a seriesof adjacentsub-volumes.Geometricfusionfor mea-

surementsin eachsub-volumeis performedusingthealgorithmpresentedin theprevious

sectionto obtainanimplicit surfacerepresentation.Sub-volumesaresetupto overlapby

exactly onevoxel. Thefield-function ������� for theoverlappingsub-voxels is exactly the

sameasidenticalinput dataareusedto evaluatethefield function. TheMarchingCubes

algorithmfor implicit surfacepolygonizationusesthefield functionvaluesat thecentre

of eachvoxel to determinethe intersectionof edgesbetweenvoxel centreswith the iso-

surface �������g�á/ . Eachintersectionwith the iso-surfaceresultsin a new vertex in the

outputfusedmeshmodel.In theoverlappingvoxelsof thesub-volumesthefield-function

valuesarethesameresultingin thesameiso-surfaceedgeintersections.MarchingCubes

generatesasetof outputmeshverticesontheboundaryof thesub-volumewhichareiden-

tical for adjacentsub-volumes.Therefore,a singlemeshrepresentationcanbeextracted

from multiple sub-volumessimply by merging outputmeshverticeson theboundaryof

thesub-volume.

This approachhasbeenusedto obtainmodelsof objectswhich requireprohibitively

large memoryfor a singlevolumetricrepresentation.Sub-volumeswereusedto recon-

structthedwarf modelshown in Figure5 wherethespacewassplit into four sub-volumes.

Thereconstructiontimeusingthesub-volumeapproachis thesameasusingasinglevol-

ume.However, thepeakmemoryrequirementis onequarterof thememoryrequirement

for reconstructingthemodelusingasinglevolume.Thereis nodifferencein theresulting

surfacemodelusingthesub-volumeapproach.

The computationalcost of multiple sub-volumesis the sameas for a single large-

volume. Therefore,this processreducesthe memoryusagewithout requiringincreased

computationalcost.Spatialsubdivisionwith geometricfusionin multiplesub-volumesal-

lowsreconstructionof arbitrarily largeobjectswithout restrictionsdueto finite computer

memorycapacity.

15

3 Conclusions

In this paperwe have introduceda new geometricfusion algorithm for reconstruction

of 3D objectsurfacemodelsfrom hand-heldsensordata. The principal advanceof this

approachis that it avoidstheassumptionsmadein previousfusionalgorithmsdeveloped

for conventionalrangeimagedatabasedon a grid structurewith a commonviewpoint.

Resultsarepresentedfor reconstructionof surfacemodelsfrom both hand-heldsensor

dataandconventionalrangeimages.

The new geometricfusion algorithm is basedon the useof the normal-volume of

a triangle. This normal-volume is usedto incrementallytransforman arbitrary trian-

gulatedmeshinto a discretevolumetric implicit field-functionrepresentation.This ap-

proachavoids any prior assumptionsof a grid structureor commonviewpoint. The

field-functionsfor overlappingsurfacemeasurementscanbeintegratedto definea single

representation.Implicit surfacepolygonizationis performedusingthe MarchingCubes

algorithmto generatea fusedmeshmodelof theobjectsurface.

Large objectscanbe reconstructedby sub-dividing the spaceinto overlappingsub-

volumes. Geometricfusion is performedindependentlyfor eachsub-volume. The re-

sulting surfacemodelsfor eachsub-volumearethenmergedinto a singlemodel. This

approachallows fusionof datafor arbitrarily largeobjectwith finite memoryresources.

The computationalcost is equivalentto fusion for a singlevolumeenclosingthe entire

space,whereasthestoragecostis equivalentto asinglesub-volume.

Acknowledgement

This researchwassupportedby theEPSRC,UK FundingCouncilon AdvancedFellow-

ship AF/95/2531held by Dr. Adrian Hilton andGrantGR/89518‘FunctionalModels:

Building RealisticModelsfor Virtual RealityandAnimation’.

References[Bloomenthal,1994] Bloomenthal,J. (1994). An implicit surfacepolygonizer. Graphics Gems

ed. Heckbert,P.S., 4:324—350.

[Boissonnat,1984] Boissonnat,J. (1984).Geometricstructuresfor three-dimensionalshaperep-resentation.ACM Transactions on Graphics, 3(4):266—286.

[Cohenetal., 1996] Cohen,J., Varshney, A., Manocha,D., Turk, G., Weber, H., Agarwal, P.,Brooks,F., andWright, W. (1996). SimplificationEnvelopes. In ACM Computer GraphicsProc., SIGGRAPH, NewOrleans, USA, pages119—128.

16

[CurlessandLevoy, 1996] Curless,B. andLevoy, M. (1996).A VolumetricMethodfor BuildingComplex Modelsfrom RangeImages.In ACM Computer Graphics Proceedings, SIGGRAPH,NewOrleans,USA, pages303–312.

[Datalabs,1996] Datalabs,V. (1996).Viewpoint Catalog. (http://www.viewpoint.com).

[Fisheretal., 1996] Fisher, R., Fitzgibbon,A., Gionis, A., Wright, M., andEggert,D. (1996).A hand-heldoptical surfacescannerfor enviromentalmodelingandvirtual reality. In Proc.Virtual Reality World, Stuttgart, Germany, pages13—15.

[Fitzgibbonetal., 1998] Fitzgibbon,A. W., Cross,G., and Zisserman,A. (1998). Automatic3D modelconstructionfor turn-tablesequences.In Koch, R. andVan Gool, L., editors,3DStructure from Multiple Images of Large-Scale Environments, LNCS 1506, pages155–170.Springer-Verlag.

[Gortleretal., 1996] Gortler, S., Grzeszczuk,R., Szeliski,R., andCohen,M. (1996). The lu-migraph. In ACM Computer Graphics Proc., SIGGRAPH, NewOrleans, USA, pages43—54.NewOrleans,USA.

[Hilton etal., 1996] Hilton, A., Stoddart,A., Illingworth, J., andWindeatt,T. (1996). Reliablesurfacereconstructionfrom multiple rangeimages.In 4th European Conference on ComputerVision, pages117—126.Springer.

[Hilton etal., 1998] Hilton, A., Stoddart,A., Illingworth, J.,andWindeatt,T. (March1998). Im-plicit surfacebasedgeometricfusion. International Journal of Computer Vision and ImageUnderstanding, Special Issue on CAD Based Vision, 69(3):273–291.

[Hoppeetal., 1992] Hoppe,H., DeRose,T., Duchamp,T., McDonald,J.,andStuetzle,W. (1992).Surfacereconstructionfrom unorganisedpoints.Computer Graphics, 26(2):71—77.

[Infografica,1996] Infografica, R. (1996). REM 3D Models Bank Catalog.(http://www.infografica.com).

[Levoy, 1999] Levoy, M. (1999).TheDigital MichelangeloProject.In Second IEEE InternationalConference on 3D Digital Imaging and Modeling, Ottawa, Canada.

[Levoy andHanrahan,1996] Levoy, M. andHanrahan,P. (1996).Light FieldRendering.In ACMComputer Graphics Proc., SIGGRAPH, NewOrleans, USA, pages31—42.

[LorensenandCline,1987] Lorensen,W. andCline,H. (1987). Marchingcubes:A high resolu-tion 3d surfaceconstructionalgorithm.Computer Graphics, 21(4):163—169.

[Mencl, 1995] Mencl, R. (1995). A graph-basedapproachto surfacereconstruction.In EURO-GRAPHICS 14(3), pages446—456.

[Niem andWingebermuhle,1997] Niem, W. andWingebermuhle,J. (1997). Automatic recon-structionof 3d objectsusinga mobilemonoscopiccamera.In Proc. IEEE International Con-ferences on Recent Advances in 3D Digital Imaging and Modeling, Ottawa, Canada, pages173—180.

[Pito, 1996a] Pito, R. (1996a). Mesh integrationbasedon co-measurements.In InternationalConf. on Image Processing, pages397—400.Lusanne,Switzerland.

[Pito, 1996b] Pito, R. (1996b). A sensorbasedsolutionto thenext bestview problem. In Inter-national Conf. on Pattern Recognition, pages941—945.Vienna,Austria.

[Rioux, 1984] Rioux, M. (1984). Laserrangefinder basedon synchronizedscanners.AppliedOptics, 23(21):3837—3844.

[RothandWibowo, 1995] Roth,G. andWibowo, E. (1995). A fastalgorithmfor makingmesh-modelsfrom multiple-view rangedata. In Proceedings of the 1995 Robotics and KnowledgeBased Systems Workshop, pages349–355.

[Rutishauseretal., 1994] Rutishauser, M., Stricker, M., andTrobina,M. (1994). Merging rangeimagesof arbitrarily shapedobjects.In Proceedings of IEEE Conference on Computer Visionand Pattern Recognition, pages573—580.

17

[Soucy andLaurendeau,1995a] Soucy, M. andLaurendeau,D. (1995a). A dynamicintegrationalgorithmto modelsurfacesfrom multiple rangeimages. Machine Vision and Applications,8:53–62.

[Soucy andLaurendeau,1995b] Soucy, M. andLaurendeau,D. (1995b). A generalsurfaceap-proachto the integrationof a setof rangeviews. IEEE Trans. Pattern Analysis and MachineIntelligence, 14(4):344–358.

[Taubin,1995] Taubin,G. (1995). Estimatingthe tensorof curvatureof a surfacefrom a poly-hedralapproximation. In Proccedings IEEE International Conference on Computer Vision,Boston, USA, pages902—907.

[Turk andLevoy, 1994] Turk, G. andLevoy, M. (1994). ZipperedPolygonMeshesfrom RangeImages.In ACM Computer Graphics Proceedings, SIGGRAPH, Orlando, Florida, pages311–318.

18

Hilton Illingworth

Author BibliographyDr. Adrian Hilton received a B.Sc.(Hons.) in MechanicalEngineeringanda D.Phil. degreefrom theUniversityof Sussex in 1988and1992respectively. In 1992hejoinedtheCVSSPat theUniversity of Surrey working on 3D shapecapturefor realisticcomputergraphicsandaccurateinspection.This work receivedBestPaperawardsfrom the journalsPatternRecognitionin 1996andIEE ElectronicsandCommunicationsin 1999.Collaborationwith 3D ScannersLtd. resultedin the first hand-heldsystemfor capturing3D objectmodelswhich wasawardeda 1996EU IT‘GrandPrize’ for innovation. In 1997hewasawardeda five yearEPSRCAdvancedFellowshipandin 1999becamelecturerin 3D Digital MediaandBroadcast.Researchinto 3D humanmod-elling from digital photographshasbeenexploitedby AvatarMeLtd. to developthefirst 3D boothfor generatinganimatedmodelsof people.His active researchinterestsincludemodel-based3Dvision for capturinghumanmovementandphoto-realistic3D modelsfor computeranimation.

Professor John Illingworth is Professorof MachineVision at Surrey University. He hasa doc-toratein Physicsbut hasbeenactive in researchin computervision, imageprocessingandpatternrecognitionfor thelast18 years.He haspublishedover 130papersin refereedjournalsandcon-ferences.He is a Fellow of the Institution of ElectricalEngineersandeditsIEE ProceedingsonVision ImageandSignalProcessing.

19

Figure CaptionsFigure1: Model reconstructionfrom hand-heldsensordata(a)3DScannersModelMaker system(b) Raw point stripes(c) Triangulatedpatches(partial)(d) Fusedmodeltriangulation(e)Fusedmodelsurface

Figure2: Volumetricsurfacerepresentation(a)Meshandvertex normals(b) Offsetsurface(c) Volumetricenvelope

Figure3:Normal-volumetrianglerepresentation(a)Trianglenormal-volume(b) Bilinear sideof normal-volume

Figure4: Fusionof field-functionsfor cross-sectionsthroughoverlappingvolumetricenvelopes(a)Overlappingfield-functions(b) Integratedfield-function

Figure5: Reconstructedmodelsfor ahand-heldsensor(top)andconventionalrangeimagesensor(bottom)

20

Corresponding AuthorAdrianHilton

Centrefor Vision,SpeechandSignalProcessingUniversityof SurreyGuildfordGU25XHUKTel: +44-1483-873956Fax: +44-1483-876031a.hilton@surrey.ac.uk

21

22

M

M

M’

M

M-

M+

23

n

nv

+ +

+

-

-

-

v v

v

v

v

v

v

v

t

t

t

s

s

s

r

r

r

r

s

nt

n

nr

s

rs

rs

-

s-

s

r

s

r+

+

rs

v v

v

v

v

v

rv

p

dn(p )

x

24

M

M1

2

M

25

26