GeometricFusionfor a Hand-held3D Sensor
AdrianHilton andJohnIllingworth
Centrefor Vision,SpeechandSignalProcessing
Universityof Surrey, GuildfordGU25XH,UK
Abstract
This articlepresentsa geometricfusionalgorithmdevelopedfor thereconstruc-
tion of 3D surfacemodelsfrom hand-heldsensordata.Hand-heldsystemsallow full
3D movementof the sensorto capturethe shapeof complex objects. Techniques
previously developedfor reconstructionfrom conventional2.5D rangeimagedata
cannotbe appliedto hand-heldsensordata. A geometricfusionalgorithmis intro-
ducedto integratethemeasured3D pointsfrom ahand-heldsensorinto asinglecon-
tinuoussurface.Thenew geometricfusionalgorithmis basedon thenormal-volume
representationof a trianglewhichenablesincrementaltransformationof anarbitrary
meshinto an implicit volumetricfield function.Thissystemis demonstratedfor re-
constructionof surfacemodelsfrom both hand-heldsensordataandconventional
2.5Drangeimages.
Key-words: Object Modelling; 3D Reconstruction; Geometric Fusion; Range Image
Integration
1
1 Capturing 3D models of real objects
Realisticobjectrepresentationis aprimarygoalof computergraphicsresearch
[CurlessandLevoy, 1996]. Model constructionis a major bottleneckin the application
of computergeneratedimagery. Applicationsdemandingrealisticmodelsof realobjects
includefilm animation,virtualmuseums,tele-shopping,tele-conferencing,immersivevir-
tual reality, military simulationandmultimediaeducation/entertainment.
Currentlymanualtechniquesareusedto build modelsof real objectsandhave been
usedto build commercial3D model-banks[Infografica,1996, Datalabs,1996]. However,
this is an expensive processrequiringan expert modelleror animatorandtakesseveral
monthsto obtain a single instanceof a complex object. Reconstructioncostsprevent
realisticmodellingof objectswith a high level-of-detail. In addition,eachnew instance
of an object must be reconstructedfrom scratch. This prohibits realistic modellingof
individualinstancesof organicobjectswhichexhibit largeshapevariationssuchaspeople
or animals.
Image-basedrepresentationshavebeenintroduced[Gortler et al., 1996, Levoy andHanrahan,1996]
whichachievephoto-realisticimagesynthesisof objects.Thisapproachencodesthepro-
jectionof all light raysin asceneasa four dimensionalfunction.Thisapproachachieves
highly realisticvisualisationbut is currentlylimited to staticobjectsandsceneswith fixed
lighting.
1.1 3D Surface Measurement
Automaticreconstructionof modelsof 3D objectshasreceived considerableinterestin
computervisionresearchusingbothactive[Hoppeetal., 1992, CurlessandLevoy, 1996,
Hilton et al., 1998, Pito,1996a,Soucy andLaurendeau,1995b, Turk andLevoy, 1994] and
passive [Niem andWingebermuhle,1997, Fitzgibbonet al., 1998] surfacemeasurement
technologies.Active surfacemeasurementtechniquesprojecta structuredlight pattern
suchasa laserstripe onto the object surfaceandby a processof optical triangulation
measurethedistanceto pointson thestripe.This approachenableshigh-accuracy dense
3D surfacemeasurement.Conventionalactive ‘2.5D range’imagesensoracquiresurface
measurementson a grid correspondingto eithera planeor cylinder. Captureof the full
surfaceshapefor 3D objectsrequirestheacquisitionof multiple2.5Dimages.Automatic
acquisitionof full surfaceshaperequirestechniquesfor viewpointplanning.For complex
2
objectsthis mayrequirethesensorto entertheobjectspaceandrisk collisionswith the
unknown objectsurface. A generalsolution to this problemremainsan openresearch
issue,see[Pito, 1996b] for a review of thiswork.
Hand-heldactive measurementdeviceshave recentlybeendevelopedto enablefull
surfaceshapemeasurementfor complex objects. The basicprinciple is to allow full
six degree-of-freedommovementof an active surfacemeasurementdevice in 3D space
aroundthe object. The freedomof movementenablesan operatorto selectviewpoints
to capturethe full objectsurface.This utilisesour expertknowledgein selectingsensor
positionswithin theobjectspacethatavoid collisions.Theusercanalsoselectandmain-
tain the sensorviewpoint to be approximatelynormal to the object surfaceto achieve
accurateand repeatablemeasurement.Providing a fast solution to acquiringsurface
shapefor complex real objects.Several hand-heldsensorsystemshave beendeveloped
[Fisheret al., 1996] andarecommerciallyavailable including systemsby 3D Scanners
(www.3dscanners.com),Polhemus(www.polhemus.com)andEOIS(www.eois.com).The
sensorpositionandorientationis measuredusinganarticulatedarm,electromagneticsen-
soror opticaldevice. Thechoiceof positionsensordeterminesthesystemcost,measure-
mentvolume,accuracy of thesurfacemeasurementsandany restrictionson movement.
An initial calibrationof thesystemregistersthesensorcoordinatesystemwith theposi-
tion sensorcoordinatesystem.Thisenablesmeasurementsof pointson theobjectsurface
with respectto a singleglobalcoordinatesystem.Theaccuracy of thesurfacemeasure-
mentsfor a hand-heldsensoraredependenton both theaccuracy of the positionsensor
andtheaccuracy of therangemeasurement.Throughoutthis work it is assumedthatthe
positionsensoris repeatablewith a zeromeanerrorover theentireobjectsurface.In this
paperwefocusprimarily onthe3D ScannersModelMakersystem(Figure1(a))for which
thetechniquespresentedwereoriginally developed.
A primary differencebetweenhand-heldsensordataand conventional2.5D range
imagedatais thatthemeasurementsarenotstructuredonagrid. With ahand-heldsensor
suchasthe ModelMaker systemthe operatormovesthe projectedlight stripebackand
forth acrossthe surfacein an actionsimilar to paint spraying. This resultsin a series
of stripesof 3D point measurementson the objectsurfacewhich arenot structuredon
a regular grid anddo not have a commonviewpoint. The raw point measurementsare
illustratedin Figure1(b). Algorithmsfor surfacereconstructionfrom conventionalrange
imagedatacannotbe appliedto handheld sensordata. This paperintroducesa new
algorithmto enablereconstructionof surfacemodelsfrom hand-heldsensordata.
3
1.2 Surface Reconstruction
Rangeimageintegration algorithmsdevelopedfor conventionalsensorscan be classi-
fied into two categories: mesh-based which integratedirectly overlappingmeshregions
into a single mesh[Pito, 1996a, Soucy andLaurendeau,1995b, Turk andLevoy, 1994,
Rutishauseret al., 1994]; andvolume-based which constructan intermediatevolumetric
implicit surfacerepresentationfor fusionof overlappingsurfacemeasurements
[CurlessandLevoy, 1996, Hilton et al., 1996, RothandWibowo, 1995].
It hasbeenshown[CurlessandLevoy, 1996, Hilton et al., 1996] thatvolumetricapproaches
achieve improved reconstructionof complex geometryand greatercomputationaleffi-
ciency. Algorithms for reconstructionof objectmodelsfrom conventionalrangeimage
sensordatahaveassumedthatthedataarestructuredonaregular2Dgrid [CurlessandLevoy, 1996,
Hilton et al., 1998, Pito,1996a,Soucy andLaurendeau,1995b, Turk andLevoy, 1994, Rutishauseret al., 1994].
Thisassumptionallowsreliableestimationof thelocalsurfacetopologybasedonthedis-
tancebetweenadjacentmeasurementonthegrid [Soucy andLaurendeau,1995b, Turk andLevoy, 1994].
In addition,many algorithmsassumethattherangeimagemeasurementsaretakenfrom a
commonview direction[CurlessandLevoy, 1996, Soucy andLaurendeau,1995b] in or-
der to evaluatemeasurementoverlap.Both of theseassumptionsprohibit theuseof pre-
viousrangeimageintegrationalgorithmsfor fusionof hand-heldsensordata.
Exceptionsto this are algorithms,which addressthe more generalproblemof re-
constructionfrom unstructureddata[Boissonnat,1984, Hoppeet al., 1992,Mencl,1995].
However, algorithmsfor unstructureddatado not achieve reliable reconstructionfrom
measurementdataof complex objectsas the Euclideandistancebetweenmeasurement
pointsis usedto estimatelocal surfacetopology. This distancemetric fails wherediffer-
entpartsof theobjectsurfacearein closeproximity suchascreaseedgesandthin sections
[Hilton et al., 1998].
In this paperwe introducea new geometricfusionalgorithmfor reliablereconstruc-
tion of objectmodelsfrom hand-heldor conventional3D sensordata. Thealgorithmis
basedon the constructionof an intermediatevolumetric implicit surfacerepresentation
for integrationof overlappingmeasurements.Thenormal-volumerepresentationof a tri-
angleis usedto incrementallytransforman arbitrarymeshinto an implicit volumetric
representation.For hand-heldsensordatawe take advantageof therelationshipbetween
adjacentmeasurementstripesto reliably estimatelocal surfacetopology. This approach
avoids prior assumptionsof a grid measurementstructureor singleviewpoint enabling
4
reconstructionfrom hand-heldsensordata.Thereconstructedobjectsurfacefor a typical
objectis illustratedin Figure1(d).
A digital colour camerain the hand-heldsensoralso enablesacquisitionof set of
multi-view colourfrom known locations.Texturemappingtechniques
[Niem andWingebermuhle,1997] areusedto projectthe imagesonto the reconstructed
3D surfaceand integrateoverlappingimageregions. This resultsin a highly realistic
colour3D objectmodelsuitablefor computergeneratedimagery.
2 Geometric fusion to reconstruct a 3d model
The goal of geometricfusion is to obtaina surfacemodel from the captured3D point
measurements.This sectionpresentsa generalalgorithmfor reconstructinga surfaceby
fusionof arbitrarysetsof surfacemeasurementsinto a singlevolumetricrepresentation.
A single triangulatedmeshmodel of the surfacecan then be obtainedfrom the fused
volumetricrepresentation.
Hoppe[Hoppeet al., 1992] introducedtheuseof anintermediateimplicit volumetric
representationfor fusion of unstructured3D point measurements.This approachwas
extended[CurlessandLevoy, 1996,Hilton etal., 1996] to achievereliablereconstruction
of detailedsurfacesfrom conventionalrangeimagesby assumingthatthemeasurements
areon a regulargrid in orderto estimatethelocal surfacetopology.
Thenew geometricfusionalgorithmpresentedhereenablesreliableandefficient fu-
sion of surfacemeasurementsfrom both conventionalrangeimagesandhand-held3D
sensordata.This is achievedby introducinganincrementalprocedurefor transformation
of an arbitrarymeshto a volumetricfield function. This approachdoesnot requirethe
measurementdatato beeitherstructuredonagrid or haveacommonview direction.The
geometricfusionalgorithmfor generatinga singleintegratedsurfacemodelproceedsin
four stages:
1. Surface Topology Estimation: Estimatethe local surfacetopologyto triangulate
themeasurements.
2. Volumetric Representation: Transformthe surfacetriangulationto an implicit
volumetricrepresentation.
3. Geometric Fusion: Integrateoverlappingvolumetricrepresentationsinto a single
implicit surface.
5
(a)3DScannersModelMaker system
(b) Raw point stripes (c) Triangulatedpatches(partial)
(d) Fusedmodeltriangulation (e)Fusedmodelsurface
Figure1: Model reconstructionfrom hand-heldsensordata
6
4. Triangulation: Triangulatetheimplicit surfaceto generateasinglemeshmodel.
2.1 Surface Topology Estimation
Conventionalrangesensordatais structuredin a2.5Dgrid enablingestimationof thelocal
surfacetopologybasedonthedistancebetweenadjacentpoints.A step discontinuity con-
strained triangulation[Hilton et al., 1998] haspreviouslybeenused[CurlessandLevoy, 1996,
Pito,1996a, Rutishauseretal., 1994, Soucy andLaurendeau,1995a,Turk andLevoy, 1994]
asa initial stepto generatea meshrepresentationwhich approximatesthelocal topology
of themeasuredsurface.
For hand-heldsensordatameasurementsaretakenin stripesacrosstheobjectsurface
asthesensoris moved. We performa stepdiscontinuitytriangulationbetweenpointsin
adjacentstripesto estimatethe local surfacetopology. As in previouswork a threshold
distance,���������
, is usedto testif the surfaceis continuousbetweenadjacentpoints
on consecutive stripes: � ����� ��� ��� ��� where�
is the known samplingresolutionof
thesensorsystemfor a givensurfacedistance.If adjacentpointssatisfythis criteriathen
they areconnectedin a localsurfacetriangulation.Discontinuitiesbetweenpatchesoccur
dueto eithera stepon the objectsurface,the sensorbeingmovedrapidly or the sensor
changingdirection. For a single patchtheremay not be a commonviewpoint as the
sensororientationis changingcontinuouslyduringacquisition.This triangulationresults
in a seriesof overlappingsurfacepatcheswhich approximatethe surfacegeometryand
topology. Theresultingtriangulationfor hand-heldsensordatais illustratedin Figure1(c)
for a subsetof thetotal patchesgeneratedfrom theraw stripedata.Theobjective is then
to integratethesesurfacepatchesinto a singlesurfacerepresentation.Dueto thelack of
a grid structureor singleviewpoint for surfacepatchesthey cannot be integratedusing
previousapproachesfor conventional2.5Drangeimagedata.
2.2 Volumetric Representation
In this sectionwe introducea generalalgorithmfor convertinganarbitrarytriangulated
mesh � to a volumetricrepresentation.An arbitrarytopologyclosedmanifold surface�canberepresentedin implicit form asan iso-surfaceof a spatialfield function, ������� ,
where ��� � "!$#%!'&)( is any point in Euclideanspace,*,+ . Thuswe canrepresenta surface
by definingthefield function ������� asthesigneddistancefrom a point, � , to thenearest
pointonthesurface.Thisgivestheiso-surface���-���.�0/ for all pointson�
and �������21�3/
7
elsewhere. A discretevolumetricrepresentationcanbe implementedby uniform spatial
subdivision into voxelscellsandlocal planarapproximationfor voxelsnearthe implicit
surface, �������4�5/ . In practice,we requirea discretevolumetricrepresentationfor com-
putationalefficiency of thefusionalgorithm[CurlessandLevoy, 1996].
Let usdefinea voxel grid basedon a uniform spatialsubdivision with voxel centres
�6 i¯
�7� ��8!$#9��!'&;:<(givenby:
=>i¯
?A@CB,DFEHGJILKMON IPD�Q �SRJT B2UVEXW�ILKMON IPU;Q �SR�T BZY�E\[]ILKMON I^Y_Q �SRa` (1)
The discreterepresentationis definedin the range� �6 Q �SRV! �6 Qcbed ( with uniform voxel
resolutionin eachspatialdimension� 6 �5�f�7�g#h�5�i& . Thenumberof voxel cells
in eachdimensionis givenby jlk �m�on d_pJq8rtsud pVvxw�yz|{ !�n~} pJq8r9s } pVvxw9yzF{ !�n�� pJq�rts � pVvxw�yzF{ (. Thevolume
occupiedby the voxel cell correspondingto centre �6 i¯
denoted���<�6 i¯
��� � ��"� zF{� !$#9���zF{� !'&;:�� z|{� ( . For agivenpoint � wecanevaluatethecorrespondingvoxel index i¯by:
i¯?A@ E�D��hD�Q �SR NB > T EHU,�hU;Q �SR NB > T EHY4�hY_Q �SR NB > `
(2)
Volumetric surfacerepresentationis achieved by local planarapproximationof the
surfacein thediscretevoxel structure.Givenaninput triangulatedmesh� composedof
a setof vertices � ��� ��a� !;������! �� ��!;������! ��)�O� sJ� ( andtriangles� ��� � � !;������!�����!;������!�� ��� sJ� ( where������� ��a� ! ��)� ! ��-� ( . We assumethat � is a simply connectedmanifold triangulationwith
no self-intersections.Thuswe canestimatethe local surfacenormalfor eachtriangleas
����x � n¢¡£�¤ s ¡£8¥ y�¦ n¢¡£8§ s ¡£8¥ y¨C¨ n¢¡£�¤ s ¡£8¥ y�¦ n¢¡£8§ s ¡£8¥ y ¨C¨ . Vertex normalscanbe estimatedby a weightedaverageof the
adjacenttrianglenormals[Taubin,1995]:
=© £8¥ ?«ª �_¬ �V � © �x ª �'¬ � � (3)
A volumetric envelope is definedaroundthe mesh � which enablesus to convert
to an implicit volumetricrepresentation.An offset surface �¯® for mesh � is givenby
displacingeachmeshvertex by a distance° � in the vertex normal direction suchthat�²±� � �-�]³ ° � ���� . If we let ° � be a constantoffset distance° Q�b�d then the distanceof
all pointson theoffset surface � ® is lessthanor equalto theoffsetdistance° Q�b�d from
the original mesh � . The offset surfaceis a continuousmeshbut may not be a simple
manifold due to self-intersection.Figure2(b) illustratesthe offset surfacefor a cross-
sectionthrougha mesh. We cannow definea volumetric envelopearound � by two
offset meshes�0´ and � s suchthat eachvertex is displacedby a distance° Qcbed and
8
M M
M’M
M-
M+
(a)Meshandvertex normals (b) Offsetsurface (c) Volumetricenvelope
Figure2: Volumetricsurfacerepresentation
� ° Qcbed in the normaldirection respectively. The spaceenclosedby �0´ and � s is a
closedvolumetricenvelopesuchthateverypoint insidethis region is lessthan ° Q�b�d from
themesh� . Thevolumetricenvelopeenclosedby theoffsetsurfacesis illustratedfor a
crosssectionthroughameshin Figure2(c).
For eachtriangle�����µ� ��-� ! ��)� ! ��a� ( in mesh � the offset meshes� ´ and � s define
a closednormal volume, � � � � � � , between� ´� �m� �� ´� ! �� ´� ! �� ´� ( and
� s� �¶� �� s� ! �� s� ! �� s� ( . The
normalvolumefor atriangularelementis illustratedin Figure3(a).Theconceptof avolu-
metric envelope andnormal volume or fundamental prism haspreviouslybeenintroduced
[Cohenet al., 1996] to definea representationfor meshsimplificationwith boundedap-
proximationerror.
Eachside of the normal volume is a surface� �8� constrainedby threevectors: the
triangleedge �·9��� � ��¸� � ��-� ; andthecorrespondingvertex normals ��%� and ��|� . If thevertex
normalsareequal ���� � ��F� thenthesurface� ��� is a plane. In generalif thenormalsare
notequalwecansatisfytheconstraintsby definingthesurface� ��� asabilinearor Bezier
patch[Cohenet al., 1996].
Thevolumetricenvelopefor mesh � betweenoffsetmeshes� ´ and � s is equiv-
alent to the union of the normal volumesfor all triangles ¹ ��ºV��» � � � � � � � � . Therefore,
transformingthemesh� to anapproximatevolumetricrepresentationcanbereducedto
anincrementalprocessof transformingthenormalvolumefor eachtriangle,� � asfollows:
1. Evaluatethenormalvolume ¼ � E�½ � N .2. Find thesetof voxel centresinsidethenormalvolume, ¾-¿X¿X¿ T => i
¯
T ¿X¿X¿SÀ,Á�¼ � E�½ � N .3. For all voxel centresinsidethenormalvolume ¾-¿X¿X¿ T => i
¯
T ¿X¿X¿SÀ constructa local planarsurface
approximationfrom thenearestpoint on ½ � andthecorrespondingnormal, @ =Â i¯
T =©i¯
`.
To ensurethat the normalvolume, � � � � � � , enclosesall points lessthan ° Q�b�d from� � we canevaluatethevertex offsetdistancefor the à �ÅÄ vertex, ° � , as: ° �Æ� � pJq8rn ¡R ¥�Ç ¡R y . It is
9
n
nv
+ +
+
-
-
-
v v
v
v
v
v
v
v
t
t
t
s
s
s
r
r
r
r
s
nt
(a)Trianglenormal-volume
n
nr
s
rs
rs
-
s-
s
r
s
r+
+
rs
v v
v
v
v
v
rv
p
dn(p )
x
(b) Bilinearsideof normal-volume
Figure3: Normal-volumetrianglerepresentation
assumedthatfor any trianglein mesh� theanglebetweenatrianglenormalandadjacent
vertex normalis ÈÊÉ /¸Ë to avoid thedegeneratecasewherethenormalvolumereducesto
zero.To obtaina closedvolumetricenvelopefor avoxel size� 6 it is necessaryto setthe
offsetdistance° Qcbed�Ì0Í ��� 6 . Thisgivesadiscretefield function ������� representationfor
all pointswith offsetdistancelessthan Î +� � 6 of the implicit surface ���-���Æ�Ï/ or mesh
� .
Efficientencodingof a trianglein avolumetricrepresentationcanbeimplementedby
first evaluatingthe boundingbox for the normalvolumeon the voxel grid� �РQ �SRJ! �РQcb�d ( .
Thentestingeachvoxel insidetheboundingbox if thevoxel center �6 ¡ � is insidethenormal
volume. Testingif a voxel centeris inside the normalvolumecanbe implementedby
evaluatingif thepoint is insideeachof thenormal-volumesides.Not all thevoxelsin the
boundingbox needto be testedaseachgrid row of voxelscanonly enterandleave the
volumeonce.
2.3 Geometric Fusion
Fusionof multipleoverlappingmeshes,� � !;������! � � , canbeachievedusingthevolumetric
surfacerepresentationintroducedin theprevioussection.For eachmesh� � wecandefine
a closedoffsetenvelopebetween� ´� and � s� with offsetdistance° Q�b�d . Thusfor each
10
triangle� � we candefinea normalvolume � � � � � � andincrementallytransformthemesh
� � to a volumetricrepresentation.In overlappingregionsof surfacemeasurementsfrom
differentmeshesthenormalvolumeswill intersectprovidedthemaximummeasurement
error · Q�b�d È0° Q�b�d . If this conditionis satisfiedwe cancombinethefield functionsfrom
differentmeshesinto a singlevolumetricrepresentation.In practicethis conditioncan
always be satisfiedfor a particularsensorby settingthe voxel size� 6 approximately
equalto themaximummeasurementerror.
Fusionof multiple meshesrequiresan‘overlap’ testto determineif surfacemeasure-
mentsfrom differentmeshesin closespatialproximity correspondto thesameor different
regionsof themeasuredobjectsurface. Definition of a robustoverlaptestis critical for
reliablesurfacereconstructionasdiscussedin previouswork [Hilton et al., 1996]. As in
previous work geometricconstraintsareusedto estimateif overlappingmeasurements
correspondto thesamesurfaceregionbasedon thefollowing criteria:
1. Spatialproximity: distancebetweenoverlappingmeasurementsis lessthanthemaximum
distanceÑ Q�b�d .2. Surfaceorientation:overlappingsurfacenormalswith thesameorientation© � ¿ © �"ÒfÓ .3. Measurementuncertainty:likelihoodof measurementoverlapbasedon estimatesof mea-
surementuncertainty.
Spatialproximity providesacoarsetestof measurementoverlapwhichhasbeenused
in previous work [CurlessandLevoy, 1996]. However, this test is unreliablefor sharp
edgesand for surfacesin closeproximity. Surfaceorientationenablesreliable recon-
structionof creaseedgesandthin surfacesectionsfor continuousimplicit surfacerep-
resentation[Hilton et al., 1996]. Someprevious discretevolumetric representationsfor
fusion of rangeimagedata[CurlessandLevoy, 1996] storedonly position information
which doesnot allow a surfaceorientationtest to be performed. The useof an ori-
entationtest increasesrobustnessfor reconstructionof complex surfaces. Surfaceori-
entationwas previously usedwith a discretevolumetric representationby Roth et al.
[RothandWibowo, 1995].
Measurementuncertaintyfor surfacemeasurementscanbeestimatedfrom therelative
orientationof thesurfaceandsensorviewpoint[Soucy andLaurendeau,1995b, Hilton et al., 1998].
If overlappingmeasurementsaredeterminedtocorrespondto thesamesurfaceregionthey
11
maybecombinedaccordingto a weightedaverage[Turk andLevoy, 1994] or maximum
confidence[Pito, 1996a]. Theweightedaveragefor the i¯
�ÅÄvoxel is evaluatedas:
�Ô i¯
� ª�
i¯� º �JÕ
i �̄ ���t� �Ô i �̄ ���t�ª�u � º �VÕ � ���<� �� i
¯
� ª�
i¯� º �VÕ
i �̄ ���t� �� i �̄ ���t�ª�u � º �JÕ
i �̄ ���<� (4)
where,j i¯
is thenumberof measurementsand Õ i �̄ ���t� is theweightbasedonthemeasure-
mentconfidencefor triangle���
.
Fusionof overlappingsurfacemeasurementsis illustratedin Figure 4. The cross-
sectionthroughoverlappingvolumetricenvelopesfor two meshesareillustratedin Figure
4(a).Theresultingvolumetricenvelopeandintegratediso-surfaceaftergeometricfusion
is illustratedin Figure 4(b). Overlappingregions of the volumetric representationare
combinedto obtainasingleiso-surfaceaccordingto thealgorithmpresentedabove.
Thecomputationalandmemorycostof thevolumetricrepresentationdependon the
numberof voxelsinsidethevolumetricenvelope.For a free-formsurfacethis is propor-
tional to thesurfaceareaor thesquareof thevoxel size.Explicit storageof all voxelshasÖ ��j +£ � memorycostwhich is prohibitively expensive. A run-lengthencodedvoxel struc-
ture [CurlessandLevoy, 1996] is usedto achieve a storagecostwhich is proportionalto
thenumberof voxels insidethevolumetricenvelope.This is proportionalto thesurface
arearesultingin a relatively efficient storagecostofÖ ��j �£ � . Therepresentationaccuracy
is inverselyproportionalto voxel size[Hilton et al., 1998].
2.4 Triangulation
Having, constructedafusedvolumetricimplicit surfacerepresentationfrom themeasure-
ment dataan explicit triangulatedmeshrepresentationcan be extractedusing an iso-
surfacepolygonizationalgorithm. Marching Cubesis appliedto obtain a fusedmesh
representationof theobjectsurface[Bloomenthal,1994, LorensenandCline,1987].
Figure5 shows reconstructedtriangulatedmodelsof complex objectusingthe geo-
metricfusionalgorithm.Thetop two objectswerecapturedusingtheModelMakerhand-
heldsensorsystem.Both datasetsconsistof approximatelyonemillion raw datapoints
which aretriangulatedto form severalhundredoverlappingsurfacepatches.Geometric
fusionof thehand-heldsensordataona200MHzPentiumPCplatformis undertwo min-
utesfor bothobjects.Theresultingmodelsconsistof theorderof onehundredthousand
polygons.Thebottomobjectswereacquiredusingconventionallaserrangeimagesensors
12
M
M1
2
M
(a)Overlappingfield-functions (b) Integratedfield-function
Figure4: Fusionof field-functionsfor cross-sectionsthroughoverlappingvolumetricen-
velopes
from multiple views usinga Cyberwarescanner(bunny) andanNRCCscanner(knight)
[Rioux, 1984]. Thedatasetsconsistof eightandtenrangeimagesrespectively andtake
of the orderof oneminuteon a SUN Sparc10platform to reconstructa fusedmodelof
severalhundredthousandpolygons.
Theseresultsdemonstratethat the geometricfusion algorithmpresentedin this pa-
per canbe usedfor both conventionalandhand-heldrangesensors.The datacaptured
usingthehand-heldrangesensorhasapproximately0.2mmrmserror resultingfrom the
accuracy of theFaroBronzearticulatedarmused.In additionthereareoffseterrorsfor
overlappingsurfacemeasurementstaken on differentsweepsof the sensorof the order
of 0.2mm. The resultingmodelswerereconstructedusinga 0.2mmvoxel resolutionto
integrateoverlappingsurfacemeasurements.A smallervoxel resolutionresultsin visible
ridgeson themodelwhereoverlappingmeasurementsfall into differentvoxels. For the
conventionalrangesensordatathe rms error is × /¸Ø|Ù or lessdueto the electro-optical
noiseof thesensor. Themodelswerereconstructedwith a 0.1mmvoxel sizeresultingin
asmoothsurfacemodelwithoutvisible artifactsdueto thediscretevoxel resolution.The
volumetricapproachintroducedin thispaperreconstructscorrectsurfacemodelsfor both
sourcesof data.
2.5 Geometric Fusion for Large Objects
Hand-held3D sensorscan be mountedon instrumentedplatformsor large articulated
armsto enablecaptureof largeobjects[Levoy, 1999]. Theproblemis thento reconstruct
asinglesurfacemodelto therequiredaccuracy.
Reconstructionof large objectsat high-resolutionis prohibitively expensive usinga
singleresolutiondiscretevolumetricrepresentation.For exampleto reconstructanobject
13
the size of a car requiresa volume of approximatelyÚ � × Ù + . However, reconstruction
accuracy mustbeof theorderof 0.2mmto accuratelyrepresentfinesurfacedetailssuchas
edges.Thenumberof voxelswouldthereforebe j +£ �ÜÛ Ú¸× /�/ + . With run-lengthencoding
this givesapproximatelyj �£ �ÝÛ ×¸Þ²� occupiedvoxels with a memoryrequirementofÛ�� ׸ßÆà . This is prohibitivelyexpensive,thereforewerequiretechniquesfor reconstruction
whichdonot requirehigh-resolutionreconstructionof objectsindependentof size.
Theproblemof reconstructingmodelsof largeobjectshasbeenaddressedby subdi-
viding theobjectspaceinto a seriesof adjacentsub-volumes.Geometricfusionfor mea-
surementsin eachsub-volumeis performedusingthealgorithmpresentedin theprevious
sectionto obtainanimplicit surfacerepresentation.Sub-volumesaresetupto overlapby
exactly onevoxel. Thefield-function ������� for theoverlappingsub-voxels is exactly the
sameasidenticalinput dataareusedto evaluatethefield function. TheMarchingCubes
algorithmfor implicit surfacepolygonizationusesthefield functionvaluesat thecentre
of eachvoxel to determinethe intersectionof edgesbetweenvoxel centreswith the iso-
surface �������g�á/ . Eachintersectionwith the iso-surfaceresultsin a new vertex in the
outputfusedmeshmodel.In theoverlappingvoxelsof thesub-volumesthefield-function
valuesarethesameresultingin thesameiso-surfaceedgeintersections.MarchingCubes
generatesasetof outputmeshverticesontheboundaryof thesub-volumewhichareiden-
tical for adjacentsub-volumes.Therefore,a singlemeshrepresentationcanbeextracted
from multiple sub-volumessimply by merging outputmeshverticeson theboundaryof
thesub-volume.
This approachhasbeenusedto obtainmodelsof objectswhich requireprohibitively
large memoryfor a singlevolumetricrepresentation.Sub-volumeswereusedto recon-
structthedwarf modelshown in Figure5 wherethespacewassplit into four sub-volumes.
Thereconstructiontimeusingthesub-volumeapproachis thesameasusingasinglevol-
ume.However, thepeakmemoryrequirementis onequarterof thememoryrequirement
for reconstructingthemodelusingasinglevolume.Thereis nodifferencein theresulting
surfacemodelusingthesub-volumeapproach.
The computationalcost of multiple sub-volumesis the sameas for a single large-
volume. Therefore,this processreducesthe memoryusagewithout requiringincreased
computationalcost.Spatialsubdivisionwith geometricfusionin multiplesub-volumesal-
lowsreconstructionof arbitrarily largeobjectswithout restrictionsdueto finite computer
memorycapacity.
15
3 Conclusions
In this paperwe have introduceda new geometricfusion algorithm for reconstruction
of 3D objectsurfacemodelsfrom hand-heldsensordata. The principal advanceof this
approachis that it avoidstheassumptionsmadein previousfusionalgorithmsdeveloped
for conventionalrangeimagedatabasedon a grid structurewith a commonviewpoint.
Resultsarepresentedfor reconstructionof surfacemodelsfrom both hand-heldsensor
dataandconventionalrangeimages.
The new geometricfusion algorithm is basedon the useof the normal-volume of
a triangle. This normal-volume is usedto incrementallytransforman arbitrary trian-
gulatedmeshinto a discretevolumetric implicit field-functionrepresentation.This ap-
proachavoids any prior assumptionsof a grid structureor commonviewpoint. The
field-functionsfor overlappingsurfacemeasurementscanbeintegratedto definea single
representation.Implicit surfacepolygonizationis performedusingthe MarchingCubes
algorithmto generatea fusedmeshmodelof theobjectsurface.
Large objectscanbe reconstructedby sub-dividing the spaceinto overlappingsub-
volumes. Geometricfusion is performedindependentlyfor eachsub-volume. The re-
sulting surfacemodelsfor eachsub-volumearethenmergedinto a singlemodel. This
approachallows fusionof datafor arbitrarily largeobjectwith finite memoryresources.
The computationalcost is equivalentto fusion for a singlevolumeenclosingthe entire
space,whereasthestoragecostis equivalentto asinglesub-volume.
Acknowledgement
This researchwassupportedby theEPSRC,UK FundingCouncilon AdvancedFellow-
ship AF/95/2531held by Dr. Adrian Hilton andGrantGR/89518‘FunctionalModels:
Building RealisticModelsfor Virtual RealityandAnimation’.
References[Bloomenthal,1994] Bloomenthal,J. (1994). An implicit surfacepolygonizer. Graphics Gems
ed. Heckbert,P.S., 4:324—350.
[Boissonnat,1984] Boissonnat,J. (1984).Geometricstructuresfor three-dimensionalshaperep-resentation.ACM Transactions on Graphics, 3(4):266—286.
[Cohenetal., 1996] Cohen,J., Varshney, A., Manocha,D., Turk, G., Weber, H., Agarwal, P.,Brooks,F., andWright, W. (1996). SimplificationEnvelopes. In ACM Computer GraphicsProc., SIGGRAPH, NewOrleans, USA, pages119—128.
16
[CurlessandLevoy, 1996] Curless,B. andLevoy, M. (1996).A VolumetricMethodfor BuildingComplex Modelsfrom RangeImages.In ACM Computer Graphics Proceedings, SIGGRAPH,NewOrleans,USA, pages303–312.
[Datalabs,1996] Datalabs,V. (1996).Viewpoint Catalog. (http://www.viewpoint.com).
[Fisheretal., 1996] Fisher, R., Fitzgibbon,A., Gionis, A., Wright, M., andEggert,D. (1996).A hand-heldoptical surfacescannerfor enviromentalmodelingandvirtual reality. In Proc.Virtual Reality World, Stuttgart, Germany, pages13—15.
[Fitzgibbonetal., 1998] Fitzgibbon,A. W., Cross,G., and Zisserman,A. (1998). Automatic3D modelconstructionfor turn-tablesequences.In Koch, R. andVan Gool, L., editors,3DStructure from Multiple Images of Large-Scale Environments, LNCS 1506, pages155–170.Springer-Verlag.
[Gortleretal., 1996] Gortler, S., Grzeszczuk,R., Szeliski,R., andCohen,M. (1996). The lu-migraph. In ACM Computer Graphics Proc., SIGGRAPH, NewOrleans, USA, pages43—54.NewOrleans,USA.
[Hilton etal., 1996] Hilton, A., Stoddart,A., Illingworth, J., andWindeatt,T. (1996). Reliablesurfacereconstructionfrom multiple rangeimages.In 4th European Conference on ComputerVision, pages117—126.Springer.
[Hilton etal., 1998] Hilton, A., Stoddart,A., Illingworth, J.,andWindeatt,T. (March1998). Im-plicit surfacebasedgeometricfusion. International Journal of Computer Vision and ImageUnderstanding, Special Issue on CAD Based Vision, 69(3):273–291.
[Hoppeetal., 1992] Hoppe,H., DeRose,T., Duchamp,T., McDonald,J.,andStuetzle,W. (1992).Surfacereconstructionfrom unorganisedpoints.Computer Graphics, 26(2):71—77.
[Infografica,1996] Infografica, R. (1996). REM 3D Models Bank Catalog.(http://www.infografica.com).
[Levoy, 1999] Levoy, M. (1999).TheDigital MichelangeloProject.In Second IEEE InternationalConference on 3D Digital Imaging and Modeling, Ottawa, Canada.
[Levoy andHanrahan,1996] Levoy, M. andHanrahan,P. (1996).Light FieldRendering.In ACMComputer Graphics Proc., SIGGRAPH, NewOrleans, USA, pages31—42.
[LorensenandCline,1987] Lorensen,W. andCline,H. (1987). Marchingcubes:A high resolu-tion 3d surfaceconstructionalgorithm.Computer Graphics, 21(4):163—169.
[Mencl, 1995] Mencl, R. (1995). A graph-basedapproachto surfacereconstruction.In EURO-GRAPHICS 14(3), pages446—456.
[Niem andWingebermuhle,1997] Niem, W. andWingebermuhle,J. (1997). Automatic recon-structionof 3d objectsusinga mobilemonoscopiccamera.In Proc. IEEE International Con-ferences on Recent Advances in 3D Digital Imaging and Modeling, Ottawa, Canada, pages173—180.
[Pito, 1996a] Pito, R. (1996a). Mesh integrationbasedon co-measurements.In InternationalConf. on Image Processing, pages397—400.Lusanne,Switzerland.
[Pito, 1996b] Pito, R. (1996b). A sensorbasedsolutionto thenext bestview problem. In Inter-national Conf. on Pattern Recognition, pages941—945.Vienna,Austria.
[Rioux, 1984] Rioux, M. (1984). Laserrangefinder basedon synchronizedscanners.AppliedOptics, 23(21):3837—3844.
[RothandWibowo, 1995] Roth,G. andWibowo, E. (1995). A fastalgorithmfor makingmesh-modelsfrom multiple-view rangedata. In Proceedings of the 1995 Robotics and KnowledgeBased Systems Workshop, pages349–355.
[Rutishauseretal., 1994] Rutishauser, M., Stricker, M., andTrobina,M. (1994). Merging rangeimagesof arbitrarily shapedobjects.In Proceedings of IEEE Conference on Computer Visionand Pattern Recognition, pages573—580.
17
[Soucy andLaurendeau,1995a] Soucy, M. andLaurendeau,D. (1995a). A dynamicintegrationalgorithmto modelsurfacesfrom multiple rangeimages. Machine Vision and Applications,8:53–62.
[Soucy andLaurendeau,1995b] Soucy, M. andLaurendeau,D. (1995b). A generalsurfaceap-proachto the integrationof a setof rangeviews. IEEE Trans. Pattern Analysis and MachineIntelligence, 14(4):344–358.
[Taubin,1995] Taubin,G. (1995). Estimatingthe tensorof curvatureof a surfacefrom a poly-hedralapproximation. In Proccedings IEEE International Conference on Computer Vision,Boston, USA, pages902—907.
[Turk andLevoy, 1994] Turk, G. andLevoy, M. (1994). ZipperedPolygonMeshesfrom RangeImages.In ACM Computer Graphics Proceedings, SIGGRAPH, Orlando, Florida, pages311–318.
18
Hilton Illingworth
Author BibliographyDr. Adrian Hilton received a B.Sc.(Hons.) in MechanicalEngineeringanda D.Phil. degreefrom theUniversityof Sussex in 1988and1992respectively. In 1992hejoinedtheCVSSPat theUniversity of Surrey working on 3D shapecapturefor realisticcomputergraphicsandaccurateinspection.This work receivedBestPaperawardsfrom the journalsPatternRecognitionin 1996andIEE ElectronicsandCommunicationsin 1999.Collaborationwith 3D ScannersLtd. resultedin the first hand-heldsystemfor capturing3D objectmodelswhich wasawardeda 1996EU IT‘GrandPrize’ for innovation. In 1997hewasawardeda five yearEPSRCAdvancedFellowshipandin 1999becamelecturerin 3D Digital MediaandBroadcast.Researchinto 3D humanmod-elling from digital photographshasbeenexploitedby AvatarMeLtd. to developthefirst 3D boothfor generatinganimatedmodelsof people.His active researchinterestsincludemodel-based3Dvision for capturinghumanmovementandphoto-realistic3D modelsfor computeranimation.
Professor John Illingworth is Professorof MachineVision at Surrey University. He hasa doc-toratein Physicsbut hasbeenactive in researchin computervision, imageprocessingandpatternrecognitionfor thelast18 years.He haspublishedover 130papersin refereedjournalsandcon-ferences.He is a Fellow of the Institution of ElectricalEngineersandeditsIEE ProceedingsonVision ImageandSignalProcessing.
19
Figure CaptionsFigure1: Model reconstructionfrom hand-heldsensordata(a)3DScannersModelMaker system(b) Raw point stripes(c) Triangulatedpatches(partial)(d) Fusedmodeltriangulation(e)Fusedmodelsurface
Figure2: Volumetricsurfacerepresentation(a)Meshandvertex normals(b) Offsetsurface(c) Volumetricenvelope
Figure3:Normal-volumetrianglerepresentation(a)Trianglenormal-volume(b) Bilinear sideof normal-volume
Figure4: Fusionof field-functionsfor cross-sectionsthroughoverlappingvolumetricenvelopes(a)Overlappingfield-functions(b) Integratedfield-function
Figure5: Reconstructedmodelsfor ahand-heldsensor(top)andconventionalrangeimagesensor(bottom)
20
Corresponding AuthorAdrianHilton
Centrefor Vision,SpeechandSignalProcessingUniversityof SurreyGuildfordGU25XHUKTel: +44-1483-873956Fax: [email protected]
21
n
nv
+ +
+
-
-
-
v v
v
v
v
v
v
v
t
t
t
s
s
s
r
r
r
r
s
nt
n
nr
s
rs
rs
-
s-
s
r
s
r+
+
rs
v v
v
v
v
v
rv
p
dn(p )
x
24
Top Related