3D Semi-Landmarks Based Statistical Face Reconstruction

Journal of Computing and Information Technology - CIT 14, 2006, 1, 31–43 31

3D Semi-Landmarks-Based StatisticalFace Reconstruction

Maxime Berar1, Michel Desvignes1, Gerard Bailly2 and Yohan Payan3

1Laboratoire des Images et des Signaux, Saint Martin d’Heres, France2Institut de la Communication Parlee, Grenoble, France3Laboratoire TIMC/IMAG, Faculte de Medecine, La Tronche France

The aim of craniofacial reconstruction is to estimatethe shape of a face from the shape of the skull. Fewworks in machine-assisted facial reconstruction havebeen conducted so far, probably due to technical �poormachine performance and data availability� and theoret-ical �complexity� reasons. Therefore, the main works inthe literature consist in manual reconstructions. In thispaper, an original approach is first proposed to build a 3Dstatistical model of the skull�face set from 3D CT scans.Then, a reconstruction method is introduced in order toestimate, from this statistical model, the 3D facial shapeof one subject from known skull data.

Keywords: facial reconstruction, statistical model, elas-tic registration, missing data reconstruction.

1. Introduction

Craniofacial reconstruction is usually consid-ered when confronted with an unrecognisablecorpse and when no other identification evi-dence is available. In such cases, the skeletalremains are the only available information forcreating a picture of that person. The aim ofcraniofacial reconstruction is then to produce alikeness of the face using the skeletalized re-mains. This reconstruction may hopefully pro-vide a route to a positive identification.

Several 3D manual methods for facial recon-struction have been developed up to now andare currently used in practice. They consist ofmodeling a face on the remaining skull by useof clay and plasticine. However, manual re-construction methods have several fundamen-tal shortcomings, such as being highly subjec-tive, time-consuming and requiring artistic tal-ent. Computer-based methods have been devel-oped to try to complement or even provide ananswer to these shortcomings.

Some current machine-aided techniques fit atemplate skin surface to a set of interactivelyplaced virtual dowels on a 3D digitised modelof the remaining skull �1� — �5�. Other workspropose to deform a reference skull in order tomatch the remaining skull, thanks to crest lines�lines of maximal local curvature� �6�, controldata sets �7� or feature points �8�. Then theyapply an extrapolation of the calculated skulldeformation to the template skin surface asso-ciated to the reference skull. For both tech-niques, the template skin or reference skull caneither be a generic surface or a specific bestlook-alike according to the skull. However, thefacial reconstruction is biased by the choice ofthe reference skull and the template skin. Re-cent works using multiple reference skulls �9�,or a combined statistical deformable model offacial surfaces and tissue thickness �10�, bothaddressed the facial reconstruction problem anddiscussed these biases.

Several works have addressed the problem ofconstruction of statistical models. Input dataare first registrated in a common reference sys-tem by minimization of a cost function. Then,the model, often based on PCA, is computed.The cost function can be modeled on voxel in-tensities, �11�, voxel labels �12�, manual land-marks �13�, features �14�, or nearest points �15�in 2D images or 3D density maps. In cranio-facial reconstruction, two objects, the skull andthe skin, have to be registrated. It is of utmostimportance that the registrationmethod does notmodify the relationship between these objects.Most methods based on voxel intensities use

32 3D Semi-Landmarks-Based Statistical Face Reconstruction

an implicit model of elastic deformation in thecost function, which can bias the relationshipbetween skin and skull objects in the registratedimage. Other methods often use meshes to rep-resent surfaces in 3D. The meshes must havethe same connectivity �same number of verticesand same relationships among them� to build astatistical model. This is achieved by a parame-terisation of the object �16�, by an optimisationof the resulting statistical model �17�, or by con-structing template references �18�. In �12�, theauthors decimate the meshes to reduce the in-fluence of noise and the processing time. Ourmodel is closely related to these last methods�17, 18� and also follows tracks from �20�wherethe model is built from labelled images.

In this paper, a method to build a joint statisti-cal 3D model of the skull and face is presented.This model is then used to reconstruct a facefrom available skull data. The idea is similarto �6-8�, but uses a statistical shape model ofboth the skull and the face for the reconstruc-tion task, instead of a sole extrapolation of thedeformation field. A 3D-to-3Dmatching proce-dure propagates pseudo-landmarks from refer-ence surfaces to the various surfaces of the skulland face of our database. Therefore, when ap-plied to several individuals, a statistical modelof the cross-variability of the skull and the faceis built. The reconstruction of the face is thensolved using the direct statistical relationshipbetween skin and skull surface shapes given bythe model. Face reconstruction can therefore beseen as a missing data problem.

This paper is organized as follows. Section2 describes the elaboration of the normalizedskull and face geometries obtained by a 3D-to-3D matching procedure. Section 3 presentsthe statistical model built upon the normalizedfaces and skulls. Finally, section 4 introducesthe facial reconstruction method and presentsresults. Some open research lines for furtherimprovements are also presented.

2. Skull and Face Database

2.1. Method

An entry �i.e. a sample� in our database consistsof a skull surface coupled with a skin surface.For facial reconstruction, only the skull surface

is known. These surfaces are represented by3D meshes �vertices and triangles�. In orderto construct the statistical model, each skull orskin shape must share the same mesh connec-tivity. This particular connectivity arises froma subject-shared reference mesh �also denotedas generic mesh in the following�. For eachindividual in our database, original meshes arereconstructed from CT data of the subject �Fig-ure 1�. Each of these subject meshes has its ownconnectivity. The main problem is then to es-tablish correspondences between the differentmeshes of the training set, so as to match theanatomically equivalent features. Each of thesesubject meshes needs to be registrated in thesubject-shared reference system. Like Fleute etal. �21� correspondence is established by elasticregistration of template shapes �mandible, skulland face� with all the subject shapes �Figure 2�.Joint propagation of mesh connectivity and ge-ometry is performed from the generic mesh tomatch the subject shapes. The triangles for aregion of the skull or the face are therefore sup-posed to be the same for all samples, while thevariability of the position of the vertices will re-flect the anatomical characteristics of each sam-ple. These vertices can be considered as semi-

Fig. 1. Generation of the subject meshes.

Fig. 2. Generation of the subject-specific genericmeshes.

3D Semi-Landmarks-Based Statistical Face Reconstruction 33

Fig. 3. Building the statistical model from thesubject-specific generic meshes.

Fig. 4. �Top� 3D raw scan data �only axial slices werecollected; midsagital and coronal have been

reconstructed here by image processing�, �Bottom�subject-specific mesh reconstrusted using the marching

cube algorithm �23�.

landmarks �or pseudo-landmarks�, i.e. pointsthat do not have names, but that match acrossall the samples of a data set under a reasonablemodel of deformation �22�. As each skull or skinshape �also denoted as subject-specific genericmeshes� shares the same mesh connectivity, astatistical model can be built �Figure 3�.

2.2. Generation of Subject Meshes

Axial CT slices �see Figure 4� were collectedfor the partial skulls and faces of 15 subjects�helical scan with a 1-mm pitch and slices re-constructed every 0.31 mm or 0.48 mm�. Sincethese data were collected during regular medi-cal exams, excitation of the brain volume wasavoided if not necessary. So nearly all theskulls are partially scanned, and only two com-plete skull and face volume data were available.Bones and skin image volumes are first sepa-rated using intensity thresholding and morpho-

logical operators. Face volumes are then filledup, with metal artefacts, if any, being manuallyremoved. The mandible and the skull have to beseparated during the segmentation process be-cause the subjects have different mandible aper-tures. Skull and mandible are semi-manuallyseparated using seed-growing regions. At theend of the segmentation process, three binaryvolumes are obtained. The subject meshesare reconstructed for each volume �face, skull,mandible� using a standard Marching Cube al-gorithm �23� and a smooth decimation algorithmis applied to the resulting meshes. Each subjectshapes are now described with a different con-nectivity. Moreover, undesired holes �such asthe orbita wall, foramina, etc.� are still presentin the subject meshes.

2.3. Subject-specific Generic MeshesGeneration

Subject-specific generic �SSG� meshes are ob-tained by matching the subject meshes withgeneric meshes. Generic meshes �see Figure5� have been taken from the Visible WomanProject �24� �skull, 3473 verts, and mandible,1100 verts� and from �25� �face, 5828 verts�.These meshes have been semi-manually ob-tained by their respective authors. The verticesof these meshes are located on crest lines andin �26� they are regularly distributed followingfacial animation needs. Moreover each of thesegeneric meshes has no undesired holes.

Fig. 5. Skull mesh from �24� �Top� and face mesh from�25� used as generic meshes �Bottom�


The matching procedure we use is the sameas described in �26�: the elastic registration ofthe generic meshes to the subject meshes usesthe matching algorithm proposed by Lavalleeet al. �27� with a minimisation of the distancesbetween the two shapes. It basically consistsof the deformation of the initial 3D space byseveral trilinear transformations. These trans-formations are applied to all vertices of ele-mentary cubes of the generic mesh towards thesubject mesh. The problem of matching sym-metry �12,19� is encountered, due to the verticesdensity dissimilarity between the subject andgeneric meshes. Indeed, the number of verticesis 30 to 70 times larger in the subject meshesthan in the subject-specific meshes. Therefore,a symmetrized minimization function is used�28�.

The distance computed for the quantitative anal-ysis of the SSG and subject meshes is a point-to-surface distance from the subject-specific genericmeshes to the subject meshes.

Maximal matching errors between the SSGmandible meshes and the subject mandiblemeshes are located on the teeth and on the coro-noid process. The mean distance can be consid-ered as the registration noise, partly due to thedensity dissimilarity �see Table 1.�. Teeth willnot be part of our model, due to the frequentmetal artefacts in CT scans.

Distances �mm� mean MaxMandible 2 8

Skull 4 36Face 1 5

Table 1. Distance between the subject �subject CT datareconstructed through Marching Cube� and SSG

meshes.

Subject skull meshes are registered on the cor-responding parts of the generic skull mesh, asmost of the skulls were partially scanned. Themaximal matching errors in the resulting SSGskull meshes are located in the spikes beneaththe skull, where the individual variability andthe surface noise are large due to segmentationerrors. Only the minimum common subset ofshapes will be used to build the statistical model�Figure 6�.

Finally, SSG face meshes are obtained usingthe same procedure. In this case, the maxi-mal matching errors between SSG and subject

Fig. 6. Minimum common subset of shapes used tobuild the statistical shape model.

Fig. 7. Each vertex of the SSG meshes is considered asbeing at the same location reflecting thus the

inter-individual variations of shape.

meshes are located around the eyes, that are partof the original data, but not part of the genericmesh �see Figure 7 for a distance map betweena SSG mesh and a subject mesh�. Again, onlythe minimum common subset of shapes will beused to build the statistical model. Figure 8shows the 15 normalized shapes of the commonsubset of the face database.

The 15 subjects are now registrated in a com-mon shape space. These subset meshes have3780 face vertices and 2900 skull vertices.


Fig. 8. The 15 face subsets forming the database.

3. Statistical Modelling

3.1. Building the Statistical Model

Each vertex of the SSG mesh is supposed to be asemi-landmark of the 3D surfaces — see Figure9 — reflecting thus the inter-individual varia-tions of shape. The statistical model is basedon this assumption and is computed on the min-imum common subset of the original data. The15 matched skulls and faces are first fitted onthe mean configuration of the skull using Pro-crustes normalization �29�. Seven degrees offreedom due to initial location and scale are re-trieved by this fit �three due to translation alongthe axes, three due to rotations, one for scaleadjustment�. As the fitting is based on meanskull configuration, the relationships betweeneach face and skull are conserved. A statisti-cal model of the subset of skulls and faces isthen built using Principal Component Analysis�PCA�. The result of the PCA is a geometri-cally averaged skull and face template, whichis computed together with a correlation-ranked

Fig. 9. Each vertex of the SSG meshes is considered asbeing at the same location reflecting thus the

inter-individual variations of shape.

set of modes of principal variations based oninter-subjects variations.

LetfTi; i � 1 � � � lgdenote l shapes �l�15�. Eachshape Ti � �xi1� yi1� zi1� �� yin�m� zin�m��R3�n�m�

consists of the 6680 vertices �n�2900 skullverts., m�3780 face verts.� of the subset ofmeshes. Using PCA, we can write :

T � T �Φb �1�

where Tis the average shape vector, Φ is a ma-trix whose columns are the eigenvectors of thecovariance matrix S of the centered data and bis the shape parameter vector of the model. IfΦcontains the t � min fl� 3�m � n�g eigenvectorscorresponding to the largest nonzero eigenval-ues of S, we can approximate any shape of thetraining set using �1�, whereΦ � �φ1j��jφt�andb is a t-dimensional vector given by b � Φt�Ti�T�. Any points of the training set can be repre-sented or retrieved with the t values of the vectorb by T � T � Φb.

By varying the parameters b, different instancesof the skull and face can be generated. Assum-ing that the cloud of the meshes vertices followsa multidimensional Gaussian distribution andthat shape parameters lie within the statisticalboundaries of the model, the skulls and facesgenerated by varying the shape parameters aresimilar to those contained in the training set, re-sulting in new synthetic but plausible skulls andfaces.


3.2. Results

In our case, with 15 subjects, a total of 13 varia-tionsmodes can be computed, since a leave-one-out approach is used to test the generalizationof the modeling procedure. Only the first eightmodes of variations �see Table 2� are significantin terms of represented variance.

Modenumber

1 2 3 4 5 6 7 8

Cumulativevariance

36 51 64 73 79 84 88 91

Table 2. Percentage of cumulative variance explained foreach part of the model �face, skull� for the first 6 modes.

The accuracy of this model is tested by recon-struction: for a given mesh, variation modes�b� are computed by minimization of the dis-tance between the true real mesh �T� and thereconstructed mesh �T � Φb�. The mean re-construction errors �Figure 10� for the last threemodes are below the millimeter for samples ofthe learning database. So the reconstructionis quite accurate with samples in the learningdatabase. Reconstruction error for a test sam-ple i.e. a sample which is not in the learningdatabase, is around 3.85 mm for the skull and3.25 mm for the face using the first four modes.The skull reconstruction is mostly determinedby the first variation, as the reconstruction er-ror is then around 4.2 mm. These two results

Fig. 10. Mean reconstruction errors of the skull and faceusing an increasing number of modes.

demonstrate that this method seems promising,but that the number of samples in the learningdatabase is too small.

Except for the first variation mode, the principalvariations of the shape explained by the modelare a little more descriptive of the variation ofthe face shape than those of the skull shape.This can be linked to the greater number of ver-tices belonging to the face �3780� than to theskull �2900� �Table 3�.

Mode number�Cumulativevariance

1 2 3 4 5 6

face 36 50 64 75 82 86skull 39 48 59 66 72 79

Table 3. Variations of the skull shape according to thefirst 3 modes for parameters varying between �3 and

�3 times the standard deviation.

Figures 11 and 12 present the variations of theskull and face shape according to the first modesfor parameters varying between �3 and �3

Fig. 11. Variations of the face shape according to thefirst 5 modes for parameters varying between �3 and



Fig. 12. Variations of the skull shape according to thefirst 3 modes for parameters varying between �3 and


times the standard deviation. The first param-eter influences variations of the face and skullwidth, while the second parameter models theface and skull height. The third parameter actsupon the shape of the nose, as well as the ratiobetween the upper and the lower parts of theface. Parameter four influences the shape of thenose and parameter five is linked to the shapeof the jaw. The first five modes of variationsrepresent 73 percent of the cumulative variance�Table 2�. As the mandible position is differentfor each subject, each mode of variation modelsalso the jaw aperture �Figure 12�.

4. Statistical Reconstruction

4.1. Missing Data Extension

The linear PCA model can be extended in anelegant way in order to take into account spatialrelations between landmarks and to estimate anunknown part of a partially visible or occludedmodel �30�.

��

C1...

CnX1...

Xm

��

��

C1...

CnX1...

Xm

��

�� Φ1�1 � � � Φ1�n�m

... . . ....

Φn�m�1 � � � Φn�m�n�m

��

��

b1...

bnbn�1

...bt

��

Under this hypothesis, if some points �saysn points� are known, the remaining unknownpoints �says m points� are determined usingPCA. The shape parameter vector b �of di-mension t � n � m� will also be determined.Without any approximations, we can write theunknown vector �b1� � � � � bt� X1� � � � � Xm� in thefollowing system:

�C0

��

�ΦC 0ΦX �Id

� �bX

�or T � MZ

whereΦC �

��Φ1�1 � � � Φ1�t

... . . ....

Φn�1 � � � Φn�t

��

andΦX �

�� Φn�1�1 � � � Φn�1�t

... . . ....

Φn�m�1 � � � Φn�m�t

��2�

This is a linear system with n � m equationsand n � 2m unknowns that cannot be resolved.Since PCA can represent the dataset with t �n+m values, if we suppose t � n, the systemhas a direct solution. Notice that if we chooset�n, the system becomes overdetermined and aleast square method can be used to resolve thesystem:

min

�

C0

��

�ΦC 0ΦX �Id

� �bX

� 2 �3�

The cost function is:J�b� X� ��C�ΦC � b �X�ΦX � b�

�C�ΦC � b�X�ΦX � b

�

�4�In matricial form, J can be rewritten as :

J�Z� � J�b� X� � kT � M � Zk2

�T �T � T �MZ � Z�M�Y � Z�M�MZ

��T �T � 2T �MZ � Z�M�MZ�

�5�

As the matrix �ΦC ΦX �� is an orthonormal ba-

sis, M�M takes the following form and the costfunction becomes:

M�M �

�Φ�

C Φ�

X0 �Id

��ΦC 0ΦX �Id

�

�

�Id �Φ�

X�ΦX Id

�

J�b� X��C�C�2b�Φ�

CC � X�X�2b�Φ�

XX � b�b�6�


The derivatives with respect to X and b are null:

�J� b� X��b

� �2Φ�

CC � 2b � 2Φ�

XX � 0

b � Φ�

CC � Φ�

XX�7�

�J� b� X��X

� �2ΦXb � 2X � 0

X � ΦXb�8�

Reporting �6� in �5�, the solution is :

b �Id � Φ�

XΦX��1 Φ�

CC

X � ΦXId �Φ�

XΦX��1 Φ�

CC�9�

Note that �Id-ΦX’ΦX� is always invertible, sinceit is a symmetric positive defined matrix.

In this framework, a linear approximation ofspatial relations between known and unknownpoints is explicitly determined from the eigen-vectors of the covariance matrix. The determi-nation of the unknown points is, in fact, the de-termination of the shape parameters given theknown points �8�. The determination of theshape parameters can be linked to the influenceof each part on each shape parameter, describedby ΦC and ΦX. As ΦC and ΦX depend onlyon the training set, new models must be builtto act upon ΦCand ΦX. One way is to build anew model with a larger �or smaller� trainingset. Another way is to change the ratio betweeneach part.

4.2. Results: Synthetic Data

A synthetic skull and face database is first builtusing one of the original individuals and a setof elastic transformations defined as an octree.Random transformations of the cube enclosingthe two meshes were provided, thus transform-ing the two meshes. Five parameters are usedto deform the meshes: three scaling parameters,and variations of the center of the X face andof the central axis of the cube �Z direction�. Ofcourse, these variations do not simulate the real-ity of the skulls and faces variability. However,it is a way to artificially verify the missing dataformulation and the semi-landmark hypothesis.

Using the extension of the linear PCA definedabove, the face of a synthetic subject can be re-constructed from his skull and from the statisti-cal model built using synthetic data. The known

Fig. 13. Examples of synthesic meshes.

part �Ci� contains the skull vertices while theunknown part �Xi� contains the face vertices.

A set of one hundred meshes is generated us-ing these random transformations �see Figure13 for examples of generated faces�. To furtherincrease the variability on the face, a Gaussiannoise is added to each point. The level of thisnoise �2 mm� is chosen so that we still remainin the semi-landmark paradigm. Indeed, a largelevel of noise could change the relative positionsof the vertexes of the mesh, thus making theconcept of semi-landmark not valid any more.

Figures 14 and 15 plot the reconstruction re-sults. Test samples are reconstructed with amean accuracy of 1 mm. The missing data er-ror is in the same range. It is important to note

Fig. 14. Mean facial reconstruction errors of the sculland face using an increasing number of modes for the

synthetic database model.


Fig. 15. Mean facial reconstruction errors using anincreasing number of modes for the synthetic database

model.

that the missing data error converges to the re-construction error for the known part of the testsample �the skull� as well as for the unknownpart �the face. If the subject-specific meshes areused as test samples on the synthesis database,bad reconstructions are obtained for each indi-vidual as the variations of the shapes used aretoo simple

4.3. Statistical Facial Reconstruction

The face of a subject can be reconstructed fromhis skull and from the statistical model definedpreviously �in 3.2�, using the missing data ex-tension of the model. The known part �Ci�contains the skull vertices while the unknownpart �Xi� contains the face vertices.

Fig. 16. Mean facial reconstruction errors using anincreasing number of modes.

Again, a leave-one-out approach is used to testthe accuracy of the facial reconstruction. Thelearning database is composed of all subjectsminus one, which is the test sample. Every sub-ject becomes the test sample in turn. Figure 16gives the mean reconstruction error of the testsample. It also gives the reconstruction errorfor the samples of the learning database.

In all cases the global reconstruction is correct.The face and skull are reconstructed with an ac-curacy of 0.5 mm for the samples in the learningdatabase. Test face sample is reconstructed witha mean accuracy of 6 mm. Clearly, these resultsshow that the method is promising, but suffersfrom the size of the learning database. The firstparameter offers a better approximation of thereconstructed face with a mean reconstructionerror of 5.2 mm. As the skull provides essen-tially the first mode of variation �see Figure 10�and the other modes are mostly related to vari-ations of the face for our test sample, only theperformance for the first parameter should beconsidered. For each additional variation modethe prediction should not be considered as it in-fers variations of the face from variations nottaken into account for the skull, the values ofthe variations modes being not accurate. Asthese parameters do not correspond to variationmodes with null eigenvalues, a large error intheir prediction results in a large error in recon-struction.

The repartition of the missing data errors onthe face is shown Figure 17. Large errors arelocated on the cheeks, on the neck and on thesides of the nose. It is important to note that thecheeks are not attached to the skull and that thedatabase provides different mandible positions.So, it is very difficult to predict correctly the po-sition of the vertexes of the cheeks. Moreover,the density of vertexes for the cheeks region isquite low, which authorizes a possible slidingof those points on the skin surface. The neck isunconnected to the skull, so large errors are in-escapable. Finally, it is known that prediction ofthe shape of the nose from the shape of the skullis very difficult �31, 32�. The links between thetwo organs are complex. These errors locatedon the sides of the nose are probably due to thislack of regularity. The good reconstruction ofthe tip of the nose can be conversely associated


Fig. 17. Distance maps and histograms of the facialreconstruction error for 3 reconstructed faces.

to the template used during the creation of thedatabase.

When using a smaller bounding box that ex-cludes the tip of the nose and the neck, we gainhalf a millimeter in the accuracy of the predic-tion �to 4.6 mm�. The maximal error is reducedto 3 mm as seen in Figure 18.

Two limitations of the current database are itssmall size, the non homogeneity of the facemesh �the regions of the nose and the lips aremuch more dense than the rest of the mesh� andthe coarseness of the skull mesh. The follow-ing section presents a way of compensating forthese limitations. Using a decimated mesh ofthe face �with a more homogenous distributionof the vertices�, we indirectly give more weightto the skull vertices in the statistical model. Theskull is then more accurately parameterised bythe model and errors on the estimation of theskull shape interfere less with the prediction ofthe face.

Fig. 18. Distance maps and histograms of the facialreconstruction error for a reconstructed face using theenclosed model �no neck or tip of the nose� �left� and

orginal model �right�.

4.4. Decimated Facial Reconstruction

A decimated face mesh �929 vertices� is ex-tracted from the original mesh �3780 vertices��Fig. 19�. As the decimated mesh is a subpartof the original mesh, every entry of the databasecan be expressed with only the vertices belong-ing to the decimated mesh. Each vertex of thedecimated mesh represents a larger area of theface. The skull vertices now represent 75% ofthe vertices of the model. They have now moreinfluence on the eigenvectors emerging fromstatistical modelling.

The accuracy of this decimated model is firsttested by global reconstruction. The mean re-construction errors �Figure 20� for the last threemodes are below the millimeter for samples ofthe learning database as with the original mesh.The reconstruction is quite accuratewith samplein the learning database. Reconstruction errorfor a test sample is around 3.6 mm for the lastfour modes. These results are similar to thoseof the non-decimated mesh. The reconstructionof the skull test sampled is now determined es-sentially by the first three modes of variations:the cumulative explained variance is now more

Fig. 19. Original and decimated face meshes.


Fig. 20. Mean reconstruction errors of the skull and faceusing an increasing number of modes for the decimated

model.

Fig. 21. Mean facial reconstruction errors using anincreasing number of modes for the decimated model.

descriptive of the skull shape variations than theface shape variations �Table 4�.

Mode number�Cumulativevariance

1 2 3 4 5 6

Face � skull 38 55 65 72 78 83face 31 40 50 63 71 76skull 41 59 68 72 78 83

Tab. 4. Percentage of cumulative variance explained foreach part of the model �face, skull� for the first 6

parameters.

Here again, the reconstruction of the face us-ing the missing data extension of the PCA ispromising �Fig. 21�. The face is reconstructed

Fig. 22. Distance maps and histograms of the facialreconstruction error for 3 reconstructed decimated faces.

with an accuracy of 0.6 mm for the samplesin the learning database. Test samples are re-constructed with a mean accuracy of 6.0mm.As with the original model, the first variationmode offers a better approximation of the re-constructed face with an error of 5.1 mm. Butnow the second parameter also gives an ade-quate information for the prediction of the face.The distribution on the face of the facial recon-struction error is similar to the original model�see Figure 22�.

In conclusion, the “decimated” model givessimilar results to the “global” model �the gainis 0.1 mm for the facial reconstruction with 2valid modes�. However these results show thatthe more accurately the skull will be parame-terised by the model �i.e. the greater the numberof valid variation parameters�, the more accu-rately the face will be predicted, as the error inthese parameters determining the skull will notinterfere with the prediction of the face.


5. Conclusion

In this paper, a face and skull statistical modelis proposed for 3D machine-aided facial re-construction. To build this statistical model, a3D-to-3D matching procedure delivers subject-specific meshes of the skull and face with thesame number of vertices. A shared normalizedspace for the faces and skulls is therefore built.The direct statistical relationships between theface and the skull included in the statisticalmodel are used to reconstruct the missing dataof the face when the skull is the only availableinformation. For this, a missing data extensionof the Principal Component Analysis is used.

Results are visually correct and mean measurederrors show that the method is promising as itwill be probably more efficient for larger learn-ing database. One way of increasing the effi-ciency of the model is presented. It consists indecimating the face mesh in order to adjust itsdensity to the skull mesh density, thus givinga higher weight to the known part of the prob-lem, i.e. skull data. The corresponding resultsare similar to the ones provided by the originalmodel �at least in terms of facial reconstruc-tion�, but a slightly more efficient modeling ofthe skull was observed.

We will test this statistical approach using dataacquired with a more adequate experimentalprotocol and using data from subjects with var-ied age and sex. The approach presented heremay be extended towards relevant covariationbetween shape and appearance as well as be-tween shape and range of motion. Aesteheticand rehabilitation surgery may thus also benefitfrom such anatomy-aware and subject-informedstatistical models.

6. Acknowledgment

CT scans data have been collected for sev-eral subjects thanks to the maxillofacial depart-ment of Toulouse Hospital, with a grant fromthe BQR�INPG “Vesale”. We acknowledgePraxim-Medivision SA for the use of the ini-tial 3D-to-3D matching software.

References

�1� P. VANEZIS, Application of 3-D computer graph-ics for facial reconstruction and comparison withsculpting techniques, Forensic Science Interna-tional, 42 �1989�, pp. 69–84.

�2� P. VANEZIS, M. VANEZIS, G. MCCOMBE AND T.NIBLET, Facial reconstruction using 3-D com-puter graphics, Forensic Science International, 108�2000�, pp. 81–95.

�3� R. EVENHOUSE, M. RASMUSSEN AND L. SADLER,Computer-aided Forensic Facial Reconstruction,Journal of BioCommunication, 19.2 �1992�, pp.22–28.

�4� A.W. SHAHROM, P. VANEZIS, R.C. CHAPMAN, A.GONZALES, C. BLENKINSOP, M.L. ROSSI, Tech-niques in facial identification : computer-aidedfacial reconstruction using laser scanner and videosuperimposition, International Journal of LegalMedicine, 108 �1996�, pp. 194–200.

�5� A.J. TYRELL, M.P. EVISON, A.T. CHAMBERLAINAND M.A.GREEN, Forensic three-dimensional facialreconstruction : historical review and contemporarydevelopments, Journal of Forensic Science; vol.42.4 �1997�, pp. 653–661.

�6� G. QUATREHOMME et al, A fully three-dimensionalmethod for facial reconstruction based on de-formable models, Journal of Forensic Science, 42.4�1997�, pp. 649–652.

�7� L.A. NELSON AND S.D. MICHAEL, The applicationof volume deformation to three dimensional facialreconstruction : a comparison with previous tech-niques, Forensic Science International, 94 �1998�,pp. 167–181.

�8� M.W. JONES, Facial reconstruction using volumet-ric data, Proceedings of the 6th International VisionModeling andVisualisationConference, �2001�, pp.21–23, Stuttgart, Germany.

�9� D. VANDERMEULEN, P. CLAES, P. SUETENS, S. DEGREEF AND G. WILLEMS, Volumetric deformableface models for cranio-facial reconstruction, Pro-ceedings of the 4th international symposium onimage and signal processing and analysis - ISPA,�2005�, pp. 353–358, Zagreb, Croatia.

�10� P. CLAES, D. VANDERMEULEN, S. DE GREEF, G.WILLEMS AND P. SUETENS, Statistically deformableface models for cranio-facial reconstruction. Pro-ceedings of the 4th international symposium onimage and signal processing and analysis - ISPA,�2005�, pp. 347–352, Zagreb, Croatia.

�11� R.P. WOODS, S.T. GRAFTON, C.J. HOLMES, S.R.CHERRY AND J.C. MAZZIOTTA, Automated imageregistration: I. General methods and intrasubject,intramodality validation, Journal of Computer As-sisted Tomography, 22 �1998�, pp. 141–154.

�12� A.D. BRETT AND C.J. TAYLOR, A method of auto-mated landmark generation for automated 3D PDMconstruction, Image and Vision Computing, 18.9�2000�, pp. 739–748.

�13� F.L. BOOKSTEIN, Morphometric tools for landmarkdata, Cambridge Press, 1991.


�14� G. SUBSOL AND G. QUATREHOMME, Automatic 3DFacial Reconstruction by Feature-based Registra-tion of a Reference Head in Computer-GraphicFacial Reconstruction �J.G. Clement, M.K. Marks,Eds.�, �2005�, pp. 79–102.ElsevierAcademicPress,Burlington.

�15� V. BLANZ, A. MEHL, T. VETTER AND H.-P. SEIDEL,A Statistical Method for Robust 3D Surface Recon-struction from Sparse Data, 3DPVT, �2004�, pp.293–300.

�16� C. BRECHBUHLER, G. GERIG AND O. KUBLER, Pa-rameterization of Closed Surfaces for 3-D ShapeDescription, CVGIP: Image Understanding, 61�1995�, pp. 154–170, 1995.

�17� R.H. DAVIES, C. TWINING, T. COOTES, J. WATERTONAND C. TAYLOR, 3D Statistical Shape Models UsingDirect Optimisation of Description Length. ECCV,3 �2002�, pp. 3–20.

�18� S. MELLER, E. NKENKE AND W. KALENDER, Statis-tical Face Models for the Prediction of Soft-TissueDeformations After Orthognathic Osteotomies.MICCAI, 2 �2005�, pp. 443–450

�19� S. ZACHOW, H. LAMECKER, B. ELSHOLTZ AND M.STILLER, Reconstruction of mandibular dysplasiausing a statistical 3D shape model, Computer-Assisted Radiology and Surgery; Int CongresSeries, 1281, �2005�, pp. 1238–1243.

�20� A. FRANGI, F.D. RUECKERT, J.A. SCHNABEL ANDW.J. NIESSEN, Automatic construction of multipleobject three-dimensional statistical shape models:Application to cardiac modeling, IEEE Trans-actions on Medical Imaging, 21.9 �2002�, pp.1151–1166.

�21� M. FLEUTE, S. LAVALLEE AND R. JULLIARD, Incor-porating a statistically based shape model into asystem for computer-assisted anterior cruciate liga-ment surgery, Medical Image Analysis, 3.3 �1999�,pp. 209–222.

�22� F.L. BOOKSTEIN, Landmarkmethods for formswith-out landmarks: morphometrics of group differencesin outline shape, Medical Image Analysis, 1 �1997�,pp. 225–243.

�23� W.E. LORENSEN AND H.E. CLINE, Marching cubes:A high resolution 3D surface construction algo-rithm, Computer Graphics, 21 �1987�, pp. 163–169.

�24� R.A. BANVARD, The Visible Human ProjectrImage Data Set from Inception to Completionand Beyond, CODATA 2002 : Frontiers of scien-tific and technical data, Track I-D-2: Medical andHealth Data, �2002�, Montreal, Canada.

�25� F. PIGHIN, J. HECKER, D. LISCHINSKI, R. SZELISKIAND D.H. SALESIN, Synthesizing Realistic FacialExpressions from Photographs, Proceedings of Sig-graph, �1998�, pp. 75-84, Orlando, FL, USA.

�26� S. BUCHAILLARD, S.H. ONG, Y. PAYAN AND K.W.C.FOONG, Reconstruction of 3D Tooth Images, Pro-ceedings of ICIP 2004, �2004�, pp. 1077–1080.

�27� R. SZELISKI AND S. LAVALLEE, Matching 3-Danatomical surfaces with non-rigid deformationsusing octree-splines, International Journal of Com-puter Vision, 18 �1996�, pp. 171–186.

�28� M. BERAR, M. DESVIGNES, G. BAILLY AND Y.PAYAN, 3D meshes registration: application toskull model, Proceedings of the International Con-ference on Image Analysis and Recognition - ICIAR,�2004�, vol. 2. pp. 100–107, Porto, Portugal.

�29� I.L. DRYDEN AND K.V. MARDIA; Statistical ShapeAnalysis, John Wiley and Sons, London, UnitedKingdom, 1998.

�30� B. ROMANIUK , M. DESVIGNES, M. REVENU ANDM.-J. DESHAYES, Shape Variability and Spatial Re-lationships Modeling in Statistical Pattern Recog-nition, 25 �2004� Pattern Recognition Letters, pp.239–247.

�31� C. BASSO AND T. VETTER, Statistically Motivated3D Faces Reconstruction. Proceedings of the 2ndInternational Conference on Reconstruction of SoftFacial Parts, �2005�, pp. 71, Remagen, Germany.

�32� C. RYNN AND C. WILKINSON, An appraisal of estab-lished and recently proposed relationships betweenthe hard and soft dimensions of the nose in profile,Proceedings of the 2nd International Conferenceon Reconstruction of Soft Facial Parts, �2005�, pp.35–49, Remagen, Germany.

Received: February, 2006Revised: February, 2006

Accepted: February, 2006

Contact address:

Maxime BerarLaboratoire des Images et des Signaux961, rue de la Houille Blanche BP 46

38402 Saint Martin d’HeresFrance

e-mail: berar�lis�inpg�fr

Michel DesvignesLaboratoire des Images et des Signaux961, rue de la Houille Blanche BP 46

38402 Saint Martin d’HeresFrance

e-mail: michel�desvignes�lis�inpg�fr

MAXIME BERAR is a doctoral student at LIS, Grenoble. He received anengineer degree in electronics form ENSERG in 2002. His scientificinterests include statistical modeling and Kernel methods.

MICHEL DESVIGNES is currently professor of computer science at theENSERG and researcher at LIS. His scientific interests include imageprocessing and pattern recognition methods.

GERARD BAILLY is CNRS researcher since 1986, senior researcher since1991 and head of the Talking Machines team at the Institute of SpeechCommunication �ICP�, Grenoble.

YOHAN PAYAN is CNRS researcher at The Computer-Aided Surgerygroup �GMCAO: Gestes Medico-Chirurgicaux Assistes par Ordina-teur� of TIMC laboratory, and coordinates the modeling operations thatare conducted in TIMC.

3D Semi-Landmarks Based Statistical Face Reconstruction

Documents

Transcript of 3D Semi-Landmarks Based Statistical Face Reconstruction