3D Face Recognition using ICP and GeodesicComputation Coupled Approach

Karima Ouji‡, Boulbaba Ben Amor§, Mohsen Ardabilian§,Faouzi Ghorbel‡, and Liming Chen§

§LIRIS, Laboratoire d’InfoRmatique en Image et Systmes d’information,36, av. Guy de Collongue, 69134 Ecully, France.

‡GRIFT, Groupe de Recherche en Images et Formes de Tunisie,Ecole Nationale des Sciences de l’Informatique, Tunisie.

{boulbaba.benamor,Mohsen.ardabilian,liming.chen}@ec-lyon.fr

{karima.ouji,faouzi.ghorbel}@ensi.rnu.tn

Key words: 3D face recognition, Iterative Closest Point, Geodesics computa-tion, biometric evaluation

Abstract. In this paper, we present a new face recognition approachbased on dimensional surface matching. While most of existing methodsuse facial intensity images, a newest ones focus on introducing depth in-formation to surmount some of classical face recognition problems suchas pose, illumination, and facial expression variations. The presentedmatching algorithm is based first on ICP (Iterative Closest Point) thatalign rigidly facial surfaces and provides perfectly the posture of thepresented probe model. Second, the similarity metric consists in com-puting geodesic maps on the overlapped parts of the aligned surfaces.The general paradigm consists in building a full 3D face gallery usinga laser-based scanner (the off-line phase). At the on-line phase of iden-tification or verification, a captured 2 1

2D face model (range image) is

performed with the whole set of 3D faces from the gallery or comparedto the 3D face model of the genuine, respectively. This probe model canbe acquired from arbitrary viewpoint, with arbitrary facial expressions,and under arbitrary lighting conditions. Finally, We discuss some exper-imental results done on the ECL− IV 2 new 3D face database.

1 Introduction

Over the past few decades, biometrics and particularly face recognition andauthentication have been applied widely in several applications such as recogni-tion inside video surveillance systems, and authentication within access controldevices. However, as described in the Face Recognition Vendor Test report pub-lished in [1], as in other reports, most commercial face recognition technologiessuffer from two kinds of problems. The first one concerns inter-class similaritysuch as twins’classes, and fathers and sons’classes. Here, people have similar ap-pearances which make their discrimination difficult. The second, and the more

2-9525435-1 © SITIS 2006 - 435 -

important complexity, is due to intra-class variations caused by changes in light-ing conditions, pose variations (i.e. three-dimensional head orientation), andfacial expressions. On the one hand, lighting conditions change dramatically the2D face appearance ; consequently approaches only based on intensity imagesare insufficient to employ. On the other hand, pose variations present also a con-siderable handicap for recognition by performing comparisons between frontalface images and changed viewpoint images. In addition, compensation of facialexpressions is a difficult task in 2D-based approaches, because it significantlychanges the appearance of the face in the texture image.

In this paper we describe a new face recognition/authentication methodbased on new face modality: 3D shape of face. The remainder of the paper is or-ganized as follows: Section (2) reviews the recent progress in 3D face recognitiondomain. Section (3) describes an overview of the proposed approach. In Sections(4) we focus on developed works for 2 1

2D vs. 3D face matching via ICP. Thesection (5) describes the geodesics computation technique for facial expressioncompensation. In sections (6), we emphasize the evaluations of the developedmethod on ECL− IV 2 3D face database.

2 Recent progress on 3D face recognition

Current state-of-the-art in face recognition is interesting since it contains workswhich aim at resolving problems regarding this challenge. The majority of theseworks use intensity faces’images for recognition or authentication, called 2Dmodel-based techniques. A second family of recent works, known as 3D model-based, exploits three-dimensional face shape in order to mitigate some of thesevariations. Where some of them propose to apply subspace-based methods, oth-ers perform shape matching algorithm. As described in [6][9][8], classical linearand non-linear dimensional reduction techniques such as PCA and LDA areapplied to range images from data collection in order to build a projection sub-space. Further, the comparison metric computes distances between the obtainedprojections. Shape matching-based approaches rather use classical 3D surfacealignment algorithms that compute the residual error between the surface ofprobe and the 3D images from the gallery as already proposed in our works[13][14] and others as [11] and [3]. In [2], authors present a new proposal whichconsiders the facial surface (frontal view) as an isometric surface (i.e. length pre-serving). Using a global transformation based on geodesics, the obtained formsare invariant to facial expressions. After the transformation, they perform oneclassical rigid surface matching and PCA for sub-space building and face match-ing. A good reviews and comparison studies of some of these techniques (both2D and 3D) are given in [10] and [7]. Another interesting study which comparesICP and PCA 3D-based approaches is presented in [5]. Here, the authors showa baseline performance between these approaches and conclude that ICP -basedmethod performs better than a PCA-based method. Their challenge is expressionchanges, particularly ”eye lip open/closed” and ”mouth open/closed”.

- 436 -

In the present paper, we discuss accuracy of a new 3D face recognition methodusing the ICP -based algorithm with a particular similarity metric based ongeodesic maps computation. A new multi-view and registered 3D face databasewhich includes full 3D faces and probe images with all these variations is col-lected in order to perform significant experiments.

3 Overview of the proposed 3D face matching method

Our identification/authentication approach is based on dimensional surfaces offaces. As illustrated by figure 1, we build the full 3D face database with neutralexpressions. The models inside includes both shape and texture channels: the off-line phase. Second, a partial probe model is captured and compared to all full 3Dfaces in the gallery (if identification scenario) or compared to the genuine model(if authentication scenario): the on-line phase. The main goal of the availabilityof full 3D face models in the gallery is to allow comparison of the probe modelfor all view point acquisition.

Geodesic maps

via Fast Marching2

Surface matching

via ICP1

Probe model

(2½D model)

Similarity

metric3

DECISION

Accepted

Rej

ecte

d

Identified

"identity"

Ga

llery of fu

ll 3D

faces

...

Reco

gn

ition

e: match

ing 1

-to-N

Au

then

tica

tion

: m

atc

hin

g 1

-to-1

Phase off-line (partial model fusion

for full 3D face modeling)

Phase on-line (ICP based alignment,

geodesics maps )

Fig. 1. overview of the proposed 3D face matching method.

- 437 -

The core of our recognition/authentication scheme consists of aligning thencomparing the probe and gallery facial surfaces. The first step, approximates therigid transformations between the presented probe and the full 3D face, a coarsealignment step and then a fine alignment step via ICP (Iterative Closest Point)algorithm [4] are applied. This algorithm is an iterative procedure minimizing theMSE (Mean Square Error) between points in partial model and the closest pointsin the 3D full model. One of the outputs of the algorithm result is two matchedsets of points in the both surfaces. For the second step, two geodesic maps arecomputed for the pair of vertices in the matched surfaces. The recognition andauthentication similarity score is based on the distance between these maps.

4 ICP for 3D surface alignement

One of interesting ways for performing verification/identification is the 3D shapematching process. Many solutions are developed to resolve this task especiallyfor range image registration and 3D object recognition. The basic algorithm isthe Iterative Closest Point developed by Besl et al. and published in [4]. In ourapproach we consider first a coarse alignment step, which approximates the rigidtransformation between models and bring them closer. Then we perform a finealignment algorithm which computes the minimal distance and converges to aminima starting from the last initial solution. The last alignment step is based onthis well-known Iterative Closet Point algorithm [4]. It is an iterative procedureminimizing the Mean Square Error (MSE) between points in one view and theclosest vertices, respectively, in the other. At each iteration of the algorithm, thegeometric transformation that best aligns the probe model and the 3D modelfrom the gallery is computed. Intuitively, starting from the two sets of verticesP = {pi}, as a reference data, and X = {yi}, as a test data, the goal is to find therigid transformation (R, t) which minimizes the distance between these two sets.The target of ICP consists in determining for each vertex pi of the reference setP the nearest vertex in the second set X within the meaning of the Euclideandistance. The rigid transformation, minimizing a least square criterion (1), iscalculated and applied to the each point of P :

e(R, t) =1N

N∑

i=0

‖(Rpi + t)− yi‖2 (1)

This procedure is alternated and iterated until convergence (i.e. stability ofthe minimal error). Indeed, total transformation (R, t) is updated in an incre-mental way as follows: for each iteration k of the algorithm: R = RkR andt = t + tk. The criterion to be minimized in the iteration k becomes (2):

e(Rk, tk) =1N

N∑

i=0

‖(Rk(Rpi + t) + tk − yi‖2 (2)

The ICP algorithm presented above always converges monotonically to alocal minimum [4]. However, we can hope for a convergence to a global minimum

- 438 -

if initialization is good. For this reason, we perform the previous coarse alignmentprocedure before the fine one.

5 Geodesic computation for 3D surface comparison

3D surface alignment via ICP does not have succeeded in curing the problem offacial expressions which present non-rigid transformations, not able to be mod-elled by rotations and translations. Thus, in a second stage, we propose to com-pute geodesic distances between pairs of points on both probe and gallery facialsurfaces since this type of distances is invariant to both rigid and nonrigid trans-formations, as concluded in [2]. Therefore, an efficient numerical approach calledthe fast marching method [16] is applied for geodesic computations. A geodesicis a generalization of straight line notion into curve spaces [15]. A geodesic is ashortest path between two points on the considered surface (as shown by figure2).

Geodesic distance vs. euclidian distance

Geodesic distance on full 3D face model

Geodesic distance on 2½D face model with

facial expression

Fig. 2. Geodesic distance vs. euclidian distance computations on full 3D and partialface models with facial expressions.

5.1 Fast Marching on triangulated domains

The fast marching method, introduced by Sethian [15] is a numerically consistentdistance computation approach that works on rectangular grids. It was extendedto triangulated domains by Kimmel & Sethian in [16]. The basic idea is anefficient numerical approach that solves the Eikonal equation |∇u| = 1, where

- 439 -

at the source point s the distance is zero u(s) = 0, namely. The solution u is adistance function and its numerical approximation is computed by a monotoneupdate scheme that is proven to converge to the ’viscosity’ smooth solution.

The idea is to iteratively construct the distance function by patching togethersmall plans supported by neighboring grid points with gradient magnitude thatequals one. The distance function is constructed by starting from the sourcespoint, S, and propagating outwards. Applying the method to triangulated do-mains requires a careful analysis of the update of one vertex in a triangle, whilethe u values at the other vertices are given. For further details in this theory, werefer to [16].

5.2 Application to 3D face recognition

After 2 12D vs. 3D ICP alignment step, we propose to compute geodesic maps

on overlapped surfaces in both probe and gallery facial surfaces. We dispose ofa list of the corresponding point already provided by ICP. We consider for moresignificantly and robust computation the zones limited by a pair of verticesin probe model and 3D full face model. As shown in figure 3 (C) the sourcevertex in probe model is noted S1, the limit vertex L1 and their correspondantsin 3D face model: the source vertex S2 and the limit vertex L2 (see figure 3(B)). Computing geodesic distances in the zone limited by these vertices is lesssensitive to errors. In fact, the mouth region can present a hole if the mouth isopened, this introduces errors in geodesic computation algorithms, as shown byfigure 3 (A).

We propose to calculate a geodesic distance map on the 2 12D probe model

via the extension of fast marching method to triangulated domains proposed bySethian & Kimmel [16]. In fact, the algorithm starts on the source vertex S1 andpropagates along all the facial surface saving on every met vertex the geodesicdistance which separate him to the source vertex S1. All these distances makeup the vector V1 (the first geodesic map). Each line of V1 contains the geodesicdistance separating S1 and the vertex having the index i of the 2 1

2D mesh. Then,we compute geodesic distance map on the 3D mesh with the same principle. Inthis case, the source vertex is a vertex S2 and geodesic distance map is a vectorV2 . Each line of V2 contains the geodesic distance separating S2 from the vertexof the 3D mesh corresponding to the vertex having the index i of the 3D mesh.Finally, we compute the vector V as V = |V2 − V1| and the similarity metric isthe standard deviation of V and which we use for the recognition process.

In our implementation, we consider only vertices which are situated abovemouth. In other words, V1 and V2 contain only geodesic distance from sourcepoint to points in the higher part of the face. In fact, we compute all geodesicdistances on facial mesh surface 2 1

2D and we get rid of all distances which value ismore than the distance separating S1 from L1. Moreover, we compute all geodesicdistances on 3D facial mesh surface and we get rid of all distances which value ismore than the distance separating S2 and L2. We justify our choice by the factthat if the probe person is laughing for example, the 21

2D mesh contains a holein mouth. Thus, all geodesic distances computed for vertices situated under the

- 440 -

Probe model

Source point

(B) Computing geodesic map on full 3D face model

limit point

3D full face

modelConsidered zone between

source and limit points

Geodesic map on 3D full face model

Geodesic map on partial face modelConsidered zone between

source and limit points

(C) Computing geodesic map on probe model (2½D model)

Point source

Point limite

(A) The opened mouth

problem for computing geodesic

distance between points

Fig. 3. Geodesic maps computation: (A) the opened mouth problem, (B) computinggeodesic map on fill 3D face model, and (C) computing geodesic map on partial facemodel.

lower lip are very different from geodesic distances of their correspondants onthe 3D mesh which have no hole in mouth as illustrated by figure 3.

6 Experiments and evaluations

In this section we present some experimental results of the presented approachperformed on ECL−IV 2 3D face database. Figure 4 illustrates this rich databasewhich contains about 50 subjects (about 50 full 3D face and 400 probe models[13]). It includes full 3D face gallery (figure 4 (A)), and eight partial modelincluding variations in facial expressions, poses and illumination (figure 4 (D)).

We produce for each experiment, labelled by the considered variation, boththe error trade-off curve (for authentication scenario) and the rank-one recog-nition rates (for identification scenario). Figure 5 presents the recognition ratesfor the elementary experiments and the global one (all probe images are consid-ered). It shown that less the facial deformation is, more invariant the method is.In addition, the proposed paradigm is invariant to both illumination and posevariation problems (see figure 6). This is done by, respectively, cancelling texturedata in our processing scheme and the availability of the full 3D models in thegallery dataset. The global Recognition Rate (RR) is equal to 92.68% and theEqual Error Rate (EER) about 12% .

- 441 -

(A) 2½D for full 3D face building

(B) Full 3D face model obtained by association of pastial ones(C) Samples from 2½D probes models

(D) All probe variations for experiments

(f): frontal (l): left profile (r): left profile (e): closed eyes (s): surprised (d): disgasting (h): happy (i): illuminatione

Fig. 4. ECL− IV 2: new multi-view registred 3D face database.

Experiments (d) (e) (f) (h) (i) (l) (r) (s) (all)

rank-one rate (%) 80.48 92.68 97.56 90.24 97.56 97.56 97.56 87.80 92.68

Legend:(d) disgusting, (e) closed eyes, (f) frontal with neutral expressions, (h)happy, (i) uncontrolled illumination, (l) left profile, (r) right profile, (s) surprise, and

(all) all probe images.

Fig. 5. Rank-one recognition rates for elementary experiments and global one.

- 442 -

This approach shows more significant rates in absence of expression, this isbecause in presence of expression, the ICP -based alignement method providesless accurate matched points. In order to surmount this problem, it is moresignificant to perform alignment only on static regions of the face. We planto combine the proposed approche with our region-based method for mimicssegmentation already presented in [12]. In this approach the alignment onlyconcerns the static regions of the face.

0 5 10 15 20 250

5

10

15

20

25

False Accept Rate (%)

Fal

se R

ejec

t Rat

e (%

)

All experimentsExperiment (d)Experiment (e)Experiment (f)Experiment (h)Experiment (i)Experiment (l)Experiment (r)Experiment (s)

Fig. 6. DET (Error trade-off) curves for elementary and all experiments.

In conclusion, the presented approach is quasi-invariant to variations in poseand illumination. However, it shows some limits in presence of important facialexpression. As described in the last paragraphe, it can be enhanced by performingalignment and geodesic computation on static region of the face.

Acknowledgments. The authors thank the Biosecure participants for theircontribution to collect this 3D face database and their significant remarks. Au-thors thanks also Gabriel Peyre, who provide us his implementation of the FastMarching algorithm.

- 443 -

References

1. P.J. Phillips, P. Grother, R.J Micheals, D.M. Blackburn, E. Tabassi, J.M. Bone:Technical report NIST. FRVT 2002: Evaluation Report (Mars 2003).

2. A. M. Bronstein, M. M. Bronstein, R. Kimmel: Three-dimensional face recognition.In: International Journal of Computer Vision (IJCV): pages 5-30 (august 2005).

3. Xiaoguang Lu and Anil K. Jain: Integrating Range and Texture Information for3D Face Recognition. In: Proc. 7th IEEE Workshop on Applications of ComputerVision pages 156-163 (2005).

4. Paul J. Besl and Neil D. McKay: A Method for Registration of 3-D Shapes. In:Proc. IEEE Trans. Pattern Anal. Mach. Intell vol. 14 pages 239-256 (1992).

5. Chang, K. J. and Bowyer, K. W. and Flynn, P. J.: Effects on facial expression in3D face recognition. Proceedings of the SPIE, Volume 5779, pp. 132-143 (2005).

6. Gang Pan and Zhaohui Wu and Yunhe Pan: Automatic 3D Face Verification FromRange Data. Proc. IEEE International Conference on Acoustics, Speech, and SignalProcessing, pp. 193-196 (2003).

7. Chenghua Xu and Yunhong Wang and Tieniu Tan and Long Quan: Depth vs. In-tensity: Which is More Important for Face Recognition?. Proc. 17th InternationalConference on Pattern Recognition, 2004.

8. Sotiris Malassiotis and Michael G. Strintzis: Pose and Illumination Compensationfor 3D Face Recognition. Proc. IEEE International Conference on Image Processing,pp 91-94, 2004.

9. Thomas Heseltine and Nick Pears and Jim Austin: Three-Dimensional Face Recog-nition: An Eigensurface Approach. Proc. IEEE International Conference on ImageProcessing, pp 1421-1424, 2004.

10. Kyong I. Chang and Kevin W. Bowyer and Patrick J. Flynn: An Evaluation ofMulti-modal 2D+3D Face Biometrics. IEEE Transactions on PAMI, vol. 27 pp 619-624, 2005.

11. Charles Beumier and Marc Acheroy: Automatic 3D Face Authentication. Imageand Vision Computing, vol. 18 pp 315-321, 2000.

12. B. BenAmor and M. Ardabilian and L. Chen: Enhancing 3D Face Recognition ByMimics Segmentation. Intelligent Systems Design and Applications (ISDA2006), pp.150-155, 2006.

13. B. BenAmor and M. Ardabilian and L. Chen: New Experiments on ICP-based3D face recognition and authentication. Proceeding of International Conference onPattern Recognition (ICPR2006), pp. 1195-1199, 2006.

14. B. BenAmor and K. Ouji and M. Ardabilian and L. Chen: 3D Face recognition byICP-based shape matching. roceeding of IEEE International Conference on MachineIntelligence (2005).

15. J.A. Sethian: A Fast Marching Level Set Method for Monotonically AdvancingFronts. Proc. Nat. Acad. Sci, 93, 4, 1996.

16. J.A. Sethian et R. Kimmel: Computing Geodesic Paths on Manifolds. Proc. Proc.Natl. Acad. Sci., 95(15):8431-8435, 1998.

- 444 -