Pattern Recognition 36 (2003) 781–798www.elsevier.com/locate/patcog

Function-described graphs for modelling objects representedby sets of attributed graphs

Francesc Serratosaa ; ∗, Ren*e Alqu*ezarb, Alberto SanfeliucaDepartment d’Enginyeria Inform�atica i Matem�atiques, Universitat Rovira i Virgili, Campus Sescelades,

Avinguda dels Pa"#sos Catalans 26, 43007 Tarragona, SpainbDept. de Llenguatges i Sistemes Inform�atics, Universitat Polit�ecnica de Catalunya, SpaincInstitut de Rob�otica i Inform�atica Industrial, Universitat Polit�ecnica de Catalunya, Spain

Received 19 November 2001; accepted 23 May 2002

Abstract

We present in this article the model function-described graph (FDG), which is a type of compact representation of a set ofattributed graphs (AGs) that borrow from random graphs the capability of probabilistic modelling of structural and attributeinformation. We de4ne the FDGs, their features and two distance measures between AGs (unclassi4ed patterns) and FDGs(models or classes) and we also explain an e6cient matching algorithm. Two applications of FDGs are presented: in theformer, FDGs are used for modelling and matching 3D-objects described by multiple views, whereas in the latter, they areused for representing and recognising human faces, described also by several views.? 2002 Pattern Recognition Society. Published by Elsevier Science Ltd. All rights reserved.

Keywords: Attributed graphs; Error-tolerant graph matching; Function-described graphs; Random graphs; Clustering; Synthesis; 3D-objectrecognition and face identi4cation

1. Introduction

A function-described graph (FDG) is a model that con-tains probabilistic and structural descriptions of a set ofattributed graphs (AGs) to maintain, to the most part, thelocal features of the AGs that belong to the set and otherAGs that are “near” them, as well as to allow the rejectionof the AGs that do not belong to it or are “far” from them[1]. Let us consider, as an example, the 3D-object modellingand recognition problem. Fig. 1 shows schematically theFDG learning and classi4cation processes. The basic idea isthat only a single FDG is synthesised from the graphs thatrepresent several views of a 3D-object. Therefore, in the

∗ Corresponding author. Tel.: +34-977-55-85-07; fax: +34-977-55-97-10.

E-mail addresses: francesc.serratosa@etse.urv.es (F. Ser-ratosa), alquezar@lsi.upc.es (R. Alqu*ezar), asanfeliu@iri.upc.es(A. Sanfeliu).

recognition process, only one comparison is neededbetween each model represented by an FDG and theunclassi4ed object (view of a 3D-object) represented by agraph.

The FDG techniques have been applied on the project “ac-tive vision system with automatic learning capacity for in-dustrial applications” in which an autonomous mobile robothas been developed [2].

Random graphs [3–5] are one of the earliest approachesused to represent a set of AGs. In this approach, AGs areextended to include probabilistic information. Wong et al.4rst de4ned the general random graphs (GRGs) for mod-elling classes of patterns described by AGs through a jointprobability space of random variables ranging over patternprimitives (vertices) and relation (arcs). Due to the com-putational intractability of GRGs, caused by the di6cultyin estimating and handling the high-order joint probabil-ity distribution, 4rst-order random graphs (FORGs) wereproposed for real applications [4]. Strong simpli4cationswere made in FORGs to allow the use of random graphs in

0031-3203/02/$22.00 ? 2002 Pattern Recognition Society. Published by Elsevier Science Ltd. All rights reserved.PII: S0031 -3203(02)00107 -3

782 F. Serratosa et al. / Pattern Recognition 36 (2003) 781–798

Fig. 1. FDG learning and classi4cation processes applied to the 3D-object recognition.

practical cases; more precisely, the following assumptionswere made [4]:

(1) the random vertices are mutually independent;(2) given values for the random vertices, the random arcs

are independent;(3) the arcs are independent of the vertices except for the

vertices that they connect.

FDGs can be seen as a type of simpli4cation of the GRGs,diGerent from FORGs, in which some structural constraintsare recorded. A drawback of FORGs is that the strong as-sumptions about the statistical independence of nodes andarcs may lead to an excessive generalisation of the samplegraphs when synthesising an FORG. To alleviate this weak-ness, a qualitative information of the joint probabilities oftwo nodes is incorporated into FDGs, thus improving therepresentional power of FORGs with a negligible increaseof computational cost.

Other diGerent approaches that de4ne a model to repre-sent a set of AGs are mentioned here. Winston [6] presentedone of the earliest works in which a set of examples wasused to learn the structures that represent the class. Later on,Stein [7] presented a system to identify the instances of theobjects in the image. Other authors presented systems ap-plied as well to the problem of 3D-object recognition fromsingle-view graphs: for instance, the interesting approach tograph matching and classi4cation [8], in which, from a train-ing set of AGs corresponding to several classes, a singlemodel graph called rulegraph is learnt (using evidence-basedsystems methods) which describes the full training set. Inthis rulegraph, vertices do not represent object parts butrules which depict attribute bounds and evidence for dif-ferent classes. A new AG is then classi4ed by rulegraphmatching (subgraph isomorphism), with the advantage of areduced search space, since the number of vertices in therulegraph can be much lower than the number of nodes inthe original AGs. And also, Kim and Kak [9] represented a

3D-object as an assembly of several parts, each representedas a node in a graph carrying information characterisingthe part. Sossa [10] indexes a library of graph-based mod-els using the coe6cients derived from the Laplacian matrixthat describes the graph. However, the graphs are not at-tributed and the system has been shown to work only withsimplistic objects comprising planar surfaces. Shapiro andHaralick [11] organise models in a tree according to an in-termodel distance metric. In the approach by Messmer andBunke [12–14], the model graphs are pre-processed gener-ating a symbolic data structure, called network of models.This network is a compact representation of the models inthe sense that multiple occurrences of the same sub-graphare represented only once. Consequently, such sub-graphswill be matched only once with the input and the computa-tional eGort will be reduced. A recent approach is the gen-eralised attributed relational graph (GARG) [15], in whichthe set of AGs is represented by only one model or proto-type, but diGerently from the network of models, the modelhas the same structure and attribute domain as the AGs thatrepresent, except that a null value is introduced. GARGsare synthesised by a prototyping algorithm, based on in-ductive learning methodologies. Moreover, Chan proposedthe fuzzy attributed graphs (FAGs) and the hard FAGsto represent objects and templates, respectively, [16]. Aneural networks method [17] was proposed to encode anAG into a real-valued feature vector using recurrent neu-ral network architectures and a generalised recursive neuronmodel. A feedforward network classi4es the feature vectorsyielded by the encoder network while returning an error sig-nal to the encoder, which is then used to adapt the encoderweights. Note however that, in this case, there is no learntmodel graph that represents a set of AGs, but a collectionof learnt feature vectors, one for each AG, together with atrained neural classi4er. Finally, Sengupta [18] presented ahierarchically structured approach to organising large struc-tural model bases using an information theoretic criterion.Objects or patterns are modelled in the form of random

F. Serratosa et al. / Pattern Recognition 36 (2003) 781–798 783

parametric structural descriptions and objects in the scenesare represented as parametric structural descriptions.

The structure of the rest of the paper is as follows. InSection 2, all the notation used throughout the paper is in-troduced, including the formal de4nitions of AG and FDG.The fundamental ideas behind FDGs are discussed in Sec-tion 3, which also includes an illustrative example. In Sec-tion 4, two distance measures between an AG and an FDGare presented, whereas the algorithms for computing orapproximating them are sketched in Section 5. The ex-perimental results of two applications of FDGs for objectmodelling and recognition are presented in Section 6:3D-object modelling and recognition [19] and face identi4-cation [20]. Finally, some conclusions are given in Section 7.

We de4ned some methods and algorithms to synthesisean FDG using a set of classi4ed AGs and also to cluster AGsand synthesise several FDGs that represent the clusters inRefs. [21,22]. Due to lack of space, these methods are outof the scope of this article.

2. Formal de�nitions and notation

De�nition 1. Let �v and �e denote the domains of possiblevalues for attributed vertices and arcs; respectively. Thesedomains are assumed to include a special value � that rep-resents a null value of a vertex or arc. An AG G over(�v; �e) is de4ned to be a four-tuple G = (�v; �e; �v; �e);where �v = {vk |k = 1; : : : ; n} is a set of vertices (or nodes);�e = {eij|i; j∈{1; : : : ; n}; i �= j} is a set of arcs (or edges);and the mappings �v : �v → �v and �e : �e → �e assignattribute values to vertices and arcs; respectively.

De�nition 2. A complete AG is an AG with a completegraph structure (�v; �e); but possibly including null ele-ments. An AG G with n vertices can be extended to forma complete AG G′ with k vertices k¿ n; by adding ver-tices and arcs with null attribute values �. We call G′ thek-extension of G.

De�nition 3. Let�v and�e be two sets of random variableswith values in �v (random vertices) and in �e (random arcs);respectively. A function-described graph F over (�v; �e) isde4ned to be a tuple (�v; �e; �v; �e; P; R);where �v={!k |k=1; : : : ; n} is a set of vertices; �e={�ij|i; j∈{1; : : : ; n}; i �= j}is a set of arcs; the mapping �v : �v → �v associates eachvertex !k ∈�v with a random variable �k=�v(!k) with val-ues in �v; and �e : �e → �e associates each arc �ij ∈�e witha random variable �k=�e(�ij) with values in �e. P=(Pv; Pe)are two sets of marginal (or 4rst-order) probability densityfunctions for random vertices and edges; respectively. Thisis; Pv = {pi(a); 16 i6 n} and Pe = {qj(b); 16 j6m}; mbeing the number of edges; where pi(a) = Pr (�i = a) forall a∈�v and qj(b) ≡ Pr (�j = b|�j1 �=� ∧ �j2 �=�) for allb∈�e such that �j1; �j2 refer to the random variables for theendpoints of the arc associated with �j . Note that; due to the

structural consistency requirements; there is no need to storethe conditional probabilities Pr(�j = b|�j1 = � ∨ �j2 = �);since by de4nition Pr(�j = �|�j1 = � ∨ �j2 = �) = 1.R = (Av; Ov; Ev) is a collection of boolean functions

de4ned over pairs of graph vertices (i.e. relations on theset of vertices) that allow the incorporation of qualitativesecond-order probability information. Thus, Av : �v×�v →{0; 1}, de4ned by Av(!i; !j)=1 ⇔ Pr(�i �=�∧�j �=�)=0,is the so-called vertex antagonism function, which can beseen as a symmetric binary relation on the set �v. In addi-tion, Ov : �v × �v → {0; 1}, de4ned by Ov(!i; !j) = 1 ⇔Pr(�i �=�∧ �j =�) = 0, is the so-called vertex occurrencefunction, which can be seen as a reMexive and transitive re-lation (partial order) on the set �v. Finally, �v : �v ×�v →{0; 1}, de4ned by Ev(!i; !j)=1 ⇔ Pr(�i=�∧�j=�)=0,is the so-called vertex existence function, which can alsobe seen as a symmetric binary relation on �v.

A random element � or � of an FDG is a null randomelement if its probability of instantiation to the null value isone, i.e. Pr(� = �) = 1 or Pr(� = �) = 1.

De�nition 4. A complete FDG is an FDG with a completegraph structure (�v; �e); but possibly including null randomelements. An FDG F with n vertices can be extended toform a complete FDG F ′ with k vertices k¿ n; by addingnull random vertices and null random arcs and extendingappropriately the boolean functions of antagonism; occur-rence and existence. We call F ′ the k-extension of F . Notethat both F ′ and F represent the same model.

De�nition 5. Any AG obtained by instantiating all ran-dom vertices and random arcs of an FDG in a way thatsatis4es all the relations of antagonism; occurrence andexistence speci4ed by the FDG is called an outcome graphof the FDG. Hence; similar to the case of random graphs;an FDG represents the set of all possible AGs that can beoutcome graphs of it; according to an associated probabilitydistribution.

3. A function-described graph as a model of a class ofattributed graphs

FDGs are proposed here for modelling classes of patternsdescribed by attributed graphs. An FDG is a probabilisticstructural model that contains some functions to maintain,to the most part the local features of the AGs that belong tothe class as well as to allow the rejection of the AGs thatdo not belong to it. In general, an FDG can be derived asa synthesis of a cluster or set of individual AGs, similar tothe case of random graphs. A comparison between FDGsand FORGs is sketched in Section 3.1 to introduce the novelfeatures that are included in FDGs to represent a set of AGs.Then, a simple example of representing a 3D-object usingFDGs is provided for illustrative purposes in Section 3.2.

784 F. Serratosa et al. / Pattern Recognition 36 (2003) 781–798

3.1. FDGs: a model lying between FORGs and generalrandom graphs

In order to attain a compact representation of a set ofAGs by means of a model, a probabilistic description of theensemble is desirable to account for the variations of struc-tural patterns in the reference set or sample. As recalled inSection 1, random graphs (RGs) provide such a represen-tation. Nevertheless, when estimating the probability distri-bution of the structural patterns from an ensemble of AGs,it is impractical to consider the high-order probability dis-tribution where all the graph elements are taken jointly. Itwas the random graph authors believe (and also ours), thatgeneral RGs cannot be applied in real applications, and forthis reason, they proposed the FORGs.

The FORG approach, although simpli4es the representa-tion considerably, continues to be di6cult to apply in realproblems where there is a large number of vertices in theAGs and their attributes have an extensive domain. The maincause of this problem is the dependence of the arc attributeswith respect to the attributes of the vertices that the arc con-nects (assumption 3 in Section 1). Although this supposi-tion is useful to constrain the generalisation of the givenset of AGs, it needs a huge amount of data to estimate theprobability density functions and bears a high computationalcost. On the other hand, an important drawback of FORGs,which is due to the probability independence assumptions1 and 2 in Section 1, is that the structural information in asample of AGs is not well preserved in the FORG synthe-sised from them. This is because an FORG represents anover-generalised prototype that may cover graph structuresquite diGerent from those in the sample. For example, if Cis a set of AGs describing diGerent perspective views of anobject O, many of the outcome graphs of the FORG syn-thesised from C will represent impossible views of O (justfrom the topological point of view, without further consid-eration of the attributes of primitives and relations).

With the aim of oGering a more practical approach,function-described graphs (FDGs) can be seen as a diGer-ent type of simpli4cation of the GRGs, in which anotherapproximation of the joint probability P of the random el-ements is proposed. On the one hand, some independenceassumptions (1) are considered, but on the other hand, someuseful second-order functions (2) are included to constrainthe generalisation of the structure.

(1) Independence assumptions in the FDGs:

1. The attributes in the vertices are independent of the othervertices and also of the arcs.

2. The attributes in the arcs are independent of the otherarcs and also of the vertices. However, it is mandatorythat all non-null arcs be linked to a non-null vertex ateach extreme in every AG covered by an FDG. In otherwords, any outcome AG of the FDG has to be structurallyconsistent.

With these assumptions, the probability density functionsare themselves independent since the attributes in the arcsdo not depend on the attributes in the vertices that they con-nect, but only on the existence of the extreme vertices. Con-sequently, associated with each graph element in an FDG,there is a random variable that represents the semantic in-formation distribution of the corresponding graph elementsin the set of outcome AGs. A random variable has a prob-ability density function de4ned over the same attribute do-mains of the AGs, including the null value �, that denotesthe non-instantiation of an FDG graph element in an out-come AG.

(2) Second-order functions in the FDGs:In order to tackle the problem of the over-generalisation

of the sample, we introduce the antagonism, occurrence andexistence relations in FDGs, which apply to pairs of vertices.In this way, random vertices are not assumed to be mutuallyindependent, at least with regard to the structural informa-tion. These second-order relations, that suppose a little in-crease of the amount of data to be stored in the prototype,are useful for two reasons. Firstly, they constrain the set ofoutcome graphs covered by the prototype and tend to cutdown notably the structural over-generalisation. Secondly,they reduce the size of the search space of the AG-to-FDGmatching algorithm, decreasing the global temporal cost ofthe recognition [1]. Antagonism, occurrence and existencerelations in the FDGs represent a qualitative informationof the second-order joint probability functions of a pair ofvertices. To de4ne these second-order relations it is neces-sary to split the domain of the joint probabilities in four re-gions (see Fig. 2a). The 4rst one is composed of the pointsthat belong to the Cartesian product of the sets of actualattributes of the two elements, corresponding to the caseswhere both elements are de4ned in the initial non-extendedAG and therefore their value is not null. The second andthird regions are both straight lines in which only one of theelements has the null value. This covers the cases when oneof the two elements does not belong to the initial AG and hasbeen added in the extending process. Finally, the fourth re-gion is the single point where both elements are null, whichincludes the cases when none of them appear in the initialAG. The second-order relations are de4ned as follows:Antagonism relations: Two vertices of the FDG are an-

tagonistic if the probabilities in the 4rst region are all zero,Pr(�i �=� ∧ �j �=�) = 0, which means that, although thesevertices are included in the prototype as diGerent elementaryparts of the covered patterns, they have never taken placetogether in any AG of the reference set used to synthesisethe FDG. Fig. 2b shows the joint probabilities of the vertices!i and !j de4ned as antagonistic.Occurrence relations: There is an occurrence relation if

the joint probability function equals zero in the second re-gion, Pr(�i �=�∧�j=�)=0. That is, it is possible to assurethat if the element �i does appear in any AG of the refer-ence set, then the element �j must appear too. The case ofthe third region is analogous to the second one with the only

F. Serratosa et al. / Pattern Recognition 36 (2003) 781–798 785

Fig. 2. (a) Split of the joint domain of two random vertices in four regions. Second-order density function of (b) two antagonistic vertices,(c) two occurrent vertices and (d) two existent vertices.

diGerence of swapping the elements. Fig. 2c shows the jointprobabilities of the vertices!i and!j , there is an occurrencerelation from !i to !j .Existence relations: Finally, there is an existence relation

between two vertices if the joint probability function equalszero in the fourth region, Pr(�1 =�∧�j=�)=0, that is, allthe objects in the class described by the FDG have at leastone of the two elements. Fig. 2d shows the joint probabilitiesof the vertices !i and !j .

3.1.1. Mapping joint probabilities to second-orderrelations

Fig. 3 shows 16 diGerent combinations of the joint proba-bility of two vertices if it is considered that the probabilitiesof the four regions can be zero or greater than zero. An Xis written on a region if and only if the sum of the proba-bilities in that region is greater than zero. For each one ofthe 16 cases, the second-order relations obtained from thecorresponding joint probability density function are shown.

Note that it is not “logical” for both occurrence and an-tagonism relations to be satis4ed at the same time betweentwo elements. If two elements cannot exist in the same AG(antagonism), one of them cannot always exist when theother exists (occurrence). This combination only appears incases in which one of the elements is null (cases 1–5); thatis, it is a synthetically created element. Moreover, the 16thcombination is impossible in a correct FDG since the sumof the joint probability throughout the four regions equals 1.Therefore, the four second-order relations cannot appear be-tween two graph elements at the same time.

3.2. An example of modelling 3D-object described byviews using FDGs

In this simple example, FDGs are used to represent3D-objects with planar faces. The knowledge of the3D-object represented by an FDG depends on the viewstaken from it. The representation of 3D-objects by meansof FDGs is a useful application to show the meaningof the probabilities in the vertices and arcs and also ofsecond-order relations. Fig. 4 shows object A, four diGerentperspective views and their corresponding AGs. Verticesand arcs represent faces and adjacency between faces, re-spectively. The attribute of the vertices is chosen to be theaverage hue of the planar face discretised in 7 colours and

the attribute of the arcs is chosen to be the cyclic distancebetween hues discretised in 4 categories. The direction ofthe arcs (from tail to head) is set by the combination of bothnodes that gives lower cyclic distance. In this example, wewill suppose that the extraction of graph elements and theirattributes is robust enough to obtain the same discrete value(colour) in all the vertices that represent the same faceand the same discrete value (colour similarity of adjacentregions) in all the arcs that represent the same edge.

Similar to AGs, vertices and arcs of FDGs represent inthis case faces and edges of the 3D-object. We assume thatthere is a given labelling between vertices and arcs of theAGs such that same faces in diGerent views are matched.Then, vertices or arcs in the AGs that represent the sameface or the same edge are used to build only one vertex orarc in the FDG (the synthesis of FDGs given a set of AGswith a common labelling is explained in Ref. [21]). We haveto take into account that faces that cannot be seen togetherin a view appear as diGerent vertices in the FDG. This fea-ture of the 3D-object representation is characterised throughthe probability density functions and the second-order rela-tions. When a face is not visible in a view, then its AG doesnot include its related vertex, but a null vertex, called v�, isintroduced in the extended AG with a null value, � (Sec-tion 2). Similarly, a null arc e� is extended when at leastone of the adjacent faces of the corresponding edge is oc-cluded in a view. Hence, the probability of a face or edge tobe occluded is represented by the probability of its relatedvertex or arc of being null. Two faces are antagonistic whenit is not possible to see both in the same view. Translatedto representations, two vertices are antagonistic when thereis no AG that contains both of them. On the other hand, aface is occurrent with respect to another if always the for-mer is visible (its vertex appears in the AG) and the latteris visible too. Finally, there is an existence relation betweentwo faces (vertices) if one of them or both appear in all theviews (in all the AGs).

The FDG Fa (Fig. 5) represents object A if we consideronly View1 and View2. Because only six faces of the objecthave been seen, the object is considered to have only thesefaces and thereforeFa has six vertices. Some faces are visiblein both views (i.e. face1 and face7), and thus, the probabilityof the related vertices in the FDG of taking their colour is 1.Other faces are only visible in one of both views (i.e. face2)and for this reason, the probability of being null is 1

2 . There

786 F. Serratosa et al. / Pattern Recognition 36 (2003) 781–798

Fig. 3. The sixteen combinations of the joint probability. Antagonism, occurrence and existence relations are represented by ↔;→ and•—-•, respectively.

Fig. 4. (a) Object A. Dashed lines are crossed out numbers represent non-visible edges and faces, respectively. (b) Four prespectiveviews of A and the obtained AGs {G1; G2; G3; G4}. (c) Attribute values of edge and nodes of the AGs: G1; G2; G3; G4. Attributeson the edges are: VS = very similar; S = similar;D= dissimilar and VD= very dissimilar. Attributes on the nodes are: C = cyan;B= blue;R = red;G = green;O = orange;Y = yellow and M = magenta.

Fig. 5. FDG Fa: (a) structure, (b) second-order relations. A(row, col)=antagonism (!row; !col), E(row, col)=existence (!row; !col) andO(row, col)= occurrence (!row; !col). (c) Labelling between the two AGs: G1 and G2 and the obtained probabilities in the FDG. The 44rst columns are related to vertices and the other 4 are related to edges. The non-existence of a vertex or an edge in the AGs is representedby the null vertex vR or the null edge eR, respectively. The column “lab” shows the labelling between the vertices and edges of the twoAGs:G1 and G2 and the ones in the FDG.

F. Serratosa et al. / Pattern Recognition 36 (2003) 781–798 787

Fig. 6. FDG Fb generated by 4 views of object A or AGs: G1; G2; G3 and G4: (a) second-order relations; (b) structure; (c) probabilitiesof FDG nodes; (d) probabilities of FDG arcs.

is an antagonism between face2 and face6 because they havenot been seen at the same time (they cannot be seen togetherindeed). There is an occurrence from face2 to face7 becausewhen face2 has not been occluded, neither face7. There areexistence relations between all the vertices, except !2 and!4 because both are occluded at view2. Note that !2 and !6

are related with both antagonism and existence relations. Itis proved in Ref. [1] that it is not possible that two verticesbe related through the three second-order relations.

The FDG Fb (Fig. 6) represents again object A but con-sidering now the four views (View1–View4). Fb incorpo-rates a new vertex representing face8 of the object, whichis visible in View3 and View4. Face5 is not visible in anyof the four views, and therefore there is no vertex in theFDG representing this face. Face7 is visible in View1 andView2 but it is occluded in View3 and View4. For this rea-son, Face7 has a zero probability of being null in Fa but theprobability of being null in Fb is 1

2 .Moreover, some occurrence and existence relations disap-

pear, e.g. the occurrences in which !7 is the second element.It is easily proved that, when new AGs are incorporated inFDGs, relations between previously existing vertices can bekept or disappear, but they cannot be incorporated. New re-lations appear only when at least one of the involved verticeshas just been incorporated in the FDG.

4. Distance measure between AGs and FDGs

Two distance measures are presented in this section toprovide a quantitative value of the match between an AG G

(data graph) and an FDG F (model graph). They are some-how related to the maximum a posteriori probability P(f|G)of a labelling function f :G → F given the measurementsand structure of the data graph G. The Bayes theorem al-lows the maximisation of the product P(G|f)P(f) insteadof the posterior probability P(f|G) to arrive at an optimalmatching f∗. And assuming a uniform probability distribu-tion among the valid labelling functions (those that satisfythe required structural constraints), the problem is reducedto that of 4ltering the invalid mappings and maximisingP(G|f) within the remaining set H of allowable con4gura-tions. Alternatively, we may attempt to minimise a globalcost measure Cf of a morphism f in the set H , by takingthe cost as a monotonic decreasing function of the condi-tional probability of the data graph given the labelling func-tion, Cf = func(P(G|f)). For instance, Cf =−ln(P(G|f))would be a possible choice.

Once a cost measure Cf is de4ned, a measure of dissim-ilarity between G and F (from here, distance measure) andthe optimal labelling f∗ are de4ned respectively as

d=minf∈H

{Cf} and f∗ = argminf∈H

{Cf}: (1)

Note that the joint probability P(G|f) cannot be estimateddirectly and has to be approximated by the product of the4rst-order probabilities of the elements,

P(G|f) =∏∀x

Pr(�(y) = �(x) |f(x) = y); (2)

where x and y are graph elements in the AG and the FDGrespectively, �(y) is the random variable associated with

788 F. Serratosa et al. / Pattern Recognition 36 (2003) 781–798

y; �(x) is the attribute value in x, and all the elements ofboth graphs have to appear in the productory (possibly byextending the mapping with null elements).

Thus, the previous choice for the global cost Cf gives

Cf =−∑∀x

ln(Pr(�(y) = �(x) |f(x) = y)): (3)

However, only if that one graph element had a probabilityof zero, the joint probability would be zero and Cf wouldbe in4nite. Since this may happen due to the noisy pres-ence of an unexpected element (insertion) or the absenceof a model’s element (deletion), only if that one graph ele-ment were not properly mapped, the involved graphs wouldbe wrongly considered to be completely diGerent. We musttherefore admit the possibility of both extraneous and miss-ing elements in the data graphs, since the data extracted fromthe information sources (e.g. images) will usually be noisy,incomplete or uncertain. As a consequence, the matches forwhich P(G|f)=0 should not be discarded since they couldbe the result of a noisy feature extraction and graph for-mation. In addition, a model (FDG) should match to a cer-tain degree not only the objects (AGs) in its learning setbut also the ones that are “near”. Hence, for practical pur-poses it is more appropriate to decompose the global costCf into the sum of some bounded individual costs, one foreach of the graph element matches. Although it has the ma-jor Maw that Cf is no longer a monotonically decreasingfunction of the probability P(G|f) considered as a whole, ithas the advantage that clutter aGects only locally the globalcost. An individual cost C(x; y) represents the dissimilar-ity between two mapped elements x and y, and it could bebased still on the 4rst-order probabilities of the graph ele-ments, C(x; y) = func(Pr(�(y) = �(x)|f(x) = y)), as far asit is bounded by some 4xed constant, C(x; y)6Max, forinstance C(x; y)6 1.Moreover, for the sake of robustness, the mapping will

not be de4ned from the initial AG which represents the pat-tern to the initial FDG which represents the class or model,but from the k-extended AG to the k-extended FDG, to con-template the possibility of some missing graph elements orsome extraneous graph elements introduced by noisy eGects.A missing element in the AG will be represented by a nullelement in the extended AG, and an extraneous element inthe AG should be mapped to a null element in the extendedFDG. The number of vertices k in the extended graphs istheoretically set to the sum of the number of vertices in bothinitial graphs. Hence, the limit situations in which all thegraph elements in the FDG are missing in the AG or all thegraph elements in the AG are extraneous are covered.

Let G′ be a k-extension of the AG G with an underlyingstructure (�v; �e) and F ′ be a k-extension of the FDG Fwith an underlying structure (�!; ��). Then, G′ and F ′ arestructurally isomorphic and complete with the same numberof vertices k, and they also share a common attribute domain(�v; �e). The labelling function f :G′ → F ′ is actuallyde4ned as a pair of morphismsf=(fv; fe), wherefv :�v →

�! is a mapping de4ned on the vertices and fe :�e → ��is a mapping de4ned on the arcs.

Now, we de4ne the individual cost of matching a pair ofelements as a normalised function depending on their dis-similarity, as given by the negative logarithm of the proba-bility of instantiating the random element of the FDG to thecorresponding attribute value in the AG,

C(x; y) =

−ln(Pr(�(y) = �(x) |f(x) = y))

−ln(KPr)if Pr(�(y) = �(x)|f(x) = y)¿KPr ;

1 otherwise;(4)

where the cost C(x; y) is bounded by [0; 1], and the positiveconstant KPr ∈ [0; 1] is a threshold on low probabilities thatis introduced to avoid the case ln(0), which gives negativein4nity. Hence, C(x; y) = 1 will be the cost of matchinga null element of the FDG to a non-null element of theAG or matching an FDG element to an AG element whoseattribute value has a very low probability of instantiation,i.e. Pr(�(y) = �(x)|f(x) = y)6KPr .

In the case of the vertices, the cost is de4ned using theprobabilities stored in the FDG as

Cfv (vi; !q) =

−ln(pq(ai))−ln(KPr)

if pq(ai)¿KPr;

1 otherwise;(5)

where fv(vi) = !q and �v(vi) = ai. And in the case of thearcs, we de4ne the cost as

Cfe (eij ; �ab) =

−ln(qn(bm))−ln(KPr)

if qn(bm)¿KPr;

1 otherwise;(6)

wherefe(eij)=�ab; �e(eij)=bm in the AG arc and ��(�ab)=�nin the matched FDG arc.

However, if either vi or vj is a null extended vertex in theAG, then the conditional probability qn(bm) is not applicable,since it depends on the existence of the two extreme vertices.But if we consider that the AG is structurally correct, eij hasto be a null arc in that case, then we apply Pr(�n = �|�a =� ∨ �b = �) = 1, which gives Cfe (eij ; �ab) = 0.

The global cost of a given mapping f based onlyon 4rst-order information is therefore computed as the sumof the individual costs of all the matches between graphelements,

Cf =∑∀xC(x; f(x)) =

∑∀vi∈�v of G′

Cfv (vi; fv(vi))

+∑

∀eij∈�e of G′Cfe (eij ; fe(eij)): (7)

Now, in order to take into account the second-order infor-mation included in the FDG boolean functions, there are twooptions, which lead to the two diGerent distance measuresthat are presented in the following subsections.

F. Serratosa et al. / Pattern Recognition 36 (2003) 781–798 789

4.1. Distance measure between AGs and FDGs usingsecond-order constraints

The 4rst approach is to consider the antagonism, occur-rence and existence relations in the FDG as second-orderconstraints that must be ful4lled by any valid morphismf∈H . We say that f∈Fv;e, if the morphism is bijectiveand structurally correct, and f∈FA; f∈FO and f∈FE ,if the antagonism, occurrence and existence constraints aremet, respectively. More precisely, the three second-orderconstraints are de4ned as follows:

f∈FA⇔ (Av(!p; !q) = 1 ∧ fv(vi) = !p ∧ fv(vj) = !q)⇒ (ai = � ∨ aj = � ∨ pp(�) = 1 ∨ pq(�) = 1); (8)

f∈FO⇔ (Ov(!p; !q) = 1 ∧ fv(vi) = !p ∧ fv(vj) = !q)

⇒ (ai = � ∨ aj �=� ∨ pp(�) = 1 ∨ pq(�) = 1); (9)

f∈FE⇔ (Ev(!p; !q) = 1 ∧ fv(vi) = !p ∧ fv(vj) = !q)

⇒ (ai �=� ∨ aj �=� ∨ pp(�) = 1 ∨ pq(�) = 1); (10)

where �v(vi)= ai and �v(vj)= aj refer to the attribute valuesof nodes vi and vj in the AG. The two right-most terms in theconsequent of the above rules allow the match of a non-nullvertex of the AG to an extended null vertex of the FDG(insertion) when a second-order relation involving the FDGnull vertex is not satis4ed. Otherwise, the FDG null verticeswould never be matched to actual vertices in the AG and,consequently, they would be of no help in bringing someMexibility to the matching process.

Finally, the 4rst distance measure between an AG and anFDG is de4ned as the minimum cost achieved by a validmorphism f∈Fv;e ∩ FA ∩ FO ∩ FE , i.e.df = min

f∈{Fv;e∩FA∩FO∩FE}{Cf};

where (11)

Cf =∑

∀vi∈�v of G′Cfv (vi; fv(vi))

+∑

∀eij∈�e of G′Cfe (eij ; fe(eij)): (11)

A comparison of the distance above with a distance based onedit operations [14] is presented in Ref. [22]. In summary,substitution and deletion costs are variable depending onthe probability density functions of the random elements,whereas the insertion cost is constant.

4.2. Distance between AGs and FDGs relaxingsecond-order constraints

In real applications, AGs can be distorted or noisy, andtherefore, the constraints associated with the second-order

relations have to be relaxed to avoid a noisy AG being mis-classi4ed due to non-ful4lment of any of them. For instance,the deletion of a strict non-null vertex of the FDG (with azero probability of being null) will almost always imply thenon-ful4lment of some of the existence or occurrence con-straints induced by that strict non-null vertex.

To gainmore Mexibility and robustness, some non-negativecosts can be added to the global cost of the labelling whensecond-order constraints are not met. Given that fv(vi)=!pand fv(vj) = !q, Eqs. (12)–(14) show these additionalcosts, which can be only 1 or 0, associated with the threerelations of antagonism, occurrence and existence betweenpairs of vertices. These equations cover, respectively, thethree following qualitative cases: presence of two verticesin the AG, presence of only one of them, and absence ofboth vertices:

CAv (vi; vj; !p; !q) =

Av(!p; !q) if (ai �=� ∧ aj �=� ∧ pp(�) �=1 ∧ pq(�) �=1);

0 otherwise;

(12)

COv (vi; vj; !p; !q) =

Ov(!p; !q) if (ai �=� ∧ aj = �∧pp(�) �=1 ∧ pq(�) �=1);

0 otherwise;(13)

CEv (vi; vj; !p; !q) =

Ev(!p; !q) if (ai = � ∧ aj = �

∧pp(�) �=1 ∧ pq(�) �=1);

0 otherwise:(14)

Now, the global cost Cf of the labelling function f is re-de4ned by CR

f with two terms that depend on the 4rst-orderprobability information, which are the original ones, andthree more terms that depend on the second-order costs:

CRf = Kv ∗

∑∀vi∈�v of G′

Cfv (vi; fv(vi))

+Ke ∗∑

∀eij∈�e of G′Cfe (eij ; fe(eij))

+KA ∗∑

∀vi ;vj∈�v of G′CAv (vi; vj; fv(vi); fv(vj))

+KO ∗∑

∀vi ;vj∈�v of G′COv (vi; vj; fv(vi); fv(vj))

+KE ∗∑

∀vi ;vj∈�v of G′CEv (vi; vj; fv(vi); fv(vj)): (15)

The 4ve terms are weighted by non-negative constantsKv; Ke; KA; Ko and KE , to compensate for the diGerent num-

790 F. Serratosa et al. / Pattern Recognition 36 (2003) 781–798

Fig. 7. View’1, which is similar to View1, with the spurious face6 and the resulting AGG’1.

Fig. 8. Labellings f and g between G′1 and Fb with their associated probabilities and costs.

ber of elements in the additions as well as to balance theinMuence of second-order costs with respect to 4rst-ordercosts in the overall value. Finally, the second distancemeasure proposed between an AG and an FDG, dRf, is de-4ned as the minimum cost CR

f achieved by a bijective andstructurally valid function f:

dRf = minf∈Fv;e

{CRf}: (16)

Note that both distance measures df and dRf will be equiv-alent in practice if we set KA = KO = KE = BigNumber,since in this case, the optimal labelling is forced to satisfythe second-order constraints, i.e. f∗ ∈{FA ∩ FO ∩ FE}.

4.3. An example of matching a 3D-object using theproposed distance measure

We are interested in matching the AG G′1, correspond-

ing to View′1 (Fig. 7) against the FDG Fb (Fig. 6). This

view is similar to View1 (Fig. 4) but the segmentationprocess has generated a new region represented by thespurious face6. Its hue has been assigned to the averagehue of their adjacent faces. Since this region has neverappeared in any view used to learn the object B, it shouldbe matched to an FDG null vertex in the optimal isomor-phism.

When one attempts to recognise a 3D-object from a singleview, the occluded parts of the object should not have aninMuence on the distance value (i.e. we are looking for asubgraph isomorphism rather than a graph isomorphism). Inour approach this can be accomplished by not taking intoaccount the costs of deleting vertices of the FDG (matchingAG vertices to null FDG vertices).

Fig. 8 shows two sub-graph isomorphisms between G′1

and Fb. Labellingf is represented by arrows→ and andlabelling g is represented by → and . The diGerencebetween both functions is in the matching of v6. In f, it

F. Serratosa et al. / Pattern Recognition 36 (2003) 781–798 791

1

2

3

45

1

2

3

4

67

8

VIEW1 Object A

ARG1 FDGbf

Antagonism

1

2

3

45

1

2

3

47

6 8Search tree

f(v1) w1

f(v2)

w6 w8

w2 w6 w8w1

Fig. 9. Tha mapping (v1; !6) and (v2; !2) is rejected due to the antagonism relation, and the search tree is pruned.

is matched to !6, although p6(brown) = 0, and in g, it ismatched to a null FDG vertex.

The table shows the probabilities and costs of the match-ing elements for the sub-graph isomorphisms f and g. Thethreshold on the probabilities used to calculate the costs isKpr = 0:1 (Eqs. (5) and (6)). The arcs e1;5 and �6;2 are ex-tended null arcs since the edge between face1 and face5 inView′

1 and the edge between face6 and face2 in the object,do not exist. In the case of the labelling f, vertices !2 and!6 are matched and then the antagonism between both ver-tices has to be considered with a cost KA. If antagonismsare not considered, KA = 0, then the optimal labelling is fwith a cost of 3.903. But, if KA¿ 1 then the cost of f isbigger than the cost of g and so f is discarded as the bestlabelling and the spurious Face6 is rejected. In the case ofKA = 1, the computed labelling depends on the implemen-tation of the algorithm since the minimum cost is 4.903 inboth labellings.

5. Algorithms for computing the distance measures

An algorithm for error-tolerant graph matching that cal-culates the distance measure and the optimal morphism be-tween an AG and an FDG was presented in Ref. [23]. Wede4ned an approach based on a tree search by A∗ algorithmreminiscent of the algorithm presented in Ref. [5], wherethe search space is reduced by a branch and bound tech-nique. We showed that by incorporating some second-orderconstraints (antagonism relations), the search tree can besomewhat pruned, although the exponential combinatorialcomplexity remains. To overcome this drawback, we pre-sented in Ref. [19] an e6cient algorithm that computes asub-optimal distance between an AG and an FDG, togetherwith its corresponding morphism, in polynomial time.

Nevertheless, we refer again to the 3D example to showthat antagonism relations not only are useful for recogni-tion purposes but also to prune the search tree computed inthe matching algorithm and thus to decrease the time usedto compute the distance. Fig. 9 shows a partial labellingbetween G1 (Fig. 4) and Fb (Fig. 6) and the search treecomputed by the matching algorithm. When the algorithmconsiders the mapping (v1; !6), the mapping (v2; !2) is

Fig. 10. Expanded vertex of an AG (a) and an FDG (b).

rejected and the matching algorithm backtracks, since thereis an antagonism between !2 and !6 and therefore, it isnot possible that these two FDG vertices be matched withtwo AG vertices. One of the FDG vertices does not have tobe matched (it is considered to be matched with a null AGvertex).

The e@cient algorithm to compute the distance betweenAGs and FDGs reduces enormously the search space ofour branch and bound algorithm [23] by initially discard-ing most of the mappings between vertices, using a func-tion based on the distance between the sub-units of thegraphs to be matched. The match of a pair of vertices isdiscarded if the distance between their related sub-units ishigher than a threshold. Thus, the graphs are broken downinto manageable sub-units that overlap, so that two adja-cent sub-units contain information about each other throughthe mutual graph elements they contain, similar to the dis-crete relaxation method presented in Ref. [24]. We referto these sub-units as expanded vertices (Fig. 10) and theyare structured by a central vertex and all the adjacent ver-tices connected to it by an outgoing arc (of the centralvertex).

Fig. 11 shows the basic scheme of the method proposed tocompute a sub-optimal approximation of the distance mea-sure dRf(G; F) between an AG G and an FDG F , that willbe denoted dR ˆf (G; F). The method consists of three main

792 F. Serratosa et al. / Pattern Recognition 36 (2003) 781–798

Fig. 11. Scheme of an e6cient algorithm to compute the distancebetween AGs and FDGs.

modules. The 4rst one computes all the distances be-tween expanded vertices using the distance measure,df(EVi; EWj). Note that an expanded vertex of an AG oran FDG is also an AG or an FDG and for this reason, thedistance between AGs and FDGs can be computed. Thethresholding module decides which matches of verticesare discarded depending on an externally imposed thresh-old /. The last module computes the sub-optimal distancedR ˆf (G; F) using the branch and bound algorithm presentedin Ref. [23] but only on the set of vertex mappings allowedby the thresholding module.

6. Application of FDGs for modelling and recognitionof objects

We present two diGerent applications of FDGs to assessthe validity of our approach: 3D-object recognition and faceidenti4cation. In both applications, there is a set of imagesof each class (of a 3D-object or a person) and an adjacencygraph is extracted from each image. The learning processis based on synthesising an FDG per each class (or a setof adjacency graphs) and the classi4cation process matchesthe unclassi4ed pattern (represented by an adjacency graph)with a class (represented by an FDG).

6.1. Application of FDGs to 3D-object recognition

FDGs are applied here to 3D-object representation andrecognition. Similar to the example that we have been us-ing throughout the paper, views are structured by adjacencygraphs. The attribute of the vertices is the average hue ofthe region (cyclic range from 0 to 49) and the attribute ofthe edges is the diGerence between the colours of the twoneighbouring regions.

We 4rst present an experimental validation of FDGs us-ing arti4cial 3D-objects in which the adjacency graphs havebeen extracted manually and afterwards we present a realapplication on an image database in which the graphs havebeen extracted automatically. The advantages of the exper-imental validation are that the results do not depend on thesegmentation process and that we can use a supervised syn-thesis [21], since we know which vertices of the AGs repre-sent the same planar face of the object. Thus, we can discernbetween the eGects of computing the distance measure usingdiGerent values of the costs on the second-order relations.

In the real application, we show the capacity of FDGs tokeep the structural and semantic knowledge of an object de-spite the noise introduced by the segmentation process andan automatic synthesis [22].

In both experiments, we want to show the inMuence ofthe antagonism and occurrence costs on the recognition ra-tio and the time spent to classify the objects. Thus, we 4rsttested combinations of the antagonism and occurrence costsconsidering three cases (Tables 1 and 3): with cost null(K3=0 or K4=0), with high costs (K3=1000 or K4=1000)and with moderate costs (K3 = 1 or K4 = 1). When thesecosts are null, the second-order relations are not consideredand the system is similar to FORGs. When a high cost isimposed, the non-ful4lment of only one antagonism or oc-currence restriction causes the labelling to be rejected. Con-versely, when a moderate cost is used, the non-ful4lmentof a second-order restriction causes only a slight increaseof the global cost. The cost associated with the existencerelations is not involved because no existence relations aresynthesised in these experiments (there is no pair of nodessuch that one or the other can be seen in all the views). Fromthe recognition ratio results, we arrived at the conclusionthat the best combination is the one that has both moder-ate costs. Then we compared FDGs to two other classicalmethods (Tables 2 and 5). The 4rst one was the 3-nearestneighbours (3-NN) classi4er: The edit-operations distance[25] was used as the distance between elements. In thiscase, the costs of insertion and deletions were set to 1 andthe cost of substitution was 0 if cyclic dist(hue1; hue2)= 0;cost = 1

2 if cyclic distance(hue1; hue2) = 1 and cost = 1if cyclic dist(hue1; hue2)¿ 1. And the second classicalmethod compared was random graphs.

6.1.1. Supervised synthesis on artiAcial objectsWe designed 4ve objects using a CAD program (Fig. 12).

After that, we took 4ve sets of views from these objectsand from these views we extracted a total of 101 adjacencygraphs. To obtain the AGs of the test set and of the refer-ence set, we modi4ed the attribute values of the vertices andarcs of the adjacency graphs by adding zero-mean Gaus-sian noise with diGerent variances. Moreover, some verticesand arcs were inserted and deleted randomly in some cases.The FDGs were synthesised using the AGs that belongedto the same 3D-object and using the synthesis given a com-mon labelling from a set of AGs described in Ref. [21].Table 1 shows the ratio of correctness (%) for diGerentlevels of noise and applying several costs on the antago-nisms and occurrences. We see that the best results appearalways when we use moderate second-order costs. Further-more, when noise increases, the recognition ratio decreasesdrastically when we use high costs but there is only a slightdecrease when we use moderate costs. Moreover, in Ta-ble 2 we compare the FDGs (with second-order costs) withother two methods. The FDG classi4er always obtains bet-ter results than the 3-NN and the random graph classi4ers.

F. Serratosa et al. / Pattern Recognition 36 (2003) 781–798 793

Table 1FDGs ratio of correctness %

No. of vertices ins. and del. 0 0 0 0 0 1 2 1Standard deviation 0 2 4 8 12 0 0 8Cost on antag. Cost on occu.

Moderate Moderate 100 98 97 95 92 89 85 83High Null 100 92 89 87 84 61 54 57Null High 100 91 89 88 85 62 59 59High High 100 95 90 86 80 60 53 56Moderate Null 100 92 91 91 87 80 75 75Null Moderate 100 95 92 91 86 81 77 76Null Null 100 90 89 88 86 70 67 68

Table 2FDGs (moderate second-order costs), FORGs and 3-NN ratio of correctness (%)

No. of vertices ins. or del. 0 0 0 0 0 1 2 1Standard deviation 0 2 4 8 12 0 0 8FDGs (moderate costs) 100 98 97 95 92 89 85 83Random graphs (FORGs) 100 90 89 88 86 70 67 683-NN (edit op. distance) 100 98 82 62 52 90 58 58

Fig. 12. Two views of 5 arti4cial objects designed by a CAD program.

The diGerence of the ratio between FDGs and the other twomethods increases when the noise also increases. FORGsobtain better results than the 3-NN only when the noiseis high.

6.1.2. Unsupervised synthesis on real life objectsImages were extracted from the database COIL-100

from Columbia University (www.cs.columbia.edu=CAVE=research=softlib=coil-100.html). It is composed of 100 iso-lated objects and for each object there are 72 views (oneview each 5 degrees). Adjacency graphs are obtained bythe segmentation process presented in Ref. [26]. Fig. 13shows 20 objects at angle 100 and their segmented imageswith the adjacency graphs. The test set was composed of36 views per object (taken at the angles 0,10, 20 and soon), whereas the reference of set was composed of the 36remaining views (taken at the angels 5, 15, 25 and so on).FDGs were synthesised automatically using the AGs in thereference set that represent the same object. The method ofincremental synthesis, in which the FDGs are updated whilenew AGs are sequentially presented, was applied [22]. We

made 6 diGerent experiments in which the number of FDGsthat represents each 3D-object varied. If the 3D-objectwas represented by only one FDG, the 36 AGs from thereference set that represent the 3D-object were used to syn-thesise the FDG. If it was represented by 2 FDGs, the 184rst and consecutive AGs from the reference set were usedto synthesise one of the FDGs and the other 18 AGs wereused to synthesise the other FDG. A similar method wasused for the other experiments with 3, 4, 6 and 9 FDGs per3D-object.

Table 3 shows the ratio of correctness of the FDG clas-si4er varying the number of FDGs per each object. Similarto the previous experimental results (Table 2), the correct-ness is higher when second-order relations are used with amoderate cost. The best result appears when each object isrepresented by 4 FDGs, that is, each FDG represents 90◦

of the 3D-object. When objects are represented by 9 FDGs,each FDG represents 40◦ of the 3D-object and 4 AGs perFDG, there is poor probabilistic knowledge and therefore thecosts on the vertices and arcs are coarse. Moreover, whenobjects are represented by only 1 or 2 FDGs, there are too

794 F. Serratosa et al. / Pattern Recognition 36 (2003) 781–798

Fig. 13. The 20 selected objects at angle 100 and the segmented images with the AGs.

Table 3Ratio of correctness (%) using several costs on antagonisms and occurrences

Number of FDGs per object 9 6 4 3 2 1Number of AGs per FDG 4 6 9 12 18 36Cost on antag. Cost on occu.

Moderate Moderate 65 69 79 70 60 54High Null 60 51 47 41 32 12Null High 59 49 45 37 25 10High High 52 41 43 33 18 8Moderate Null 62 60 65 61 52 45Null Moderate 61 59 63 57 45 42(FORGs) 40 44 45 53 27 143-NN (edit op. distance) 63

much spurious regions (produced in the segmentation pro-cess) to keep the structural and semantic knowledge of theobject.

Moreover, Table 4 shows the average time in millisec-onds spent to compute the classi4cation of a new AG. Thealgorithms were implemented in Visual C++ and run on aPentium 800 MHz. The higher the cost on the antagonisms(KA), the search tree is more pruned and so, the faster is thecomputation of the distance. Moreover, when the numberof FDGs per object decreases, the number of comparisonsalso decreases but the time spent to compute the distanceincreases since the FDGs are bigger.

Table 4Average time in milliseconds for each AG

No. of FDGsper object 9 6 4 3 2 1

No. of AGs per FDG 4 6 9 12 18 36FDG (high, high) 125 89 32 44 64 59FDG (moderate,

moderate) 150 101 47 56 72 82FORG 187 123 253 392 814 12033-NN 93

F. Serratosa et al. / Pattern Recognition 36 (2003) 781–798 795

Fig. 14. Identi4cation and location of the pre-established points and four selected faces.

Table 5Correctness (%) for diGerent representations of the 3D-object applying FORGs, FDGs and 3-NN. We used 3 levels of occlusion of the testimages: 0%, 20% and 30%

Occlusion 0% 20% 40%

Number of FDGs per object 9 6 4 3 2 1 9 6 4 3 2 1 9 6 4 3 2 1Number of AGs per FDG 4 6 9 12 18 36 4 6 9 12 18 36 4 6 9 12 18 36FDGs (moderate costs) 65 69 79 70 60 54 62 65 77 60 61 45 51 59 62 58 49 34Random graphs (FORGs) 40 54 45 33 27 14 33 49 40 37 19 10 29 24 25 23 20 103-NN (edit op. distance) 63 54 33

And 4nally, Table 5 shows that FDGs with moderatesecond-order costs obtain the best results independent ofthe occlusion on the test image. The best results usingFORGs are obtained when 6 AGs are used to synthesisethe FDGs, in contrast to the FDGs where the best resultsare obtained when 9 AGs are used. This is because FORGsdo not have the second-order information. Note that FDGsobtained much better results than FORGs and the 3-NN.

6.2. Application of FDGs to face identiAcation

In our approach, faces were de4ned by AGs, in whichvertices and arcs represented pre-established points and thedistances between them, respectively. Fig. 14 shows theidenti4cation of the pre-established points in one of the im-ages and the location of 36 points, and also, four faces ofthe database.

In the learning process, FDGs were obtained by thesupervised synthesis [21] and the AGs obtained by thefaces of the same person. To carry out the identi4cationexperiments, we used images of 100 people and 10 im-ages per person. Half of the images were composed of thereference set and the other half the test set. These imageswere taken from the database at the address http://cswww.essex.ac.uk/projects/vision/allfaces/faces94. See Ref. [20]for more details. We compared the FDG approach with theRG and the 5-NN approach, where the distance between el-ements was de4ned as the edit-operations distance betweenAGs [25]. Table 6 shows the correctness ratio varying the ra-tio of occlusion of the faces in the test set. In all the cases, the

Table 6Correctness ratio (%) using FDGs, random graphs and 5-NN

Occlusion Full face 20% 40% 60% 80%

FDGs (moderate second- 95 92 88 82 64order costs)Random graphs (FORGs) 89 85 70 63 475-NN (edit op. distance) 91 90 83 78 61

FDG classi4er obtained better results than the other classicalapproaches.

7. Conclusions

FDGs are a type of compact representation of a set of AGsthat borrow from Random Graphs the capability of prob-abilistic modelling of structural and attribute information,while improving the capacity of FORGs to record structuralrelationships that consistently appear through the data. Thisis achieved by incorporating some qualitative knowledge ofthe second-order probabilities of the elements that are ex-pressed as relations (boolean functions) between pairs ofvertices in the FDG.

If only the relations between vertices are considered, thespace complexity of the FDG representation does not aug-ment with respect to a 4rst-order random graph, n2, since

796 F. Serratosa et al. / Pattern Recognition 36 (2003) 781–798

all the relations de4ned (antagonism, occurrence, existence)can be obtained easily from the 4rst-order marginal proba-bilities and just one second-order probability for each pair ofvertices (namely, the probability of both vertices being nullat the same time). However, if arc relations were consid-ered as well, the space complexity obviously would rise toa2, where a is the number of arcs, and for dense graphs, thiswould imply a storage (and a time complexity of arc relationcreation, maintenance and veri4cation) of order n4 insteadof n2, which would be a severe drawback in practice. This isthe reason why second-order constraints between arcs havenot been discussed in this work.

A distance measure between an AG and an FDG has beenproposed to be used in the problem of matching an AG to anFDG for recognition or classi4cation purposes. Although,in the origin, a distance measure was derived from the prin-ciple of maximum likelihood in a Bayesian framework [1],a more robust measure has been described by consideringthe eGects of extraneous and missing elements only locallyand relaxing the second-order hard constraints into moder-ate costs, at the expense of losing the theoretical link withthe maximum likelihood source [27]. These second-orderrelations are useful not only for increasing the correctnessin the classi4cation process but also to reduce the computa-tional time of the labelling process. Nevertheless, the com-binatorial complexity of computing the presented distancemeasure remains. For this reason, an e6cient algorithm thatyields a suboptimal distance value is presented [23] to re-duce dramatically the computational time at the expense oflosing the optimal labelling.

In this work, two diGerent methods to synthesise an FDGfrom a set of AGs were used in the experiments. These meth-ods, together with some others to synthesise an FDG froma set of unlabelled AGs and to cluster, in a non-supervisedmanner, a set of AGs into a set of FDGs representing sub-classes, have been reported elsewhere [21,22].

Some experimental tests were carried out in a 3D-objectrecognition problem for two diGerent scenarios, an arti4cialone with a varying controlled noise level in feature extrac-tion and a real-world one, in which the 4nal results are heav-ily inMuenced by the image segmentation process. In bothcases, an FDG model has been synthesised for each objectfrom a set of views (AGs), including antagonism and oc-currence relations between vertices (representing faces ofthe object). Moreover, some tests on face identi4cation havealso been carried out, where an FDG is built from a set ofviews of a person’s face.

In all the experiments, FDGs obtained better results thanthe 3-NN classi4er, which means that it is useful to representthe set of AGs (or object) by only one structure since thisobject is seen as a whole. However, in the real 3D-objectrecognition application, it has been shown that it is moreeGective and robust to partition a single 3D-object into afew FDGs for recognition purposes. This is due to the rep-resentation of the joint probability of vertices and arcs byonly the 4rst-order probabilities and second-order relations.

Furthermore, FDGs obtained also better results than FORGsdue to the use of the second-order relations. Since these re-lations are a simpli4cation of the second-order probabilities,results show that it is better to use moderate second-ordercosts than high costs.

Finally, some arti4cial experiments were carried outin Ref. [1] to show the sensibility of the method to failuremismatch. In these experiments AGs were generated arti-4cially and FDGs were built using the methods presentedin Ref. [22].

References

[1] F. Serratosa, Function-described graphs for structural patternrecognition, Ph.D. Thesis Dissertation, Universitat PolitWecnicade Catalunya, Barcelona, Spain, 2000.

[2] A. Sanfeliu, R. Alqu*ezar, J. Andrade, J. Climent, F. Serratosa,J. Verg*es, Graph based representations and techniques forimage processing and image analysis, Pattern Recognition 35(2002) 639–650.

[3] A.K.C. Wong, J. Constant, M. You, Random graphs,in: Syntactic and Structural Pattern Recognition: Theoryand Applications, World Scienti4c, Singapore, 1990, pp.197–234.

[4] A.K.C. Wong, M. You, Entropy and distance of randomgraphs with application to structural pattern recognition, IEEETrans. Pattern Anal. Mach. Intell. 7 (1985) 599–609.

[5] A.K.C. Wong, M. You, S.C. Chan, An algorithm for graphoptimal monomorphism, IEEE Trans. Systems Man Cybern.20 (1990) 628–636.

[6] P.H. Winston, Learning structural descriptions from examples,Technical Report MAC-TR-76, Department of ElectricalEngineering and Computer Science-MIT, 1976.

[7] F. Stein, G. Medioni, Structural indexing: e6cient 3D objectrecognition IEEE Trans. Pattern Anal. Mach. Intell. 14 (1992)125–145.

[8] A. Pearce, T. Caelly, W.F. Bischof, Rulegraphs for graphmatching in pattern recognition, Pattern Recognition 27 (9)(1994) 1231–1247.

[9] W. Kim, A. Kak, 3D object recognition using bipartitematching embedded in discrete relaxation, IEEE Trans. PatternAnal. Mach. Intell 13 (1991) 224–251.

[10] K. Sossa, R. Horaud, Model indexing: the graph-hashingapproach, Proceedings of the IEEE Conference on ComputerVision and Pattern Recognition, 1992, pp. 811–815.

[11] L.G. Shapiro, R.M. Haralick, Organization of relationalmodels for scene analysis, IEEE Trans. Pattern Anal. Mach.Intell. 4 (1982) 595–602.

[12] A. Sanfeliu, K. Fu, A distance measure between attributedrelational graphs for pattern recognition, IEEE Trans. SystemsMan Cybern. 13 (1983) 353–362.

[13] A. Sanfeliu, F. Serratosa, R. Alquezar, Clustering of attributedgraphs and unsupervised synthesis of function-describedgraphs, in: Proceedings of ICPR’2000, 15th InternationalConference on Pattern Recognition, Barcelona, Spain, Vol. 2,2000, pp. 1026–1029.

[14] B.T. Messmer, H. Bunke, A decision tree approach to graphand subgraph isomorphism detection, Pattern Recognition 32(1999) 1979–1998.

F. Serratosa et al. / Pattern Recognition 36 (2003) 781–798 797

[15] P. Foggia, R. Genna, M. Vento, Learning in structureddomains by attributed relational graphs, Proceedings of theIAPR Workshops SSPR’00 and SPR’00, Alacant, Spain,2000.

[16] K.P. Chan, Learning templates from fuzzy examples instructural pattern recognition, IEEE Trans. Systems ManCybern. 26 (1996) 118–123.

[17] A. Sperduti, A. Starita, Supervised neural networks for theclassi4cation of structures, IEEE Trans. Neural Networks 8(3) (1997) 714–735.

[18] K. Sengupta, K. Boyer, Organizing large structuralmodelbases, IEEE Trans. Pattern Anal. Mach. Intell. 17 (1995)321–332.

[19] F. Serratosa, A. Sanfeliu, Function-described graphs appliedto 3D object recognition, Ninth International Conferenceon Image Analysis and Processing, Italy, Vol. I, 1997, pp.701–708.

[20] J. Verg*es, A. Sanfeliu, F. Serratosa, R. Alqu*ezar, Facerecognition: Graph matching versus neural techniques, EighthNational Symposium on Pattern Recognition and ImageAnalysis, Vol. I, 1999, pp. 259–266.

[21] R. Alqu*ezar, A. Sanfeliu, F. Serratosa, Synthesis offunction-described graphs, Proceedings of SSPR’98, Sydney,Australia, Lecture Notes in Computer Science, Vol. 1451,Spinger, Berlin, 1998, pp. 112–121.

[22] A. Sanfeliu, F. Serratosa, R. Alquezar, Clustering of attributedgraphs and unsupervised synthesis of function-describedgraphs, Proceedings of ICPR’2000, 15th InternationalConference on Pattern Recognition, Barcelona, Spain, Vol. 2,2000, pp. 1026–1029.

[23] F. Serratosa, R. Alquezar, A. Sanfeliu, E6cient algorithmsfor matching attributed graphs and function-described graphs,in: Proceedings of ICPR’2000, 15th International Conferenceon Pattern Recognition, Barcelona, Spain, Vol. 2, 2000, pp.871–876.

[24] R.C. Wilson, E.R. Hancock, Structural matching by discreterelaxation, IEEE Trans. Pattern Anal. Mach. Intell. 19 (1997)634–648.

[25] A. Sanfeliu, K. Fu, A distance measure between attributedrelational graphs for pattern recognition, IEEE Trans. SystemsMan Cybern. 13 (1983) 353–362.

[26] P.F. Felzenswalb, D.P. Huttenlocher, Image segmentationusing local variation, Proceedings of the IEEE Conference onComputer Vision Pattern Recognition, 1998, pp. 98–104.

[27] R. Alqu*ezar, F. Serratosa, A. Sanfeliu, Distance betweenattributed graphs and function-described graphs relaxing2nd-order restrictions, Proceedings of the IAPR InternationalWorkshops SSPR’2000 and SPR’2000, Barcelona, Spain,Lecture Notes in Computer Science, Vol. 1876, Springer,Berlin, 2000, pp. 277–286.

About the Author—Dr. SANFELIU was born in Barcelona in 1954. He got the Industrial Engineering degree (speciality in ElectricalEngineering) in 1978, and the Ph.D. degree in 1982 from the Universitat Polit*ecnica de Catalunya (UPC). He is the General Manager in theCognivision Research S.L. and Professor in the UPC. He has received the Prize to the Technology given by the Generalitat de Cataloniaand he is Fellow of the International Association for Pattern Recognition. Dr. Sanfeliu has been the Scienti4c Co-ordinator and researcherin more than 18 scienti4c projects, 4ve of them European Projects (RAIM (BRITE-EURAM BE96-3004), HITEX (BRITE-EURAMBE96-3505), ACIL (BRITE-EURAM E-7137), MUVI (BRITE-EURAM E-4330)) and TEXTERM (GRD1-2000-26817).

He is the co-author of 4 international patents in quality control techniques based on computer vision for the 4bre and textile industry. Hehas been managing several industrial projects in the automation of classi4cation and packaging lines for textile bobbin packages (BobbinVision) and also several projects in the automation of electronic management of documents.

Dr. Sanfeliu has published (edited) 10 books, 22 book chapters, 19 journal papers and around 100 conference papers. He is an associatededitor of three scienti4c journals and reviewer of several top scienti4c journals in pattern recognition and computer vision (“IEEE Transactionson Pattern Analysis and Machine Intelligence”, “Computer Vision and Image Understanding”, “Image and Vision Computing”, “PatternRecognition”, “Pattern Recognition Letters”, “International Journal on Document Analysis and Recognition (IJDAR)”, “Computaci*on ySistemas”).

From 1979 to 1981, he was visiting researcher in the Department of Prof. K.S. Fu in Purdue University, USA. He joined the faculty ofthe Polytechnical University of Cataluyna in 1981 as associate professor. He is now professor in the department of “Ingenier*Ya de Sistemas,Autom*atica e Inform*atica Industrial” in the UPC. From 1975 to 1995 he has been a researcher in the “Instituto de Cibern*etica”, researchinstitute which belongs to the National Council of Scienti4c Research (CSIC) and to the UPC. He is now researcher of the “Institut deRob*otica i Inform*atica Industrial (CSIC-UPC)”.

Dr. Sanfeliu has been the President of the Spanish Association for Pattern Recognition and Image Analysis from 1984 to 2000,the Chairman of the Technical Committee on Syntactic and Structural Pattern Recognition of the International Association for PatternRecognition and the General Co-Chairman of the International Conference on Pattern Recognition which was held in Barcelona in the year2000.

About the Author—RENE ALQUEZAR received the licentiate and Ph.D. degrees in Computer Science from the Polytechnical Universityof Catalonia (UPC), Barcelona, Spain, in 1986 and 1997, respectively. From 1987 to 1991 he was with the Spanish company NTE astechnical manager of R& D projects on image processing applications for the European Space Agency (ESA). From 1991 to 1994 he waswith the “Institut de Cibernetica”, Barcelona, holding a pre-doctoral research grant from the Government of Catalonia and completing thedoctoral courses of the UPC on arti4cial intelligence. He joined the Department of “Llenguatges i Sistemes Informatics”, UPC, in 1994 asan associate professor, and since March 2001 he has been professor in the same department. He has been co-chair of the local organisingcommittee of the 15th ICPR held in Barcelona in September 2000.

His current research interests include neural networks, syntactic and structural pattern recognition and computer vision.

798 F. Serratosa et al. / Pattern Recognition 36 (2003) 781–798

About the Author—FRANCESC SERRATOSA was born in Barcelona, 24 may 1967. He received his Computer Science Engineeringdegree from the Universitat PolitWecnica de Catalunya (Barcelona) in 1993. Since then, he has been active in research in the areas of computervision, robotics and structural pattern recognition. He received his Ph.D. from the same university in 2000. He is currently associate professorof computer science at the Universitat Rovira i Virgili (Tarragona), he has published more than 30 papers and he is an active reviewer insome congresses and journals.