An Object Oriented Fuzzy Data Model for SimilarityDetection in Image DatabasesDr. A. K. Majumdar�, Indrajit Bhattacharya�, Amit K. Saha�Abstract: In this paper we introduce a fuzzy set theoretic approach for dealing withuncertainty in images in the context of spatial and topological relations existing among theobjects in the image. We propose an object oriented graph theoretic model for representing animage with a view to assessing similarity between images using the concept of (fuzzy) graphmatching. Su�cient exibility has been provided in the similarity algorithm so that di�erentfeatures of an image may be independently focussed upon.Index Terms : - image databases, fuzziness in images, spatial and topological rela-tions, similarity retrieval.1 IntroductionWith the advancement of multimedia applications, image databases have emerged as amajor �eld of research [7, 9, 10, 11]. Unlike a record in a conventional database, the size ofan image is usually very large. This has necessitated the development of e�cient storagerepresentation techniques for image databases. Moreover, the queries in an image databaseoften require retrieval based on the similarity with respect to given images or based oncertain attributes of the objects present in the images. Thus it is necessary to supportqueries based on image semantics rather than mere pixel-to-pixel matching. The imagedatabase systems should therefore present adequate abstraction mechanisms for capturing�Dept. of Computer Sc. and Engg., Indian Institute of Technology, Kharagpur1

higher level semantics of the images with a view to supporting content addressability asfar as possible.Over the last decade there has been considerable interest in representing spatial andtopological relations among objects present in a scene. Chang and Jungert[6] have pro-posed 2-D Strings and Interval Projection Strings to represent spatial relationships amongobjects in a scene. Similarly, V.N. Gudivada[1] has suggested a geometric approach called�R Strings for capturing spatial relationships. The representation schemes using 2-D or�R Strings, however, are not e�cient for capturing topological relations like meet, over-lap, cover, contain etc. Considerable work has been done in this particular direction byEgenhofer et.al.[2] with GIS design and developments.In real images, it is often not possible to precisely identify the boundaries of objectsin a scene. Thus, object identi�cation by image processing techniques will be di�cultand identi�cation of binary relations, like left-of, contains, or overlaps, inexact. In imageprocessing literature, statistical or fuzzy set theoretic approaches have been advocatedto deal with noise or uncertainty in images. Several algorithms have been developed to�lter out noise and identify boundaries of objects in such unfavorable environments[14].In this paper, we would be using a fuzzy set theoretic approach to represent objects andtheir spatial and topological relationships. This will enable us to deal with semantics ofreal-life images with objects having non-crisp boundaries. Moreover, object relationshipssuch as left-of or covered-by, with fuzzy quali�ers like more-or-less, almost, etc., can alsobe handled.2 Spatial and Topological RelationsSpatial relations like left, right, above and below de�ne the spatial orientations of thedomain objects with respect to each other, and help in ascribing meaning to a scene.On the other hand, topological relations deal with the nature of overlap between objectsand are invariant under rotations of the scene. Egenhofer[2] has de�ned interior(Ao)2

and boundary(�A) of a region (n-cell) A. The topological relations between two regions(objects) A and B are de�ned in terms of the intersections (or non-intersections) of theinterior and boundary of the regions under consideration. This model is called the 4-intersection model[2] and can be concisely represented by a 2�2 matrix:R(A;B) = 264 Ao \Bo Ao \ �B�A \ Bo �A \ �B 375Egenhofer used this approach to represent eight basic topological relations among theregions without holes, namely disjoint, cover, meet, inside, contains, coveredby, overlapand equal. Some pairs such as cover and coveredby are duals of each other (same is truefor inside and contains). Further, disjoint, meet, overlap and equal are symmetric. Thefour empty/non-empty intersections describe a set of relations that provides a completecoverage. These relations are mutually exclusive so that the union(OR) of all speci�cationsis true, and the intersection (AND) of any two speci�ed relations is identically false.3 A Fuzzy Object Data Model for ImagesWe have already mentioned, that with imprecise or noisy data, it is often not possible tosay de�nitely whether a pixel de�nitely belongs to an object. Rather, an object may belooked upon as a fuzzy subset of the set of pixels. Accordingly, let O be the set of objectspresent in a two dimensional scene. An object A 2 O is treated as a fuzzy subset of pixelsof the scene with membership function �A. Thus for a pixel located at a point z in thescene, the possibility of this pixel belonging to the object A is �A(z). In the following,we use the notations zX and zY to denote the X and Y coordinates of a pixel z. Let z+denote the set of 8-neighbors of a pixel at point z. Then the boundary (�A) of an objectA may also be de�ned as a fuzzy subset with membership function��A(z) = minf�A(z); maxu2z+f1� �A(u)gg........ (a)Following this approach, the de�nition of membership function for the interior Ao ofan object A is given by�Ao(z) = minf�A(z); 1� ��A(z)g........ (b)3

In the proposed framework, the spatial and topological relations among the objectswill also be imprecise and will be treated as fuzzy subsets.Fuzzy Spatial Relations: Since objects are treated as fuzzy subsets, the spatial re-lations left-of, right-of, above or below between a given pixel and an object would beimprecise and will be treated as fuzzy subsets. The corresponding relations between anytwo given pixels are, however, precise, i.e. a pixel a is either to the left or right or havethe same X-coordinate as another pixel b. Thus we de�ne�leftof (a; b) = 8><>: 0 if ax � bx1 else (1) �rightof (a; b) = 8><>: 0 if ax � bx1 else (2)The spatial relations above and below between two pixels can be similarly de�ned byconsidering the Y coordinates of the pixels.We now de�ne the possibility of a pixel z being to the left-of an object A, by the fuzzypredicate `no pixel of A is to the left-of the pixel z'.�left�of (z; A) = 1�maxvfmin(�left�of (v; z); �A(v))g (3)Similarly, �right�of (z; A) = 1�maxvfmin(�right�of (v; z); �A(v))g (4)Based on these de�nitions, it may be seen that the possibility of an object A beingleft-of a pixel z, has the same value as that of the pixel z being right-of the object A. Thisfollows from the observation that�left�of (A; z) = 1�maxvfmin(�left�of (z; v); �A(v))g (5)and for any two pixels v and z, �right�of(v; z) = �left�of (z; v). Hence for any pixel z andan object A, we have�right�of(z; A) = �left�of (A; z)....... (c)However, unlike the classical case, the fuzzy left-of and right-of relations between a pixeland an object are not complementary, i.e.�left�of (z; A) 6= 1� �right�of(z; A) .......... (d)4

We can now treat the spatial relations left-of and right-of between two objects (A and B)as fuzzy subsets of O � O. Accordingly, the fuzzy relation `A left-of B' is de�ned by thefuzzy predicate `no pixel of B is to the left-of A'. Thus,�left�of (A;B) = 1�maxzfmin(�B(z); �left�of (z; A))g (6)Similarly, �right�of (A;B) = 1�maxzfmin(�B(z); �right�of (z; A)) (7)The fuzzy spatial relations above or below between two objects can be similarly de�nedby extending the same relations between two pixels.Lemma 3.1 : The left-of(right-of) relation between two objects is transitive.�left�of (A;C) � maxBfmin(�left�of (A;B); �left�of (B;C))g (8)The proof could not be included due to space crunch. The transitivity claim can also bemade for the fuzzy relation above (below) between two objects. Note that to enforce sym-metry for the left-of and right-of relations between objects, we may modify the de�nitionas follows. �left�of (A;B) = min[ f1�maxz(min(�B(z); �left�of (z; A)))g;f1�maxz(min(�A(z); �right�of (z; B)))g] (9)Fuzzy Topological Relations: We are now in a position to de�ne fuzzy extensions ofthe topological relations discussed in Section 2.� disjoint : Two objects A and B are said to be disjoint if all pixels in A are not inB, and vice versa. �disjoint(A;B) = 1�maxzfj�A(z) + �B(z)� 1jg (10)Note that the disjoint relation is symmetric. Moreover, �disjoint(A;Ac) = 1, where Ac isthe fuzzy complement object of A, i.e., �Ac(z) = 1� �A(z). However, �disjoint(A;A) maynot be always zero. 5

� meet :�meet(A;B) = minf(1�maxz(j�Ao(z) + �Bo(z)� 1j)); maxv(min(��A(v); ��B(v)))g (11)Here the �rst term in the right hand side expression corresponds to the condition thatAo and Bo are disjoint and the second term is associated with the requirement that thereexists some common boundary point. The fuzzy relation `meet' thus de�ned is symmetric.� equal : Two objects A and B are said to be equal if the pixels have equal possibilityof belonging to either of them.�equal(A;B) = 1�maxz j�A(z)� �B(z)j (12)The fuzzy equal relation thus de�ned is a resemblance relation i.e. it is re exive andsymmetric, but not always transitive.� overlap : An object A overlaps another object B if there exists a pixel in Ao andBo, AND there exists a pixel in Ao and �B, AND there exists a pixel in �A and Bo, ANDthere exists a pixel in �A and �B. Hence�overlap(A;B) = minfmaxz(min(�Bo(z); �Ao(z))); maxz(min(�Ao(z); ��B(z)));maxz(min(�Bo(z); ��A(z))); maxz(min(��A(z); �deltaB(z)))g(13)The fuzzy relation overlap is symmetric.� contains : An object A contains another object B if the presence of the pixel in Bimplies that it is in Ao. Accordingly,�contains(A;B) = min(1; minz(1 + �Ao(z)� �B(z))) (14)Note, �contains(Ao; Ao) = 1 and �contains(A;B) = 1, whenever 8z; �Ao(z) � �B(z).Also, if there exists a pixel in the boundary of A, with ��A(z) = 1; �contains(A;Ao) = 0.� inside : This follows from contains. �inside(B;A) = �contains(A;B)� covers : An object A is said to cover another object B if, for any pixel z, presencein B implies presence in A, and there exists a pixel in �A which is not in B.�covers(A;B) = minfmin(1; minz(1 + �A(z)� �B(z))); maxv(min(��A(v); 1� �B(v)))g(15)6

� covered-by : This is the dual of covers.It may be noted that with precisely de�ned objects, i.e. with binary membershipfunctions, the fuzzy topological relations reduce to their classical counterparts discussedin Section 2.4 Graph Model for an Image and SimilaritySo far we have considered spatial and topological relations between a pair of objects inisolation. In an image many objects may be present. For a proper description of theimage, it is therefore necessary to consider the spatial and topological relations amongall these objects. With this in view we treat an image as a labeled graph with a vertexcorresponding to each object. An edge between two nodes, say A and B, is labeled by thespatial and topological relations that hold between the objects corresponding to A and B.Graphical representation of objects in an image has been proposed by Faloutsos et.al.[8].The proposed graphical model of an image allows fuzzy spatial and topological relationsbetween objects to be represented.An image (a two dimensional scene) consists of a collection of objects, which correspondto the nodes of the image graph. The edges capture inter-object relationships through thefollowing information:1: Spatial Information: The spatial relations that hold between the object pair associatedwith the edge. It stores the fuzzy membership values for these relations.2: Topological Information: The topological relations and their associated membershipvalues that hold between the objects.3: Euclidean distance between the centroids of the two objects (edge length).We would like to emphasize that the proposed model allows more than one spatialand topological relation between objects to be represented. Thus one may have a scenein which a car is left-of and below a house. Again from the picture it may not be clearwhether a tumor is inside the pancreas or is covered by it. In this case both the topological7

relations inside and covered-by between the objects tumor and pancreas will have nonzeromembership values.Similarity Measure: With the logical representation of images thus de�ned, we arenow in a position to examine the problem of associative retrieval of images from imagedatabases. In order to support such retrievals, it is necessary to de�ne a suitable similaritymeasure between the query image and the target image stored in the database. In imagedatabase literature, several such similarity measures have been proposed for associativeretrieval of images[3, 4, 5, 1]. However, not adequate importance has been given to thespatial and topological relations while de�ning such similarity measures. Based on thegraph model of an image proposed above, we de�ne the fuzzy similarity measure (SIM)between two images, I1 (query image) and I2 (an image stored in the database) as follows.We use S and T to be the sets of spatial and topological relations respectively betweentwo objects, G(OI; EI) is the graph of an image I, where OI is the set of nodes(objects)and EI is the set of edges. An edge e 2 EI between two objects A;B 2 OI is associatedwith the membership values of the spatial Se � S, and topological relations Te � T thathold between A and B, and the Euclidean distance De between their centroids.Given two images I1 and I2 with associated sets of graphs G(O1; E1) and G(O2; E2),to �nd the similarity between them, we �rst need to a �nd suitable association amongthe objects in the two images. Such an association can be expressed in terms a graphhomomorphism: : G(O1; E1) �! G(O2; E2). maps a node (object) A 2 O1 to a node (A) 2 O2 and thereby an edge e1 2 E1 to an edge (e1) 2 E2. Once a mapping isselected, we can estimate the similarity between I1 and (I1). Such a similarity estimateis assumed to consist of four components:� Object Similarity: determines the extent of similarity between the objects A and (A), 8A 2 O1. Accordingly, we write,Obj Sim (O1; (O1)) = XA2O1(wA�obj(A; (A)))= XA2O1wA (16)where �obj(A;B) is a fuzzy similarity between two objects A and B. Such a similarity8

measure can be de�ned based on the similarity in color and other attributes of the objectsunder consideration, like distance computation between color histograms, as used in [12,13]. The weight value wA; A 2 O1 indicates the importance given to the object A whilecomputing object similarity.� Spatial Similarity: determines to what extent spatial relations between the objectsin I1 match with those between the corresponding objects in (I1). This is estimated asfollows:Spatial Sim (E1; (E1)) = Pe2E1 Ps2Se we;smin(�s(A;B); �s( (A); (B)))=Pe2E1Ps2Se we;s (17)where the edge e 2 E1 is assumed to be between the nodes(objects) A and B in O1 and�s(A;B) is the membership value of the spatial relation s 2 Se between the objects A andB. The weight we;s represents the importance given to the spatial relation s associatedwith the edge e 2 E1.� Topological Similarity: determines to what extent topological relations between theobjects in I1 match with those between the corresponding objects in (I1). Using a similarnotation as above,Topo Sim (E1; (E1)) = Pe2E1 Pt2Te we;tmin(�t(A;B); �t( (A); (B))=Pe2E1 Pt2Te we;t (18)� Distance Similarity: gives the similarity in distance between a pair of objects in I1and that between the corresponding objects in (I1). This is estimated as follows:Dist Sim (E1; (E1)) = Xe2E1we�dist(e; (e))= Xe2E1we (19)where for a pair of edges e1 and e2, �dist(e1; e2) = e�d and d is the absolute value of thedi�erence in the length of the two edges.These four similarity measures are combined to provide an estimate of similarity betweenthe image I1 and its homomorphic map (I1) according to the following expression.SIM (I1; (I1)) = � Obj Sim (O1; (O1))+ � Spatial Sim (E1; (E1))9

+ Topo Sim (E1; (E1))+ � Dist Sim (E1; (E1)) (20)where � + � + + � = 1.Finally, the similarity between any two pair of images I1 and I2 is de�ned with respectto the best mapping by SIM(I1; I2) = max (SIM (I1; (I1)))The problem of associative retrieval of images has been thus reduced to �nding a bestpossible mapping between the graph of the query image and that of the target images inthe database using the fuzzy membership functions for the relations. Though the problemof graph isomorphism is in NP, in the present case it is simpli�ed by the fact that thevertices are labeled by the names of the objects. Further, if we consider cases wheremultiple instances of the same object are not allowed in an image, it essentially becomesa �(n2) problem because then there exists just one possible mapping.Implementation: The associative retrieval is based on the image similarity measurediscussed in Section 4 and is computed using eqn (20). The module that estimates thesimilarity measure between two image graphs is integrated with the overall query pro-cessing package. Each image in the database is associated with a graph representationwhich is identi�ed by a unique graph id. While estimating the image similarity, the graphmodels of the associated images are fetched �rst. The graph ids of the images, whichhave similarity with the query image (above a given threshold value), are used to fetchthe target images, in order of their similarity. Since a graph representation requires muchless storage space than the original image, usually a single disk access will be su�cient tofetch the graph of an image.Indices are built to support faster retrieval of image graphs. The indices are basedon attribute values (e.g. color histogram) of the objects(nodes) present in the image, aswell as spatial and topological relations that hold among the objects. Since same spatial/topological relations may hold between multiple nodes, the index based on such relationswill only identify the graph ids of the images where such a relation holds among some pairof objects in the image. It may be noted that the triangle inequality does not necessarily10

hold for the similarity measure between image graphs since the topological relations arenot transitive.The system is being used for associative retrieval from a medical image database whichcontains X'Ray, ultra-sonogram and images of blood slides and skin patches. A medicaldatabase system developed using the proposed scheme is being used in a telemedicineproject for supporting online teleconsultation among doctors at remote health centersand referral hospitals. Detailed performance evaluation of the proposed scheme is underprogress.Conclusion : We have proposed a fuzzy object-oriented data-model for images thatextends Egenhofer's 4-intersection model to deal with objects having imprecise boundariesfound in many real-life images. The proposed graph model has been backed up by suitablefuzzy similarity measures involving object attributes, spatial and topological relationsamong the objects to support associative retrieval from image databases. We have shownhow, by appropriate selection of the weights, the user can focus on di�erent features ofthe image.References[1] Venkat N. Gudivada, \A Geometry-based Representation for E�cient and E�ective Re-trieval of Images by Spatial Similarity", IEEE TKDE Vol. 10 p504-512 No. 3 May/June1998[2] M. Egenhofer, J. Herring, \Categorizing Binary Topological Relations between Regions,Lines and Points in Geographic Databases", Technical Report, Department of SurveyingEngineering, University of Maine, 1990.[3] S. Santini, R. Jain, \Similarity Matching", IEEE Transactions on Pattern Analysis andMachine Intelligence, 1999 (in press)11

[4] S. Santini, R. Jain, \Similarity Queries in Image Databases", Proceedings of CVPR 96, IEEEInternational Conference on Computer Vision and Pattern Recognition, San Francisco, June1996[5] Y. Cheng, V. Wu, R. Collins, A. Hanson, E. Riseman, \Maximum-weight Bipartite MatchingTechnique and its Application in Image Feature Matching", SPIE Conference on VisualCommunication and Image Processing, 1996.[6] S. K. Chang, E. Jungert, \Symbolic Projection for Image Information Retrievaland Spatial Reasoning", Academic Press, 1996 (ISBN 0-12-168030-4).[7] M. Flickener, H. Swahaney, \Query by Image and Video Content : The QBIC System",IEEE computer, Vol. 28, No. 9, Sep. 1995[8] E. Petrakis, C. Faloutsos, \Similarity Matching in Large Image Databases", University ofMaryland, Computer Science Technical Report, December '94[9] V. Ogle, M. Stonebraker, \Chabot : Retrieval from a relational database of images", IEEEComputer, Vol. 28, p40-45, No. 9, September 1995[10] A. Pentland, R. Picard, S. Sclaro�, \Photobook : Tools for Content-based Manipulationof Image Databases", International Journal of Computer Vision (IJCV), Vol. 18, No 3,p233{254, Jun., 1996.[11] M. Christel, S. Stevens, H. Wactlar, \Informedia Digital Video Library." Proceedings of theACM Multimedia '94 Conference. New York: ACM, pp. 480-481, October, 1994.[12] Smith J.R., and S.-F. Chang, \Tools and Techniques for Color Image Retrieval," ColumbiaUniversity CTR Technical Report, June 1995.[13] Swain M., and D. Ballard, \Color Indexing," International Journal of Computer Vision,7:1, 1991, p. 11 { 32.[14] \Home Page of Fuzzy Image Processing", http://ipe.et.uni-magdeburg.de/ hamid/FuzzyImageProcessing.htm.12