ADDING COLOR TO GEODESIC INVARIANT FEATURES

Pier Paolo CampariPolitecnico di Milano, Milan, Italy

pc666272@stud.polimi.it

Matteo Matteucci, Davide MigliorePolitecnico di Milano, Milan, Italy

matteucci@elet.polimi.it, migliore@elet.polimi.it

Keywords: Feature Extraction, Feature Description, Color Pattern Recognition.

Abstract: Geodesic invariant feature have been originally proposed to build a new local feature descriptor invariant notonly to affine transformations, but also to general deformations. The aim of this paper is to investigate thepossible improvements given by the use of color information in this kind of descriptor. We introduced colorinformation both in geodesic feature construction and description. At feature construction level, we extendedthe fast marching algorithm to use color information; at description level, we tested several color spaces onreal data and we devised the opponent color space as an useful integration to intensity information. Theexperiments used to validate our theory are based on publicly available data and show the improvement, inprecision and recall, with respect to the original intensity based geodesic features. We also compared thiskind of features, on affine and non affine transformation, with SIFT, steerable filters, moments invariants, spinimages and GIH.

1 INTRODUCTION

In this paper we face the issue of feature descriptionat the base of automatic correspondence matching be-tween images from different views of the same sceneor images of the same objects in different poses. Inparticular we are interested in exploiting color infor-mation to enrich feature descriptors invariant with re-spect to generic transformation/deformations.

In (Ling and Jacobs, 2005), Ling and Jacobs in-troduced such kind of descriptor, together with thegeodesic framework, for deformation invariant fea-tures extraction and matching. To cope with genericimage deformations, this kind of descriptor is basedon a histogram built out of pixel intensities from re-gions at the same geodesic distance with respect toa given interest point. Geodesic invariant descrip-tors have been experimentally proved to be covariantwith respect to generic deformations, but have littledescriptive capabilities with respect to other classi-cal feature descriptors such as SIFT. This weaknessis mainly due to the smoothing effect induced by thehistogram and the sensitivity to image gradient ingeodesic distance calculation.

The aim of our work is to investigate how color in-formation can be used to improve geodesic imagedescriptors by reducing this sensitivity to the imagegradient and extending histogram description. In theoriginal paper Ling and Jacobs (Ling and Jacobs,2005) use the fast marching algorithm (Sethian, 1999)to expand geodesic borders based on intensity values;in this paper we propose to use the RGB componentsto consider edges in color components instead of in-tensity. Color information is also used to complementintensity histogram with an appropriate color descrip-tion based on opponent color space (van de Weijer andSchmid, 2006).In the next section we briefly review related worksin feature description and evaluation, Section 3 intro-duces the geodesic invariant framework and the nov-elties introduced by our work. Section 4 presents theexperimental setup and results to assess the improve-ment given by the use of color information, while abrief discussion about future work is presented in thefinal section.

227

2 RELATED WORKS

A considerable amount of work has been done on ro-bust local feature descriptors, studying invariance toorientation, scale, affine transformation, and, recently,also to generic deformation.

In the work of Mikolajczyck and Schmid (Miko-lajczyk and Schmid, 2005; Mikolajczyk et al., 2005)a performance evaluation of several of these localdescriptors is performed both with respect to view-points and lighting conditions. As final result, thiswork reports that the SIFT descriptor, proposed byLowe (Lowe, 2004), has the best performance withimages of flat scenes and affine transformations.These conclusions have been supported also in thepaper by Moreels and Perona (Moreels and Perona,2005), which generalizes these results to 3D scenesusing images of 3D objects viewed under differentscales, viewpoints, and lighting conditions.

All the local invariant descriptors, investigated sofar, are based on the hypothesis of perspective de-formation being properly approximated, locally, byaffine transformation; recently, Ling and Jacobs (Lingand Jacobs, 2005) demonstrated that it is possible alsoto construct descriptors invariant to generic deforma-tion of image subject (e.g., a moving flag). In theirproposal, they suggest to treat the intensity image asa surface embedded in 3D space, with the third coor-dinate being proportional to the intensity values, andthen build the descriptor by deformation invariancegeodesic distance in this 3D space.

Stimulated by their work, in this paper we wantdemonstrate that it is possible to improve the perfor-mance of the Geodesic Intensity Histogram (GIH) de-scriptor by introducing color information. Althoughcolor seems to be a fundamental clue for object recog-nition in everyday life only few color invariant de-scriptors have been proposed in the literature. Thework of Van De Weijer and Schmid (van de Weijerand Schmid, 2006) is an important example from thispoint of view. Their results lead to the encouragingconclusions that a pure color-based approach outper-forms a shape-based approach only for colorful ob-jects, while, for the general case, it is possible anyway to outperforms a pure shape-based approach us-ing a combination of shape and color.

In our work we verify the results achieved by VanDe Weijer and Schmid also for non affine transforma-tions with the idea that shape-based descriptors canfail when dealing with generic deformations and com-bining it with color can improve recognition rates..

3 COLORING GEODESICINVARIANT FEATURES

In the geodesic framework, an image can be inter-preted as a 2D surface in a 3D space, with the thirdcoordinate being proportional to the pixels intensityvalue, with an aspect weight α→ 1, and the first twocoordinates proportional to (x,y) (image pixel coordi-nates) with weight 1−α.

We define a geodesic level curve as the set ofpoints at the same geodesic distance from a given in-terest point; it is possible to capture the joint distribu-tion of intensity and geodesic distances and summa-rize it into the so called GIH histogram-based descrip-tor by sampling pixels with constant geodesic step ∆..

An efficient scheme for the geodesic level curvescomputation on discrete pixel grids, was providedby Sethian with the name of fast marching algo-rithm (Sethian, 1999). A marching speed F(x,y) isassociated to each pixel x,y and the geodesic distanceT (x,y) can be estimated solving locally the equation|∇T |F = 1, where

F(x,y) =1

f (x,y)=

1√(1−α)2 +α2I2

x +α2I2y

. (1)

Although the shape of the resulting region is irregular,it is covariant with deformation and it has shown in-teresting results for generic continuous deformations.Sometimes worst behaviors may occur in correlationwith the presence of isotropic and anisotropic scaletransformations, causing a resampling of pattern con-tours, but for uniform intensity region, the expansionis independent from image gradient, mainly depend-ing on 1−α value.

3.1 Fast Marching Algorithm in RGBSpace

The first improvement proposed in this paper aims atmodifying region expansion in the fast marching al-gorithm considering color information1. We take intoaccount each RGB channel separately, computing 3different inverse marching speeds one for each chan-nel:

fr(x,y)2 = (1−α)2 +α2R2x +α2R2

y , (2)

fg(x,y)2 = (1−α)2 +α2G2x +α2G2

y , (3)

fb(x,y)2 = (1−α)2 +α2B2x +α2B2

y . (4)

1We present in this paper only the RGB implementationfor the new fast marching algorithm; we tested also othercolor spaces with no meaningful improvements thus we de-cided to use the most efficient one.

VISAPP 2008 - International Conference on Computer Vision Theory and Applications

228

Assuming there is no particularly strong photometrictransformation (e.g. Lambertian surface), these func-tions are deformation covariant, as with the geodesicdistance on the intensity surface, and, under thesame hypothesis, the ordering of fr(x,y), fg(x,y) andfb(x,y) is preserved as well. The geodesic region, inour proposal, is thus computed starting from a new in-verse marching speed based on these three color chan-nels:

fmin(x,y) = min( fr(x,y), fg(x,y), fb(x,y)), (5)

and the resulting new geodesic distance T is invari-ant, being the sum of the same invariant stretches.This choice, being somehow counterintuitive, has tworationalities behind it and has been confirmed by ex-perimental results:

• The growth of geodesic region is bounded by allchannels and slows down only for high values offmin(x,y). The moving front speed decreases onlyif contour are thus found on all channels, cor-responding to a strong change in luminance orcrominance, less sensitive to noise with respect togeometric transformations

• Weak contours found on a single channel have mi-nor influence on the geodesic region and let thefront go on. In this way, given a geodesic levelcorresponding to a quantization interval on thegeodesic distance, this will show the salient in-tensity and color varieties separated by weak con-tours.

Table 1 shows an example of geodesic distance Texpansion on a monodimensional grid of pixel i ∈{1..n}. Pixel updating order is deterministic, in factthe moving front can proceed in only one direction.The updating formula becomes:

T (i+1) = T (i)+ fmin(i+1) (6)

We can identify weak contour on pixels i = 2 andi = 3 and a strong contour on pixel i = 6. Comput-ing geodesic distance in this fashion, it puts in thesame interval of T more color varieties, representedby weak contours. For instance, let be the samplinggap ∆ = 0.1 and the histogram quantization intervalof T to δT = 0.8. The geodesic level segment (ringin the bidimensional space) starts at the first pixel andstops between pixel i = 5 and i = 6. Sampling pointscoming from this segment shows a lot of color varia-tions, the same variations causing the weak contourson pixels i = 2 and i = 3.

The results obtained by using Equation (5) istwofolds, on one hand we have that strong contoursare less likely to be smoothed, if affected by geo-metric deformations, and, on the other one, a more

Table 1: Evolution of geodesic distance T on a monodimen-sional grid of pixel i ∈ {1..n}., computed using fr, fg, fb.

i 1 2 3 4 5 6 7fr 0.1 0.5 0.1 0.1 0.2 0.9 0.1fg 0.1 0.1 0.1 0.1 0.1 0.8 0.1fb 0.1 0.1 0.4 0.1 0.2 0.8 0.2T 0.1 0.2 0.3 0.4 0.5 1.3 1.4

deformation-covariant placement of sampling regionsis obtained.

3.2 Building the Geodesic ColorDescriptor

Low distinctiveness is a disadvantage of the originalgeodesic descriptor. The GIH descriptor summarizesthe geodesic region content as an histogram Hp(k,m)built on two variables: geodesic distance g and nor-malized intensity I (g is quantized in M intervals, I inK intervals). For each geodesic interval m, a normal-ization has to be performed such as ∑K

k=1 Hp(k,m) = 1to compare inner and outer geodesic “rings” with thesame weight.

Given a ring of geodesic level curves, correspond-ing to a quantization interval on the geodesic distance,GIH stores no spatial informations about sample loca-tion along the ring thus the resulting numeric vector isless informative than other descriptors such as SIFT(on affine covariant trasformation). We propose topartially overcome this distinctiveness problem, with-out losing rotation invariance, by adding color infor-mation into the descriptor.

To build a richer descriptor, we consider 3 or moredimensions, together with their respective quantiza-tion intervals: the geodesic distance g (quantized inM bins), the normalized intensity I (quantized in Kbins), and one or more photometric invariants Invn(quantized in Qn bins).

The implemented descriptor keeps the geodesicinformation associated to color invariants, extendingthe original M×K GIH descriptor with N additionalM×Qn histograms one per invariant Invi. The finaldimension of the descriptor is thus M(K +∑N

n=1 Qi).The matching distance between two Geodesic

Histogram & Histograms descriptor (GHH hereafter)p and q is computed starting from the χ2 distancesbetween each couple of histograms:

dI = χ2(HIp,H

Iq) (7)

dInv1 = k1 ·χ2(HInv1p ,HInv1

q ) (8)

dInv2 = k2 ·χ2(HInv2p ,HInv2

q ) (9). . . (10)

ADDING COLOR TO GEODESIC INVARIANT FEATURES

229

where k1 and k2 are weighting parameters, and thanthe descriptors distance as a whole is obtained as

d(Hp,Hq) = max(dI ,dInv1 ,dInv2 , . . .). (11)

This implies that a match between two point is al-lowed only if all pairs of histograms are fairly sim-ilar, up to a given threshold, both in luminance andchrominance.

3.3 Color Invariants Selection

GHH descriptor can be seen as a general construc-tion method for geodesic descriptors including inten-sity and color information; effective invariant selec-tion, then, should be performed taking into accountspecific assumptions about image formation and thespecific application.

We can distinguish between zero-order invariantsand first-order invariants, i.e., derivative based invari-ants. For the use with geodesic regions, characterizedby generic deformations, the main challenge is to pre-serve geometric robustness, then we can have focusedthe work on zero-order invariants computable fromcommon color spaces. Another practical requirementis stability, since known photometric invariant haveinherent instabilities.

The measured color information for a camera sen-sor can be modeled as:

C(~x) = mb(~x)bC(~x)eC +mi(~x)eC, (12)

where the first part describes the light which is re-flected after interaction with surface albedo b andthe second-one is related to the light immediately re-flected at the surface, causing specularities. The aimof color space selection is to provide invariance withrespect to mb and mi, which depend on scene and illu-mination geometry. In this paper we considered threeinvariant for color representation:

RGB normalized RGB are invariant to the shadingmodeled by mb term in absence of specular effects(i.e. mi = 0);

O1,O2 opponent colors, that, for white illuminant,are invariant with respect to specularities;

H,S the hue information is invariant to both mb andmi, but this is known to be unstable with low sat-uration also; weighting each sample by its satura-tion is only a partial solution.

We have tested these invariants on the realdataset provided in the paper of Mikolajczyck andSchmid (Mikolajczyk and Schmid, 2005) with GHHdescriptor, after normalization obtained subtracting

0

0.001

0.002

0.003

0.004

0.005

0.006

0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6

P

k

P(dI=k|corr)P(dr=k|corr)P(dg=k|corr)P(dI=k|!corr)P(dr=k|!corr)P(dg=k|!corr)

Figure 1: Histograms distance distributions for I, r and g.

0

0.001

0.002

0.003

0.004

0.005

0.006

0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6

P

k

P(dI=k|corr)P(dO1=k|corr)P(dO2=k|corr)

P(dI=k|!corr)P(dO1=k|!corr)P(dO2=k|!corr)

Figure 2: Histograms distance distributions for I, O1 andO2.

0

0.001

0.002

0.003

0.004

0.005

0.006

0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6

P

k

P(dI=k|corr)P(dH*0.2=k|corr)

P(dI=k|!corr)P(dH*0.2=k|!corr)

Figure 3: Histograms distance distributions for I and Hue(scaled by 0.2 for better illustration).

the mean and dividing by standard deviation. For an-gular invariant hue, illuminant normalization is pre-viously performed on RGB channels, dividing eachchannel value by its spatial average. Values withinthe (−3σ,+3σ) interval have been segmented in 13bins and Figures 1, 2 and 3 compare the intensity his-tograms HI respectively with normalized r and g his-tograms Hr Hg, opponent color histograms HO1 HO2

and hue histograms Hhue.

VISAPP 2008 - International Conference on Computer Vision Theory and Applications

230

0

0.2

0.4

0.6

0.8

1

0 2 4 6 8 10

dete

ctio

n ra

te (

74)

N

RGB GIHRGB GHH

SIFTLing GIH

SPINMOM.INV.

STEER.FILT.

(a) Graffiti image (graffiti 3)

0

0.2

0.4

0.6

0.8

1

0 2 4 6 8 10

dete

ctio

n ra

te (

91)

N

RGB GIHRGB GHH

SIFTLing GIH

SPINMOM.INV.

STEER.FILT.

(b) Flag image

0

0.2

0.4

0.6

0.8

1

0 2 4 6 8 10

dete

ctio

n ra

te (

88)

N

RGB GIHRGB GHH

SIFTLing GIH

SPINMOM.INV.

STEER.FILT.

(c) T-Shirt image

0

0.2

0.4

0.6

0.8

1

0 2 4 6 8 10

dete

ctio

n ra

te (

34)

N

RGB GIHRGB GHH

SIFTLing GIH

SPINMOM.INV.

STEER.FILT.

(d) Candle image

Figure 4: Performace evaluation on real datasets.

These plots underline the relationship between his-tograms distance χ2 and matching correctness interms of conditional probabilities. Let be (a,b) a cou-ple of features; we estimate the probabilities that thehistograms distance d(a,b) falls into a small intervalk−δ,k, given that (a,b) is a correspondence

P(k−δ < d(a,b)≤ k|T (a) = b) (13)

and given that (a,b) is not a correspondence

P(k−δ < d(a,b)≤ k|T (a) 6= b) (14)

being T the transformation between the images. In or-der to make a distinctiveness comparison for intensityand color histograms, the weighting parameters ki areset to 1.

From the plots it can be noticed that correspondingand non-corresponding features distances for inten-sity show a higher discrimination power with respectto color information. Nevertheless, opponent colorsshow a more symmetric and regular behavior (com-pared to normalized RGB), allowing the assumptionkO1 = kO2 = kO. Mostly due to saturation weight, thehue histograms have low distinctiveness (overlappeddistributions). For this reasons, we decided to use op-ponent color in the following of the paper as colorinvariants; moreover this analysis suggests a value forthe color histograms weighting parameter, that shouldbe adjusted according to the real working conditions.In the case of this dataset kO = 0.3 can be used; thisvalue roughly overlaps correct matches distributions

of dI , dO1 and dO2, allowing histograms comparisonwith respect to the same threshold dth, and thus im-proving matching precision.

4 EXPERIMENTAL RESULTS

In this section we present the experimental evalua-tion done to test the capabilities of the new descriptorsproposed in this paper, using as interest point detec-tor the Harris-Affine2 proposed by Mikolajczyck andSchmid (Mikolajczyk and Schmid, 2004). In partic-ular we compare the performances of our approacheswith the descriptors: GIH, GIH with RGB expansion,SIFT (Lowe, 2004), spin images (Lazebnik et al.,2003), moments invariants (Gool et al., 1996), andsteerable filters (Freeman and Adelson, 1991).

The characteristics we are interested in evaluatingare robustness, i.e., the capability to describe in thesame manner two correspondent regions, and preci-sion, i.e., the distinctiveness of the descriptors. Weuse the criterion proposed by Ling and Jacobs to eval-uate the GIH performances. For each pair of imageswe select the 200 interest points with the higher cor-nerness and, for each of them, we manually estimatethe ground truth matching. Each interest point inthe first image is compared with all interest points in

2Code available athttp://www.robots.ox.ac.uk/vgg/research/affine/

ADDING COLOR TO GEODESIC INVARIANT FEATURES

231

(a) T-shirt images

(b) Candle images

Figure 5: Examples of matches on real images.

the second image according to the distance describedin Section 3.2 and we considered the detection rateamong the top N matches used to study the perfor-mance

r =#correct matches

#points f irst image. (15)

Experiments have been done on the dataset providedby Ling and Jacobs, on the dataset by Mikolajczyckand Schmid, and on other real images with non affinedeformation. Due to space limits we present here onlytwo plots with performances on public available dataand we details the results on the data we collected thatare more significative for non affine trasformation.

The plots in Figure 4 show that GHH descriptioncapability improves the GIH intensity-based descrip-tion in all the cases and that also the GIH with RGBexpansion achieve better performances with respectto classical GIH even if the descriptor is still basedonly on intensity information. GHH descriptors out-perform most of other descriptors when non-affine de-formation is present and are able to describe local fea-tures also when these are not on a planar surface.

In Figure 5 we report the new (with respect ofpublic available datasets) images used in this paperand the matches found by the use of GHH. To bet-

ter understand the rationale behind GHH we plottedgeodesic levels and descriptor histograms for a fewinterest points on the T-shirt images in Figure 6.

When dealing with affine transformation (See plotin Figure 4(a)) SIFT are still better. We investigatedthis and devised two possible reasons: the SIFT de-scriptor is more rich than a histogram based one, andthe affine property is not exploited in the geodesic dis-tance calculation. We have coped with the latter issue,but this is out of the scope of this paper, while the for-mer issue is still an ongoing work.

In Figure 4(a) it is also possible to notice thatsometimes the SPIN descriptors are better than SIFT,GIH and GHH, this is not a surprising result be-ing the latters based on the “SPIN image” idea: atwo-dimensional histogram encoding of the bright-ness distribution in an affine-normalized patch.

5 DISCUSSION AND FUTUREWORKS

In this paper we introduced a method to extendgeodesic invariant feature with color information.Experimental results confirm the promising perfor-

VISAPP 2008 - International Conference on Computer Vision Theory and Applications

232

Figure 6: Examples of GHH descriptors on the T-shirt image.

mances of geodesic descriptors and the improvementgiven by the use of color information.

Althought we succeded in adding color informa-tion to the geodesic framework some open issuesstill remain: the improvement obtained by GHHon deformed images is paid with a reduced perfor-mance with respect to SIFT descriptors on real pla-nar patches. To better understand the reasons for this,we made a deep analysis of geodesic expansion. Itturns out that adding the affine information carried bythe Harris-affine detector to the expansion mechanismimproves the GHH performance on planar patches,but this results are not reported here since somehowreduce the generality of GHH and GHI descriptors.Another reason for SIFT better performance is due tothe smoothing effect induced by the histogram rep-resentation; we are currently working on SIFT likerepresentation of geodesic levels able to capture localgradient information.

A final note should be made about interest pointdetectors. In this paper we used the Harris-affine de-tector, but intuitively it is not optimal for geodesicdistance computation since it selects corner points lo-cated on singular positions of the image geodesic rep-resentation. We have done preliminary work withMSER detectors (Matas et al., 2002), but this needsfurther investigation since they are located on the bari-center of uniform regions and with non affine defor-mation this might not be in a repeatable positions.This is a problem since in performance evaluation we

do not consider area matching, but points, and this canfail even if we perform the correct match.

ACKNOWLEDGEMENTS

This work has partially been supported by Italian Isti-tute of Tecnology (IIT) grant.

REFERENCES

Freeman, W. and Adelson, E. (1991). The design and useof steerable filters. IEEE Trans. on PAMI, 13(9):891–906.

Gool, L. V., Moons, T., and Ungureanu, D. (1996). Affinephotometric invariants for planar intensity patterns.ECCV, 642-651:1996.

Lazebnik, S., Schmid, C., and Ponce, J. (2003). A sparsetexture representation using affine-invariant regions.CVPR, 2:319–324.

Ling, H. and Jacobs, D. W. (2005). Deformation invariantimage matching. ICCV, 1:1466–1473.

Lowe, D. (2004). Distinctive image features from scale in-variant keypoints. IJCV, 60(2):91–110.

Matas, J., Chum, O., Urban, M., and Pajdla, T. (2002). Ro-bust wide baseline stereo from maximally stable ex-tremal regions. BMVC, pages 384–393.

Mikolajczyk, K. and Schmid, C. (2004). Scale and affineinvariant interest point detectors. IJCV, 60(1):63–86.

ADDING COLOR TO GEODESIC INVARIANT FEATURES

233

Mikolajczyk, K. and Schmid, C. (2005). A performanceevaluation of local descriptors. PAMI, 27:1615 – 1630.

Mikolajczyk, K., Tuytelaars, T., Schmid, C., Zisserman,A., Matas, J., Schaffalitzky, F., Kadir, T., and Gool,L. V. (2005). A comparison of affine region detectors.IJCV, 65(1/2):43–72.

Moreels, P. and Perona, P. (2005). Evaluation of featuresdetectors and descriptors based on 3d objects. ICCV,1:800–807.

Sethian, J. (1999). Efficient Schemes: Fast Marching Meth-ods, chapter 8, pages 87–100. Cambridgr UniversityPress.

van de Weijer, J. and Schmid, C. (2006). Coloring localfeature extraction. ECCV, 2:334–348.

VISAPP 2008 - International Conference on Computer Vision Theory and Applications

234