Unsupervised Segmentation of Objects using Efficient Learning

{Himanshu Arora, Nicolas Loeff}∗, David A. Forsyth, Narendra AhujaUniversity of Illinois at Urbana-Champaign,

Urbana, IL, 61801{harora1,loeff,daf,n-ahuja}@uiuc.edu

Abstract

We describe an unsupervised method to segment objectsdetected in images using a novel variant of an interest pointtemplate, which is very efficient to train and evaluate. Oncean object has been detected, our method segments an im-age using a Conditional Random Field (CRF) model. Thismodel integrates image gradients, the location and scale ofthe object, the presence of object parts, and the tendency ofthese parts to have characteristic patterns of edges nearby.We enhance our method using multiple unsegmented imagesof objects to learn the parameters of the CRF, in an iterativeconditional maximization framework. We show quantitativeresults on images of real scenes that demonstrate the accu-racy of segmentation.

1. Introduction

Keypoint based object templates are efficient and ac-curate [6, 7, 14, 12], and can be used to localize objects.This paper presents a novel model that integrates a keypointbased template into a random field to achieve unsupervisedobject segmentation. We show that the information gener-ated while learning the template is rich enough so that seg-mentation can be learned without supervision. In particular,coherence of edges within the frame of reference of a partin the template and across the training examples is a strongcue for segmentation.

Towards this, we extend the template based object modelintroduced in [14] to estimate scale. We introduce a newunsupervised edge model that aids in the segmentation ofthe object. This model encodes object boundaries in thelocal coordinate system of the parts in the template.

We integrate the template and image gradient informa-tion into a Conditional Random Field model. To the bestof our knowledge, it is the first attempt to unite keypoint-based methods with random field segmentation, in an unsu-pervised setting. We learn the CRF using an iterative mech-anism, alternating between estimating segmentation andparameters, imposing coherence among the unsegmented

training images. The final detection-segmentation system isfast, and can tolerate large intra class variations. Both quan-titative and qualitative results on real images demonstratethat this iterative procedure produces good segmentations.

Object dependent segmentation. There has been ex-tensive work recently on model driven object segmentationin images. Kumaret al. [11] use a bayesian approach tointegrate a layer based object model with Markov RandomFields. While this method produces good segmentations,it is very demanding of training data, requiring a video foreach category, and also cannot tolerate large intraclass vari-ations. Winnet al. [21] use a Conditional Random Fieldbased model to segments and handle multiple objects withocclusion. Training the model requires already segmentedimages. Leibeet al. [12] use a supervised part based modelto resolve conflicting segmentation of objects in images.Their model uses only keypoint features and not image tex-ture information. Levin and Weiss [13] present a super-vised framework for learning bottom-up and top-down cuessimultaneously. Borensteinet al. [2, 1] present a formu-lation to integrate top-down and bottom-up cues for unsu-pervised segmentation, using image fragments. Winnet al.[20] present a pixel based deformable object model that usescolor cues for segmentation. The model in [16] presents analgorithm to segment and detect people, using a superpixelrepresentation. Carbonettoet al. [4] integrate a bag of key-points model with a CRF to output a per superpixel con-fidance map of object localization, but no segmentation ofthe object. Russell [17]et al. learn object classes as a topicmodel with superpixels as features. Heet al. use a mixtureof CRFs [9] to obtain a per superpixel labelling of objectclasses. Many of these works pose the final segmentationestimation problem as a graph cut energy minimization [3].

2. Object Model

We extend the generative template model of [14] to scaleinvariance and introduce new learning algorithms. An im-age is represented as a set ofF features{fj}Fj=1, describedby their location in the image (f lj) and appearance (faj ). The

model generates the image by first choosing an object con-figuration oc = (ol, os), whereol is the object location,andos is the object scale. Conditioned on the object con-figuration, the model produces features independently. Foreach feature the model chooses a parti ∈ {1, . . . , P} withprobabilityπi. This conditional independence assumptionmakes inference very efficient in comparison to a fully con-nected constellation model [6]. The parti generates the fea-ture appearancefaj from a multinomial distributionpAi (faj )

with parameterhi on a dictionary of patches, and locationf lj from a gaussianpLi (f lj |oc) ∼ N (µi ·(os)−ol,Σi ·(os)

2).The mean and variance of this distribution are corrected forthe scale of the object. The variance term also compensatesfor the fact that smaller objects tend to produce lower num-ber of feature detections. In addition, there is a backgroundpart with uniform distribution in location.

Let us denote byωij the event that the featurefj wasgenerated by parti. This is a binary random variable. Bythe above discussion, we have

∑

i ωij = 1,∀j. We thushavep(fj |ωij = 1, oc) = pLi (f lj |oc)p

Ai (faj ).

The features{fj}F1 in an image make up the setvof observed variables. The set of parameters isΘ =({µi,Σi}

Pi=1, {hi, πi}

P+1i=1 ). The object configuration,oc,

together with part-feature associations,ωij ’s, are the sethof hidden random variables. Thus, the complete data likeli-hood is

P objΘ (h, v) =∏

j,i

{

piL(f lj |oc)piA(faj )πi

}[ωij=1]P (oc) (1)

where [expr] is 1 if expr is true, andP (oc) =P (ol)P (os). We assume thatol and os are a priori uni-formly distributed across their respective domains.

Learning: In appendix A we introduce and analyze twovariational approximations to learn this object model in anefficient way. One (H-EM) is a hybrid which uses Expecta-tion Maximization to estimate the location distribution andpoint estimates to determine the scale of the object. Theother (H-V) is a hybrid which uses a mean-field approxima-tion for location instead.

Template Evaluation: To estimate the configuration ofthe object, we need to maximize the posterior likelihoodp(ol, os|{fj},Θ) with respect to(ol, os).

3. Object Segmentation

The model described in the previous section can detectand estimate the location and scale of the object, as well asthe assignment between features in the image and parts ofthe object. Given a set of unsegmented images, this infor-mation can help rectify them both globally (i.e. location andscale of object) and locally with respect to each part of themodel. In this way, by matching several unsegmented im-ages we can compensate for the lack of supervision; we can

learn boundaries of object and segment (i.e. label regions asbelonging to the object, or background). We approach theproblem in an iterative fashion, first segmenting the objectsbased on the current parameter estimates, and then updat-ing the parameters to match the segmentation in the wholestack of images.

Motivated by [16], we use a superpixel representationobtained by oversegmenting the image intoR = 50 re-gions using the normalized cuts algorithm [18]. The su-perpixel representation not only is computationally advan-tageous due to small number of nodes, but also is percep-tually relevant. In addition, computing features over super-pixels allows for capturing more global phenomena whencompared to interaction between only adjacent pixels. Themain drawback of superpixels is that their boundaries maynot align with the object boundaries.

For the object segmentation task, i.e. labelling superpix-els in the image as belonging to object or background, weincorporate the following information:

1. Object location and scale estimates provided by the ob-ject model (M ) to localize the object in the image,

2. Shape and extent of the object for which we introducea) Model Edgeresponse (mE), which captures the lo-cations of edges with respect to a local reference framegiven by the model, and b)Object/Part Overlap(mO),which captures the extent of overlap of each part in themodel with the inside of the object. We explain thesebelow in Sec. 3.1 and 3.2.

3. The location of image edges (IE) with respect to theobject edges estimated by the model.

To integrate all this information, we use a random fieldmodel. With each superpixelv in the imageI, we associatea binary labellv which takes a value of 1 ifv lies inside theobject and 0 if it is in the background. The random fieldmodel is then expressed for the graphG = (V, E), whereVis the set of all superpixels in the image andE is the set ofall pairs of neighboring superpixels, as follows:

log(P ({lv}|I)) + logZ(λM , λE , IE) =Xv∈V

φ(lv|mE ,mO)| {z }Model Unary

+X

{v,u}∈ E

�λM ψM (lv, lu|mE)| {z }

Model Edge Interaction

+λE ψE(lv, lu|IE)| {z }Image Edge Interaction

�. (2)

where,φ(lv|mE ,mO) is a local estimate of the probabilitythatv is in the object. Superpixels that belong to the objecttend to include features assigned to the objectmO as wellas model inferred edgesmE . The interaction terms tend toseparate superpixels when both model inferred edgesψMand image edgesψE align.Z is the normalization term.

3.1. Model edgemE

The template model in Sec. 2 only accounts for thelocal appearance of the object, and not its boundaries.

Figure 1.Computation of per part mean edge response. Weobtain the dominant edges as the boundaries of normalized cut al-gorithm. Top row shows the detected feature locations overlaidon the normalized cut segmentation for the training images.Bot-tom row shows edge response computedcentered in the featureframe of reference. Weighted average of these give the per partmean edge response as shown by the final figure. This procedureis described in section 3.1

Here we propose a novel mechanism to enhance the modelfor encoding object boundaries. Given the set of trainingimages, we can infer the configuration of objects in them,and the assignments between features in the image andparts in the model. We expect that in thelocal coordinateframe of each model part, object edges will be consistent,while background edges will not. This suggests that amodel for edge generation can be added to each part.

In addition to the local appearance of a feature, partinow also generates a local edge response, as a gaussian withmean edge responseµep

i , for a pointp centered at the fea-ture locationf lj . Assume that the observed dominant edgeresponse for thenth image is given byen(p). Also assumethat the parametersθ of the model have been learnt, and ob-ject configuration(ol, os) has been inferred. Then,µepi canbe estimated as

µep

i =X

n,j

αnijen((p− f l

j)/os), αnij , P (wn

ij = 1|ol, os, fnj ) (3)

i.e., an average of the feature centered edge responses scaledby the estimated scaleos, weighted by the posterior of partigenerating that featureαij . The entire process is depicted inFig. 1. We use normalized cuts segmentation withR = 10regions as the image edge response during learning.

Given a new image, the (local) per feature inferred edgesare aggregated over the entire image to produce a mapof model edges on the imagemE . This edge generationprocess is illustrated in Fig. 2 . Note that this edge responseis independentof the actual edges in the image. We usemE

to define both the unary and interaction terms in our model(Eq. (2)).

3.2. Overlap between features and objectmO

Each part in the object model has a position and extent inlocation described by its mean location and variance. Due tothis extent, a part can generate features that lie completelyinside the object, or on its periphery. For instance, consideran airplane model: window features usually lie completely

Figure 2.Template based segmentation. Top left image showsan original face image with the estimated configuration of the ob-ject using the keypoint model.Top right image shows the modelbased edge response computed using locally rectified templateedges weighted by votes from image keypoints. Note that thisdoes not dependon theedges of the imageitself, which are en-coded by the superpixels.Bottom left image shows the superpixelgraph. The width of edges here represents the total intraction terms(λMψM (lu, lv|mE) +λEψE(lu, lv|IE)). The final segmentationis shown inbottom right image.

inside the airplane, while wing, tail and specially landinggear features tend to overlap to a lesser extent.

Thus for each parti, we introduce an object-part overlapparameterqi ∈ [0, 1], which denotes the fraction of extentof a feature lying inside the object, given that the featureis generated by parti. Note that this information is vitalfor using a keypoint template based model for segmenta-tion, since the object model does not explicitly model eachpixel’s probability of lying inside the object. This is anothernovel contribution of this paper.

3.3. Unary potentialφ

Using these parameters, we can compute a unary perpixel classifier by averaging over the observed features as

P (lv|mO) =1

Av

∑

p∈v

1

Np

∑

i,j

(qi)lv (1− qi)

(1−lv)αijδp∈fj

(4)where,Np is the total number of features which includepixel p (i.e.

∑

j δp∈fj), Av is the number of pixels in su-

perpixelv andαij is the posterior of parti given featurefj and inferred location and scale of the object, as definedin Eq. (3). Pixels where no features exist are given equalprobabilities of belonging to object and background.

Since consistent model edges should only lie on theboundary or inside that object, a model edge response ina superpixel should indicate the presence of an object. Con-sidering this, we choose to estimateP (lv = 1|mE) as themean of the normalized object model edge response insidesuperpixelv.

We define the unary termφ in the random field (Eq. (2))

as a weighted average of these estimates as follows:

φ(lv|mE ,mO) = log(γP (lv|mE) + (1 − γ)P (lv|mO))

3.4. Interaction Termsψ

The presence of a strong inferred model edge at theboundary between neighboring superpixelsu and v indi-cates that they might have different labels. We compute theinteraction potential between superpixelsψM in Eq. (2) asthe inverse of the integral of model edge response across theboundary betweenu andv for lv 6= lu and zero otherwise.

The image edge dependent termψE is motivated by thefact that image edges should align with object boundaries.We thus want to penalize two neighboring superpixels hav-ing different labels if there is no image edge between them.We computeψE as the normalized-cut cost (sum of intraregion affinity as defined in [18] for both superpixels overinter region affinity between superpixels) forlv 6= lu andzero otherwise.

3.5. Inference and parameter estimation

Since we do not use supervised training data with groundtruth segmentations for training,Q = {qi}

P1 is not known

a priori and needs to be estimated. We learn these para-meters in an iterative fashion. We first segment all trainingimages, labeling every superpixel as object or background.We then fix the labels and reestimate the parameters, pool-ing information from all training images simultaneously. Asthe segmentations are significantly influenced by the modelestimates, the algorithm converges to a meaningful solution(figure 3). The procedure stops when no segmentation oftraining images changes for one loop.

Inference: Given the parameters, the globally optimallabeling that maximizes the log likelihood in Eq. (2) canbe posed as a min-cut problem on the graphG [3]. Thesolution to this problem is very fast.

Parameter Estimation: Given the segmentation, weadopt a piecewise approximation [19] to the normalizationtermZ, which is a bound that decouples the unary and in-teraction potentials. Consider a set ofN training imagesIn

with object model inferredmE , mO, and assumed labels{lnv }

Rv=1, wheren = 1 . . . N . Then the piecewise approxi-

mation islogZ≤log(

P{lv} exp

Pv φ(lv|mE ,mO)) + (5)

logP

{lv} exp{P

{u,v}∈E(λMψM (lu,lv|mE)+λEψE(lu,lv|IE))}

Sinceφ(·) is a log probability, the first term in above ex-pression is zero, and thus this upper bound is independentof Q. Using eq. 2 and 3.3, the parameter estimate becomes

Q← argmaxqi

∑

n,v

log

(

γP (lnv |mE) + (1− γ)∑

i

qiβnvi

)

(6)

Figure 3.Effect of overlap between parts and object. (Topimages): Before learning the overlap between parts and object(Q), object area is overestimated. This is because features on theboundary of the object result in regions close to the objects beingassigned to it during segmentation. (Bottom images): After reesti-mation, the problem is diminished.All images are taken fromtestsets.The superpixel representation is shown inred and the final segmentation isshown ingreen. The brightness of the estimated background superpixelsis diminished to aid visibility. This figure is most easily interpreted whenviewed in color.

whereβvi = 1Av

∑

p∈v1Np

∑

j αijδp∈fj.

Lower bounded using Jensen’s inequality; maximizingthis givesqk ←

∑

n,i lnvβ

nik/∑

n,i βnik.

This iteration between the estimating of labelslnv (seg-menting the images) and parametersQ can be seen as akinto learning with partially labeled data. The termP (lv|mE),which does not depend onQ, biases the search towardmeaningful local maxima.

4. Experiments and Results

We test our algorithm on standard datasets [6]. Each cat-egory has an object image dataset and a background dataset.The scale-saliency keypoint detector [10] is used to extractF features from each image in the dataset. The SIFT [15]representations of features in the object images are clus-tered into representative vectors. The appearance of eachfeature is vector quantized using this dictionary. We trainthe hybrid-EM (H-EM) and hybrid-Variational (H-V) mod-els usingP = 30 parts andNxNy = 238. We initializedthe appearance and location of the parts using randomly se-lected features in the training set. The stopping criterionused is the change in the free energyFe (c.f. appendix A).

Template performance: The median equal error ratedetection performance for10 random test/train splits (50%training/50% testing) of the dataset for each category ofboth inference algorithms (Airplanes: H-EM: 96.6, H-V:96.5,Motorbikes: H-EM: 97.1, H-V: 97.1,Faces: H-EM:98.8, H-V: 98.7) is significantly better than the constellationmodel [6] and comparable to the best performing discrimi-native approach [5]. The H-V model has virtually the sameperformance as the H-EM, but is more than 50 times faster,converging in less than 5 minutes for hundreds of trainingimages.

Qualitative segmentation resultsare shown in figure4 on grayscale images for the caltech faces, airplanes,motorbikes, and caltech 101 carstest sets. Images inthese datasets have large intraclass variations in appearance.

Faces Airplanes Cars MotorbikesTot Obj Bg Tot Obj Bg Tot Obj Bg Tot Obj Bg

Best 97.2 92.2 98.4 96.2 77.4 98.6 96.9 89.3 98.0 89.6 85.6 92.2Algorithm 92.4 67.1 99.1 93.1 75.9 95.9 95.1 87.0 96.1 83.1 79.8 88.9

Table 1.Segmentation performancefor different datasets. The measures used are percentage of total correctly classified pixels (Tot),percentage of object pixels classified as object (Obj ), and percentage of background pixels classified as background (Bg). We comparethe performance of segmentation achieved by theCRF against thebestpossible performance given the superpixels. To compute the bestpossible segmentation, a superpixel is assigned to the object if more than50% of its area is inside the object. Even for anunsupervisedapproach, performance is close to the optimal, with the possible exception of the faces dataset. The reason is that due to high variability,hair is not learnt to be included inside the object (see figure 4).

There are many images for which it is difficult to segmentthe grayscale object without a model. We useγ = 0.5 andλM = λE = 10/3 for our experiments.

Quantitative segmentation results: Table 1 shows theperformance of the algorithm on the test sets in terms ofcorrectly labeled pixels after the CRF parameter learning.The performance measures are 1) Total performance, thepercentage of correctly classified pixels, 2) Object perfor-mance, the percentage of object pixels classified as object,and 3) Background performance, the percentage of back-ground pixels classified as background. For instance, for100% object performance means that the estimated seg-mentation always encloses the ground truth segmentation,but the segmentation might overestimate the object. Hencethis does not mean perfect segmentation. The results arecompared with ideal (best) results, which are obtained bylabelling all superpixels with> 50% overlap with thegroundtruth as object. Even for an unsupervised approach,performance is close to the optimal, with the possible ex-ception of the faces dataset. Hair tends to change from sub-ject to subject, and therefore the algorithm learned a lowvalue for object/part overlapqi for the hair parts (figure 4).As a result, face segmentations do not include hair for mostimages (and thus the background performance is better thanthe ‘optimal’). This behavior can be expected in an unsu-pervised framework.

5. Conclusion

We present a novel framework for integrating keypoint-based template object models with segmentation, in anunsupervised fashion. We enhance a generative objectmodel, and introduced veryefficient learning algorithms.We extend the object model to produce boundary informa-tion based on keypoint features independently of the actualedges in a given image. We feed this all this informationinto a random field model and introduce an iterative para-meter reestimation. The result is verygood segmentationeven for images with very low object-background contrast.

Future Work : One of the main drawback of the currentmodel is that the information flows in one way from objectmodel to segmentation. We are working on a new modelfor multiple object detection/segmentation where the object

model proposes several candidate configuration and feedsthe information into the random field, which solvesboththe detection and segmentation problems simultaneously.

References

[1] E. Borenstein, E. Sharon, and S. Ullman. Combining Top-Down and Bottom-Up Segmentation. InProc. of CVPR.,pages 46–46, 2004.

[2] E. Borenstein and S. Ullman. Class-Specific, Top-DownSegmentation. InProc. of ECCV, pages 109–124, 2002.

[3] Y. Boykov, O. Veksler, and R. Zabih. Fast approximateenergy minimization via graph cuts.IEEE Trans. PAMI,23(11):1222–1239, 2001.

[4] P. Carbonetto, G. Dorko, C. Schmid, H. Kuck, and N. de Fre-itas. A semi-supervised learning approach to object recog-nition with spatial integration of local features and segmen-tation cues. InToward Category-Level Object Recognition,pages 277–300, 2006.

[5] G. Dorko and C. Schmid. Object class recognition using dis-criminative local features. Technical Report RR-5497, IN-RIA, 2005.

[6] R. Fergus, P. Perona, and A. Zisserman. Object Class Recog-nition by Unsupervised Scale-Invariant Learning.Proc. ofCVPR, 2003.

[7] R. Fergus, P. Perona, and A. Zisserman. A sparse object cate-gory model for efficient learning and exhaustive recognition.In CVPR, volume 1, pages 380–387, 2005.

[8] B. Frey and N. Jojic. A Comparison of Algorithms for Infer-ence and Learning in Probabilistic Graphical Models.IEEETrans. PAMI, 27(9):1392–1416, 2005.

[9] X. He, R. S. Zemel, and D. Ray. Learning and incorporatingtop-down cues in image segmentation. InECCV (1), pages338–351, 2006.

[10] T. Kadir and M. Brady. Scale saliency : A novel approach tosalient feature and scale selection.International ConferenceVisual Information Engineering, pages 25–28, 2003.

[11] M. P. Kumar, P. Torr, and A. Zisserman. Objcut. InProc. ofCVPR, pages 18–25, 2005.

[12] B. Leibe, A. Leonardis, and B. Schiele. Combined object cat-egorization and segmentation with an implicit shape model.In Proc. of the Workshop on Stat. Learning in Comp. Vision,Prague, Czech Republic, May 2004.

[13] A. Levin and Y. Weiss. Learning to combine bottom-up andtop-down segmentation. InECCV (4), pages 581–594, 2006.

Figure 4.Segmentation resultsof our algorithm on example test images in standard datasets. Grayscale images are used. The superpixelrepresentation is shown inred and the final segmentation is shown ingreen. The brightness of the estimated background superpixels isdiminished to aid visibility. We test the algorithm on caltech faces, airplanes, motorbikes, and caltech 101 cars. Note that the quality ofsegmentation is affected due to bad approximation of object boundaries by the superpixel boundaries. For most images the segmentationis quiteaccurate, even in the prescence of intra-class variation and poor object-background contrast. Examplarfailures are shown in thelast column. Performance for faces is low in table 1 because the unsupervised algorithm learned to segment faces without hair. This figureis most easily interpreted when viewed in color.

[14] N. Loeff, H. Arora, A. Sorokin, and D. Forsyth. Efficientunsupervised learning for localization and detection in objectcategories. InProc. of NIPS. 2005.

[15] D. Lowe. Distinctive image features from scale-invariantkeypoints.IJCV, 60(2):91–110, 2004.

[16] X. Ren, C. Fowlkes, and J. Malik. Cue Integration in Fig-ure/Ground Labeling. InProc. of NIPS, 2005.

[17] B. C. Russell, W. T. Freeman, A. A. Efros, J. Sivic, andA. Zisserman. Using multiple segmentations to discover ob-

jects and their extent in image collections. InCVPR (2),pages 1605–1614, 2006.

[18] J. Shi and J. Malik. Normalized cuts and image segmenta-tion. IEEE Trans. PAMI, 22(8):888–905, 2000.

[19] C. Sutton and A. McCallum. Piecewise training for undi-rected models. InProc. of UAI, pages 568–575, 2005.

[20] J. Winn and N. Jojic. LOCUS: Learning Object Classes withUnsupervised Segmentation. InProc. of ICCV, pages 756–763, 2005.

[21] J. Winn and J. Shotton. The Layout Consistent Random Fieldfor Recognizing and Segmenting Partially Occluded Objects.In Proc. of CVPR, pages 37–44, 2006.

A. Learning as Variational Inference

Learning the model from a set of images of an object cat-egory involves estimating the parametersΘ of the model.The background distribution parameter (hP+1) is estimatedfrom the background dataset; this distribution is fixed be-fore estimatingΘ. Obtaining the maximum likelihood es-timate forΘ involves maximizing the likelihood ofPΘ(v),for which no closed form solution exists. Many algorithmsfor iterative inference such as EM, ICM, BP can be viewedas instances of variational inference [8]. Variational infer-ence involves minimizing free-energyFe with respect to anapproximationQ(h) to the posterior distributionPΘ(h|v).LetDKL be the K-L divergence, thenFe is defined as

Fe(Q,Θ).= DKL

{

Q(h)∣

∣

∣

∣PΘ(h|v)}

− logPΘ(v)

=

∫

h

Q(h) logQ(h)

PΘ(h, v)dh (7)

EM update: Minimizing Fe without restrictions onQ(h) = Q(ωij |ol, os)Q(ol)Q(os) leads to E step of the EMalgorithm, withQ(h) = PΘ(h|v). This can be viewed astaking expectation with respect to the hidden random vari-ables. Thus the complexity of the approach is the dimensionspace of the hidden variables. Let us assume, for simplicity,that the object center takes values on aNx×Ny grid on theimage, and scale takes values on a grid of sizeNs. Thenthe complexity of the approach isO(FPNxNyNs). Unlike[6], this is linear in the number of features per imageF , andparts in the modelP . However, as we include more typesof 2D transformations, this number explodes.

Variational (mean-field) update: Another possi-ble approach is to minimizeFe restricting Q(h) =Q(ωij)Q(ol)Q(os). This assumption basically makes theassignments (ωij) conditionally independent of the objectconfigurationoc given the observations. Following [14],this makes inference much faster, as the complexity is nowO(FP ) +O(NxNy) +O(Ns).

ICM (point estimate) update: Complexity can be re-duced even further if we consideroc to be continuous vari-ables and restrictQ(ol) = δo∗

landQ(os) = δo∗s . Mini-

mizingFe now involves finding the most likelyocnew as afunction of the data, parameters and previous iteration esti-mate ofocold. This makes inference even faster (O(FP )),and we do not constrain the solution to lie on the grid, butas the search space is more limited, the approach is prone tolocal minima.

Hybrid EM update : In experiments we observed themodel is much better behaved in scale (see figure 5), as ithasonly one minimum, than in location, where there are

Figure 5.Model behavior in scale. The green ellipses representthe distribution in location of object parts of the model superim-posed on the image according to the estimated object scale andlocation (black dot). The dots show the detected feature loca-tions. Blue features are associated by the model to the object, andred features to the background.Last column plots negative loglikelihood vs. the scale estimate for the images in the other twocolumns. There is only one local minimum, thus the use of pointestimates for scale.

multiple local minima. Thus we consider a hybrid approachwith EM updates forQ(wij |oc) andQ(ol) and a point esti-mate update ofQ(os). The updates of the parameters are:

µi ←

Pn,j,ol

αnijol

ons (fnl

j−ol)

Pn,j,ol

αnijol

Σi ←

Pn,j,ol

αnijol

(ons )2(fnl

j−ol)(fnl

j−ol)

T

Pn,j,ol

αnijol

−µiµTi

hi ←

Pn,j,ol

αnijol

faj

Pn,j,ol

αnijol

Here,αnij,oc = Qn(ol)Qn(ωnij |ol, os), and superscriptn

represents the attributes for then-th training image. Maxi-mization with respect to the object scale leads to solving thefollowing quadratic equation

a(ons )2 + b(ons ) + c = 0, where (8)

a = −P

i,j,olαn

ijol(fnl

j −ol)T Σi(f

nlj −ol)

b =P

i,j,olαn

ijolµT Σi(f

nlj −ol)

c =P

i,j,olαn

ijol

It can be easily shown that the above quadratic hasa unique positive solution, which forms the update ofthe object scale. The complexity of these updates isO(FPNxNy), equal to that of the non-scale invariant ver-sion [14].

Hybrid Variational update : We consider variational(mean-field) updates forQ(wij) and Q(ol) and a pointestimate update ofQ(os). The updates of the para-meters vary from the hybrid EM approach only in thatαnij,oc = Qn(ol)Q

n(ωnij). The complexity of these up-dates isO(FP + NxNy), equal to that of the non-scaleinvariant variational approach of [14]. We further refinethe resulting model by applying point estimate updates, toremove the conditional independencies introduced by thevariational updates.