Download - Extracting Buildings from Aerial Images Using Hierarchical Aggregation in 2D and 3D

Extracting Buildings from Aerial Images using

Hierarchical Aggregation in �D and �D

Andr�e Fischer�� Thomas H� Kolbe�� Felicitas Lang�

Armin B� Cremers�� Wolfgang F�orstner�� Lutz Pl�umer�� Volker Steinhage�

Computer Science Department I��III�� Institute for Photogrammetry�� University of BonnInstitute for Environmental Sciences�� University of Vechta

Abstract

We propose a model�based approach to automated �D extraction of buildings fromaerial images� We focus on a reconstruction strategy that is not restricted to a small classof buildings� Therefore� we employ a generic modeling approach which relies on the well�de�ned combination of building part models� Building parts are classi�ed by their roof type�Starting from low�level image features we combine data�driven and model�driven processeswithin a multi�level aggregation hierarchy� thereby using a tight coupling of �D image and�D object modeling and processing� ending up in complex �D building estimations of shapeand location� Due to the explicit representation of well de�ned processing states in termsof model�based �D and �D descriptions at all levels of modeling and data aggregation� ourapproach reveals a great potential for reliable building extraction�

Keywords� explicit �D�modeling� coupling of �D� and �D�modeling� multilayer aggrega�tion� building modeling� multi�image correspondence analysis� mid�level feature aggregates�aspect hierarchies� constraint logic programming�

� Introduction

Due to the fact that more than about �� of the world population lives in urban or suburbanenvironments the automation of �D building extraction is an issue of great importance andshows an increasing need for various applications including geo�information systems� townplanning or environmental related investigations�

Aerial images contain on the one hand a certain amount of information not relevant forthe given task of building extraction like vegetation� cars and building details� On the otherhand� there is a loss of relevant information due to occlusions� low contrast or disadvantageousperspective� To compensate for these properties of image data as well as for being able to handlethe complexity of building types and building structures� a promising concept of automatedbuilding extraction from aerial images must incorporate a suciently complete model of theobjects of interest and their relations within the whole process of image interpretation andobject reconstruction ��

We propose a model�based approach to automated �D extraction of buildings from aerialimages� The knowledge about buildings is used to control and to evaluate building extraction inall stages of the process� It is encoded by means of a generic �D object model� which describesspatial appearances of buildings and characterizations of their components� Furthermore� a�D image model� which is capable to integrate sensor and illumination modeling� describes theprojective appearances of buildings speci�c for the given aerial imagery�

�

�� Related work

Related work on �D building extraction � or in general on �D scene reconstruction � revealsdi�erent modeling schemes� Polyhedral models show a long tradition as approximative objectdescriptions �� Obviously polyhedral descriptions are too general for theuse within �D building extraction and therefore move the burden of building modeling toadditional representation schemes to represent and organize domain speci�c heuristics andconstraints� like in the MOSAIC system �� or in the approach of Braun �� Parameterizedmodels are restricted to describe the most common building types in the sense of prototypes�� but represent few variations and combinations of their shapes as wellas other relations� Prismatic models can describe arbitrary complex polygonal ground plansof buildings� but reveal a strong restriction to buildings with only �at roofs �� CADmodels are used to describe objects with �xed geometry and topology in object recognition

tasks� especially for controlling industrial processes �� The use of CAD modelsin building extraction is therefore restricted to the identi�cation of a priori known buildings��

Generic modeling approaches promise on the one hand the greatest modeling power� buton the other hand demand e�ective constraints and heuristics to restrict modeling to buildingspeci�c shapes� Fua and Hanson �� employ simple box�type primitives but propose anexplicit representation of legal primitive combinations to more complex building aggregates�The approaches of Dickinson et al� �� and of Bergevin and Levine �� are of outstandingimportance due to the integration of �D generic object models and an explicit modeling of�D projective object appearances within a recognition�by�components strategy �� Both ap�proaches employ volumetric primitives instead of simple box�types but neglect the descriptionof elaborated schemes for domain dependent primitive combinations� Furthermore� the reli�able extraction of their primitives from real images is the concern of current research �� Bignone et al� �� propose a generic roof model which assumes planar roof surfaces� The roofpatches are extracted by combining photometric and chromatic attributes of image regionswith spatial edge geometry� The �D patches are grouped by an overall optimization accordingto the simplicity� compactness and completeness of the resulting roof shape� To complete thebuilding shape� vertical walls are assumed� Frere et al� �� adopt this modeling approach� butextract image regions with homogeneous photometric and chromatic properties by navigatingthrough a constraint triangulation network where each extracted line segment coincides withedges of the triangles� Henricsson et al� �� present impressive results of their approach onsome test data but show no explicit modeling of building types and building speci�c aggre�gation schemes� Groups of planar �D patches that are optimized according to the criteria ofsimplicity� compactness and completeness� are not necessarily real roof shapes�

For a general and up�to�date overview on the topic of building reconstruction we highlyrecommend the proceedings of the two workshops on �Automatic Extraction of Man�MadeObjects from Aerial and Space Images� �� and the proceedings of the workshop on�Semantic Modeling for the Acquisition of Topographic Information from Images and Maps��

�� Overview

In earlier papers �� we proposed in detail the concepts and processes which have to be takeninto account for a suciently complete modeling framework for �D building extraction� This

�

modeling framework integrates interrelations between image data and model descriptions atdi�erent aggregation levels and in terms of corresponding �D object and �D projective objectdescriptions� In this paper we present a strategy for a well de�ned path from the unstructuredimage data to the model�based and highly structured �D reconstruction of buildings�

The overall strategy follows the paradigm of hypotheses generation and veri�cation andcombines bottom�up �data�driven� and top�down �model�driven� processes� Domain knowledgeconstrains even the early stages of hypotheses generation due to an elaborated part�of�hierarchyof �D building parts and their corresponding projective descriptions� The reconstruction pro�cess is already carried out for local �D feature aggregates to allow an early domain speci�cclassi�cation as �D local building feature aggregates� A step�wise and strongly model�drivenaggregation process combines �D local building feature aggregates to well de�ned parame�terized �D building parts and then to more complex �D building aggregates� The resultingcomplex �D building hypotheses and their components are reprojected into the images to allowa component�based and robust hypothesis veri�cation applying constraint solving techniques��

First results of our approach were presented at the SMATI workshop in Bonn �� Thispaper is a substantially extended version which describes our models and operations in moredetail�

Due to the explicit representation of well�de�ned processing states in terms of model�based�D and �D descriptions at all levels of modeling and data aggregation our approach showsgreat potential for reliable building extraction�

� Concept

In this section we present the proposed building model and discuss its implications on thedeveloped strategy�

�� Models

For coping with the complexity of natural scenes we propose an application speci�c modelingof the domain buildings� Our general concept was presented in an earlier paper �� It containsa close interaction of bottom�up and top�down strategies within an aggregation hierarchy in�D and �D� In this aggregation hierarchy the concept of building parts plays a fundamentalrole for our approach to a generic building model� On the one hand� this model is not based ona polyhedral scheme �� which proved to be too general in order to restrict reconstructionresults to building speci�c shapes in terms of well�de�ned roof types and ground plans etc� Onthe other hand� our approach overcomes the limitations of parametrized CAD models due tothe explicitly de�ned combinations of domain speci�c volumetric and parametrized primitivesin terms of building parts�

Another crucial aspect of the approach is the explicit representation of building appearancesin aerial images in terms of an image model covering all levels of the aggregation hierarchy�This allows a tight coupling of �D and �D reasoning based on a coherent representation ofobject and image model� We start with the modeling of the �D shape of buildings yielding theobject model� The sensor model will transfer many of the concepts of the object model to theimage model describing the expected appearance of the objects�

�

Corner

Wing

Terminal

Region

Point

Line

Whole

Part

part-of

Aggregation

Generalization

Superclass

is-a is-a

Subclass Subclass

Building

Face

Building Part

Feature Aggregate

Feature

Connector

2D 3D

Figure �� Building model� The di�erent semantic levels of the part�of hierarchy areshown in vertical direction� the di�erent levels of abstraction of the is�a hierarchy inhorizontal direction� which are only shown for the �D�model� The �D�image modeldescribes the expected appearance of the building in the di�erent levels of the part�ofhierarchy� which is indicated by not showing the hidden lines�

�� Object Model

Buildings reveal a high variability in structure which suggests to represent them as an aggrega�tion of several simple building parts� Furthermore� this modeling approach meets the problemsof incomplete results of low�level feature extraction� caused by occlusions� low contrast� noiseand disturbances�

We therefore propose a multi�level part�of hierarchy �cf� Figure �� It re�ects di�erent levelsof the envisaged semantic abstraction� The primitives of each aggregation level are specializedby an is�a hierarchy into subclasses�

Each primitive is described by its geometry� its domain speci�c role in terms of classmemberships and its relations to other primitives� The geometry is described by pose andshape parameters�

We currently employ four description levels for modeling complex buildings� which seemsto be sucient for a large class of buildings�

The �rst level �feature level� contains features F � namely attributed points P � lines Land regions R� Attributes for lines and regions� for instance� are the orientation classi�cationshorizontal �h�� oblique �o� and vertical �v�� Regions have an additional attribute describing itsrole� valid values among others are wall� roof and �oor� In general� the set of parameters isdivided into positional parameters on the one hand� describing position and orientation� andon the other hand form parameters� like width� height and length�

The second level �feature aggregate level� contains feature aggregates A which are inducedby points� lines and regions� and contain all their direct neighbors� Each aggregate is de�ned bya feature graph� given by a set F � ff�� fkg of features and adjacency relations R � F �F �A corner C� for instance� contains one point and all its adjacent lines and regions �cf� Figure ��

�

2l

r1

l1

0r

0l

r2

p

0

0

11

2

2

r

r

l

rl

l

P

Figure �� A corner is a feature neighborhood of a point �left�� Drawn as a graph� thearcs express the adjacency relation �center�� Assuming no occlusions and disturbances�its expected appearance in the image reveals the same neighborhood relations betweenthe corresponding �D�features �right��

The third level �building part level� contains building parts BP � Currently� we concentrateon the reconstruction of �D corners� Therefore� the building part models are de�ned as cornergraphs given by a set C � fc�� cng of corners and adjacency relations R � C�C� In futurework we will describe the building part models as graph structures which will also integrateline and region induced feature aggregates �s� the discussion in �� Each building part modelis also described as a parameterized volumetric object and has at least one so�called plug face

which is used for connecting building parts to each other� We discriminate terminals havingexactly one plug face and connectors with two or more plug faces �cf� Figure ��

Terminals: Connectors:

Figure �� Some examples of building parts� Plug faces� which are used to connect them�are drawn dashed�

The fourth level �building level� contains complete buildings� Buildings are de�ned asgraphs with building parts as nodes� the arcs representing pairs of building parts connected bycorresponding plug faces� Thus� the most simple building consists of two connected terminals�

�� Image Model

The �D image model describes the expected appearance of the building at the same levels ofaggregation as the corresponding �D�structures�� This guarantees a maximum of coherencefor representation and processing of �D� and �D�information�

The image model contains all properties which are invariant under projection and is taken tode�ne constraints� This especially holds for all class memberships� neighborhood and geometricrelations� as far as they are not disturbed by self occlusions� These self occlusions can bepredicted by an appropriate representation of the image model following the concept of an

�Actually the image model contains the raster image as the lowest� say �th level� from which the image

features are extracted� This lowest level is not shown as we do not explicitly refer to it�

�

aspect�based scheme �� E�g� a �D�corner� being a point induced image aggregate inheritsthe class membership of the corresponding �D gable corner� The neighborhood relation betweentwo faces of a roof in general can be expected to be transferred to a pair of image regions�whereas � assuming weak perspective � parallel �D�roof lines map to parallel �D�roof lines�Constraints of the higher aggregation levels are transferred to the primitives of the lower ones�if necessary�

Thus� the transformation from the object model to the image model needs at least thede�nition of an appropriate projective model� i�e� perspective or parallel projection� dependingon the distance between the sensor and the objects of interest� In fact� to take into account alle�ects which contribute to the process of image formation and feature extraction� there is aneed for models of sensor characteristics� illumination and low�level feature extraction methods�The results� presented in this paper� are derived by using a pinhole camera as sensor modeland assuming weak perspective projection� The potential role of the integration and usage ofan illumination model is discussed elsewhere �� and will only be sketched in section ��

The image model is currently used for both� the multi�view reconstruction of �D featureaggregates and the �D veri�cation of complete building hypotheses� But of course� an enhancedimage model can be used at each aggregation level� to utilize photometric attributes� to verifyintermediate aggregation results and to estimate certain parameters by monocular analysis�e�g� of shadows and wall projections ��

�� Strategy

Our input data is given as digital raster images with multiple overlap� Further informationabout the aerial image �ight like exterior and interior camera orientation and time stamp areused� The starting point of our analysis is the extraction of a polymorphic image descriptionconsisting of points P �D� lines L�D� and regions R�D and their mutual relations �� Itallows to derive point� line� and region neighborhood aggregates A�D� where the point inducedvertices V �D are the most promising ones for starting our analysis�

To cope with the combinatoric complexity of interpretation and reconstruction processes�a tight integration of �D and �D reasoning has proven to be successful �� Therefore�we decide to carry out the �D reconstruction process already for local feature aggregates� tomeet projective ambiguities and to allow an early domain speci�c interpretation utilizing �Dgeometry� Due to their relative stability against partial occlusions we select corners �classi�edvertices� as the basic class of feature aggregates for the reconstruction process� This selectionshould be regarded as a �rst choice� Future work will utilize a polymorphic reconstructionapproach on the feature aggregate level� which will also include line and region induced featureaggregates� i�e� wings and faces ��

Now� our current strategy consists of three main tasks� which are executed in sequence �cf�Figure �� and will be described in detail in section ��

�� D � �D� reconstruction of �D corners� To cope with the combinatorics of featureand building aggregation� we aim at an early integration of �D and �D reasoning� in this wayreducing the overall number of future hypotheses� This is done by extracting feature groups�namely vertices in the images� which are evaluated as projections of building corners with thehelp of a �D corner model� These �D corner hypotheses of di�erent images are used to derive �Dcorner reconstructions by multi�image parameter estimation� The �D corner reconstructionswill be interpreted by a domain speci�c �D corner model�

�� D� �D� generation of building hypotheses� The �D corners are used for indexing

�

Building

Viewparts

Vertices

Features 3D-Features

Corners

Buildingparts

3D2D

Views

Figure �� Information transfer within the wholeprocess� The close integration of �D and �D rea�soning is performed by an iteration loop� Thedashed arrow marks the initialization step� �D�reconstruction� generation and veri�cation ofhypotheses for building parts and buildings is re�peated� Veri�cation is based on generated viewsusing matching on the feature level�

into a library of parameterized building parts� which explain these corners� Within this indexingprocess� the parameter set of an indexed building part is instantiated due to the measured �Dgeometry of the corner reconstructions� Generally� more than one building part is needed toexplain all reconstructed �D corners� Thus� the indexed building parts must be aggregated tocomplete �D building hypotheses� This aggregation step is implemented by successive steps ofmerging and connecting building parts�

�� D � �D� veri�cation of building hypotheses� For veri�cation the �D buildinghypotheses are projected back into the original aerial images� Due to incomplete results offeature extraction� in general not all �D corners can be reconstructed� Therefore� not allparameters of building parts and the aggregated building hypotheses may be �xed� Thus� wecall the projected hypotheses parameterized views and implement these views in terms of amodi�ed aspect graph notation� Hypotheses are con�rmed respectively rejected by a votingprocess according to the degree of similarity resulting from the relational matching betweenthe parameterized views and the originally extracted image features� Furthermore� geometricreasoning in the images by applying projective geometry may derive previously undetectedimage features and determine free object parameters of the building hypotheses�

The successful sequence of matching steps results in an iteratively improved gain in knowl�edge� These three steps are repeated until no further hypotheses can be generated� The veri�edbuilding hypotheses lead to predicted unobserved �D primitives on the lower levels� giving ad�ditional information for reconstructing previously undetected corners� This initiates a furtheriteration of �� D�reconstruction� �� generation of building hypotheses and �� veri�cation�

� Models and Operations

In this section we describe in detail the three main steps of our approach to building recon�struction as proposed in �� i�e� �� D � �D� reconstruction of �D corners� �� D � �D�generation of building hypotheses� �� D � �D� veri�cation of building hypotheses �cf� Fig�ure �� Each step is presented by describing the underlying models and the operations ofreconstruction� aggregation and veri�cation�

�� D � �D� Reconstruction of �D Corners

This section describes the �rst two levels of our hierarchical aggregation approach� namelythe extraction and reconstruction of features and feature aggregates �cf� Figure �� We select

�

point induced corner features as the �rst choice for feature aggregates to be processed� Corners�a� reveal a high stability against occlusions� �b� their projections into the images give strongrestrictions during the correspondence analysis using epipolar geometry� and �c� corners haveproved to be suitable components for the proposed �D aggregation process�

Our experience with fully automatic subprocesses for the purpose of �D reconstruction hasshown a high stability while simultaneously using image patches with multiple overlap �cf�� Furthermore� we use the commonly available orientation data which de�ne geometricrestrictions during the correspondence analysis�

�� Models of Corner Features

The corner reconstruction is guided top�down by the �D corner model� with the corners beingparts of the hierarchical building model as described in chapter �� For reconstruction� weperform the transition from image space to object space on the level of feature aggregates �cf�Figure �� Therefore� we need to model corners in object space as well as their appearance inthe images by using the corner model in �D and �D�

�D Corner Model� The representation of corners is subdivided into the geometric infor�mation and the domain speci�c class labels� Each corner c�Di � �v�Di � �c� is geometricallydescribed by the vertex v�Di which is represented by its components on the feature level� beinga corner point� several lines and planar faces �cf� Figure �� and a corner class label �c � �C

out of the set of all possible corner classes �C �The corner classes are given by a two�level specialization hierarchy� which distinguishes

corners into subclasses �c� The partitioning into classes �c depends on di�erent geometricclass inherent constraints ��D�c � ��D

�c from the set of all possible constraints ��D�c of the class

�c� The �D corner c�D � �v�D� �c� results from implying the class dependent constraints ��D�c

onto the vertex geometry v�D�On the �rst level of the specialization hierarchy we use unary constraints for classi�cation�

They refer to single components of a corner� especially the corner components of type line� Weuse line attributes� given by the qualitative geometric labels horizontal �h�� vertical �v�� and�v�� and oblique �o�� and �o�� cf� �� depending on the slope of the lines� The sign denotesa positive �� or negative �� slope with respect to the corner point� We exclude meaninglesscorners like e�g� a corner with line attributes fh�h�hg� which make no sense in the context ofbuildings and their functionality�

This description is further re�ned by the second level of the specialization hierarchy� Ituses binary constraints which refer to the geometric relationship of pairs of corner compo�nents� Examples of binary constraints are symmetry of two lines with respect to the vertical�orthogonality of two lines and verticality of the plane� spanned up by two lines�

If there are no constraints attached to the corner� we call it the unconstrained corner withthe class label ��

Examples of di�erent corner classes are given in Figure �� The typical corner at the ridgeof a hip roof �corner no� �� is given by the line attributes fh�o��o��symmetry �o��o��g� Thedistinction between corner no� � and corner no� � is due to the constraint verticality�o��o�� onthe second level of the specialization hierarchy� whereas on the �rst level both corner classesare described by the line attributes fh�o��o�g�

�

gable roof

o- o-

h

v-

h

6

5

o+ho-

h

o-

v-

o+h

7

8

hip roofh

h

4

3 v-h

h

v+

prismatic roof

hh

h h

v+

v-1

2

rectangular flat roof

Figure �� examples of cor�ner classes �c� which are su�cient for describing the buildingtypes rectangular �at roof� pris�

matic roof� gable roof and hip

roof� are shown�

�D Corner Model� The �D corner model is fundamental for the access to vertices in theimages� which are suitable for the corner reconstruction� Because of the higher expressivenessof the �D vertex geometry in contrast to �D� the assignment of a corner class label �c to thegeometric vertex elements can be much more easily be performed in object space than in imagespace�

Therefore� we propose using a �D corner model without any distinction into subclasses to�nd point induced vertex aggregates V �D which have the structural characteristics of cornerprojections into the images �cf� Figure � �right�� These characteristics de�ne the vertex class�v� We statistically model the probability of a vertex being a projected corner� using theconditional probabilities P ��v j v

�D� of the vertex class �v given the vertex observation v�D�The conditional probabilities are used for classifying point induced feature aggregates being avertex� Please note that the transition of vertices to corners in addition to the transition toobject space� requires the assignment of a class label �c of a corner class�

�� Operations� Construction and Veri�cation of Corner Hypotheses

The construction of corner hypotheses starts with the analysis of �D vertices V �D� The verticescan be directly derived by using the feature adjacency graph �cf� section ��

By correspondence analysis using the images i � I� we derive the vertex correspondenceset �v�Di � �v�D� � � � � � v�DI �� which is then used for the transition to �D vertices V �D� Theyrepresent the geometry of corners C�D and thus serve for the subsequent interpretation byassigning the vertex v�D to a corner class �c� The result of this step are corner hypothesesc�Di � �v�Di � �c�i��

In the following� we give a detailed description of the �D vertex� and �D corner generationsteps�

�D Vertex Generation� In the �rst step the search for corresponding vertices is guidedby a priority list of �D vertices V �D which is build up by evaluating their suitability forthe correspondence analysis and reconstruction� The evaluation is performed by statisticalclassi�cation� based on the �D corner model� The priority list is given by minimizing theinformation I��v j v

�D� for the vertex class �v while observing the vertex v�D� The informationI��v j v�D� is derived using the probability P ��v j v�D� with I��v j v�D� � � lnP ��v jv�D�� ln ��

The conditional probability results from P ��v j v�D� � P ��v j b� c� using discrete and

continuous vertex characteristics b and c � Bayes theorem breaks it down into

P ��v j b� c� �

Qnbk�� P �bk j �v� � p �c j �v�Qnb

k�� P �bk� � p �c�� P ��v��

�

With the discrete characteristics bk with k � �� nb and the continuous characteristics cjwith j � �� nc� The latter are assumed to be uncorrelated with mean �cj and variance ��cj �This leads to

I��v j b� c� ��

� ln �

�� nbXk��

�� lnP �bk j �v� !

ncXj��

�cj � �cj�cj

��

The characteristics consider the stability� uniqueness and structural richness of the vertex�Actually� we use the number of lines and regions as discrete characteristics and the length ofcorner lines as continuous characteristics�

The second step during the search for corresponding vertices evaluates the structural sim�ilarity of matching candidates by cost minimization� In addition to the information of singlevertices� a score value describing the similarity of vertex components using epipolar geometryis introduced into the cost function�

Based on the correspondence tuple �v�Di � the transition to object space is performed by ajoint forward intersection of the corresponding image feature components points �p�Di and lines�l�Di of the vertex correspondence tuple �v�Di using all images i � I simultaneously� Epipolargeometry once again gives geometric restrictions� which facilitate the matching of the features�

predicted vertex

vertex acces by following reconstructed lines

reconstructed corner

Figure �� The information propagation for selecting and predicting vertices for the nextreconstruction step is shown� The upper left image visualizes one reconstructed corner�The other images show the reduced search space for vertices which are used for the nextcorner reconstruction� The access to these vertices is given by the point�line adjacencyrelation de�ned by image lines which are used for the reconstruction of the previouscorners� The right image shows the top�down prediction of a vertex� generated by lineintersection where no vertex access via the feature adjacency graph was possible�

The search space for the next correspondence set is de�ned by the neighborhood relationsof the corresponding image features of the reconstructed features F �D using the prolongationof lines and the neighborhood of regions� For instance� we have to expect further corners inthe prolongation of reconstructed �D lines� Therefore� we �rst follow line features of recon�structed vertices for directly selecting a set of vertex aggregates V �D which serves for the nextreconstruction step �cf� Figure �� In case no appropriate vertices are found� the neighborhoodof regions are used for vertex access by analysing the region�induced aggregates�

��

In addition to vertices� which are directly derived by using the feature adjacency graph�we also perform a top�down prediction of additional vertices� Depending on a selected pair ofpossibly corresponding vertices �vi� vj�� we derive the �D position of the vertex point by forwardintersection which we propagate to the expected vertex positions in the remaining images byreprojection� At these positions we can then perform a well�aimed search for line intersectionsto generate additional �D vertices� The prediction helps to bridge the incompleteness of theextracted feature adjacency relations without unnecessarily enlarging the search space�

�D Corner Generation� Establishing corner hypotheses C�D uses the �D corner modelby following the specialization hierarchy of corners� We perform a hierarchical interpretation�starting with the set of all corner classes �C � It is reduced in two steps to ��

C and ��C

by testing possible class inherent constraints ��D�c � ��D�c � The reason for performing a two�

step interpretation� using the corner specialization hierarchy� is to prevent the instantiationof mutually dependent� redundant or contradicting constraints� which is fundamental for thesubsequent veri�cation by parameter estimation �cf� ��

The �rst interpretation step tests the generated �D vertices V �D for unary constraints�Therefore� the corner components of type line are analysed for their qualitative geometriclabels being h� v�� v�� o� or o�� They are used to obtain the subset ��

C � �C of possiblecorner classes� In the second interpretation step� for each element of �c � ��

C possible binaryconstraints like e� g� symmetry and orthogonality are tested� Thus� we yield a further reduced��C � ��

C � where each element in ��C is an admissible interpretation of the vertex v�D� The

corner specialization hierarchy for example ensures that we do not test the line pair �h� v��

for orthogonality because the orthogonality constraint is implicitly contained within the linelabels and may lead to redundant or even contradicting constraints�

If the test accepts no constraints� ��C is the empty set and the underlying vertex represents

an unconstrained corner of class ��Depending on the selected constraints� we complete the reconstructed vertices to corners

by using the most probable corner class �c which contains the selected constraints� In thesimplest case we just index into the corner classes �cf� Figure �� In general� we may evenhypothesize additional corner components of feature type line L to complete the vertex� Forexample� if we observe a reconstructed vertex of node degree � with line labels h and o� and theconstraint orthogonal�h�o�� we perform a model�based prediction of a corner of node degree� of class fh�o��o��symmetry�o��o��verticality�o��o��g� as it occurs for the corner at the ridge ofa gabled roof building �cf� corner no� �� Figure �� For a more general approach� a prioriprobability distributions for the di�erent corner classes can be learned as proposed in �� andcan be introduced to decide for the most probable corner class �c�

The �D corner generation may lead to several corner hypotheses c�Dj � �v�D� �c�j� for each

vertex reconstruction v�D which have to be resolved in the veri�cation step�

Veri�cation of Corner Hypotheses� For veri�cation of the corner hypotheses� we performa second rigorous classi�cation by statistical analysis� This is formulated as an optimizationproblem for �nding the best interpretation "c of the data �v�Di from all possible corner interpre�tations cj � �v�D� �c�j� with �c�j � ��

C � Using Bayes theorem and neglecting the denominatorby normalization� we can break down the conditional probabilites� which leads to

"c � argmaxcj

P �cj j �v�Di � � argmaxcj

P ��v�Di j cj�P �cj� ��

��

Figure �� Examples of �D corner reconstruction� The left column shows the featureaggregates marked in one image� The right column visualizes the reconstructed �Dcorners with each corner plane shown as triangle �please note that the corner planesare open ended� the drawn lines solely support the visualization��

The a priori probability P �cj� for the corner class �c�j can be obtained empirically by learning�� and can in principle be integrated to evaluate the model�dependent in�uence on the result�

The conditional probability P ��v�Di j cj� evaluates how good the corner class instantiation�ts the observed image features points P �D and lines L�D which are contained in the vertex�v�Di � Introducing the unknown class speci�c parameters �j which depend on the geometricconstraints ��c of the corner class �c�j� equation � is transformed to�

"c� "� � argmaxcj

P ��v�Di j cj��j�P ��j j cj�P �cj� ��

� argmaxcj

P ��v�Di j �j�P �cj� ��

As we suppose the unknowns �j to be prede�ned by the model� the probability P ��j j cj� isconstant and eq� � can be reduced to eq� ��

For each hypothesis� we estimate the geometric parameters by a maximum likelihood pa�rameter estimation using all supporting image features� The matching features are selectedby backprojecting the instantiated corner model into the images� Thus� we may get accessto features which were not contained in the selected vertices� Assuming the expected valuesE�y� of the observations being a function E�y� � f�� of the geometric parameters �� theevaluation can be derived from the residuals y� "y of the optimal estimation "y � f�� We usethe probability density function p��v�Di j �� in case the features exist and have been successfullymatched to the model�

p��v�Di j ��

��u��det�yy��e��

�

��y� �y�T��

yy �y� �y��

where u is the number of unknowns in � and #yy is the covariane matrix of the observationsy�

��

To decide for the optimal interpretation and its corresponding constraints which de�ne theunknowns �� we �rst test for acceptance of the reconstruction of the unconstrained corner ofclass �� with ��D

�� If this test is successful� we test for the class speci�c constraints ��D

�c �Both tests use a Fisher distributed test value� depending on the weighted sum of the squaredresiduals� � � �y � "y�T��

yy �y � "y� of the maximum likelihood parameter estimation�Table � gives two examples of the estimation using � images simultaneously� Example �

uses constraints of the corner class no� �� example � of the corner class no� � �cf� Figure �� asselected during corner hypotheses generation�

n u r �� x�m �y�m �z�m

Example � no constraints �� constraints ��

Example � no constraints �� constraints ��

Table �� The results of the parameter estimation for two examples using the classdepending constraints or no constraints alternatively are given in comparison� In bothcases the constraints are accepted� The precision of the reconstructed vertex point isincreased when using the constraints for stabilization� �n is the number of observations�u the number of unknown parameters and r the redundancy of the estimation system��

According to the second test the increase of the internally derived precision � is notsigni�cant� Thus� in both cases the class speci�c constraints ��D

�c are accepted� The constraintsare useful for geometric stabilization and for improvement of the accuracy of the reconstructionas shown with the decrease of the empirical standard deviation �x� �y and �z of the three cornercoordinates �x� y� z� �cf� Table ��

Further details of the corner reconstruction approach can be found in �� The result of the estimation are evaluated corner reconstructions c�D � �v�D� �c� con�

strained by ��D�c � They form the basis for the �D aggregation and generation of building

hypotheses as presented in the next section� After the �rst iteration of the whole buildingreconstruction process� we want to use newly generated �D corners C�D �cf� section ��in addition to the original vertex data and thus expect an increasing number of reconstructedcorners�

�� D � �D� Generation of Building Hypotheses

This section describes the last two levels of our hierarchical aggregation approach in �D� namelythe reconstruction of building parts and building aggregates� The indexing into a library ofparameterized volumetric building parts uses the �D corner hypotheses derived at the featureaggregate level� Building parts are merged and combined successively to complex buildingaggregates describing complete building hypotheses�

In this section we describe in detail our models of �D building parts and building aggregatesand the operations for the selection and aggregation of building parts to complete �D buildinghypotheses�

�Please note that � in this context is used for a statistical measure in contrast to �c being the set of possible

corner classes �c � �c�

��

�� Models of Building Parts and Buildings

Given a set C�D � fc�D� � � � � � c�Dn g of n reconstructed corner aggregates� one or more buildinghypotheses have to be generated that explain all corners in the following sense� A corner iseither part of the building hypotheses or is explicitly classi�ed as not being part of it� If acorner is part of a hypothesis� this corner must match the hypothesis geometry and structure�

We use a component based approach for the modeling and construction of buildings� Build�ing part models are selected and instantiated according to the reconstructed �D corners� Sub�sequent aggregation operations combine these building parts to more complex ones and �nallyto complete buildings� In the current state of our work the set of building part models is buildup manually� In the future� learning schemes as described in �� will be used�

Plug Faces Terminals Connectors� Each building part contains one or more so�calledplug faces where it can be connected to other building parts� According to the number ofplug faces the set of building part models is classi�ed into terminals having one plug face andconnectors having two or more plug faces �cf� Figure �� A type is assigned to each plug face�which represents its geometry and topology� Two building parts may be connected to eachother via two plug faces only if they are of the same type� A building part is called open�if it contains one or more plug faces and closed if it contains no plug face� A �nal buildinghypothesis consists of one closed building part�

Parameterization of Building Parts� Building part models are parameterized by poseparameters determining the location and orientation and by shape parameters determiningfeatures like slope of the roof� width and height �cf� Figure �� If a parameter has an assignedvalue it is called �xed� otherwise free� Constraints on the parameters restrict them to validvalues�

v5

l5

l4

l3l2

l1

v4

v3

v2

hrv1

w

hs

zy

x vi �

� xi

yizi

A

vi xi yi ziv� !w � �v� !w !hs �v� � !hs ! hr �v� �w !hs �v �w � �

Figure �� Parameterised description of a hip roof terminal� The table on the rightshows the coordinates of the vertices v�� v�

Closed building parts that cover all corner observations are the requested building hy�potheses� In general� there will be more than one building hypothesis for a given set of cornerobservations�

�� Operations� Selection and Aggregation of Building Parts

The generation of building hypotheses is performed with three basic types of operations�Indexing selects and instantiates building part models from a library of building parts� Thetwo aggregation operations of Merging and Connection combine existing building parts to

��

more complex ones� Closed building parts generated in this process� which cover all cornerobservations� are called building hypotheses�

First we describe these three types of operations in detail� Then we propose our strategyof organizing these operations to derive �D building hypotheses in an ecient way�

Indexing Merging and Connection�

Indexing�c� into the library of building part models yields one partially instantiated buildingpart for the reconstructed �D corner c� Location and orientation of the reconstructedcorner c will also �x position and orientation of the indexed building parts� Also� itsparameters of shape will be determined as far as possible� In general there are severalcompeting building part models and several di�erent instances of each of these modelsthat match a single corner� Therefore� subsequent invocations of Indexing�c� yielddi�erent building parts�

We say that the corner reconstruction c is covered by the instantiated building part�

Indexing �nds for a given corner observation ci all possible mappings to all corners cjof all building part primitives� such that one such maps ci s edge indices onto cj s edgeindices with the following properties�

Every edge of ci is mapped onto one edge of cj�

The model corner cj may have more edges than ci� but not less� due to possiblyunobservable edges in the images�

The labels of a model edge and an observed edge identi�ed by must be the same�

If for a pair of model corner cj and corner observation ci such a mapping is found� it im�plies a set of equations� which determine location� orientation and some form parametersof the instantiated building part template�

Merging�b��b�� joins two building parts b� and b� of the same type� which show the samelocation and orientation and a consistent setting of the �xed parameters of shape �cf�Figure �� All corners covered by b� and b� are also covered by the resulting buildingpart�

Connection�b��b��f��f�� is the aggregation of the two building parts b� and b� via their plugfaces f� and f�� The two plug faces must have the same type and their orientation

o-

o-

h

o-o-

h

Figure �� Indexing establishes a link between corner observations �left� and buildingpart primitives �center� resulting in an instantiated building part �right��

��

must be opposite to each other� The aggregation is performed by �glueing� f� to f��cf� Figure� �� Edges of the two building parts� which meet at f� and f�� are joinedtogether�

A new length parameter is introduced� that allows these new edges to shrink or expand�Parameters that de�ne the shape of the plug faces are uni�ed� If parameters which are tobe uni�ed have �xed values� they may di�er only by a small amount� This condition maybe used for an early rejection of the aggregation of two building parts� if for example theposition and orientation of two building parts do not �t� they must not be aggregated�

All corners covered by b� and b� are also covered by the resulting building part�

Figure �� Left� Two saddle roof terminals are merged together� Parameters areadjusted� Right� Two saddle roof terminals are connected together� The resultingbuilding part is closed�

Each operation is followed by a maximum likelihood parameter estimation in form of aleast�squares �tting� In case of a merging or connection operation� this parameter estimationdetermines whether the parameters of the two participating building parts are compatible ornot� In case of an indexing operation it determines initial values for position and orientationand some of its form parameters�

Strategy for Selection and Aggregation� The straightforward procedure would startwith a set B that contains all instantiated building parts resulting from indexing� Iteratedaggregation steps generate from a set Bi a new set Bi�� Bi $�Ni� where Ni contains all possiblebuilding parts that are constructed by merging and connection of building parts in Bi� Note�that only building parts will be constructed which are not already in Bi� Therefore Bi and Ni

are disjoint� The iterations stop after n steps� because there are only n corner observations anda building hypothesis may not contain more than one building part per reconstructed corner�The number of hypotheses contained in Bn is exponential in n�

We use an approach that employs a heuristics that typically allows the construction of thebest building hypothesis in a few steps while the construction of all closed building hypothesescovering all corner observations is still exponential� This is achieved by using a priority queueto order the operations Indexing� Merging� and Connection� As long as the queue is notempty� the �rst operation in it is performed and removed from the queue� Afterwards newoperations are inserted into the queue� The process terminates if the priority queue is emptyand no new operations can be inserted� All closed building parts generated in this process thatcover all corner reconstructions are called building hypotheses�

Organization and Execution of the Priority Queue� The strategy which leads to a fastconstruction of hypotheses is expressed by the following construction principles�

��

S� Indexing is performed before any aggregations take place� This provides the starting pointfor the algorithm�

S� Aggregation is done depth �rst� Construct building aggregates with as much covered cornerobservations and minimal values for the least�squares �tting as soon as possible�

S� The growth of aggregations shall be as uniform as possible� This avoids having one ker�nel steadily growing by connecting small building parts to it and thus neglecting otherpossible hypotheses�

These construction principles de�ne how the operations are ordered within the priorityqueue and which operation is to be performed next� Let � and � be two operations�

�� If � and � have di�erent types� Indexing comes before Merging before Connection �S��

�� If � and � are both of type Indexing� then

�a� if the operations belong to di�erent corner observations� the corner having lessbuilding part interpretations is preferred �S��

�b� else if the operations belong to the same corner observation� the operation with ana priori better expected result is performed �rst �S��

�� If � and � are both of type Merging or Connection� with the argument building partsbi�� and bi�� of i� the following attributes are considered�

�a� The operation i with the higher number of corner observations covered by bi�� andbi�� is performed �rst� building aggregates with as much covered corner observationsas possible are generated �rst �S��

�b� The operation i with the smaller di�erence in corner observations covered by bi��and bi�� is performed �rst �S��

�c� Finally� the operations i are sorted according to the sum of the values of thefunction minimised in the least�squares �tting performed after the construction ofbi�� and bi�� building aggregates that �t the observed corners best are constructed�rst �S��

Algorithm� The set of building hypotheses H and the set of building parts B are initiallyempty� The priority queue is initialised with all indexing operations for each corner� As longas the queue is not empty� the �rst operation in it is removed and executed� If its result� thebuilding part b� is closed� then if it covers all corner observations it is a new building hypothesisand is added to H� If b does not cover all observations� it is removed� If b is open� new entriesfor the priority queue are constructed�

If b is an instance of a building part model� then for the set fb�� bkg � B of existingbuilding parts� which are instances of the same building part model and have the sameorientation �with respect to uncertainty�� the operations Merging�b�bi�� i k� areinserted into the priority queue�

For each plug face f of b and for each plug face f � of another building part b� � B� iff and f � are of the same type and their orientation is opposite to each other� insert theoperation Connection�b�b��f�f �� into the priority queue�

��

An operation is inserted only if its result is not yet a member of B or H and if it covers everycorner observation at least once� After all possible operations are inserted into the queue� b isadded to B�

If no new operations can be inserted and the priority queue is empty the algorithm termi�nates� If H is not empty� it contains all the generated building hypotheses� Otherwise� it hasnot been possible to generate a closed building hypothesis� If there exist closed building partsin B which do not cover all corner observations� the one which covers the most observations ischosen as building hypothesis� If all building parts in B are open� one building part is selectedfrom B and its open plug faces are closed by connection to default terminals� The buildingpart selected� is the one which consists of the most instantiated building models and the fewestplug faces and �ts the corner observations best�

Example� Figure �� shows an example� Six corner reconstructions are shown on the left� Af�ter all indexing operations and two merging operations are performed� the best scored buildingparts are shown in the middle� The building part in the front covering three corner recon�structions is the result of the two merging operations� The visible three terminals and the oneconnector are aggregated to the building hypotheses shown on the right in three connectionoperations� Please note that there are no corner reconstructions at ground level� and thereforethe height parameter is undetermined� Only for visualisation a default value has been chosen�

Figure �� On the left the given corner observations are shown� The �gure in the middleshows the best �tting building parts after all indexing steps and some merging steps�building part in the front�� These building parts are aggregated in three operationsresulting in the building hypothesis shown on the right�

�� D � �D� Veri�cation of Building Hypotheses

This section describes the last two levels of our hierarchical aggregation approach in �D� namelythe image models of building parts and building aggregates� Each aggregation level can beaccompanied by corresponding aspect descriptions �cf� �� for intermediate hypotheses veri��cation and parameter estimation by image analysis� Due to the high complexity of hypothesesveri�cation for complex building shapes� we employ constraint satisfaction and propagationto verify hypotheses in an ecient way� We have currently implemented our veri�cation pro�cedure for complete�but also only partially instantiated�building hypotheses� Obviously�our veri�cation procedure can also be applied for the veri�cation of intermediate aggregationresults�

��

Thus� in this section we describe the generation of the image models for the �D hypothesesof buildings and building parts and the constraint�based reasoning for hypotheses veri�cationand the derivation of new image features�

�� Image Modeling by Parameterized View Hierarchies

To generate image models of building parts and aggregates� we employ a modi�ed version ofthe aspect hierarchies �� Aspect hierarchies describe qualitative image models of volumetricprimitives in terms of aspects� Each aspect is decomposed into a hierarchical feature�baseddescription� to facilitate the recognition of primitives even in the case of partial occlusion bythe detection of its visible features�

While aspects represent qualitatively di�erent object appearences due to di�erent viewpointpositions� di�erent appearences of our building hypotheses are caused by di�erent settings offree shape parameters� The parameters of location and orientation are usually �xed due to the�tting with the �D corner reconstructions� If all parameters of a building hypothesis are �xed�exactly one view is generated due to the �xed camera position and orientation� If one or moreparameters are still undetermined� the valid parameter space is sampled and several views areinserted �see Figure �� Due to the reliable processes of feature extraction and reconstruction�the number of free parameters is generally low�

To facilitate hypotheses veri�cation� even in the case when occlusions and noise causeincomplete feature extraction� we adopt the hierarchical feature�based representation of theapproach of Dickinson et al� �cf� �� The image models of the hypotheses of buildingsand building parts are called parameterized view hierarchies due to their independency fromviewpoint� For each aerial image one parameterized view hierarchy is generated� It consists ofthree levels�

Level �� The highest level contains all three�dimensional building hypotheses� Note that ahypothesis may still have undetermined parameters�

Level �� The medium level contains all relevant two�dimensional views of these hypotheses�

Level �� Each view is decomposed into its image features and their relations like parallelisms�symmetries etc� which describe the lowest level of the hierarchy�

Figure �� shows an example of a view hierarchy for four di�erent building hypotheses forthe same set of corner observations �cf� Figure �� left��

The image model currently employed for hypotheses veri�cation� utilizes a pinhole sen�sor model with weak perspective projection of the visible object contours� Obviously� moreenhanced models of sensor characteristics� illumination and physical properties of building ma�terials allow further analysis for determining shape parameters and verifying intermediate andcomplete results of the aggregation hierarchy� Currently we are investigating view representa�tions which employ a standard lighting model including ambient light and di�use re�ection�where the sun vector is computed from sensor model and time stamp� The analysis of illu�mination and shadows allows to derive �D object parameters and to test the consistency ofshadow�ground transitions and inter�surface intensities �cf� ��

�� Operations for Veri�cation and Parameter Estimation

Hypotheses veri�cation relies on the matching between the image models of complex buildinghypotheses on the one hand and extracted image features of the aerial images on the other

��

Figure �� Example of a view hierarchy� For one set of corner reconstructions fourdi�erent building hypotheses have been generated due to two di�erent indexing resultsinto the library of building parts for the leftmost ridge corner and the free parameterof height �no ground corner was reconstructed��

hand� Since this matching process considers objects and their interrelationships� this task isan instance of relational matching �� We employ constraint solving methods �� to handlethe exponential combinatorial complexity� permitting exhaustive search for the optimal match�

Matching between image features and image model yields a grouping of fragmented lines anda model�based interpretation of each matched image feature� Then� previously not extracted�D corners are detected by geometric reasoning with robust estimations of intersections ofidenti�ed lines� These �D corners are additional information that is used to reconstruct furtherpreviously undetected �D corners in the next iteration of the whole building reconstructionprocedure�

The transformation of building hypotheses to a constraint satisfaction problem �CSP��the matching and its implementation using constraint logic programming �CLP� �� isexplained in detail in ��

Transformation of View Hierarchies to Sets of Constraints� A view hierarchy enu�merates the possible views for the building hypotheses with respect to one image� To evaluatethe correspondence of these image models with the originally extracted image features� thematching between the view hierarchy and the image features is done on the lowest level ofour model hierarchy shown in Figure �� The entities considered for matching are objects of

��

the three feature classes points� lines� and regions and di�erent relations� e�g� adjacency� lineparallelism� region symmetry� and region contrast� The line parallelism and region symmetry re�lations re�ect the respective �D properties in �D� For each pair of adjacent regions the contrastrelation expresses their expected intensity ratio�

Thus� we employ a representation of the image model� which describes each view as a setof �D image features and a set of relations between them� These relations may be regarded asa set of constraints which have to be satis�ed simultaneously by the corresponding objects� Aconsistent match� also called consistent labeling �� is an identi�cation of a set of extractedimage features which satisfy all constraints� The decomposition of the matching problem intothe simultaneous satisfaction of di�erent constraints leads to a structure CSP�V�D� C� whichis adequately represented by the translation of hypotheses into conjunctions of constraints Cwhere the features are represented by variables V with restricted domains D� Figure �� il�lustrates the constraint representation of views of building hypotheses using constraint logicprogramming� For each view of a hierarchy� one corresponding constraint representation willbe employed�

house_aspect(F1,F2,F3,L1,...,L10,Sum) :-

extracted_blobs(BlobDom), [F1,F2,F3 ] ∈ BlobDom ∪ {*},

extracted_lines(LineDom), [L1,...,L 10] ∈ LineDom ∪ {*},

[B1,...,B19] ∈ [-1,0,1],all_distinct([F1,F2,F3]),

all_distinct([L1,...,L10]),

adjacnt(F1,L1,B1), adjacnt(F1,L2,B2), adjacnt(F1,L3,B3),



adjacnt(F3,L5,B10),adjacnt(F3,L7,B11),adjacnt(F3,L9,B12),

adjacnt(F3,L10,B13),

line_parallel(L2,L3,B14), line_parallel(L6,L7,B15),



Sum = B1 + B2 + ... + B18 + B19.

F1

F2 F3

L1

L3 L2

L4 L5

L8 L9

L6 L7

L10

Figure �� Constraint representation of a building hypothesis for a saddleback roofhouse� The adjacnt�constraints express the topological adjacencies between lines andregions� The indicator variables B�� B�� re�ect the status of each constraint� ��means violated� � relaxed� and � satis�ed �see chapter ��

Matching of Constraint Sets to the Image Data� The matching is done separatelyfor every aerial image and view model by searching for a valid assignment of extracted im�age features to the variables of the corresponding CSP such that all constraints are satis�ed�Consistency techniques� like forward checking and look ahead� as described in �� exploit the structured representation of the matching problem as a constraint network� Eachconstraint is used to restrict the domains of its variables and these reductions are propagatedthrough the network� This causes early prunings of the search tree and in most cases dramat�ically reduces the computational e�ort�

These standard techniques for constraint solving demand that a match satisi�es everyconstraint� Due to occlusions and disturbances� often neither every predicted model feature ofa view can be found� nor are all their incident constraints satis�able� Thus� the resulting CSPs

��

are in general over�constrained �� The approaches of MaxCSP as proposed by Freuder andWallace �� or inexact graph matching described by Haralick and Shapiro �� handlethis by allowing the relaxation of constraints� The best matching then is de�ned as the onewhich satis�es the maximum number of constraints�

The main problem of this metric lies in the inappropriate representation of unobservabilityand is discussed in detail by Vosselman �� If an expected model feature is missing in theimage� the corresponding variable has to be assigned a wildcard value� and all its incidentconstraints have to be relaxed in order to get a solution for the CSP� However� these costs arenot correlated to the importance of the model feature�

To overcome this problem� we also permit the explicit elimination of variables� The domainof every variable is extended by the wildcard � � which is assigned to a variable if the respectivemodel feature could not be found in the image� We further distinguish between the relaxationof constraints due to the unobservability of an incident feature and due to single constraintviolation� Therefore� we extend every constraint c�t�� tn� of the model by a variable b withthe three possible values �� or � to a constraint c��t�� tn� b� where

c��t�� tn� b� � �b � �� b � �� c�t�� tn�� b � �� c�t�� tn��

In the framework of constraint logic programming �CLP�� variable b is used both as an indicatorand a switch� If the original constraint c is satis�ed� b is set to �� If it is violated� b is set to�� If b is set to � �or �� c has to be satis�ed �or violated respectively� to ful�ll c� and thusc� will be replaced by c �or �c�� If a wildcard is assigned to a variable vi � V� every incidentconstraint c�j�� vi� � � � � bj� � C

� has to be relaxed by setting bj � �� If the wildcard is removedfrom the domains of all variables v�� vn of a constraint c��v�� vn� b�� the value � can beremoved from the domain of b� Vice�versa� if b is set to �� the wildcard has to be removed fromthe domains of the variables v�� vn�

So the original constraint satisfaction problem CSP�V�D� C� is transformed to a problemCSP �V�D�� C�� with D� � D � f�g and C� � fc��t�� tn� b� j c�t�� tn� � C� b � f�� gg�To �nd the best mapping we maximize

f�C� �

nXi��

bi with c�i�� bi� � C� ��

using a branch�and�bound algorithm embedded in the CLP system �� During the search�the lower bound � is added as a constraint f�C� � � to the constraint system� From thisconstraint the CLP system can infer additional knowledge about the values of the variables bi�For example� if we have eight constraints� six of which have already been �xed �four to � andtwo to �� and � is �� the �nal two constraints have to be satis�ed to reach the lower boundand so the bi for the remaining two constraints can be set to �� enforcing both constraints�

The solution of the constraint satisfaction problem consists of valid assignments of ex�tracted features to the variables� On one hand� these assignments determine the free geometricparameters of the features of the image model and thus determine free parameters of the wholebuilding hypothesis� On the other hand� every assigned image feature is identi�ed as a speci�ccomponent of the building hypothesis� These matched image features are labeled with respectto the building model� Figure �� shows highlighted on the left the image features that weresuccessfully matched with the image model of building hypothesis of a saddle roof house�

Since lines in general are fragmented into several line image features� corresponding linefeatures are grouped and �tted using the labeling information to determine the lines of theimage model of the building hypothesis�

��

Figure �� Left� Image features which are successfully matched with the image modelof the building hypothesis� Right� �D corners� predicted by the matched image modeldepicted by circles and line intersections�

Generation of New �D Corners� The solution of the CSP in general �xes further param�eters of the building hypotheses� It should be noted� that free parameters may only remain� ifsome constraints or variables had to be relaxed� Corners of the building hypotheses with �xedpositions that have no corresponding vertex structure in the image are �nally detected in �D�as illustrated by the right picture in Figure �� to provide new information for the �D cornerreconstruction in the next iteration of the reconstruction process�

As mentioned above� the loop of �D reconstruction� generation and veri�cation of buildinghypotheses will be terminated if no further hypotheses can be generated� After the �nal veri��cation step� the parameters of the �D building model will be determined simultaneously basedon the observed image features and taking all geometric and possibly radiometric constraintsinto account�

� Results

The current result of the reconstruction procedure is presented for the international test dataset which was distributed from the ETH Z%urich for the Ascona Workshop �� on Automatic

Extraction of Man�Made Objects from Aerial and Space Images �cf� �� The image scale is� � �� Due to the fact that our feature extraction currently yields too many spurious imagefeatures �e�g� roo�ng tiles� at a resolution of ��m� we use a resolution of ��m pixel size�As building h�� is under construction� only �� out of the �� buildings which are contained inmultiple images� are relevant for the analysis� To test the feasibility of the concept in a �rstinstance the number of building primitives is reduced to some few building part terminals andtheir possible connectors� which occur in the data set� To reconstruct corners at the footprintof the buildings� higher image resolution is necessary� Therefore� we propose to derive thebuilding heights by using a digital terrain model �� For visualization� we actually set thebuilding heights to �xed values� Figure �� shows the result after the �rst iteration loop �cf�Figure �� Results of the intermediate steps are listed in Table � and re�ect the followingaspects of the di�erent reasoning steps�

Reconstruction of �D corners I�� The number of reconstructed corners RC on averagecompounds �� of the buildings corners �C�� whereas � of them are incorrectly classi�ed� We

��

h12

h5

h4

h2 h8

h7

h9

h11

h6

h10

h3h1

Figure �� Visual�ization of the recon�struction result for theAscona dataset afterone iteration loop �cf�Fig� �� is performed�The images contain ��buildings in multipleoverlap� As buildingh�� is under construc�tion� we excluded itfrom our analysis� Weare able to completelyreconstruct all of the�� buildings whichwere relevant for theanalysis�

are not able to completely reconstruct every corner� because not all corners are observable inthe symbolic image descriptions� An application independent grouping as presented in �� may improve the initial symbolic image descriptions� Possibly still remaining unreconstructedcorners have to be identi�ed during the veri�cation step of building hypotheses� The reasonfor incorrect classi�cation is given by a weak intersection geometry for the line reconstruction�which is not yet considered in the modeling process� Neglecting the in�uence of the uncertaintyof the line reconstruction� the corner point is correctly reconstructed� Currently� the identi�ca�tion of wrong corner interpretations is performed by �nding global inconsistencies during thegeneration of building hypotheses� The parameter estimation for correctly classi�ed corners�using � images simultaneously� achieves an accuracy of the reconstructed corner point about�x � �y � ��cm and �z � ��cm� The accuracy of the orientation of the corners is about�� o� the accuracy of the slope of the roof is about �sl � �o�

Generation of building hypotheses II�� The column BPH in Table � gives the numberof building part hypotheses which are generated for each of the reconstructed single corners�The combination of these building parts results in the number BH of hypotheses of completebuildings which are consistent with all reconstructed corners� If only one corner is given� as itis the case for h� and h�� the corresponding terminals are each completed to a closed buildinghypothesis with a second terminal of the same type� For each of the BH building hypotheses theview graph is generated� Currently� the view graph of the best building hypothesis is passedto the veri�cation of building hypotheses� The best decision is given by using the MinimumDescription Length Principle following ��

Veri�cation of building hypothesesIII�� For each building in each image one viewgraph� which is a �D building hypothesis� is available and matched to the extracted image

��

I II IIIBuilding C RC CE BPH BH UV GV FP

h� � �� h� � � � � � � h� �� h� � �� h� � �� h� � � �� h� � � �� h � �� h � �� h�� h��

C number of corners of the buildingRC number of reconstructed cornersCE number of corner classi�cation errorsBPH number of building part hypotheses

for each corner

BH number of building hypothesesUV predicted corners �not previously

reconstructed� before veri�cationGV predicted and veri�ed vertices

per imageFP remaining free parameters

Table �� Detailed results of the intermediate reconstruction processes Reconstructionof �D corners �I�� Generation of building hypotheses �II� and Veri�cation of buildinghypotheses�III� for the Ascona dataset� All of the �� buildings can be reconstructed asshown in column FP�

features and mutual relations� Column UV lists the number of corners which initially could notbe �D reconstructed but were predicted by the generated building hypotheses� The columnGV lists for each of the � images the number of correctly matched vertices for the previouslyunknown corners� It shows that �� of the undetermined building corners could be correctlyidenti�ed by the veri�cation procedure� Since in general the building parameters are redun�dantly determined by the geometrical and topological constraints between the model features�all free parameters for every building hypotheses have been determined �see column FP�� Pleasenote that for buildings h� and h� the matching process also determined the formerly unknownlength parameter�

� Conclusions and Outlook

We have proposed a model�based approach to �D building extraction from aerial images� Ourmodeling concept reveals a tight coupling of generic �D object modeling and an explicitlyrepresented image model� Object and image model show corresponding part�of hierarchiesdescribing aggregation states within a recognition�by�components strategy�

The strategy aims at a step by step increase of knowledge during the reasoning� As thedomain speci�c object models of buildings and building parts de�ne the maximum achievableknowledge about an actual scene and its granularity determines the resolution for knowledge�our model spans hierarchically from the smallest observable features to complex building shapesand allows an adequate evaluation of hypotheses within all di�erent reasoning steps�

We have presented �rst experimental results� Currently� we are investigating and developingon

��

the measurement and propagation of uncertainty within the overall reconstruction pro�cess&

the extension of the current building modeling by more sophisticated knowledge aboutbuildings� esp� functional aspects&

the use of enhanced image models including sensor characteristics and lighting modelswithin hypotheses veri�cation&

the derivation of domain dependent heuristics to constrain search spaces&

the handling of incomplete observations within all processes and stages of hypothesesgeneration and veri�cation�

Especially� we have to consider the integration of logical and statistical knowledge into theframework of constraint logic programming� Furthermore� the approach has to be compared todi�erent automatic algorithms for segmentation� �D reconstruction and classical photogram�metry� In order to achieve this goal� an attempt is made to �nd a set of standardized repre�sentations about basic building models� parts of buildings and classes of buildings to ease suchcomparisons on a conceptual level�

Acknowledgments�

This work was largely done within the the project �Semantic Modeling and Extraction ofSpatial Objects from Images and Maps� especially in the subproject �Building Extraction�which is supported by the Deutsche Forschungsgemeinschaft� DFG� We thank the DFG forsupporting our work�

References

�� R� Bergevin and M�D� Levine� Generic Object Recognition� Building and Matching Coarse De�scriptions from Line Drawings� IEEE T�PAMI� � ��

�� I� Biederman� Recognition�by�Components� A Theory of Human Image Understanding� Psycho�logical Review� ��

�� F� Bignone� O� Henricsson� P� Fua� and M� Stricker� Automatic Extraction of Generic House Roofsfrom High Resolution Aerial Images� In Computer Vision � ECCV �� number �� in LectureNotes in Computer Science� pages � �� Springer Verlag� Cambridge� ��

�� K� Bowyer and C� Dyer� Aspect graphs� An introduction and survey of recent results� Int� Journalof Imaging Systems and Technology� ��

� � C� Braun� Interpretation von Einzelbildern zur Geb�audeerfassung� PhD thesis� Institut f�ur Pho�togrammetrie� Universit�at Bonn� ��

�� C� Braun� T� H� Kolbe� F� Lang� W� Schickler� V� Steinhage� A� B� Cremers� W� F�orstner�and L� Pl�umer� Models for Photogrammetric Building Reconstruction� Computer � Graphics��

�� A� Brunn� E� G�ulch� F� Lang� and W� F�orstner� A hybrid concept for �d building acquisition�ISPRS Journal of Photogrammetry and Remote Sensing� ��

��

�� A� Brunn� F� Lang� and W� F�orstner� A Procedure for Segmenting Surfaces by Symbolic and IconicImage Fusion� In J�ahne B�� editor� Mustererkennung �� DAGM�Symposium Heidelberg�Informatik aktuell� pages �� Springer Verlag� Berlin� ��

�� M� Clowes� On Seeing Things� Arti�cial Intelligence� ��

�� R� Dechter and J� Pearl� Network�Based Heuristics for Constraint Satisfaction Problems� Arti�cialIntelligence� ��

�� S� J� Dickinson� A� P� Pentland� and A� Rosenfeld� ��D Shape Recovery Using Distributed AspectMatching� IEEE Transactions on Pattern Analysis and Machine Intelligence� ��

�� S� J� Dickinson� A� P� Pentland� and A� Rosenfeld� From Volumes to Views� An Approach to ��DObject Recognition� CVGIP Image Understanding� ��

�� R� Englert� Systematic Acquisition of Generic �D Building Model Knowledge� In W� F�orstner andL� Pl�umer� editors� Semantic Modeling for the Acquisition of Topographic Information from Imagesand Maps� pages �� Birkh�auser Verlag� Basel� Switzerland� ��

�� A� Fischer and V� Steinhage� On the Computation of Visual Events in Aspect Graph Generation� InMustererkennung �� DAGM�Symposium Braunschweig� Informatik aktuell� pages � ��Springer Verlag� Berlin� ��

�� A� Fischer� T� H� Kolbe� and F� Lang� Integration of �D and �D Reasoning for Building Recon�struction Using a Generic Hierarchical Model� In W� F�orstner and L� Pl�umer� editors� SemanticModeling for the Acquisition of Topographic Information from Images and Maps� pages � ��Birkh�auser Verlag� Basel� Switzerland� ��

�� P� J� Flynn and A� K� Jain� CAD�Based Computer Vision� From CAD�Models to RelationalGraphs� IEEE T�PAMI� ��

�� W� F�orstner� A Framework for Low Level Feature Extraction� In J��O� Eklundh� editor� ComputerVision ECCV �� Vol� II� number �� in Lecture Notes in Computer Science� pages ��Springer Verlag� Berlin� ��

�� W� F�orstner� Mid�Level Vision Processes for Automatic Building Extraction� In A� Gr�un�O� K�ubler� and P� Agouris� editors� International Workshop on Automatic Extraction of Man�Made Objects from Aerial and Space images� pages �� Birkh�auser� Basel� ��

�� W� F�orstner and L� Pl�umer� editors� Semantic Modeling for the Acquisition of Topographic Infor�mation from Images and Maps� Birkh�auser Verlag� Basel� ��

�� E� C� Freuder and R� J� Wallace� Partial Constraint Satisfaction� In M� Jampel� E� Freuder� andM� Maher� editors� Over�Constrained Systems� number �� in Lecture Notes in Computer Science�pages �� Springer Verlag� Berlin� ��

�� P� Fua and A� J� Hanson� Resegmentation Using Generic Shape� Locating General CulturalObjects� Pattern Recognition Letters� ��

�� P� Fua and A� J� Hanson� An Optimization Framework for Feature Extraction� Pattern RecognitionLetters� ��

�� C� Fuchs and W� F�orstner� Polymorphic Grouping for Image Segmentation� In Proceedings �thInternational Conference on Computer Vision� pages �� IEEE Computer Society Press� ��

�� A� Gr�un� � E� P� Baltsavias� and O� Henricsson� editors� Automatic Extraction of Man�Made Objectsfrom Aerial and Space Images �II�� Birkh�auser� Basel� ��

�� A� Gr�un� O� K�ubler� and P� Agouris� editors� Automatic Extraction of Man�Made Objects fromAerial and Space Images� Birkh�auser� Basel� ��

��

�� E� G�ulch� A Knowledge�based Approach to Reconstruct Buildings in Digital Aerial Imagery�In L� W� Fritz and J� R� Lucas� editors� International Archives of Photogrammetry and RemoteSensing� volume �� B�� pages ��

�� C� Hansen and T� C� Henderson� CAD�Based Computer Vision� IEEE T�PAMI� ��

�� R� M� Haralick and G� L� Elliott� Increasing Tree Search E�ciency for Constraint SatisfactionProblems� Arti�cial Intelligence� ��

�� R� M� Haralick and L� G� Shapiro� Computer and Robot Vision� volume II� Addison�WesleyPublishing Company� New York� ��

�� M� Hendrickx� J� Vandekerckhove� D� Fr�ere� T� Moons� and L� Van Gool� On the �D Reconstructionof House Roofs from Aerial Images of Urban Areas� In International Archives of Photogrammetryand Remote Sensing� volume �� Part ��W�� pages �� ISPRS� ��

�� O� Henricson� F� Bignone� W� Willuhn� F� Ade� O� K�ubler� E� P� Baltsavias� S� Mason� andA� Gr�un� Project AMOBE� Strategies� Current Status and Future Work� In International Archivesof Photogrammetry and Remote Sensing� volume �� B�� pages ��

�� O� Henricsson and E� P� Baltsavias� ��D Building Reconstruction with ARUBA� A Qualitativeand Quantitative Evaluation� In A� Gr�un� E� P� Baltsavias� and O� Henricsson� editors� AutomaticExtraction of Man�Made Objects from Aerial and Space Images �II�� pages �� Birkh�auser�Basel� ��

�� M� Herman and T� Kanade� The �D MOSAIC Scene Understanding System� Incremental Recon�struction of �D Scenes from Complex Images� In M� A� Fischler and O� Firschein� editors� Readingsin Computer Vision Issues Problems Principles and Paradigms� pages �� Morgan Kauf�mann Publishers� Los Altos� California� ��

�� A� Heyden� Consistency and Correction of Line�Drawings� Obtained by Projections of PiecewisePlanar Objects� In Computer Vision � ECCV �� Proc� of the �rd Europ� Conf� on ComputerVision� number � in Lecture Notes in Computer Science� pages �� Springer Verlag� Berlin��

�� A� Huertas� M� Bejanin� and R� Nevatia� Model Registration and Validation� In A� Gr�un� O� K�ubler�and P� Agouris� editors� Automatic Extraction of Man�Made Objects from Aerial and Space Images�pages �� Birkh�auser� Basel� ��

�� D� A� Hu�man� Impossible Objects as Nonsense Sentences� Machine Intelligence� ��

�� K� Ikeuchi and P� J� Flynn� Recent Progress in CAD�Based Vision� CVGIP Image Understanding��

�� J� Ja�ar and M� J� Maher� Constraint Logic Programming� A Survey� Journal of Logic Program�ming� ��

�� M� Jampel� E� Freuder� and M� Maher� editors� Over�Constrained Systems� Number �� in LectureNotes in Computer Science� Springer Verlag� Berlin� ��

�� K� Kanatani� Group�Theoretical Methods in Image Understanding� Springer Verlag� Berlin� ��

�� J� J� Koenderink and A� J� van Doorn� The Internal Representation of Solid Shape with Respectto Vision� Biological Cybernetics� ��

�� T� H� Kolbe� L� Pl�umer� and A� B� Cremers� Using Constraints for the Identi�cation of Buildingsin Aerial Images� In Proceedings of the �� Int� Conf� on Practical Applications of ConstraintTechnology PACT�� in London� pages �� The Practical Application Company Ltd�� London��

��

�� T� L�abe and K��H� Ellenbeck� �D�Wireframe Models as Ground Control Points for the AutomaticExterior Orientation� In International Archives of Photogrammetry and Remote Sensing� volume�� B�� pages ��

�� F� Lang and W� F�orstner� �D�City Modelling with a Digital One�Eye�Stereo System� In Interna�tional Archives of Photogrammetry and Remote Sensing� volume �� B�� pages ��

�� F� Lang and W� F�orstner� Surface Reconstruction of Man�Made Objects using Polymorphic Mid�Level Features and Generic Scene Knowledge� Zeitschrift f�ur Photogrammetrie und Fernerkundung��

�� C� Lin� A� Huertas� and R� Nevatia� Detection of Buildings Using Perceptual Grouping and Shad�ows� In Proceedings of CVPR�� pages ��

�� C� Lin� A� Huertas� and R� Nevatia� Detection of Buildings from Monocular Images� In A� Gr�un�O� K�ubler� and P� Agouris� editors� Automatic Extraction of Man�Made Objects from Aerial andSpace Images� pages �� Birkh�auser� Basel� ��

�� C� L� Lin� Q� Zheng� R� Chelappa� L� S� Davis� and X� Zhang� Site Model supported Monitoringof Aerial Images� In Proceedings of CVPR�� pages ��

�� A� K� Mackworth� Consistency in Networks of Relations� Arti�cial Intelligence� ��

� � D� M� McKeown� Towards Automatic Cartographic Feature Extraction from Aerial Imagery� InH� Ebner� D� Fritsch� and Ch� Heipcke� editors� Digital Photogrammetric Systems� Wichmann�Karlsruhe� ��

� �� P� Meseguer� Constraint Satisfaction Problems� An Overview� AICOM� ��

� �� D� Metaxas� S� J� Dickinson� R� C� Munck�Fairwood� and L� Du� Integration of Quantitative andQualitative Techniques for Deformable Model Fitting from Orthographic� Perspective and StereoProjection� In Proceedings of the �th Int� Conf� on Computer Vision ICCV �� pages ��

� �� O� Munkelt� Aspect�Trees� Generation and Interpretation� CVGIP� ��

� �� R� Nevatia� C� Lin� and A� Huertas� A System for Building Detection from Aerial Images� InA� Gr�un� E� P� Baltsavias� and O� Henricsson� editors� Automatic Extraction of Man�Made Objectsfrom Aerial and Space Images �II�� pages �� Birkh�auser� Basel� ��

� � M� Pilu and R� B� Fisher� Snakes and Splines for Tracking Non�Rigid Heart Motion� In ComputerVision � ECCV �� number �� in Lecture Notes in Computer Science� pages �� SpringerVerlag� Cambridge� ��

� � L� Quam and T� M� Strat� SRI Image Understanding Research in Cartographic Feature Extraction�In H� Ebner� D� Fritsch� and C� Heipke� editors� Digital Photogrammetric Systems� pages ��Wichmann� Karlsruhe� ��

� �� J� Rissanen� Minimum�Description�Length Principle� Encyclopedia of Statistical Sciences� � ��

� �� M� Roux and H� Maitre� ��D Description of Urban Areas Using Maps and Aerial Images� InA� Gr�un� E� P� Baltsavias� and O� Henricsson� editors� Automatic Extraction of Man�Made Objectsfrom Aerial and Space Images �II�� pages �� Birkh�auser� Basel� ��

� �� W� Schickler� Towards Automation in Photogrammetry� Geodetical Information Magazine� ��

�� M� Sester and W� F�orstner� Object Location Based on Uncertain Models� In H� Burkhardt� K��H� H�ohne� and B� Neumann� editors� Mustererkennung �� DAGM�Symposium Hamburg�volume �� of Informatik Fachberichte� pages � �� Springer Verlag� Berlin� ��

��

�� L� G� Shapiro and R� M� Haralick� A Metric for Comparing Relational Descriptions� IEEE Trans�actions on Pattern Analysis and Machine Intelligence� ��

�� V� Steinhage� On the Integration of Object Modeling and Image Modeling in Automated BuildingExtraction from Aerial Images� In A� Gr�un� E� P� Baltsavias� and O� Henricsson� editors� AutomaticExtraction of Man�Made Objects from Aerial and Space Images �II�� pages �� Birkh�auser�Basel� ��

�� P� Suetens� P� Fua� and A� J� Hanson� Computational Strategies for Object Recognition� ACMComputing Surveys� ��

�� K� Sugihara� Machine Interpretation of Line Drawings� MIT�Press� ��

� � P� van Hentenryck� Constraint Satisfaction in Logic Programming� Logic Programming Series�MIT Press� Cambridge� MA� ��

�� G� Vosselman� Relational Matching� Number �� in Lecture Notes in Computer Science� SpringerVerlag� Berlin� ��

�� R� J� Wallace and E� C� Freuder� Heuristic Methods for Over�Constrained Constraint SatisfactionProblems� In M� Jampel� E� Freuder� and M� Maher� editors� Over�Constrained Systems� number�� in Lecture Notes in Computer Science� pages �� Springer Verlag� Berlin� ��

�� D� L� Waltz� Understanding Line Drawings of Scenes with Shadows� In P� H� Winston� editor�Psychology of Computer Vision� pages �� McGraw�Hill� New York� ��

�� U� Weidner� An Approach to Building Extraction from Digital Surface Models� In InternationalArchives of Photogrammetry and Remote Sensing� volume �� B�� pages ��

�� U� Weidner and W� F�orstner� Towards Automatic Building Extraction from High ResolutionDigital Elevation Models� ISPRS Journal of Photogrammetry and Remote Sensing� ��

��