The management of environmental and industrialdisasters, search and rescue operations, surveillanceof natural scenarios, environmental monitoring,and many other field robotics applications requirehigh mobility and the need to reach locations that

are difficult to access with ground vehicles. In many cases,the use of aerial vehicles is the best way to approach theobjective to get information or to deploy instrumentation.

Unmanned air vehicles (UAVs) have significantly increasedtheir flight performance and autonomous onboard processingcapabilities in the last ten years. But a single aerial vehicleequipped with a large array of different sensors of variousmodalities is limited at any time to a single viewpoint. A teamof aerial vehicles, however, can simultaneously collect infor-mation from multiple locations and exploit the informationderived from multiple disparate points. Furthermore, having ateam with multiple heterogeneous aerial vehicles offers addi-tional advantages due to the possibility of beneficial comple-mentarities of the vehicles.

The COMETS ProjectIn the past few years, several research and development pro-jects on multiple UAVs have been conducted. In many cases,these projects consider homogeneous UAV teams (UAVformations) and emphasize control aspects [1]–[3]. Further-more, some projects also deal with cooperation between aeri-al and ground vehicles and involve environment perceptionactivities [4].

The objective of COMETS (Real-Time Coordination andControl of Multiple Heterogeneous Unmanned Aerial Vehi-

cles) is to design andimplement a system forcooperative activities using het-erogeneous UAVs. Particularly, both helicopters and airshipsare considered in COMETS.

In order to achieve this general objective, a new controlarchitecture has been designed and implemented. COMETSalso involves cooperative environmental perception, includingfire detection and monitoring as well as terrain mapping.Research and development (R&D) on autonomous flightcontrol techniques for helicopters and airships as well as faultdetection techniques are also considered within the project;however, those aspects are not depicted in this article.

The heterogeneity of the UAVs is twofold. On one hand,complementary platforms are considered. Helicopters havehigh maneuverability and hovering ability and are thereforesuited for agile target-tracking tasks and inspection and moni-toring tasks that require the vehicle to maintain a position andobtain detailed views. Having much less maneuverability, air-ships can be used to provide global views or to act as commu-nications relays. They also offer a graceful degradation in caseof failure. On the other hand, the considered UAVs are alsoheterogeneous in terms of onboard processing capabilities,ranging from fully autonomous aerial systems to conventionalradio-controlled systems with minimal onboard capabilitiesrequired to record and transmit information.

Thus, the planning, perception, and control functionalitiesof the UAVs can be implemented either onboard the vehiclesor on ground stations for the low-cost and light aerial vehicleswithout enough onboard processing capabilities. The global

BY ANÍBAL OLLERO, SIMON LACROIX, LUIS MERINO,JEREMI GANCET, JOHAN WIKLUND, VOLKER REMUSS,IKER VEIGA PEREZ, LUIS G. GUTIÉRREZ, DOMINGOSXAVIER VIEGAS, MIGUEL ANGEL GONZÁLEZ BENÍTEZ,ANTHONY MALLET, RACHID ALAMI, RAJA CHATILA,GÜNTER HOMMEL, F.J. COLMENERO LECHUGA, BEGOÑA C. ARRUE, JOAQUÍN FERRUZ, JOSE RAMIROMARTINEZ-DE DIOS, AND FERNANDO CABALLERO

Multiple Eyes in the SkiesArchitecture and Perception Issues in theCOMETS Unmanned Air Vehicles Project

1070-9932/05/$20.00©2005 IEEEIEEE Robotics & Automation Magazine JUNE 200546

JUNE 2005 IEEE Robotics & Automation Magazine 47

architecture of the COMETS system is shown in Figure 1. Itinvolves remotely piloted UAVs and UAVs with onboard con-trollers that are capable of performing planning and perceptionactivities. Both kinds of UAVs are equipped with radio trans-mitters and receivers in order to communicate with theground control center and with other UAVs. The ground seg-ment performs centralized planning, interaction with humanoperators, and the cooperative environment perception tasksthat require the integration of information provided by vari-ous UAVs. It should be noted that the bandwidth and rangeof the wireless communication system with fast-moving nodesimpose significant operation constraints.

In order to test and validate the developments, the projectincludes experiments and demonstrations in forest fire applica-tions. The cooperation of UAVs is very valuable in this verychallenging context. Missions involve fire alarm detection,confirmation and localization, and fire monitoring. Terrainmapping missions are also relevant because of their attempt topredict the forest fire spread and for their perception activitiesbased on aerial image processing of the ground scenes. Addi-tionally, computation of the distances of the fire to people,houses, ground firefighters, and sensible locations are alsoobjectives of the multi-UAV system.

For the exper iments, COMETS integrates theautonomous helicopter Marvin, the airship Karma, and theteleoperated helicopter Heliv. Marvin[Figure 2(a)] is an autonomous heli-copter developed by the Real-TimeSystems and Robotics Group of theTechnische Universität Berlin, Ger-many. The helicopter has a rotor diame-ter of 2 m and is driven by a two-strokeengine. It is equipped with a strap-down inertial measurement unit (IMU),a compass, an ultrasonic rangefinder, aprecision differential global positioningsystem (DGPS), and a microcontrollerfor autonomous flight. It also carries ahigh-resolution digital still camera and aPC-104+ for image acquisition, a fire

and temperature sensor for fire detection, and two redundantradio links for telemetry, commands, and image transfer. Anearlier state of the system has won the International AerialRobotics Competition 2000 held by the Association forUnmanned Aerial Vehicle Systems (AUVS).

Karma is an 18-m3, electrically propelled airship developedat Laboratoire d’Architecture et d’Analyse des Systèmes(LAAS) [5] [Figure 2(b)]. It is equipped with the necessarysensor to implement autonomous flight and carries a widebaseline stereovision bench.

The Heliv helicopter is the result of the evolution of a con-ventional remotely piloted helicopter of HELIVISION thathas been transformed by the Robotics, Vision, and ControlGroup at the University of Seville, Spain, by adding sensing,perception, communication, and control functions [Figure2(c)]. The teleoperation system of Heliv [6] is not described inthis article.

General Architecture of the COMETS SystemIn COMETS, some UAVs are directly controlled by an oper-ator. Some others are only endowed with operational auton-omy, their tasks being planned and monitored by the controlcenter. Still other may have decisional autonomy capacitiesand, hence, should achieve given missions by themselves.Moreover, depending on the situation, the central station

Figure 1. Architecture of the COMETS system.

Com

mun

icat

ion

Net

wor

kGround

Segment

Mission Planning

Environment Perception

and Representation

Com

mun

icat

ion

Net

wor

k

Monitoring and

Control

Teleoperation System

Intelligent Controller UAV 1

Intelligent Controller UAV n

Flying Segment

UAV Teleoperated

Plu

gins

laye

r

Figure 2. Unmanned air vehicles: (a) Marvin, (b) Karma, and (c) Heliv.

(a) (b) (c)

IEEE Robotics & Automation Magazine JUNE 200548

should be able to take control over UAVs endowed withdecisional capabilities. COMETS, hence, requires an archi-tecture integrating both central decision making and distrib-uted decision capabilities. Moreover, this architecture shouldenable users to dynamically configure the decisional scheme,depending on the available decisional capabilities of therobots and the operational context.

Decision Issues in Multirobot SystemsWithin a multirobot system, the “decision” encompasses sev-eral notions.

◆ Supervision and execution: The executive is a passive, reac-tive management of task execution, whereas supervisionis an active process that manages all the decisional activi-ties of the robot, whatever their extent.

◆ Coordination: Coordination ensures the consistency ofthe activities within a group of robots. It defines themechanisms dedicated to avoid or solve possibleresource conflicts that may arise during operationalactivities. This is especially related to trajectories andmultirobot cooperative task execution (e.g., simultane-ous perception of the same target by several robots).

◆ Mission refinement, planning, and scheduling: These deci-sional activities are dedicated to plan building, takinginto account models of missions and tasks as well asmodels of the world (the robot’s perception andmotion abilities, current knowledge related to theenvironment, etc.).

◆ Task allocation: Task allocation addresses the distributionof tasks among the robots. It requires the establishmentof a task assignment protocol in the system and the defi-nition of some metrics to assess the relevance of assign-ing given tasks to a specific robot.

Global ArchitectureThese decisional components can be implemented accordingto different configurations: they can be gathered within thecontrol center, or they can be partially (or even totally) dis-tributed among the robots. The COMETS architecture hastherefore been designed to support different control para-digms. Decisions related to the UAVs not endowed withautonomous abilities are by the ground central system, whileautonomous UAVs can also be integrated in the system. Thechosen architecture (Figure 3) consists of three segments.

1) The ground segment includes the mission plan-ning system (MPS), the monitoring and controlsystem (MCS), and the ground part of the percep-tion system (PS).

2) The flying segment encompasses all the UAVs. EachUAV is endowed with

◆ onboard proprietary components that gather thevarious functions specific to the UAV (flight con-trol, data acquisition, possible data process)

◆ a supervisor defined according to a genericscheme that interfaces the UAV with the otherCOMETS subsystems (ground segment and otherUAVs) and controls its activities

◆ a deliberative layer when the UAV has auto-nomous capacities.

3) Finally, the communication segment provides the soft-ware that supports the communications between thedifferent components of the COMETS system.

Control CenterThe control center is the core decisional section of the groundsegment, and its main responsibility is to provide decisionmaking for UAVs not endowed with high-level decisional

abilities. It also must assume control overany UAV if operational constraintsrequire it, while providing the requiredflexibility to deal with multi-UAV het-erogeneous fleets. Additionally, the con-trol center implements a complete anduser-friendly GUI, which is the fore-most interface with the users of theCOMETS system.

The key functionalities supplied are1) the breakdown of a user-definedabstract mission plan in sets of atomicprocedures directly executable by theUAVs and 2) the supervision of the mis-sion execution to detect conditions thatmay require corrective actions toincrease safety and success probability.The generation of the mission planincludes the usual topics, such as missiondecomposition, resource allocation, pathplanning, and conflict resolution. Noclear trends have been identified in thesefields in reference to reliable, centralizedFigure 3. Global architecture of COMETS.

Perception System

Control Center

Ground Segment

MPS

MCS of UAV 1

MCS of UAV 2

MCS of UAV 3

Communication Segment

Generic Supervisor

Generic Supervisor

Onboard Proprietary

Components

Onboard Proprietary

Components

Onboard Proprietary

Components

Generic Supervisor

Deliberative Layer

Deliberative Layer

Levels 4 and 5

Levels 4 and 5

Levels 1, 2, 3

Flying Segment

UAV 3

UAV 2

UAV 1

Levels 1, 2, 3

Levels 1, 2, 3

JUNE 2005 IEEE Robotics & Automation Magazine 49

multi-UAV planning without reliance on specific planningabilities of the robots.

The mission decomposition offers an additional degree offlexibility for defining mission objectives. The system is sup-plied with a generic framework to break down complex oper-ations into small pieces to achieve abstract goals like targetsearching and monitoring. Once the mission is fully defined interms of simple operations, the rest of the planning system isapplied to generate a task schedule for each agent of the fleet.

Among the several methods that exist for solving theresource allocation problem in bounded time, the reactivetabu search metaheuristic [7] has been applied. Unlike otherexisting approaches, this method takes into consideration themost updated picture of the environment and almost anykind of constraint. It offers a near-optimal solution fastenough to be operationally effective in a situation where thecontinuous evolution of the mission conditions is a primaryconcern. The output is piped to the conflict analyzer, whichdeals with general conflicts that arise from the resource allo-cation. The final objective is to produce a conflict-free time-line of lower-level procedures suitable for execution by thetarget platforms.

Clearly, the chief feature of the control center is the flexi-bility required for the online reconfiguration of the previoussubsystems driven by the heterogeneous nature of the availablefleet. To accomplish this, the system is designed according to ageneric plug in-based architecture that relies on models storedin database tables, allowing user customization at any time.Switching among the plug-ins that define the operations to beperformed with the parameters stored in the database, it ispossible to tune the performance of the UAV models relativeto dynamic behavior and general resource consumption. Thepath planner will rely upon those models to ensure that theprocedure schedule is feasible given the specific constraints ofthe existing UAVs.

Finally, the supervisor of the mission execution providesthe system with a certain degree of reactivity to face unex-pected situations (from a hardware failure to the sudden pres-ence of unpredicted wind), assessing its potential influenceand invoking the replanning routines if necessary.

Robots Decisional ArchitectureWe define the “level of autonomy” of a robot as the numberof decisional mechanisms it is endowed with and consider thefollowing five levels [8]:

◆ Level 1: No autonomy on board the robot. The robot isonly able to directly execute elementary tasks requestedby the control center.

◆ Level 2: Executive capabilities (operational autono-my). The robot is able to manage partially orderedsequences of elementary tasks and to return executionstatus of the tasks.

◆ Level 3: Same as Level 2 plus simple interacting capabil-ities. The robot may manage online simple interactions (synchronizations) directly with other robots endowedwith at least the same level of decisional autonomy.

◆ Level 4: Deliberative capabilities. High-level taskrequests are managed (involving autonomous task plan-ning/scheduling) and a distributed multirobot taskcoordination is ensured among robots endowed with atleast the same level of autonomy.

◆ Level 5: Same as Level 4 plus task reallocation capabili-ties. The UAV may opportunistically reallocate tasks andaccept new tasks from other UAVs that are alsoendowed with this level of autonomy.

This taxonomy is characterized by a large gap betweenLevels 3 and 4. Up to Level 3, the control center is expectedto ensure the global consistency of the system’s activity (“lowlevels” of decisional autonomy). Levels 4 and 5 introduce thepossibility of delegating coordination and mission refinementactivities in a distributed way (“high levels” of decisionalautonomy). Regarding low autonomy levels, the decisionalnode (DN) is restricted to an executive. Because this executiveis actually common to all levels, we denote it as the multilevelexecutive (MLE). At Level 1, the MLE behaves as a transpar-ent connecting point between the control center and theUAV’s functional components. However, the full features ofthe MLE are required when considering upper levels: at Lev-els 2 and 3, it manages task sequences execution and, at Level3, it enables simple interactions with other UAVs of the samelevel. For higher levels, the MLE relies on the UAV’s delibera-tive layer (DL) instead of the control center, tackling higherautonomous decisional capabilities. The DL deals with mis-sions and tasks refinements as well as coordination activities ortask reallocation when relevant (as for Level 5).

Figure 4 exhibits the general structure of the DNs for thehigher-level UAVs. The main part is the supervisor, which isresponsible for missions and tasks refinements. It elaborates onplans with the support of a task planner/scheduler and thespecialized refiners. It also triggers negotiation sessions forcoordination purposes or task reallocations when relevant (i.e.,at Level 5 only) through the negotiation manager.

Task Planning/SchedulingGiven a requested mission, the deliberative layer is expectedto build executable plans, and execution should be straight-forward in an ideal context. Actually, contingencies duringmission execution may require dynamically repairing parts ofthe plans. Hence, a plan is never considered definite: itshould be reprocessable online, taking into account the latestcontingencies. This is partially devolved to the taskplanner/scheduler, manipulating symbolic tasks models. Thespecialized refiners are intended to bridge the gap betweenabstract, symbolic plans and actual data of the world, knownthrough various models (namely, environment models, UAV,and communication models). They return expectedtime/resource consumptions and possibly further taskrefinements (e.g., waypoints for the “path” refiner) accord-ing to the refiner’s specialty. Preliminary versions of theserefiners have been developed and ongoing work deals withthe integration of a HTN-oriented (hierarchal task network-oriented) symbolic planner [9] with these refiners.

CoordinationTwo kinds of coordination can be pointed out: spatial coordi-nation and interactions-related coordination. The spatialcoordination with other UAVs is an activity that should con-tinuously work in the background since it is a vital mecha-nism that prevents conflicting trajectories. It should overcomeall other kinds of deliberative mechanisms; we are currentlyexploring the possibility of adapting the Plan-Merging Proto-col to the particular purpose of UAVs’ spatial coordination[10]. Regarding interactions-related task coordination, flexibleplans produced by the planner/scheduler should be coordinat-ed with the plans of other involved UAVs for each giveninteraction; this mechanism should rely on a relevant negotia-tion protocol, ensuring consistency of the joint plans of theUAVs. Such a protocol definition and development, based onthe incremental assessment and exchange of a critical plan’ssections, is also ongoing.

Task ReallocationTask reallocation is a distributed mechanism intended todynamically improve missions/task allocation. Task realloca-tion continuously requires identifying which tasks or parts ofplans should be reassigned to other UAVs. This can be solvedusing an opportunistic mechanism, pointing out “expensive”tasks (metrics to be defined) or tasks that look consistent withregard to other UAV plans (criteria to be defined). For the

reallocation process itself, we intend to extend the contractnet protocol paradigm [11].

CommunicationsCOMETS makes use of a custom communication system(CS). To take advantage of standard network components, thedecision was made to design the system as an overlay networkon top of existing transportation layers. It can be adapted todifferent kinds of operating systems. In this context, a net-work node (NN) is a task in either a multi- or single-taskingenvironment. A serverless, peer-to-peer approach has beenchosen for the network design; it is able to mimic varioustopologies since NNs are able to impart information betweenadjacent NNs. A connection between two NNs is a portdefined by a set of channels. Each channel is capable of trans-mitting a byte stream with an inherent bandwidth. The CScan provide reliable service using unreliable channels becauseit efficiently handles errors, such as data losses, multiples, orout-of-order reception. A port has the combined bandwidthof all its channels.

The CS is realized via a distributed shared memory, theblackboard (BB). Each NN has a local copy of the BB por-tion that it is accessing. The BB is organized in state-typeslots (an atomic, self-contained, relatively small amount ofdata) and stream-type slots (relatively large or “infinite”amount of data), where each slot has an assigned bandwidth

IEEE Robotics & Automation Magazine JUNE 200550

Figure 4. Deliberative layer of the high-level UAVs.

Other RobotsDNs

Control Center

Robot j

Robot j's DN

Task Planner / Scheduler

SupervisorNegotiation

Management

Lev. 4: Definite Mission/ Tasks Allocation

Lev. 5: Initial Mission/ Tasks Allocation

Lev. 4: CoordinationOnly

Lev. 5: Coordination + Tasks Reallocation

Functional Components

Multilevel Executive

Deliberative Layer

Perception

Path

Communication

Interaction

Specialized Refiners

Mission Formalism

Negotiation Formalism

relative to the port’s bandwidth. Each slot is associated witha specific class of information and can hold a specificamount of data. In COMETS, state-type slots are mainlyused in unreliable transmission schemes for data where onlythe latest information is important (i.e., the current positionof a UAV). Stream slots are mainly used in reliable transmis-sion schemes to send pictures and video streams. InCOMETS, a slot’s content is defined by a C programminglanguage record; this approach is much less error-prone thanindividual parsing of data streams. As in conventional mem-ory, mutual exclusion in reading (by data drains) and writ-ing (by data sources) slots must be obeyed by the user and isenforced by the CS.

The coherency protocol that ensures the consistency of theBB sends traffic-shaped data streams over the channels. Thesestreams are compiled with respect to each slot’s bandwidth.The utilized scheduling algorithm is real-time aware and usesa static binary tree approach that guarantees a maximumqueuing delay.

Routing is conducted for each slot individually from thedata source to all data drains; currently, distance vector rout-ing is used. The routing is recalculated dynamically whenev-er the visibility of NNs changes; the net adapts fast, and thecur rent performance is less than 30 ms per NN. InCOMETS, this feature is currently used to enhance the reli-ability of the radio transmission to the UAV Marvin, whichhas a low-delay DECT (digital enhanced cordless telecom-munication) radio modem for telemetry data and a wirelesslocal area network (LAN) for image transmission. If there isa DECT link failure, the teleme-try data will be routed over thewireless LAN because the CS willseamlessly bridge between thesetwo very different transportationlayers. In a future version of theCOMETS flying segment, theUAVs may set up an ad hoc net-work that can be reconfigured touse other UAVs as relays whenthe ground station is out of range.

The CS can be adapted to awide range of transportation lay-ers. TCP, UDP, and RS232 areused in COMETS. The robust-ness of the network is achievedby graceful, user-aware degrada-tion of bandwidth assignments,rerouting, and the possible useof multiple redundant physicalconnections for a single port. InCOMETS, the network includeswired and wireless links to con-nect all systems; nodes are most-ly PCs running Windows orLinux, embedded PCs, and alsomicrocontrollers.

The Perception SystemThe perception system (PS) consists of the following mainsubsystems: application-independent image processing (AIIP);detection/alarm confirmation, localization, and evaluationservice (DACLE); event monitoring system (EMS); and ter-rain mapping system (TMS). Figure 5 shows a scheme of theperception system architecture.

AIIP SubsystemAIIP deals with the image processing functions that are common to the DACLE, EMS, and TMS. The AIIP subsystem constitutes a kind of front end for the processing ofthe images gathered by the different platforms. It groups thefunctionalities that could be on board UAVs with enoughprocessing capabilities. AIIP includes functions such as imagestabilization, camera position estimation, image geo-location,and object tracking. Some of these functions are based ontechniques for the extraction of robust and repeatable features[12]. Homogeneity features, called blobs in scale-space theory

JUNE 2005 IEEE Robotics & Automation Magazine 51

Figure 5. Perception system architecture. One AIIP process per image source. DACLE,EMS, and TMS interact with AIIP modules following a client-server model using the BBCS.

Camera 1

UAV i

UAV i

UAV j

BBCS

TMS

DACLE

EMSAIIP Process

AIIP Process

AIIP Process

PS

Camera 2

Camera 1

Unmanned air vehicles havesignificantly increased their flight

performance and autonomousonboard processing capabilities in

the last ten years.

[13], are useful for applications such as wide baseline matching[14] and view-based object recognition [15]. A developedblob extraction algorithm is described in detail in [16].

A visual odometer has been developed for camera positionestimation. This is a system that computes the three-dimen-sional (3-D) displacement and change in orientation of thecamera based on estimates of local image displacements. Whenusing a calibrated camera, the camera location and orientationcan also be extracted from the estimation procedure [17].

Once an interesting object has been localized on theimage plane, it should be anchored to geographical coordi-nates by means of automatic geo-location. If the position andorientation of the camera are known and the camera is cali-brated, this can be done by projecting the detected positionover a digital terrain model [18], which can be obtainedthrough the terrain mapping function of COMETS. Objecttracking is a very useful functionality for surveillance and

monitor ing applications. The field-of-object trackinginvolves many issues, from rigid to nonrigid object tracking,multiple object tracking, tracking considering occlusions, andtracking of small objects. In the frame of COMETS, thisfunctionality is built upon a region tracking algorithm. Froman object of interest, a set of features defining the object isdynamically obtained. The features are tracked over thesequence [12] by using motion consistency to prune features.

DACLE Subsystem The objective of the DACLE service is to specify a coopera-tive alarm detection/alarm confirmation, localization, andevaluation system based on the images transmitted by the dif-ferent UAVs of the COMETS system. DACLE receivesimages of the scene from the AIIP. Once an alarm has beendetected, it uses the AIIP geo-location functionalities toobtain its position (with its associated uncertainty). Both

infrared (IR) images and visual imagesare used for fire detection. Fire is detect-ed by means of the thresholding of theblack-and-white IR images because fireusually provides the highest IR radiationintensity. A special-purpose, fuzzy-wavelet threshold selection method hasbeen applied [19]. Fire detection in thevisual images is carried out by means ofsmoke motion detection and segmenta-tion in a video sequence. Particularly, amethod based on the combination ofthe optical flow techniques and waveletanalysis has been applied in COMETS[20]. It should be noted that thismethod requires the previous applica-tion of stabilization in the AIIP.

EMS SubsystemThe objective of the EMS subsystem isto cope with the monitoring activitiesof the COMETS system. Once an eventhas been detected, a monitoring missionto obtain data on the desired events canbe scheduled. First, the EMS makes useof the AIIP scene image stabilizationfunction and then, constructs a two-dimensional (2-D)/3-D model of theevent in the image plane. In the case offorest fires, this model includes fire frontwidth and flame height. Fire segmenta-tion techniques that classify the pixelsbelonging to the fire front (over the ter-rain) and the corresponding pixels at thetop of the flames have been developed.Finally, the pixels of the fire front areautomatically geo-located. Furthermore,EMS computes the fire rate of spreadand the flame height and transmits all

IEEE Robotics & Automation Magazine JUNE 200552

Figure 6. (a) Blimp trajectory reconstructed from a sequence of 400 stereovision unreg-istered image pairs and (b) the 10-cm resolution digital elevation map reconstructed.

−20

0

20

40

60

80

100

−100−80−60−40−20020Latitude (m)

(a)

(b)

Long

itude

(m

)

this information to the ground control center to be used forforest fire fighting [18].

TMS SubsystemThe TMS subsystem provides an update of the initially availablecartographic files of the area over which the UAVs are operatingin order to provide up-to-date information at a higher resolu-tion than initially available. To build a digital terrain map basedon aerial images, the following issues must be tackled: 3-D dataperception, precise localization, and map building to structurethe registered 3-D data into a digital terrain model (DTM).Three-dimensional data perception in COMETS is carried outby means of both stereovision and sequences of monocularimages. In the case of stereovision, a simultaneous localizationand mapping approach involving an extended Kalman filter toestimate both the location of the UAV and of detected land-marks has been proposed [21] (see Figure 6).

Cooperative PerceptionOne of the key points for COMETS is cooperative perception.Cooperative perception strategies for the UAVs are useful for increasing the precision of the location of an event,decreasing the false alarm ratio in event detection, and improv-ing the measurements obtained in monitoring missions. Coop-erative perception involves data fusion, which can be applied atdifferent levels. Thus, raw sensor data can be directly combinedif the sensor data are commensurate, such as two visual imagecameras. Data registration, noncoincident sampling, and data res-olution are key points for these techniques. The result of a widebaseline image registration algorithm on blob features is shownin Figure 7. The images were captured at approximately thesame instant by different cameras on board different UAVs. Blobfeatures were calculated for each image, and matching blobsbetween the images were found using a homographic model.The matching blobs are overlaid in each image in Figure 7.

On the other hand, if the data are not commensurate (i.e., IR and visual images), a feature-level fusion approachcan be considered. Feature-level fusion involves the extrac-tion of representative features from sensor data. Then, fea-tures are extracted from the available sensors andclassification is performed based on these features [22]. Deci-sion-level fusion involves the combination of sensor infor-mation after each sensor has made a preliminarydetermination of the object location, attributes, and identity.In event-detection activities in COMETS, each sensor willproduce an estimation of the certainty of the alarm and theposition of the alarm. The position of the alarm can beimproved by fusing all the data, taking into account theiruncertainties in a statistical framework. Also, an alarm indexgiven by the different sources is used to decrease the falsealarm ratio through a similar data fusion scheme. For moni-toring applications, the UAVs are distributed around theobject of interest. Each camera will provide measures of theobject of interest. Procedures are employed to combine theresults obtained from several cameras to derive the 3-Dmodel of the fire. IR cameras are used to detect and segment

the fire base while visual cameras (placed in appropriatedlocations to avoid the smoke) are used to segment theflames. Each measure will be accompanied also by a degreeof certainty, and all will be used to obtain a better estimationof the object measurements.

JUNE 2005 IEEE Robotics & Automation Magazine 53

Figure 7. Correspondences for blobs in two images originat-ing from two different UAVs with (a) a video sensor and (b) ahigh-resolution digital camera.

(a)

(b)

A single aerial vehicle equippedwith a large array of differentsensors of various modalities

is limited at any time toa single viewpoint.

Field ExperimentsGeneral field experiments of COMETStook place in May 2003 and May 2004at Lousã and Castanheira de Pera, nearCoimbra, Portugal. The Marvin, Karma,and Heliv helicopters participated inthese experiments. The missions werefire detection and monitoring. Marvinhad a high-resolution digital still camera[see Figure 7(b)], Karma had a pair ofvideo cameras, and Heliv had an IRcamera and a visual camera.

In these experiments, the individualsubsystems and the whole integratedCOMETS system were tested. The per-formance in the execution of the gener-ated paths for both teleoperated andautonomous flights has been analyzed. Ingeneral, the autonomous executionoffered better performance than teleop-erated execution. Figure 8 shows a 7-min autonomous flight of Marvin,including start and landing. The coordi-

nates in Figure 8 are in a UAV-specific mission coordinate sys-tem. The autonomous start of the helicopter is close to 0 mnorth and -10 m west, the rightmost point in Figure 8. Thehelicopter was set to point-to-point flying, processing thetransmitted list of waypoints. On each waypoint reached, thehelicopter stayed for some seconds while changing its headingto the next waypoint. An ideal flight would produce a pathwhere line segments connect the waypoints. In real flights,there are deviations. In Figure 8, the strongest deviation is inthe first line segment after the start. The error was 5 m, whichis 250% of the rotor diameter and was not treated as missioncritical. The deviation occurred due to upcoming wind.

Figure 9 shows the visual and IR images and otherinformation depicted in the teleoperation station during a20-min experiment devoted to test teleoperation and per-ception functionalities.

IEEE Robotics & Automation Magazine JUNE 200554

Figure 8. Autonomous Marvin flight.

Flight 15 May 200320,000

10,000

0

−10,000

−20,000

−30,000

−40,000

−50,000

−60,000−80,000 −70,000 −60,000 −50,000 −40,000

North (mm)

Wes

t (m

m)

−30,000 −20,000 −10,000 0

Figure 9. Teleoperation experiment.

Figure 10. Preliminary results over Castanheira de Pera fire images: (a) stabilized image and (b) segmented fire: in blue, the pix-els of the fire base; in red, the top of the flames.

(a) (b)

The techniques presented above were also used to stabi-lize and segment the fire in the IR and visual images. Figure10 presents some results. Figure 10(a) shows the result of thestabilization procedure over an original image sequence. Fig-ure 10(b) is the result of the EMS processing. Both fire seg-mentation (based in color histogram) and fire modeling(fire-base extraction and height points assignment) algo-rithms were tested. The fire front is printed in blue, whilecorresponding height points are printed in red.

The MPS in the control center also has been tested inLousã, Portugal. Thus, MPS computed a sequence of way-points for each UAV using the environment map and trans-mitted the sequences to the UAVs through thecommunication system. Figure 11(a) shows the planned pathsfor the three vehicles. Furthermore, coordinated flights withthe Marvin, Karma, and Heliv helicopters have been per-formed [see Figure 11(b)]. The MCS allowed the operator inthe control center to monitor the execution of the mission.During the flights, the UAVs transmitted images of the firefrom different positions, allowing the application of the tech-niques mentioned in the previous sections.

Several videos and other results of the experiments can beviewed on the COMETS Web site (http://www.comets-uavs.org).

ConclusionsDistributed aerial robotics is currently in its early stages.The cooperation of several unmanned aer ial vehicles,including autonomous and teleoperated vehicles, is a chal-lenging activity with many possible applications. This articlehas presented the COMETS system, which is able to inte-grate several heterogeneous UAVs. Heterogeneity is consid-ered both in terms of the aerial vehicles (helicopters andairships are integrated) and onboard processing capabilitiesranging from fully autonomous systems to conventionalremotely piloted vehicles.

AcknowledgmentsThis research has been funded by the European Commission(IST 2001–34304) Fifth Framework Programme. Theauthors acknowledge the contributions of all the membersof the COMETS teams.

KeywordsAerial robotics, unmanned aerial vehicles, multirobot sys-tems, architectures for cooperation, cooperative perception.

References[1] F. Giulietti, L. Pollini, and M. Innocenti, “Autonomous formation

flight,” IEEE Contr. Syst. Mag., vol. 20, no. 6, pp 34–44, Dec. 2000.[2] T.W. McLain, P.R. Chandler, and M. Pachter, “A decomposition strategy

for optimal coordination of unmanned air vehicles,” in Proc. 2000 Amer-ican Control Conf., vol. 1, 2000, pp. 369–373.

[3] S. Saripally, D. Naffin, and G.S. Sukhatme, “Autonomous flying vehicleresearch at the University of Southern California,” in Proc. 1st Int. Workshopon Multi-Robot Syst., Washington D.C., USA, 2002, pp. 73–82.

[4] R. Vidal, S. Rashid, C.S. Sharp, O. Shakernia, and S. Sastry, “Pursuit-evasion games with unmanned ground and aerial vehicles,” in Proc. Int.Conf. Robotics and Automation, 2001, pp. 2948–2955.

[5] S. Lacroix, I-K. Jung, P. Soueres, E. Hygounenc, and J-P. Berry, “Theautonomous blimp project of LAAS/CNRS—Current status and re-search challenges,” in Experimental Robotics VIII, Springer Tracts inAdvanced Robotics, vol. 5. New York: Springer, 2003, pp 487–496.

[6] A. Ollero, J. Alcázar, F. Cuesta, F. López-Pichaco, and C. Nogales,“Helicopter teleoperation for aerial monitoring in the COMETS multi-UAV system,” in Proc. 3rd IARP Workshop on Service, Assistive and Person-al Robots, 2003, pp 137–142.

[7] K.P. O’Rourke, T. Glenn Bailey, R. Hill, and W.B. Carlton, “Dynamicrouting of unmanned aerial vehicles using reactive tabu search,” in Proc.67th MORS Symp., Nov. 1999, pp. 5–30.

[8] J. Gancet and S. Lacroix, “Embedding heterogeneous levels of deci-sional autonomy in multi-robots systems,” in Proc. 7th Int. Symp. Dis-tributed Autonomous Robot Syst., Toulouse, France, June 2004, pp.251–260.

[9] D. Nau, T.C. Au, O. Ilghami, U. Kuter, W. Murdock, D. Wu, and F. Yaman, “SHOP2: An HTN planning system,” J. Artificial Intell. Res.,vol. 20, no. 1, pp 379–404, 2003.

JUNE 2005 IEEE Robotics & Automation Magazine 55

Figure 11. Coordinated flight: (a) the vehicles’ paths planned by the MPS and (b) the coorrdinated flight of the three vehicles.

(a) (b)

[10] R. Alami, F. Ingrand, and S. Qutub, “A scheme for coordinatingmulti-robot planning activities and plans execution,” in Proc. 13th Euro-pean Conf. Artificial Intell. (ECAI’98), 1998, pp. 617–621.

[11] T. Lemaire, R. Alami, and S. Lacroix, “A distributed task-allocationscheme in multi-UAVs context,” in Proc. IEEE Int. Conf. Robotics andAutomation, Apr. 2004, 3622–3627.

[12] J. Ferruz and A. Ollero, “Real-time feature matching in imagesequences for non-structured environments. Applications to vehicleguidance,” J. Intell. Roboti. Syst., vol. 28, no. 1/2, pp. 85–123, Jun. 2000.

[13] T. Lindeberg, Scale-Space Theory in Computer Vision. Norwell, MA:Kluwer Academic, 1994.

[14] T. Tuytelaars and L. Van Gool, “Matching widely separated viewsbased on affinely invariant neighbourhoods,” Int. J. Comput. Vision, vol.59, no. 1, pp. 61–85, 2003

[15] G.H. Granlund and A. Moe, “Unrestricted recognition of 3-D objectsfor robotics using multi-level triplet invariants,” Artific. Intell. Mag., vol.25, no. 2, pp. 51–67, 2003.

[16] P-E. Forssén and G. Granlund, “Robust multi-scale extraction of blobfeatures,” in Proc. 13th Scandinavian Conf. Image Anal., LNCS 2749,2003, pp. 11–18.

[17] P-E. Forssén, “Updating camera location and heading using a sparsedisplacement field,” Dept. EE, Linköping Univ., SE-581 83 Linköping,Sweden, Tech. Rep. LiTH-ISY-R-2318, Nov. 2000.

[18] L. Mer ino, A. Ollero, J. Ferruz, J.R. Martínez-de-Dios, and B.C. Arrue, “Motion analysis and geo-location for aerial monitoring inthe COMETS multi-UAV system,” in Proc. ICAR 2003, Coimbra (Por-tugal), 2003.

[19] J.R. Martínez-de Dios and A. Ollero, “Automatic threshold selectionfor infrared images of fires,” in Proc. Forest Fire Research & Wildland FireSafety - IV ICFFR 2002, Luso, Coimbra (Portugal), 2002.

[20] F. Gómez-Rodríguez, S. Pascual-Peña, B.C. Arrue, and A. Ollero, “Smokedetection using imaging processing,” in Proc. Forest Fire Research & WildlandFire Safety - IV ICFFR 2002, Luso, Coimbra (Portugal), 2002.

[21] I-K Jung and S. Lacroix, “Simultaneous localization and mapping withstereovision,” in Proc. Int. Symp. Robotics Res., Siena, Italy, Oct. 2003.

[22] B.C. Arrue, A. Ollero and J.R. Martinez-de Dios, “An intelligent system for false alarm reduction in infrared forest-fire detection,” IEEEIntell. Syst., vol. 15, no. 3, pp. 64–73, 2000.

Aníbal Ollero received his electrical engineering degree in1976 and the Dr. Engineer with honors in 1980 (doctorateaward 1978–1980) from the University of Seville, Spain. Since1993, he has been a full professor at the E.S Ingenieros, Univer-sity of Seville. He participated in or led 56 research and devel-opment projects, including ten projects funded by the EuropeanCommission. He is currently the scientific and technical coor-dinator of the European IST project Real-Time Coordinationand Control of Multiple Heterogeneous Unmanned AerialVehicles (COMETS). He is chair of the Coordinating Com-mittee on Mechatronics, Robotics, and Components of theInternational Federation of Automatic Control (IFAC).

Simon Lacroix graduated from the Ecole Centrale de Parisin 1990 and obtained a Ph.D. at Laboratoire d’Architecture etd’Analyse des Systèmes (LAAS)/Centre National de laRecherche Scientifique (CNRS) on autonomous rover navi-gation in natural environments. After a one-year postdoctoralfellowship at the Center for Intelligent Machines, McGillUniversity, in 1996 he joined the Robotics and ArtificialIntelligence Group at LAAS/CNRS as a research scientist.His research concerns perception and navigation forautonomous outdoor mobile robots, including environment

modeling, localization, perception control, and explorationstrategies. Since 2000, he has led the activities related torobotics in natural unstructured environments, includingrecently initiated aerial robotics activities.

Luis Merino received the M.Sc. degree in telecommunica-tion engineering from the University of Seville, Spain, in2000. He is a Ph.D. candidate at University of Seville, wherehe has been a research assistant since 2001 under a grant of theSpanish Ministry of Education. His main research interests arecomputer vision applications for field robotics and cooperativeperception techniques.

Jeremi Gancet currently has recently held a Ph.D. position atthe Laboratoire d’Architecture et d’Analyse des Systèmes(LAAS)/Centre National de la Recherche Scientifique(CNRS) in the Robotics and Artificial Intelligence Group. Hiswork is mainly related to decisional architectures in multirobotenvironments and outdoor mobile robotics applications.

Johan Wiklund recieved the M.Sc. degree and Licentiate ofTechnology degree from Linkoeping University, Sweden, in1984 and 1987, respectively. Since 1984, he has been aresearcher in the Computer Vision Laboratory of LinkoepingUniversity, working on various image processing problems.

Volker Remuss studied at the Technische Universität (TU)Berlin, Germany. He was one of the initial members of thestudent team that developed the autonomous helicopter laterknown as Marvin, which won the international aerial roboticscompetition. In 2002, he received his degree in computerengineering and joined the Real-Time Systems and RoboticsGroup of the TU Berlin led by Günter Hommel.

Iker Veiga Perez received a degree in telecommunicationsengineering (special field: radiocommunications) from theHigher Technical School of Industrial and TelecommunicationEngineers of Bilbao, Spain, in 2001. He has been a softwareengineer at the Mission Planning Division of GMV S.A. since2002. He has actively participated in control center design anddevelopment for UAV applications and for satellite applications.

Luis G. Gutiérrez is the head of the Mission Planning Divi-sion of GMV and leads the mission planning-related projectsboth to space and aircraft. He has been working at GMVsince 1990, and since then he has taken part in projects of dif-ferent natures until 1998 when his tasks focused on missionplanning. Also, he was the technical manager of mission plan-ning development for the Eurofighter and of mission planningof the Envisat, and he is now the program manager of theMetop satellites for EUMETSAT.

Domingos Xavier Viegas is a full professor at the Mechani-cal Engineering Department of the Faculty of Science and Technology of the University of Coimbra, Portugal. Hisresearch activity is in the fields of industrial aerodynamics, wind

IEEE Robotics & Automation Magazine JUNE 200556

engineering, and forest fire propagation. He has been the headof a research unit on forest fires at the University of Coimbra,working in this area since 1986. He is the coordinator of severalresearch projects supported by national and European funds. Hehas been a lecturer at several international meetings and is theauthor of several articles on the topic of forest fires.

Miguel Angel González Benítez has 20 years of expertisein radio control of aerial vehicles, including helicopters,blimps, and airplanes He is the Spanish champion of heli-copter radio control and winner of three Open InternationalChampionships in helicopter radio control. He has developedmany different radio-controlled aircraft, including airplanes,airships, and helicopters. He has been developing numerousservices for the Spanish Army. He is the technical director ofthe company Helivision, which offers services in high-qualityfilming and photographs.

Anthony Mallet currently holds a postdoctoral fellowshipposition at the École Polytechnique Fédérale de Lausanne(EPFL). He received his Ph.D. from the Laboratoire d’Archi-tecture et d’Analyse des Systèmes (LAAS)/Centre National dela Recherche Scientifique (CNRS), where he has beeninvolved in the outdoor robotics project (EDEN) of theRobotics and Artificial Intelligence Group. His work dealswith robots autonomous localization and mapping, as well asrobotics architecture concerns.

Rachid Alami leads the human environment robotics pro-ject in the Robotics and Artificial Intelligence Group at theLaboratoire d’Architecture et d’Analyse des Systèmes(LAAS)/Centre National de la Recherche Scientifique(CNRS). His work is related to human/robot interaction,personal and service robotics, and multirobot applications. Healso has a major interest in control architectures and mobilerobotics in general.

Raja Chatila leads the Robotics and Artificial IntelligenceGroup of Laboratoire d’Architecture et d’Analyse des Sys-tèmes (LAAS)/Centre National de la Recherche Scien-tifique (CNRS), Toulouse, France. His work encompassesseveral aspects of autonomous robotics: navigation in struc-tured and natural environments, perception and map build-ing, data fusion, control architectures and real-time systems,and learning. He leads or has led several projects on fieldand service robotics.

Günter Hommel received his degree in electrical engineer-ing from Technische Universität (TU), Berlin, Germany, in1970. He first joined the Department of Electrical Engineer-ing and then the Department of Computer Science at TUBerlin, where he received his Ph.D. in computer science. In1978, he joined the Nuclear Research Center in Karlsruhe,working in the field of real-time systems. From 1980, he wasthe head a research group in software engineering at the Ger-man National Research Center for Information Technology

(GMD) in Bonn. In 1982, he was appointed professor at theInstitute for Computer Science of TU Munich, where heworked in the fields of real-time programming and robotics.Since 1984, he has been a professor at the Institute for Techni-cal Computer Science of TU Berlin, heading the Real-TimeSystems and Robotics Group. In 2002, he received an hon-orary doctorate (Dr.h.c.) from the Moscow State Academy ofInstrument Engineering and Computer Science.

F.J. Colmenero Lechuga is a computer science engineer. He received a degree in intelligence artificial from the HigherTechnical School of Computer Engineers of Granada, Spain.He has been a project engineer at the Mission Planning Divi-sion of GMV S.A. since 1999. He has collaborated in thedevelopment of several systems related to space, defense, andcivil markets.

Begoña C. Arrue is an associate professor at the Universityof Seville, Spain. She also serves as a project reviewer for theSpanish National Evaluation Agency. Her research interests arein computer vision, neural networks, wavelets, perception sys-tems, and intelligent systems. She received her B.S. in com-puter science from the University of Deusto, Spain, and herM.S. and Ph.D. degrees in electrical engineering from theUniversity of Virginia.

Joaquín Ferruz received the M.Sc. degree in industrialengineering in 1986 and the Ph.D. in 1997, both from theUniversity of Seville, Spain. He was a member of theResearch and Development Laboratory of Fujitsu EspañaS.A. from 1985–1992 as a design engineer. Since 1992, hehas been with the Systems and Automation Department ofthe University of Seville; he is currently an assistant profes-sor. His main research interests are computer vision, real-time systems, and aerial robotics.

Jose Ramiro Martinez-de Dios received his B.S. andPh.D. degrees in 1996 and 2001, respectively, in telecommu-nication engineering, from the University of Seville, Spain.He has been an ssistant professor since 1999. His researchinterests include vision segmentation techniques in unstruc-tured environments. He developed forest fire detection andfalse-alarm-reduction techniques. He has published more than20 papers and worked on more than 15 research and develop-ment projects.

Fernando Caballero received a degree in telecommunica-tion engineering in 2003 from the University of Seville,Spain. Currently, he is a Ph.D. student at the University ofSeville, where he is researching computer vision algorithmsand working on several national and European projects.

Address for Correspondence: Aníbal Ollero, Escuela Superiorde Ingenieros de la Universidad de Sevilla, Camino de losDescubrimientos, 41092 Sevilla, Spain. E-mail: aollero@cartuja.us.es.

JUNE 2005 IEEE Robotics & Automation Magazine 57