DyKnow: An Approach to Middleware for Knowledge Processing

Fredrik Heintz∗and Patrick Doherty†

Dept. of Computer and Information ScienceLinkoping University,

581 83 Linkoping, Sweden{frehe, patdo}@ida.liu.se

Abstract

Any autonomous system embedded in a dynamic andchanging environment must be able to create qualitativeknowledge and object structures representing aspects ofits environment on the fly from raw or preprocessed sen-sor data in order to reason qualitatively about the environ-ment. These structures must be managed and made ac-cessible to deliberative and reactive functionalities whichare dependent on being situationally aware of the changesin both the robotic agent’s embedding and internal envi-ronment. DyKnow is a software framework which pro-vides a set of functionalities for contextually accessing,storing, creating and processing such structures. The sys-tem is implemented and has been deployed in a deliber-ative/reactive architecture for an autonomous unmannedaerial vehicle. The architecture itself is distributed anduses real-time CORBA as a communications infrastruc-ture. We describe the system and show how it can be usedin execution monitoring and chronicle recognition scenar-ios for UAV applications.

1 Introduction

Research in cognitive robotics is concerned with endow-ing robots and software agents with higher level cognitivefunctions that enable them to reason, act and perceive ina goal-directed manner in changing, incompletely known,

∗Corresponding author, tel no +46-13-282428, fax no +46-13-142231

†Both authors are supported by grants from the Wallenberg Founda-tion, Sweden and NFFP 539 COMPAS.

and unpredictable environments. Research in robotics hastraditionally emphasized low-level sensing, sensor pro-cessing, control and manipulative tasks. One of the openchallenges in cognitive robotics is to integrate techniquesfrom both disciplines and develop architectures whichsupport the seamless integration of low-level sensing andsensor processing with the generation and maintenance ofhigher level knowledge structures grounded in the sensordata.

Knowledge about the internal and external environ-ments of a robotic agent is often both static and dynamic.A great amount of background or deep knowledge is re-quired by the agent in understanding its world and in un-derstanding the dynamics in the embedding environmentwhere objects of interest are congnized, hypothesized asbeing of a particular type or types and whose dynamicsmust be continuously reasoned about in a timely manner.This implies signal-to-symbol transformations at manylevels of abstraction with different and varying constraintson real-time processing.

Much of the reasoning involved with dynamic objectsand the dynamic knowledge related to such objects in-volves issues of situation awareness. How can a roboticsarchitecture support the task of getting the right informa-tion in the right form to the right functionalities in thearchitecture at the right time in order to support decisionmaking and goal-directed behavior? Another importantaspect of the problem is the fact that this is an on-goingprocess. Data and knowledge about dynamic objects hasto be provided continuously and on-the-fly at the rate andin the form most efficient for the receiving cognitive orreactive robotics functionality in a particular context.

Context is important because the most optimal rates andforms in which a robotic functionality receives data areoften task and environmentally dependent. Consequently,autonomous agents must be able to declaratively specifyand re-configure the character of the data received. Howto define a change, how to approximate values at time-points where no value is given and how to synchronizecollections of values are examples of properties that canbe set in the context. By robotic functionalities, we meancontrol, reactive and deliberative functionalities rangingfrom sensor manipulation and navigation to high-levelfunctionalities such as chronicle recognition, trajectoryplanning, and execution monitoring.

The paper is structured as follows. We start with sec-tion 2 where a larger scenario using the proposed frame-work is described. In section 3, the UAV platform usedin the project is briefly described. In section 4, DARA, aDistributed Autonomous Robotics Architecture for UAVsis briefly described. DyKnow is an essential module inthis architecture. In section 5, the underlying ontologyfor dynamic knowledge and object structures is described.In section 6, the basic structure and implementation ofthe DyKnow framework is described. In sections 7.2 and7.3, two deliberative functionalities which use the Dy-Know framework are considered, chronicle recognitionand execution monitoring, in addition to the dynamic ob-ject repository (DOR) described in section 7.1. We con-clude in section 8 with a discussion of the role of the Dy-Know framework and some related work.

2 An Identification and Track Sce-nario

In order to make these ideas more precise, we will beginwith a scenario from an unmanned aerial vehicle projectthe authors are involved in which requires many of thecapabilities discussed so far.

Picture the following scenario. An autonomous un-manned aerial vehicle (UAV), in our case, a helicopter, isgiven a mission to identify and track a vehicle with a par-ticular signature in a region of a small city. The signatureis provided in terms of color and size (and possibly 3Dshape). Assume that the UAV has a 3D model of the re-gion in addition to information about building structures

and the road system. These models can be provided ormay have been generated by the UAV itself. Additionally,assume the UAV is equipped with a GPS and INS1 fornavigating purposes and that its main sensor is a cameraon a pan/tilt mount.

Let’s consider the processing from the bottom up, eventhough in reality, there will be many feedback loops inthe UAV architecture. One way for the UAV to achieveits task would be to initiate a reactive task procedure (par-ent procedure) which calls the systems image processingmodule with the vehicle signature as a parameter. The im-age processing module might then try to identify coloredblobs in the region of the right size, shape and color asa first step. These object descriptions would have to besent to a module in the architecture called the dynamicobject repository (DOR) which is responsible for the dy-namic management of such objects. Each of thesevisionobjectswould contain features related to the image pro-cessing task such as RGB values with uncertainty bounds,length and width in pixels, position in the image, a sub-image of the object which can be used as a template fortracking, an estimate of velocity, etc.

From the perspective of the UAV, these objects are onlycognized to the extent that they are moving colored blobsof interest and the feature data being collected shouldcontinue to be collected while tracking those objects per-ceived to be of interest. What objects are of interest?The parent procedure might identify that or those objectswhich are of interest based on a similarity measure ac-cording to size, color and movement. In order to do this,the DOR would be instructed to create one or moreworldobjectsand link them to their respective vision objects.At this point the object is cognized at a more qualitativelevel of abstraction, yet its description in terms of its link-age structure contains both cognitive and pre-cognitive in-formation which must be continuously managed and pro-cessed due to the interdependencies of the features at var-ious levels.

A world object could contain additional features suchas position in a geographic coordinate system rather thanthe low-level image coordinate. Generating a geographiccoordinate from an image coordinate continuously, calledco-locationis a complex process that involves combining

1GPS and INS are acronyms for global positioning system and iner-tial navigation system, respectively.

2

dynamic data about features from several different objectssuch as the camera object, helicopter object and worldobjects, together with data from an onboard geographi-cal information system (GIS) module which is also partof the architecture. One would require a computationalunit of sorts that takes streamed data as input and outputsa new stream at a higher level of abstraction representingthe current geographical coordinate of the object. This co-location process must occur in real-time and continuallyoccur as the world object is tracked. This implies that allfeatures for all dynamic objects linked to the world objectin focus have to be continually updated and managed.

At this point, the parent task may want to make a com-parison between the geographical coordinate and the posi-tion of that coordinate in terms of the road system for theregion, information of which is stored in the onboard GIS.This indexing mechanism is important since it allows theUAV to reason qualitatively about its spatial surroundings.Let’s assume this is done and after some period of track-ing and monitoring the stream of coordinates, the parentprocedure decides that this looks like a vehicle that is fol-lowing the road.On-roadobjects might then be createdfor each of the world objects that pass the test and linkedto their respective world objects. An on-road object couldcontain more abstract and qualitative features such as po-sition in a road segment which would allow the parentprocedure to reason qualitatively about its position in theworld relative to the road, other vehicles on the road, andother building structures in the vicinity of the road. At thispoint, streams of data are being generated and computedfor many of the features in the linked object structures atmany levels of abstraction as the helicopter tracks the on-road objects.

The parent procedure could now use static knowledgestored in onboard knowledge bases and the GIS togetherwith this dynamic knowledge to hypothesize as to the typeof vehicle. The hypothesis would of course be based onthe linkage structure for an on-road object and variousfeatures at different levels of abstraction. Assume the par-ent procedure hypothesizes that the on-road object is a car.A car objectcould then be created and linked to the ex-isting linkage structure with additional high-level featureinformation about the car.

Whether or not the sum of streamed data which makesup the linkage structure represents a particular type ofconceptual entity will only ever remain a hypothesis

which could very well change, based on changes in thecharacter of the streams of data. Monitors, users ofthese structures, would have to be set up to observe suchchanges and alert the parent procedure if the changes be-come too abnormal relative to some criteria determinedby the parent procedure. Abnormality is a concept thatis well-suited for being reasoned about at a logical leveland the streamed data would have to be put into a formamenable to this type of processing.

How then can an architecture be set up to support theprocesses described in the UAV scenario above? This isthe main topic of this paper and in it we propose a soft-ware system calledDyKnow.2

3 The WITAS UAV Platform

The WITAS3 Unmanned Aerial Vehicle Project [3,4] is a long-term basic research project whose mainobjectives are the development of an integrated hard-ware/software VTOL (Vertical Take-Off and Landing)platform for fully-autonomous missions and its future de-ployment in applications such as traffic monitoring andsurveillance, emergency services assistance, photogram-metry and surveying.

The WITAS UAV platform we use is a slightly modi-fied Yamaha RMAX (figure 1). It has a total length of 3.6m (incl. main rotor), a maximum take-off weight of 95kg, and is powered by a 21 hp two-stroke engine. Yamahaequipped the radio controlled RMAX with an attitude sen-sor (YAS) and an attitude control system (YACS).

The hardware platform consists of three PC104 em-bedded computers (figure 2). The primary flight con-trol (PFC) system consists of a PIII (700Mhz) proces-sor, a wireless Ethernet bridge and the following sen-sors: a RTK GPS (serial), and a barometric altitude sen-sor (analog). It is connected to the YAS and YACS (se-rial), the image processing computer (serial) and the de-liberative computer (Ethernet). The image processing(IP) system consists of a second PC104 embedded com-

2”DyKnow” is pronounced as ”Dino” in ”Dinosaur” and stands forDynamic Knowledge and Object Structure Processing.

3WITAS (pronouncedvee-tas) is an acronym for the Wallen-berg Information Technology and Autonomous Systems Laboratory atLinkoping University, Sweden.

3

Figure 1: The WITAS RMAX Helicopter

puter (PIII 700MHz), a color CCD camera (S-VIDEO,serial interface for control) mounted on a pan/tilt unit (se-rial), a video transmitter (composite video) and a recorder(miniDV). The deliberative/reactive (D/R) system runson a third PC104 embedded computer (PIII 700MHz)which is connected to the PFC system with Ethernet usingCORBA event channels. The D/R system is described inmore detail in the next section.

For further discussion, it is important to note that com-putational processes are executed concurrently on dis-tributed hardware. Data flow is both synchronous andasynchronous and the concurrent distributed nature of thehardware platform contributes to diverse latencies in dataflow throughout the system.

4 DARA: A Distributed Au-tonomous Robotics Architecture

The DARA system [6] consists of both deliberative andreactive components which interface to the control ar-chitecture of the primary flight controller (PFC). Cur-rent flight modes include autonomous take-off and land-ing, pre-defined and dynamic trajectory following, ve-hicle tracking and hovering. We have chosen real-timeCORBA [15]4 as a basis for the design and implementa-tion of a loosely coupled distributed software architecturefor our aerial robotic system.

The communication infrastructure for the architecturesis provided by CORBA facilities and services. Figure 3

4We are currently using TAO/ACE.TheAceOrb is an open sourceimplementation of CORBA 2.6.

RTLINUX

RTLINUX

TCP/IP

700Mhz PIII/256Mbram/500Mbflash

700Mhz PIII/256Mbram/500Mbflash

700Mhz PIII/256Mbram/256Mbflash

GPS

serial analog

magneticcompass

pressuresensor

temp.sensors

cameracontrol

framegrabberBT878

preprocessor

IPAPI

pathplanner

taskplanner

knowledgerepository

TP exec

chroniclerecognition

GIS

DOR

Other. . .

Helicopter Control

RMAX HelicopterPlatform

YamahaAttitudeController

roll

yaw

pitch

200Hz

50Hz

Camera Platform

Color CCD Camera/PTU

mini-dv

YamahaAttitudeSensors

200/66Hz

LINUX

RS232

sonar

Figure 2: DARA Hardware Schematic

depicts an (incomplete) high-level schematic of some ofthe software components used in the architecture. Eachof these may be viewed as a CORBA server/client provid-ing or requesting services from each other and receivingdata and events through both real-time and standard eventchannels.

The modular task architecture (MTA) which is part ofDARA is a reactive system design in the procedure-basedparadigm developed for loosely coupled heterogeneoussystems such as the WITAS aerial robotic system. Reac-tive behaviors are implemented astask procedures(TP)which are executed concurrently and essentially event-driven. A TP may open its own (CORBA) event channels,and call its own services (both CORBA and application-oriented services such as path planners) including func-tionalities in DyKnow.

Given the distributed nature of both the hardware andsoftware architectures in addition to their complexity, oneof the main issues is getting data to the right place at theright time in the right form and to be able to transform thedata to the proper levels of abstraction for use by high-level deliberative functionalities and middle level reactive

4

functionalities. In the following sections we will describea very important component of the DARA system whichcontributes to achieving this.

������������ �

����� ���

�������������� ���

���������

����� ���

������������������ �� ������!��� "

��# ���

���� ��$��%�

&����������� ��$��%�

�� ���������$��%�

������������$��%�

'���� ��&�� ������

������&�����&�� ������

(����&�� ������

(�)�((�)�(���� ��

(�������������� ������!(� "

*��� � %�$�������������

&�� ������

Figure 3: DARA Software Schematic

5 An Ontological Specification forDyKnow

In any knowledge representation endeavor, it is impor-tant to provide a concise specification of the entities oraspects which are the focus of representation. That beingsaid, it is surprisingly difficult to do this in the context ofDARA and DyKnow for the following reasons:

• There are often multiple representations of the same en-tities or aspects associated with those entities. Often thismay be known by the system, but equally often it may not.

• The physical locations of representational structures asso-ciated with entities must be taken into account due to thedistributed and combined asynchronous/synchronous na-ture of the DARA architecture which often results in un-predictable and varying latencies.

• The dual requirement of not only reasoning about the enti-ties and aspects being represented, but the representationalstructures themselves since the ultimate purpose of Dy-Know is that of situational awareness, getting the right in-formation in the right form to the right place in a timelymanner.

Ontologically, we view the external and internal environ-ment of the agent as consisting of entities representingphysical and non-physical objects, properties associatedwith these entities, and relations between entities. We willcall such entitiesobjectsand those properties or relationsassociated with objects will be calledfeatures. Featuresmay be static or dynamic and parameterized with objects.

Due to the potentially dynamic nature of a feature, thatis, its ability to change value through time, afluent is as-sociated with each feature. A fluent is simply a functionof time whose range is the feature’s type. For a dynamicfeature, the fluent values will vary through time, whereasfor a static feature the fluent will remain constant throughtime.

Some examples of features would be theestimated ve-locity of a world object, thecurrent road segmentofan on-road object, or thedistance betweentwo car ob-jects. Each fluent associated with these examples implic-itly generates a continuous stream of time tagged valuesof the appropriate type.

Additionally, we will introduce locations, policies,computational unitsandfluent streamswhich refer to as-pects of fluent representations in the actual software ar-chitecture. At first glance, this may appear as an ontologi-cal level error because software architectures are intendedto implement declarative specifications of entities in ourontologies. On the other hand, it is useful to use the repre-sentational domain which the architecture implements asa domain of discourse for an autonomous agent if one isinterested in reasoning about limited reflective capabilityas to source, quantity and quality of data flowing througha system and dynamically specifying its use by variousfunctionalities to achieve tasks.

A location is intended to denote any pre-defined phys-ical or software location that hosts feature data in theDARA architecture. Some examples would be onboardor offboard databases, CORBA event channels, physicalsensors or their device interfaces, etc. In fact, a locationwill be used as an index to reference a representationalstructure associated with a feature. This structure denotesthe process which implements the fluent associated withthe feature. A fluent implicitly represents a stream of data,a fluent stream. The stream is continuous, but can onlyever be approximated in an architecture. Apolicy is in-tended to represent a particular contextual window or fil-ter used to access a fluent data stream. Particular function-alities in the architecture may need to sample the streamat a particular rate or interpolate values in the stream ina certain manner. Policies will denote such collections ofconstraints.Computational unitsare intended to denoteprocesses which take fluent streams as input, perform op-erations on these streams and generate new fluent streamsas output.

5

In summary, the ontology consists of objects, features,fluents, in addition to fluent streams, locations, policies,and computational units. Each of these entities has to berepresented either syntactically or in the form of a datastructure within the architecture and many of these datastructures are grounded through sensor data perceivedthrough the robotic agent’s sensors.

An ontologically difficult issue involves the meaning ofan object. In a distributed architecture such as DARA,information about a specific object is often distributedthroughout the system, some of this information may beredundant and it may often even be inconsistent due to is-sues of precision and approximation. For example, givena car object, we have seen that it is part of a linkage struc-ture which may contain otherobjectssuch as on-road,world and vision objects. In addition, many of the fea-tures associated with these objects are computed in dif-ferent manners in different parts of the architecture withdifferent latencies.

One candidate definition for an object could be the ag-gregate of all features parameterized with the object iden-tifier. But an object only represents some aspects of anentity in the world. To represent that several differentobjects actually represent the same entity in the world,links are created between those objects. It is these link-age structures that represent all the aspects of an entitywhich are known to the UAV agent. It can be the case thattwo linkage structures in fact represent the same entity inthe world but the UAV agent is unable to determine this.Two objects may even be of the same type but have differ-ent linkage structures associated with them. For example,given two car objects, one may not have an on-road ob-ject, but an off-road object, as part of its linkage structure.It is important to point out that objects as intended herehave some similarities with OOP objects, but many dif-ferences. For more details see [13].

6 The DyKnow Framework

In section 5, we considered a number of ontologicaldistinctions that have to be represented within DARA asdata structures. In addition, since declarative specifica-tions of both features and policies that determine viewsof fluent streams are 1st-class citizens in DyKnow, a lan-

guage for referring to features, locations, computationalunits and policies has to be provided.

6.1 The Specification Level

Let’s begin at the declarative level. We use an ex-tended version of a logic of action and change calledTAL [5] to provide the syntax for features and flu-ents. Any structured primitive featuref is denoted asf(loc, policy, arg1, . . . , argk). The parameterloc de-notes a system location from a pre-defined set of locationsassociated with DARA. The location could be a database,an event channel, a sensor, etc. The parameterpolicy de-notes a set of constraints one wishes to place on the fea-turef ’s system representation of its associated fluent. Thesystem representation of the fluent will generate a streamof fluent data of the proper type. Often, one may wantto control the character of the fluent stream in terms ofsampling rate, interpolation strategy, etc. The parametersarg1, . . . , argk denote the arguments to the feature. Forsimplicity, we assume these areobject symbols. Here area few examples:

estimatedVelocity(DOR, policy1, carObj1),sfs1 = cameraPos1(camSensor1, policy2, camObj1),sfs2 = cameraPos2(camSensor2, policy3, camObj1)sfs3 = cameraPos2(DOR, policy4, camObj1)Note that in the latter two cases, we are essentially re-

ferring to the same property of the onboard camera. Inthe first case the position is stored in the actual sensor in-terface while in the second a representation of the sensorinformation exists in the DOR. It is the same feature, buteach representation may have different properties.

Additionally, we allow structured complex features,whose intended purpose is to denote features definedin terms of others through some operation on the inputfeatures. Any structured complex featuref is denoted asf(loc, compUnit(fs1, . . . , fsl), policy, arg1, . . . , argk).In this case, the only difference between a structuredprimitive feature and a structured complex feature is thatan additional function representing a computational unit isadded, with structured primitive or complex features as ar-guments. For example, one might like to fuse stream datafrom the two different sources of data for camera position,denotedsfs1 and sfs2, above. This could be denoted ascameraPos(DOR, fuseCameraPos(sfs1, sfs2), pol5, heli)whereheli refers to the UAV platform.

6

One could argue that too many aspects of the archi-tecture’s implementation level are being lifted into thedeclarative specification language, but that is the point.Various high-level functionalities in autonomous systemsmay often be required to reason about sources, quantity,and quality of data in the absence of operator assistance.Context will also determine what kind of data and howmuch is required for the task at hand. This particularframework provides the means of reflecting on aspects ofthe underlying architecture of the system. In fact, in com-plex distributed environments, autonomous agents may infact be receiving data streams from outside sources andthe location and policy parameters could easily reflect thisand be used in the systems reasoning processes.

6.2 The Data Representation Level

The meaning of features, fluents, locations, policies,computational units, and objects is determined throughrepresentational structures in DARA. These in turn aregrounded through sensors on a continual basis during theachievement of mission tasks. The nature of these repre-sentational structures is the topic of this section.

Associated with each primitive structured feature sym-bol is aprimitive feature representation structure

〈sfs,fluent spec.,fluent generator , update spec.,

access spec., cache spec.〉

A primitive feature representation structure pro-vides information about access and update methods(access spec., update spec.) for a feature at a location(sfs); storage constraints for samples of the feature(cache spec.) and constraints (fluent spec.) on a process(fluent generator ) which can be invoked to generate astream of data representing the fluent associated with thefeature in question. From a semantic perspective, a fea-ture representation structure denotes in some sense, themeaning of the fluent associated with the feature in thesfs. The system can not represent the fluent more pre-cisely due to the inherent limitations of its implementationat this specific location in the architecture.

For example, different types of sensors may place anupper or lower bound on legal sampling capability, or par-ticular types of event channels may only allow push or

pull or both push and pull subscription to data. These par-ticular constraints can not be changed by any user of thisfeature at this location, although if there are choices asto access, these may be further constrained by the repre-sentational equivalent of a policy which we will considershortly. Thecache specification can specify how manydata values are stored or for what interval of time datavalues should be stored in the cache. For instance, the lo-cation may only allow the storage of the current value, orof the current and previous value, or of a bounded streamof values generated in the last 3 seconds.

Associated with each complex structured feature sym-bol is acomplex feature representation structure

〈sfs,fluent spec., compUnit(fs1, . . . , fsl),access spec., cache spec.〉

There is little difference between a complex and prim-itive feature representation structure other than that thefluent generator is replaced with an instantiated computa-tional unit and there is no update specification since theintention is that such structures represent nodes in a dataflow network consisting of combinations of streams andcomputational units. For example, such a structure wouldbe used to translate between coordinates in an image andgeographical coordinates.

The internal data structure used to represent a fluentstream is the sample trace, which is a set of samples.Sample traces are generated by the fluent generator orcomputational unit relative to the sum of constraints inthe different specifications. A sample is a triple〈ti, ts, v〉consisting of a value and two time-points. The first time-point, the index time, represents the time at which its cor-responding value is relevant, e.g. the time a picture wastaken from which the position of an object was extracted.The second time-point, the sample time, is the time atwhich the sample was created. This information is im-portant when reasoning about latencies. The time-pointsthemselves can be sampled either from a discrete or a con-tinuous clock.

Associated with each policy is apolicy descriptor. Apolicy descriptor instantiates an interface to a specificfeature representation and can be used to place furtherconstraints on the fluent stream generated by the specificfeature representation. Any functionality in DARA thatwants to access a specific feature must do so through a

7

policy descriptor. The policy descriptor is thus contex-tual in nature and there can be many users of a specificfeature representation, each with its own unique policy.There are many uses of policy descriptors. For instance,they can be used to implement specific sampling rates andwhen specific values are not provided by the associatedfeature representation, the policy descriptor can interpo-late stream values in different ways to assure values foreach sample. A descriptor can be used as a filter and onlygenerate values within upper and/or lower bounds, or onlygenerate values when there is a change in the stream. Inthe forthcoming examples, some of these techniques willbe demonstrated.

6.3 The Semantic Level

The specification level provides the DARA system with alogical tool consisting of an extended version of a logicof action and change, TAL (temporal action logic) whichcan be used to specify features used in DARA, data flownetworks constructed from computational units and flu-ent streams, and policies which contextually modify flu-ent streams for particular constraints associated with mis-sion tasks. In addition, TAL can be used as a basis for plangeneration, execution monitoring, diagnosis, and chroni-cle recognition, although currently, it is primarily used forspecification purposes.

The data representation level implements the DyKnowframework which relates the formal specification to sen-sory input via a suite of representational structures de-scribed in the previous section. The data representationlevel can also be used separately from the specificationlevel by the different deliberative, reactive and controlfunctionalities in the DARA system.

In fact, one can view DyKnow as implementing a dis-tributed qualitative signal processing tool where the sys-tem is given the functionality to generate dynamic repre-sentations of parts of its internal and external environmentin a contextual manner through the use of policy descrip-tors and feature representation structures. The dynamicrepresentations can be viewed as collections of time se-ries data at various levels of abstraction, each time seriesrepresenting a particular feature and each bundle repre-senting a particular history or progression. Another viewof such dynamic representations and one which is actuallyput to good use is to interpret the fluent stream bundles as

partial temporal models in the logical sense. These partialtemporal models can then be used on the fly to interprettemporal logical formulas in TAL or other temporal for-malisms. Such a functionality can be put to good use inconstructing execution monitors, predictive modules, di-agnostic modules, etc. The net result is a very powerfulmechanism for dealing with a plethora of issues associ-ated with focus of attention and situational awareness.

7 Applications using DyKnow

In the following two subsections, we will show howthe DyKnow framework can be used to generate fluentstreams for further processing by two important delibera-tive functionalities in the DARA system, chronicle recog-nition and execution monitoring. Both are implemented inthe UAV system. Before doing this, we provide a short de-scription of the Dynamic Object Repository (DOR), an es-sential part of the DARA which uses the DyKnow frame-work to provide other functionalities in the system withinformation about the properties of dynamic objects mostoften constructed from sensor data streams.

7.1 The Dynamic Object Repository

The Dynamic Object Repository (DOR) is essentially asoft real-time database used to construct and manage theobject linkage structures described in the introduction.The DOR is implemented as a CORBA server and theimage processing module interfaces to the DOR and sup-plies vision objects. Task procedures in the MTA accessfeature information about these objects via the DyKnowframework, creating descriptors on-the-fly and construct-ing linkages. Computational units are used to provide val-ues for more abstract feature properties associated withthese objects. For example, the co-location process in-volving features from the vision, helicopter and cameraobjects, in addition to information from the GIS, use com-putational units to output geographical coordinates. Theseare then used to update the positional features in worldobjects linked to the specific vision objects in question.

Objects are referenced via unique symbols which arecreated by the symbol generation module which is partof the DOR. Each symbol is typed using pre-defined do-mains such as car, world-object, vision-object, vehicle,

8

etc. Symbols can be members of more than one domainand are used to instantiate feature representations and asindexes for collecting information about features whichtake these symbols as arguments. Since domains collectsymbols whichreferencea certain type of object, one canalso conveniently ask for information about collections oraggregates of objects. For example, “take all vision ob-jects and process a particular feature for each in a certainmanner”.

7.2 An Application to Chronicle Recogni-tion

Chronicles are used to represent complex occurrencesof activity described in terms of temporally constrainedevent structures. In this context, an event is defined as achange in the value of a feature. For example, in a trafficmonitoring application, a UAV might fly to an intersectionand try and identify how many vehicles turn left, right ordrive straight through a specific intersection. In anotherscenario, the UAV may be interested in identifying vehi-cle overtaking. Each of these complex activities can bedefined in terms of one or more chronicles. In the WITASUAV, we use the CRS chronicle recognition system devel-oped by France Telecom. CRS is an extension of IxTeT[8]. Our chronicle recognition module is wrapped as aCORBA server.

As an example, suppose we would like to recognize ve-hicles passing through an intersection. Assume cars arebeing identified and tracked through the UAV’s cameraas it hovers over a particular intersection. Recall that theDOR generates and maintains linkage structures for vehi-cles as they are identified and tracked. It can be assumedthat the following structured features exist:pos = position(DOR, policy1, car1)roadseg = road segment(DOR, roadSegment(pos), pol-icy2, car1)incross = in crossing(DOR, inCrossing(roadseg), policy3,car1)

pos is a feature of a car object and its fluent stream canbe accessed via the DOR as part of its linkage structure.roadseg is a complex feature whose value is calculatedvia a computational unitroadSegment which takes thegeographical position of a world object associated with

the car object as argument and uses this as an index intothe GIS to return the road segment that the vehicle is in.Similarly, incross is a complex feature structure whoseboolean value is calculated by using a computational unitthat takes theroadseg fluent stream as input and returns aboolean output stream calculated via a lookup in the GIS.

For the sake of brevity, a car is defined to pass throughan intersection if its road segment type is not a crossingthen it eventually is in a road segment that is a crossingand then it is again in a road segment that is not a cross-ing. In this case, if the fluent stream generated byincrossgenerates samples going from false to true and then even-tually true to false within a certain time frame then the caris recognized as passing through a crossing. The chron-icle recognition system would receive such streams andrecognize two change events which match its chronicledefinition.

The stream itself requires some modification andpol-icy3 specifies this via amonotonic timeconstraint and achangeconstraint. The monotonic time constraint wouldmake sure the stream is ordered, i.e. the time stampof events increase monotonically. The change constraintspecifies how change is defined for this stream. There areseveral alternatives which can be used:

• any change policy– any difference between the previousand current value is a change;

• absolute change policy– an absolute difference betweenthe previous and current value larger than a parameterdelta is a change;

• relative change policy– a normalized difference betweenthe previous and current value larger than a parameterdelta is a change.

There are obvious variations on these policies for dif-ferent types of signal behavior. For example, one mightwant to deal with oscillatory values due to uncertainty ofdata, etc. The example used above is only intended toprovide an overview as to how DyKnow is used by othermodules and is therefore simplified.

7.3 An Application to Execution Monitor-ing

The WITAS UAV architecture has an execution mon-itoring module which is based on the use of a tempo-ral logic, LTL (linear temporal logic with intervals [14]),

9

which provides a succinct syntax for expressing highlycomplex temporal constraints on activity in the UAV’s in-ternal environment and even aspects of its embedding en-vironment. For example safety and liveness conditionscan easily be expressed. Due to page limitations wecan only briefly describe this functionality. Essentially,we appeal to the intuitions about viewing bundles of flu-ent streams as partial models for a temporal logic andevaluating formulas relative to this model. In this casethough, the model is fed piecewise (state-wise) to the ex-ecution monitor via a state extraction mechanism associ-ated with the execution monitor. A special progressionalgorithm [14] is used which evaluates formulas in a cur-rent state and returns a new formula which if true on thefuture states would imply that the formula is true for thecomplete time-line being generated.

The DyKnow system is ideal for generating suchstreams and feeds these to the execution monitor. Sup-pose we would like to make sure that two task proce-dures (all invocations) in the reactive layer of the DARA,called A and B, can never execute in parallel. For ex-ample, A and B may both want to use the camera re-source. This safety condition can be expressed in LTLas the temporal formulaalways¬(∃x∃y tp name[x]=A∧tp running[x]=true∧ tp name[y]=B∧ tp runing[y]=true),where “always” in the formula is the modal operator for“at all times”. To monitor this condition the executionmonitor requires fluent streams for each of the possibleinstantiations of the parameterized featurestp name andtp running which can be generated by the reactive layerof the DARA. These are fed to the instantiated executionmonitor which applies the progression algorithm to thetemporal formula above relative to the fluent streams gen-erated via the DyKnow framework. This algorithm is runcontinuously. If the formula evaluates to false at somepoint, an alert message is sent to a monitor set up by thefunctionality interested in this information and modifica-tions in the system configuration can be made.

8 Related Work

The DyKnow framework is designed for a distributed,real-time and embedded environment [17, 18] and is de-veloped on top of an existing middleware platform, real-time CORBA [19], using the real-time event channel

[12], the notification [11] and the forthcoming real-timenotification [9] services. One of the purposes for thiswork is in the creation of a knowledge processing mid-dleware capability, i.e. a framework for interconnect-ing different knowledge representation and reasoning ser-vices, grounding knowledge in sensor data and provid-ing uniform interfaces for processing and management ofgenerated knowledge and object structures. The frame-work is quite general and is intended to serve as a plat-form for investigating a number of pressing issues as-sociated with the processing and use of knowledge onrobotic platforms with soft and hard real-time constraints.These issues include anchoring, or more generally symbolgrounding, signal to symbol transformations, informationfusion, contextual reasoning, and focus of attention. Ex-amples of application services which use the middlewarecapabilities are execution monitoring services, anchoringservices and chronicle recognition services.

We are not aware of any similar frameworks, but theframework itself uses ideas from many diverse researchareas mainly related to real-time, active, temporal, andtime-series database [7, 16, 20], data stream management[1, 2, 10], and work in the area of knowledge representa-tion and reasoning.

The main differences between DyKnow and thedatabase and data stream approaches are that we have adifferent data model based on the concepts of features andfluents and we have many views or representations of thesame feature data in the system each with different prop-erties depending on the context where the feature is usedas described by a policy.

References

[1] D. Abadi, D. Carney, U. Cetintemel, M. Cherniack,C. Convey, S. Lee, M. Stonebraker, N. Tatbul, andS. Zdonik. Aurora: A new model and architecturefor data stream management.VLDB Journal, August2003.

[2] Brian Babcock, Shivnath Babu, Mayur Datar, Ra-jeev Motwani, and Jennifer Widom. Models and is-sues in data stream systems. InProceedings of 21stACM Symposium on Principles of Database Systems(PODS 2002), 2002.

10

[3] P. Doherty. Advanced research with autonomous un-manned aerial vehicles. InProceedings on the 9thInternational Conference on Principles of Knowl-edge Representation and Reasoning, 2004.

[4] P. Doherty, G. Granlund, K. Kuchcinski, E. Sande-wall, K. Nordberg, E. Skarman, and J. Wiklund. TheWITAS unmanned aerial vehicle project. InPro-ceedings of the 14th European Conference on Artifi-cial Intelligence, pages 747–755, 2000.

[5] P. Doherty, J. Gustafsson, L. Karlsson, and J. Kvarn-strom. (TAL) temporal action logics: Languagespecification and tutorial.Electronic Transactionson Artificial Intelligence, 2(3-4):273–306, 1998.http://www.ep.liu.se/ej/etai/1998/009/.

[6] P. Doherty, P. Haslum, F. Heintz, T. Merz, P. Ny-blom, T. Persson, and B. Wingman. A distributedarchitecture for autonomous unmanned aerial vehi-cle experimentation. InProceedings of the 7th In-ternational Symposium on Distributed AutonomousRobotic Systems, 2004.

[7] Joakim Eriksson. Real-time and active databases: Asurvey. InProc. of 2nd International Workshop onActive, Real-Time, and Temporal Database Systems,1997.

[8] M. Ghallab. On chronicles: Representation, on-linerecognition and learning. InProceedings of the In-ternational Conference on Knowledge Representa-tion and Reasoning (KR-96), 1996.

[9] Pradeep Gore, Douglas C. Schmidt, Chris Gill, andIrfan Pyarali. The design and performance of a real-time notification service. InProc. of the 10th IEEEReal-time Technology and Application Symposium,may 2004.

[10] The STREAM Group. STREAM: The Stanfordstream data manager.IEEE Data Engineering Bul-letin, 26(1), 2003.

[11] R. Gruber, B. Krishnamurthy, and E. Panagos.CORBA notification service: Design challenges andscalable solutions. In17th International Conferenceon Data Engineering, pages 13–20, 2001.

[12] Tim Harrison, David Levine, and Douglas C.Schmidt. The design and performance of a real-time CORBA event service. InProceedings ofthe ACM SIGPLAN Conference on Object-OrientedProgramming Systems, Languages and Applications(OOPSLA-97), volume 32, 10 ofACM SIGPLANNotices, pages 184–200, New York, October 5–91997. ACM Press.

[13] Fredrik Heintz and Patrick Doherty. Managing dy-namic object structures using hypothesis generationand validation. InProceedings of the AAAI Work-shop on Anchoring Symbols to Sensor Data, 2004.

[14] K. Ben. Lamine and F. Kabanza. Reasoning aboutrobot actions: A model checking approach. InAd-vances in Plan-Based Control of Robotic Agents,LNAI, pages 123–139, 2002.

[15] Object Computing, Inc. TAO Developer’s Guide,Version 1.3a, 2003. See alsohttp://www.cs.wustl.edu/˜schmidt/TAO.html .

[16] Gultekin Ozsoyoglu and Richard T. Snodgrass.Temporal and real-time databases: A survey.IEEETrans. Knowl. Data Eng., 7(4):513–532, 1995.

[17] Douglas C. Schmidt. Adaptive and reflectivemiddleware for distributed real-time and embed-ded systems.Lecture Notes in Computer Science,2491:282–??, 2002.

[18] Douglas C. Schmidt. Middleware for real-time andembedded systems.Communications of the ACM,45(6):43–48, June 2002.

[19] Douglas C. Schmidt and Fred Kuhns. An overviewof the real-time CORBA specification.IEEE Com-puter, 33(6):56–63, June 2000.

[20] Duri Schmidt, Angelika Kotz Dittrich, WernerDreyer, and Robert W. Marti. Time series, a ne-glected issue in temporal database research? InPro-ceedings of the International Workshop on TemporalDatabases, pages 214–232. Springer-Verlag, 1995.

11