Human-style interaction with a robot for cooperativelearning of scene objects!

Shuyin Li, Axel Haasch, Britta Wrede, Jannik Fritsch, Gerhard SagererFaculty of TechnologyBielefeld University

33594 Bielefeld, Germany{shuyinli, ahaasch, bwrede, jannik, sagerer}@techfak.uni-bielefeld.de

ABSTRACTIn research on human-robot interaction the interest is cur-rently shifting from uni-modal dialog systems to multi-modalinteraction schemes. We present a system for human-styleinteraction with a robot that is integrated on our mobilerobot BIRON. To model the dialog we adopt an extendedgrounding concept with a mechanism to handle multi-modalin- and output where object references are resolved by the in-teraction with an object attention system (OAS). The OASintegrates multiple input from, e.g., the object and gesturerecognition systems and provides the information for a com-mon representation. This representation can be accessed byboth modules and combines symbolic verbal attributes withsensor-based features. We argue that such a representationis necessary to achieve a robust and e!cient informationprocessing.

Categories and Subject DescriptorsH.5.2 [Information Interfaces and Presentation]: UserIntefaces—Natural language; I.4.8 [Image Processing andComputer Vision]: Scene Analysis—Tracking, Object recog-nition; H.2.5 [Heterogeneous Databases]: HeterogeneousDatabases

General TermsManagement

1. INTRODUCTIONMulti-modality is one of the most important features that

characterize human-human social interaction. Based on this

!This work has been partially supported by the EuropeanUnion within the ’Cognitive Robot Companion’ (COGN-IRON) project (FP6-002020) and by the German ResearchFoundation within the Collaborative Research Center ’Sit-uated Artificial Communicators’ as well as the GraduateProgram ’Task Oriented Communication’.

Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page. To copy otherwise, torepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee.ICMI’05, October 4–6, 2005, Trento, Italy.Copyright 2005 ACM 1-59593-028-0/05/0010 ...$5.00.

observation, designers of computer systems have been try-ing to integrate multi-modal input and output mechanismsin human-machine interactions to enable more intuitive op-erations for human users. Given the huge amount of existingmulti-modal systems with quite di"erent notions of multi-modality we first want to discuss two important aspects ofmodality intuitiveness to clarify our position. In general itis assumed that the degree of intuitiveness of a modalitywill determine the smoothness and e!ciency of the inter-action it facilitates. Thus, the question is what modalitiesare the most intuitive for users? We argue that the an-swer is of evolutionary nature and is application-dependent.For example, the mouse has become the major modality inhuman-computer interaction for many users. For these usersthe mouse is probably one of the most intuitive ways to op-erate a computer although it has little to do with the naturalcommunication channels they use in human-human commu-nication such as speech. In contrast, many other users pre-fer to write commands directly. We can imagine that futuregenerations will find other modalities more intuitive thana mouse. The feeling of intuitiveness in computer opera-tion is thus continuously changing. Even the same peoplemay judge the intuitiveness of one modality di"erently whenoperating di"erent computer systems. For example, peopletend to anthropomorphize mobile robots when interactingwith them. We argue that this is even more the case forapplication areas where a robot is supposed to assist andaccompany humans in a social environment such as privatehouseholds. Our envisioned robot is supposed to “live” ina private household. Our long term goal, therefore, is toendow it with social capabilities so that it can become somesort of “companion”. This means that we do not envisionit to serve humans in a master-slave manner as people tendto suppose. Rather, a robot companion should cooperatewith humans to achieve certain goals. One basic function ofsuch a Robot Companion is to be able to learn interactivelyabout its environment. We therefore devised the home-tourscenario within our project where a human user is supposedto show his/her home to a newly purchased robot.

Based on our goal to design a robot that can be acceptedby the user as a companion we argue that the interactionwith such machines should be in a human style. We definethe term human style modalities as multi-modal communica-tion channels that humans are biologically equipped for and(learn to) use from their birth. Typical examples are speechand gestures. These modalities di"er from other modali-ties like mouse and keyboard in that they are learned nat-

COGNIRON FP6-IST-002020 Appendix

Appendix page 94 of 101

urally and without the use of artificial devices. In contrast,we define artificial modalities that are commonly used forhuman-computer interaction as virtual modalities becausetheir e"ect is a virtual one which is only observable by hu-mans via an artificial interface (e.g., the computer display).Thus, since people tend to anthropomorphize robots by ex-pecting human-like abilities and attributes we conclude thatrobots should be endowed with human-style modalities forinteraction.

A further aspect concerns the knowledge representation.Consider our home-tour scenario where the interaction willinvolve a high degree of deictic activity as the user willpoint to diverse objects in the environment. Thus, whenthe user points to his/her computer and explains “This ismy computer”, the robot should be able to recognize theuser’s gesture, find the computer and associate the sym-bolic name “computer” with a visual representation. Thisknowledge should be stored in a multi-modal way in orderto be retrievable from di"erent modules for further interac-tions. Psychological theories of knowledge representation inhumans suggest that the symbolic name of an object is asso-ciated with its sensory features like its image or haptic char-acteristics (e.g. [3]). When activating the name of an objectother features of the object are also activated. This indi-cates that the cognitive representation of objects in humansis multi-modal and therfore allows for multi-modal process-ing of information. In order for a robot to cooperate witha human, it should therefore be able to also process multi-modal information to build a representation similar to thatof its human communication partner to support a better mu-tual understanding. We therefore developed a multi-modalrepresentation scheme.

To summarize our position shortly: The intuitiveness ofmodalities needed for operation of computers and machineshas an evolutionary aspect and depends on the individualapplications. In our Robot Companion domain we are inter-ested in human-style multi-modality that should be consid-ered for both, communication channels and the representa-tion of knowledge. Its impact is of functional and technicalnature.

In this paper we will first present the multi-modal pro-cessing strategies of the Dialog System (section 4) and theObject Attention System (section 5) followed by a detaileddescription of our multi-modal representation scheme in sec-tion 6. Results in the form of a dialog example will be givenin section 7.

2. RELATEDWORKWhile there is an increasing interest in multi-modal inter-

faces there is only a very limited number of applications thatuse human-style modalities based on an integrated multi-modal knowledge representation.

That multi-modal cues are beneficial for increasing therobustness in human-robot interaction has been shown forexample in [13] where communication errors are detected byusing not only speech recognition scores but also by includ-ing laser data to infer the presence or absence of communica-tion partners and noise sources. Repair actions also involvethe use of multiple modalities by either driving around to ac-tively search for a communication partner or by o"ering but-tons as alternative communication channels in noisy environ-ments. More human-style interactions have been suggestedin [1] by modeling a naturalness support behavior. This be-

havior includes verbal strategies by inserting filler phrases,as well as non-verbal reactions such as nodding or headturning as reactions to environmental noise. The authorsreport positive reactions by the users but extensive evalua-tions still remain to be done. The benefits of using multi-modalities in a learning scenario have been demonstrated inseveral applications. For example, the robot Leonardo [2]can learn the names of buttons when a human communi-cation partner points them out by verbal and deictic in-structions. Leonardo also learns specific interactions withthese buttons by demonstration. However, while the inter-active capabilities of Leonardo are quite realistic the under-lying representations are simple and no new objects can belearned. An impressive system running on a mobile robot ispresented by Ghidary et al. [6]. The robot is able to learnobjects by analyzing speech commands and hand posturesof the user. The user gives verbal information about theobject’s size and can describe the spatial relations betweenobjects, e.g., by phrases like ‘left of my hand’. The rect-angular views of the learned objects are stored in a maprepresenting the robot’s environment and can be used forlater interactions. Although the interaction system is verylimited and the resulting map is rather coarse, this systemcan be compared to our approach as it also builds up a long-term memory about objects in the environment. However,we focus on a more detailed representation of objects andtheir later recognition in order to support natural human-robot interaction going beyond simple navigational tasks.

In general there is a tendency to either focus on buildinga robust representation while neglecting interaction smooth-ness or vice versa. We argue that in order to build a robotthat is able to be perceived as a companion it needs both, amore natural interaction based on an integrated multi-modalrepresentation.

3. OVERALL SYSTEMThe scene acquisition system described in this paper is

being implemented on our mobile robot BIRON. BIRON’shardware platform is a Pioneer PeopleBot from ActivMe-dia with a pan-tilt camera for face tracking and object andgesture recognition, stereo microphones for speaker localiza-tion and speech recognition, and a SICK laser range finderfor locating legs of potential communication partners. Theoverall architecture [10] of BIRON is based on a hybrid con-trol mechanism and has three layers: a reactive, an inter-mediate and a deliberative layer (see Fig. 1). Modules thatare responsible for reactive feedback of the system are seton the reactive layer: the Person Attention System detectspotential communication partners and the OAS detects ob-jects that users refer to. Since these are purely data-drivenprocesses they belong to the reactive layer. Modules re-sponsible for higher-level processing that involve top-down,expectation-driven strategies such as a planner or the Dia-log System, are located on the deliberative layer. The SceneModel, which contains a multi-modal representation of theobjects that the system has observed and can be seen as anintermediator between the Dialog System and the OAS, isconsequently located on the intermediate layer. The com-munication between modules is carried out via XCF (XMLEnabled Communication Framework). The system is cen-trally controlled by the so-called Execution Supervisor onthe intermediate layer. It coordinates the module opera-tions and makes sure that neither the reactive layer mod-

COGNIRON FP6-IST-002020 Appendix

Appendix page 95 of 101

ExecutionSupervisor

UnderstandingRecognition &Speech

Person Attention System Object Attention System

Gesture Detection

Dialog Control

Player/Stage Software

Scene ModelSequencer

PlannerDeliberativeLayer

IntermediateLayer

LayerReactive

Hardware (Robot Basis, Camera, Microphones, etc.)

CameraImage

SpeechOutput

Camera

Image

SpeechSignal

SoundSignal

Figure 1: Overview of the BIRON architecture(optional modules currently not implemented aredrawn in grey).

ules control the deliberative layer modules nor vice versa.Instead, it exerts control by taking into account the overallsystem state. This architecture allows for both fast reactionto dynamic environmental changes and extensive high-levelplanning and reasoning activities.

Since we focus on the interaction capabilities of BIRON,we do currently not integrate a planner and a sequencer.The Person Attention System establishes the basis for theinteraction with users but is not involved in the process ofresolving object references. In the following we thereforeonly describe the interfacing between the Dialog System,the OAS and the Scene Model.

4. THE DIALOG SYSTEMThe dialog system of BIRON is responsible for carrying

out interactions with the user including handling miscom-munications [16], guiding the discourse, and transferringuser utterances to internal command for the robot controlsystem to execute tasks. A dialog is made up of contribu-tions from the dialog partners. Two central questions ofdialog modeling are therefore (1) how to represent individ-ual contributions represented and (2) how to represent thedynamic change of the dialog state represented which is trig-gered by individual contributions successively. In subsection4.1 and 4.2 we are going to present our answers to these twoquestions. In section 4.3 we focus on the mechanism thatthe implemented dialog model provides for the integrationof speech and visual input. A more detailed account on thisintegration will be given in section 5.

4.1 The structure of a contributionConversants contribute to a dialog in a multi-modal way.

McNeill [12] investigated the relationship between speechand simultaneous conversational gesture and claims that theproduction of them are motivated by one single semanticsource, the so-called “idea unit”. Inspired by this finding,we represent the conversants’ contribution as the so-called“interaction unit” that includes two important stages of thelanguage production process. The structure of the interac-tion unit is illustrated in Fig. 2. An interaction unit hastwo layers: a domain layer and a conversation layer. Thedomain layer mirrors the cognitive activities of a dialog par-ticipant that motivate language production: If the inter-action unit represents an utterance to be produced by the

21non−verbal unit

4

(empty)

verbal unit non−verbal unit5

3connecting to robot control system

Facial expression generatorOAS

− Conversation Layer − − Conversation Layer −

− Domain Layer −

Initiated by the user

− Domain Layer −

Initiated by the robot

verbal unit

Figure 2: Processing flow in interaction units

robot itself the domain layer is where the Dialog Systemaccesses the robot’s control system or knowledge base. Ifthe interaction unit represents an utterance of the user thedomain layer remains empty because we do not make anyassumptions about the user’s cognitive activities behind thelanguage front. In future work this may be replaced by auser model. The conversation layer transforms the intentionthat is created based on the results of these cognitive activ-ities to language. For example, based on a successful followbehavior of the robot that is reported to the domain layerby the robot control system, the conversation layer formu-lates and synthesizes a message such as “OK, I follow you”.The conversation layer consists of two units: a verbal anda non-verbal unit. Based on the intention that results fromthe domain layer they are responsible for generating outputin verbal and non-verbal way, respectively.

The precondition of language production is successful lan-guage perception. Before reacting, i.e., before creating his/herown interaction unit to produce a contribution, a conversantfirst needs to understand the semantic meaning of his/her di-alog partner’s contribution by studying the verbal and non-verbal unit on the conversation layer of the dialog partner’sinteraction unit. Therefore the processing in the interactionunit should start from this language perception phase. Fig-ure 2 illustrates how the robot processes user contributions.In our system, the user’s verbal information as delivered bythe Speech Understanding System initiates the creation ofa user interaction unit (➊). In case that the user’s intentioncan not be fully recognized by the verbal unit, the systemwill consult the visual perceptual module via OAS in thenon-verbal unit (➋) and fuse these multi-modal informationon the user conversation layer of the interaction unit. Oncethe user intention is fully recognized, the system creates aninteraction unit for itself and tries to provide acceptance:the system first formulates and sends commands to the robotcontrol system or the knowledge base on the domain layer(➌) and then generates verbal and non-verbal output onthe conversation layer (➍, ➎) after receiving the executionresults. Currently, we have implemented the visualizationof facial expressions as the only non-verbal output. Thus,the integration of speech and visual information is mainlyperformed on the conversation layer of the interaction unit.

In the whole language perception and production processproblems may occur, e.g., the semantic meaning of the userinteraction unit cannot be resolved or the desired task can-not be executed by the robot system. These problems can-not be handled in a single interaction unit, new interactionunits are necessary. Now the question arises as to how toorganize the individual interaction units.

COGNIRON FP6-IST-002020 Appendix

Appendix page 96 of 101

4.2 The grounding mechanismThe interaction units have to be organized in a dynamic

way since every new contribution that is added to the dia-log changes the dialog state. Our dialog model is inspiredby the common ground theory of Clark [5]. According tothis theory a dialog is carried out in the way that one par-ticipant presents an account (presentation) and the otherissues the evidence of understanding of the account (accep-tance). The grounding process is complete and both dialogparticipants can go on with a new account only if the ac-ceptance is available. Dialog systems that implement thispsychological model ([15], [4]) di"er in their way of defin-ing grounding units (the units of the discourse where thegrounding takes place) and the organization of these units.We take exchanges in the style of adjacency pairs [14] as thegrounding units. These exchanges consist of two interactionunits that are initiated by the two dialog partners, respec-tively. The first interaction unit is the presentation and thesecond one is the acceptance, e.g., the first interaction unitrepresents a question and the second one the answer. To or-ganize them we introduce four grounding relations betweenexchanges: (1) default : introducing a new task. A groundeddefault exchange has no further e"ect on the grounding of itspreceding exchange. (2) support : clarifying in case of an un-groundable account. After a support exchange is groundedits initiator will try to ground the preceding one again thatis updated with the new information. For example, clari-fication question in case of an incorrect speech recognitionresult. (3) correct : correct the previous account. As sup-port, if such an exchange is grounded the initiator will try toground the updated preceding one again. (4) delete: deletethe previous account. If such an exchange is grounded, allthe previous ungrounded exchanges can be deleted.

Each contribution is analyzed in terms of whether it is apresentation or an acceptance. If it is a presentation, thenwe also need to find out its grounding relation to the pre-ceding exchanges. A presentation initiates the creation ofan exchange that is put onto the top of a stack while anacceptance completes the top exchange of the stack. Whenthe top exchange is complete it is popped. Additionally, ac-tions like updating the preceding exchange can be triggeredaccording to these relations. As long as there is an incom-plete exchange on the top of the stack, the conversant otherthan the initiator of the exchange’s presentation will try toground it. The implemented dialog system enables us tohandle clarification questions (as an exchange with supportrelation to its preceding exchange) and take initiative thatis motivated by the robot control system. For example, incase of technical problems of the robot control system theimplemented dialog system initiates an interaction unit toreport this problem to the user. It does this by encoding theerror message into its domain layer and generating outputto the user on the conversation layer.

4.3 Resolving object referencesIn the following we detail how the Dialog System and

the OAS cooperate to resolve object references in the user’sutterances.

According to [9] there are three types of informationalrelations between gesture and speech: reinforcement, dis-ambiguation and adding information. In our work, we fo-cus on the “adding information” relation. When people usegestures to complete the meaning of their utterances they

mostly indicate this intention in the utterance. For exam-ple, if a user says “This is my green mug” while pointing ata mug, the word “this” serves as a cue for the listener thathe/she is using a gesture to specify the concrete location ofthe mug. But in case of the subsequent utterance “The mugis my favorite one” the listener usually does not expect agesture but will search mentally in the dialog history whichcup might be meant. According to these two di"erent casesthe Dialog System activates either the OAS or the SceneModel to resolve the object reference. This process can beillustrated as a UML activity diagram (see Fig. 3).

successful?successful?

report success report failureto DLG to DLG

consult OAS

yes no

successful?

report success report failureto DLG to DLG

create user verbal unit

Cue of gesture present?yes no

consult Scene Model

yes no

report successto DLG consult OAS

yes no

Figure 3: Resolving object references in user’s ver-bal input (OAS: Object Attention System, DLG: Di-alog System)

If there are any cues of the involvement of a gesture inthe user’s verbal input, e.g., the word “this” in the exampleabove, this will be explicitely pointed out by the Speech Un-derstanding System. The Dialog System interprets this hintas evidence that the user’s verbal unit needs to consult thenon-verbal unit. In the non-verbal unit the Dialog Systemactivates the OAS by sending it the request to resolve theobject reference “my green mug”. The OAS activates thegesture recognition module. A successful gesture recognitionresult helps the OAS to orient the camera towards the posi-tion of the user’s hand which enables the OAS to confine theRegion Of Interest for its search of a green mug. This searchis carried out, as the case may be, either by an appearance-based object recognizer or with the help of salient objectfeatures as described in section 5. Subsequently, the OASsends the search result back to the Dialog System. In case ofa positive result the OAS also updates the Scene Model withboth symbolic and visual information about the new object“green mug”. According to this result the Dialog Systemcreates a system interaction unit (cf. Fig. 2) to provide ac-ceptance for the user’s input by either acknowledging or byinitiating verbal repair.

If there is no evidence of the involvement of a gesture inthe user’s verbal input but only some objects to be identified(such as the “mug” in the example “The mug is my favoriteone”) the Dialog System will first try to find a correspondingentry in the Scene Model. The query is constructed withall the features of the object present in the current verbalinput; in this case, the owner of the cup and his/her relationto this object (favorite). If the object can be found in theScene Model the Dialog System finishes its processing on theconversation layer of the user interaction unit and creates an

COGNIRON FP6-IST-002020 Appendix

Appendix page 97 of 101

acceptance to the user’s input; if this object is not registeredin the Scene Model, the Dialog System activates the OASto find it in the current scene. This process is described indetail in the following section.

5. OBJECT ATTENTION SYSTEMIn order for a system to be able to acquire knowledge

about objects in its environment it needs a mechanism tofocus its attention on those objects that the user is currentlytalking about. In our context we define attention as the abil-ity to select and concentrate on a specific stimulus out of allstimuli that are provided by the environment while suppress-ing others. The Object Attention System (OAS) thereforeneeds to coordinate the visual processing results (which cur-rently consist of deictic gestures, object recognition results,and visual object features) and making them accessible forthe Dialog System by storing them in the multi-modal SceneModel.

The OAS is activated when the user is verbally referringto an object and the Speech Understanding System has de-termined that either a gesture is expected or that the robothas to interact with an object autonomously. In order to ac-quire visual information about objects, BIRON uses an ac-tive camera with a maximal opening angle of view of about50 degrees horizontal and 38 degrees vertical which facili-tates only a limited field of view. It can therefore be neces-sary to re-orient the camera to relevant parts of the currentscene which the user refers to. Once the robot has focusedits attention on such a so-called Region Of Interest, the ac-quisition of information about this object, like position orview, can be completed. The Region Of Interest is that partof the image that has been specified by a gesture and con-tains the distinctive feature verbally specified by the user.Our assumption is that it contains an image of the object ifthe object is known to the robot. If it is unknown the Re-gion Of Interest encloses the verbally specified visual feature(e.g. the “round thing” or a color).

The collected object data then has to be added to therobot’s knowledge base which must allow retrieving storedobject information and updating already stored data. Also,additional information given verbally by the user has tobe stored. As this knowledge base, subsequently named asScene Model, is crucial for the interaction between the Di-alog System and the OAS because it represents BIRON’slong-term memory, it is described in detail in section 6.

In addition to the maintenance of the Scene Model formemorizing tasks, the OAS needs to take care of the coor-dination of verbal information, gestures, and salient objectfeatures (e.g., color, shape, etc.) perceived by the camera,as well as the control of the hardware components like thecamera during the object attention phases. This is realizedby a Finite State Machine (see Fig. 5) where the input ismainly provided by the camera, the Dialog System (cf. sec-tion 4), the Speech Understanding System, and the gesturerecognition component [7].

A crucial distinction that has to be made during the pro-cessing of multi-modal information is that between objectsthat are already known to the robot and those that are not(see Fig. 4). This is because in the latter case the OAS willhave to establish (or ‘learn’) a first link between the ver-bal symbols describing the object and the percepts while inthe former case the object needs to be retrieved from thedatabase.

VisAtt

UCB

ObjStor

UCBObjDet

ObjAna

SelObj

AmbigObjAreaFnd

<ObjDesc><ObjDesc>

<ObjList>

ObjNotExtr<ObjData>

UCBSelArea

ObjSpecReq

ObjSpecify<ObjUpdate>

<None>

<ObjDesc> AmbigReg<ObjList>

<ObjData>ObjNotFnd

ObjFnd<ObjData>

Abort<None>

Objfnd<ObjData>

UCBIAVariedInp

ObjAlertAbort

Abort

ObjInf GstNotFnd

<None>

<ObjData><ObjData>

<None>

UnknownObj<ObjData>

Stored<None>

KnownObj<ObjData><None>

Abort

Figure 5: The Finite State Machine of the ObjectAttention System.

In both cases the OAS is activated on demand by theDialog System if a gesture is expected or an access to theScene Model by the Dialog System has failed afore. At thismoment, the Finite State Machine (see Fig. 5) will be inthe idle state Object Alertness (ObjAlert). Once the OAS isprovided with data by the Dialog System, the Finite StateMachine changes to the Input Analysis (IA) state. Now, thegesture recognition component is activated and provides theOAS with the user’s hand coordinates and the direction ofthe corresponding pointing gesture. Thus, an area withinthe camera image is selected as the Region Of Interest. Incase the Dialog System sends a description of the object(e.g., type, color, owner, etc.) to the OAS, a query to theScene Model is initiated in order to check whether the objecttype is already known. In the following, we will describe indetail the processing for the case when the object type isknown to the robot. The more complex process for the caseof unknown objects will be exemplified subsequently.

5.1 Previously known objectsSuppose the user specifies an object type that the system

has already stored in its Scene Model. In this case the SceneModel will return all object entries that match the symbolicdescription of the specified object. In order to verify if oneof the returned objects is indeed the object the user refersto, the OAS will need to search for the object in the realscene and compare it with the stored image pattern. Thissearch involves an object detection process for which we arecurrently using a simple appearance-based object recognizerthat is only suitable for a very limited object scenario. It isbased on the fast Normalized Cross-Correlation (NCC) algo-rithm described in [11] which is a simple but fast algorithmthat is su!cient for our task at hand. However, in orderfor the system to work reliably in a more unstructured en-vironment, as for example a real home-tour scenario a moresophisticated object recognizer will be needed.

After all appropriate image patterns have been retrievedfor the known object type, the Finite State Machine switches

COGNIRON FP6-IST-002020 Appendix

Appendix page 98 of 101

no Obj.

Clarificationnegativefeedback

[repetition counter]Object Recognition

RecognitionVisual feature>= 2 reiterations

known unknown<Object type description>

No featuresFeatures okSearch on image

Feature 1...successfulsuccessfulfailed

ROI Analysis> 1 Obj. 1 Obj.

<False ROI>

<Obj. notFnd> Dialog System

<Obj. Fnd>

<Add. Inf.>

failed

negativefeedback

<Obj. notFnd>

Requestadd. Feature

[repetition counter]

<Add. Inf.>

<False ROI>

<Obj. notFnd>

Consult Object Attention System

ConfirmationUpdate Scene Model

>= 2reiterations

4

3

1

5

2

67

98

Figure 4: Schematic description of distinction between known and unknown objects

from the Input Analysis state to the Object Detection (Ob-jDet) state. Within the Object Detection state the OASuses the retrieved image patterns (e.g., for cups) in orderto feed them to the object recognizer. At the same timethe camera is re-oriented based on the hand coordinatesand the pointing direction that are provided by the ges-ture recognition component. Also, the position of the handis used to determine the relevant Region Of Interest. Basedon this information, the object detection process is initiatedby scanning the Region Of Interest for patterns similar tothose provided by the database. If an object is found bythis procedure a confirmation message is sent to the Dia-log System (cf. Fig. 4 (➊)) and the Finite State Machineswitches to the Object Store (ObjStor) state. In this statethe position of the object in the scene is updated in theScene Model. Finally, the Finite State Machine returns tothe Object Alertness state and the OAS awaits new orders.

If two or more objects of the same type are found in thereal scene during the detection phase (cf. Fig. 4 (➋)) inthe Object Detection state, the Finite State Machine willswitch to the User callback (UCB) state. This means, thata message is sent to the Dialog System to clarify which ofthe found objects was meant by the user. After the DialogSystem has provided a more detailed description, the FiniteState Machine switches to the Object Analysis (ObjAna)state. In this state a new Region Of Interest is determinedbased on the information from the gesture detection and theextended verbal information. Now the Finite State Machinereturns to the Object Detection state and initiates a newsearch. This cycle is performed until the object is found, orthe user aborts the action within the User callback state butat most two times. Then, the Finite State Machine switchesto the Visual Attention (VisAtt) state (cf. Fig. 4 (➌)). If noobject is found in the Object Detection state (cf. Fig. 4 (➍))this means that the user is referring to an unknown objectwhich is supposedly similar to the description of objects pre-viously retrieved from the Scene Model. However, since the

object is unknown the Finite State Machine switches to theVisual Attention state, that is also used for the localizationof unknown objects if two reiterations have been reached (cf.Fig. 4 (➌)).

5.2 Unknown objectsIf no object detection is possible because no object en-

try matching the user’s specification has been found in theScene Model the OAS will search for salient features in thecamera image such as colors or shapes by applying di"erentfilters that detect salient visual object features as specifiedby the user. We call these filters attention maps follow-ing the terminology of [8] where a similar technique is used.The use of these attention maps is coordinated within theVisual Attention state and can help to select Regions Of In-terest. The appropriate attention map is selected based onthe verbal information (e.g., the color) given by the user (cf.Fig. 4 (➎)).

Once a region matching the search criteria (i.e., color) isfound within the Region of Interest by the attention mapit is selected within a bounding box (cf. Fig. 4 (➏)). Thisbounding box is supposed to contain a view of the retrievedobject (e.g., a blue cup) and is stored in the Scene Model(cf. Fig. 4 (➊)). Additionally, a confirmation message is sentto the Dialog System.

If the verbal information given by the user is insu!cient todetermine a Region Of Interest, that is if no visual descrip-tions are given that can be found by the attention map, (cf.Fig. 4 (➐)), the Finite State Machine changes to the Usercallback state. In the User callback state the OAS sendsa request to the Dialog System in order to get more infor-mation (e.g., shape, position, ...) about the object whichthe user refers to. When the user has given a more spe-cific description which is sent by the Dialog System to theOAS, the Finite State Machine returns to the Visual Atten-tion state (cf. Fig. 4 (➑)). The User callback state is alsoreached if more than one Region Of Interest is found (cf.Fig. 4 (➋)). Then, the OAS asks the Dialog System to re-

COGNIRON FP6-IST-002020 Appendix

Appendix page 99 of 101

solve this ambiguity. As soon as the OAS has determinedthe Region Of Interest, the Finite State Machine switchesto the Object Analysis state to acquire the position of theobject by means of the hand position of the user. Next, theFinite State Machine switches to the Object Store state andstores the extracted view and the position of the object inthe Scene Model (cf. Fig. 4 (➊)). Then, the Finite StateMachine returns to the Object Alertness state to await neworders.

If no Region of Interest is found during the search for vi-sual object features, the Finite State Machine switches tothe User Callback state and returns a negative response tothe Dialog System. In parallel, the OAS asks the Dialog Sys-tem for a more detailed object description and re-initiates asecond search on the image (cf. Fig. 4 (➒)). If for a secondtime no Region Of Interest is found, the OAS sends a mes-sage to the Dialog System, that the search for the referencedobject was not successful and returns into its idle state toawait new orders from the Dialog System.

6. REPRESENTATIONInformation acquired by the Dialog System and the OAS

in the ongoing interaction with a user must be stored in anappropriate way. Because the same information from di"er-ent modalities require di"erent ways of representation themanagement of such a multi-modal database is a non-trivialtask. Our approach to such a database, that we call SceneModel, is based on the concept of an active memory [17]since it uses intrinsic processes which allow not only a sim-ple access to the data but also provides intelligent main-tenance functionalities. One of the intrinsic processes forexample enables the autonomous removal of obsolete in-formation about objects (e.g., the position of a cup threemonths ago). This forget mechanism is quite essential forour application since the robot’s environment is continouslychanging.

Within our work we have extended the functionality ofthe active memory in order to be able to handle the dif-ferent modalities by storing the same data in di"erent for-mats. This information that might seem to be redundant atfirst glance is necessary because the Scene Model is used asBIRON’s long-term memory and both Dialog System andOAS access the data stored in it. For example, consider thecolor of an object: the Dialog System may store its value inform of a character string, e.g.,“blue”; but after finding it inthe current scene the OAS may need to store its color valuebased on the Hue Saturation Intensity (HSI) color model.The same holds true for the position of an object, which theDialog System would store symbolically as “on the table”while the OAS would store its coordinates. The coordinatesdescribing the position of an object are obviously quite use-less for the Dialog System when the user asks “where is myblue cup?”. On the other hand, the OAS would not be ableto handle the value “blue” when it has to find a blue cup inthe camera image.

Consequently, the Scene Model needs to include a compo-nent that is able to convert the format of data, the so-calledModality Converter. The Modality Converter is a simple yetpowerful mechanism that is not only able to convert singleobject features like the color. It can also search the data baseand will return whole object entries matching given descrip-tions. This may be necessary for example when the OASfails to detect an object by matching all memorized views

against the current camera image and the object’s color isnot yet known to the OAS. Then, in order to extend thesearch for visual object features, the OAS sends a requestfor the object’s color to the Dialog System. Subsequently,a search for the newly given color can be performed, afteran appropriate conversion of the Dialog System’s responseis received by the OAS.

For the conversion process the Modality Converter usesa lookup table (cf. Table 1) that contains for every storedpredicate name (e.g., color, relation, ...) two attribute fields,in particular a symbolic description as well as a visual featuredescription. Since the HSI values might vary for a distinctiveverbally named color, the corresponding value field can con-tain specific values as well as ranges of values. Depending bywhich module a query is sent the Scene Model returns auto-matically the adequate description if available. For instance,if the query origins from the Dialog System the Converterwill automatically return the attribute “Symbolic”.

Predicate name Symbolic VisualColor red 330..20,0..1.0,0..1.0

0.0,1.0,1.0Color green 135,1.0,0.7Relation O1 under O2 O1.y < O2.y

Table 1: Lookup table of the Modality Converter

Note that this converter is a very powerful tool since itdoes not only convert pre-defined symbol-value pairs but itis also able to learn new associations. In sum, three dif-ferent responses to queries are possible. ➀ The converterfinds an entry where all data fields match the attributes ofthe specified object. In this case, it will return the valuesuitable for the inquiring component. ➁ The converter findsno valid entry because there is no correspondence of the en-try in the other modality as for example for the symbolicname “transparent” which does not have an HSI equivalent.➂ The converter finds no valid entry because it is not yetcomplete. This usually occurs when after a search for visualobject features a new HSI value is stored in the Scene Modelfor which no symbolic name is yet known. Then, the DialogSystem will ask the user for the color name of the RegionOf Interest.

7. RESULTSIn order to illustrate our results we present a dialog exam-

ple where the resolution of object references is involved. Inthis example, the user asks the robot to pay attention to amug. Fig. 6 illustrates the dialog flow, the operations of themodules underlying the robot output and the content of theScene Model in the first, second and third column respec-tively. We assume that the robot has already recognized theuser as Thomas.

In the utterance U1 the word “this” indicates a possibleaccompanying gesture which can help to specify its mean-ing. The Dialog System therefore sends a request to theOAS to search for the object mug (➊). Upon this requestthe OAS will first query the Scene Model for an object ofthe type mug in order to provide a template to the objectrecognizer (➋). In our example, no such object is stored inthe Scene Model (➌). The OAS now switches to its sec-ond searching strategy: search with salient features of theobject in the current scene. But since neither salient fea-

COGNIRON FP6-IST-002020 Appendix

Appendix page 100 of 101

Module Operations

generateOutput(need(color))name: mugcolor: (blue) (250,0,0)image:

name: mugcolor: (blue) (250,0,0)image:

relation: favoriteowner: Thomas

11

Dialog

U1:

What color it is?I can’t find it.

R1:

This is a mug.

Blue!U2:

R2:Ok, I see it.It is nice.

U3:It’s myfavorite mug.

need(color)

found(mug)

search(blue mug)

generateOutput(confirmation)

Scene Model

6

5

1

43

2

9

7 10OASDLG

DLG

DLG OASnoEntry(mug)search(mug)search(mug)

color(blue) enter(mug)

update(mug)12SM

SM

SM

no gesture detected, no salient feature specified

8

DLG

DLG

Figure 6: A dialog example (U: User; R: Robot;DLG: the Dialog System; OAS: the Object Atten-tion System; SM: Scene Model)

tures of the mug were specified, nor a gesture was found inthe current scene (➍), the OAS informs the Dialog Systemabout its need of a salient feature, namely the color (➎).The Dialog System will then generate the output R1 to getthe requested information from the user (➏). Thomas an-swers the question and the value of the color is sent fromthe Dialog System to the OAS (➐). With this informationthe OAS successfully finds the object mug (➑) and informsthe Dialog System about this result (➓). At the same timethe multi-modal information about the mug is entered inthe Scene Model by the OAS (●)11 . The Dialog System cannow generate a confirmation (●)12 as feedback to the user.In U3 the user specifies two further features of the mug, theowner (“mine”) and a description (“my favorite”). This in-formation is directly entered into the Scene Model by theDialog System since there is no evidence of the involvementof a user gesture, which would indicate that a new objectis being specified, and there is only one entry in the SceneModel that matches the described object.

8. CONCLUSIONIn order for a mobile robot to be able to communicate in

a human-style it needs processing and representation strate-gies that can deal with multi-modal information. We there-fore integrated a Dialog System using multi-modal inter-action units with an Object Attention System that is ableto resolve object references. The interaction between thesemodules is based on a multi-modal representation, the SceneModel, which stores the acquired scene information and pro-vides a Modality Converter that not only converts informa-tion from one modality to another but also can learn associ-ations between data from di"erent modalities such as colornames and HSI values. This powerful mechanism allows onthe one hand to use a pre-defined knowledge base while it ison the other hand capable of adapting to new environmentsby learning new objects and salient features.

The current system still has some limitations. Firstly, therobot cannot yet learn new words, this means, the robot can

only learn objects with known symbolic names. A solutionof this problem can be adding a mechanism into the DLGthat can store and reuse new words once they are spelledby the user. Secondly, a global 3D coordinate system rep-resenting the absolute position of an object in relation tothe room is not yet integrated into the OAS. The conse-quence is, the robot can only find the object again if it isin the position as it learned the object. Thirdly, no naviga-tion system is implemented for the robot so that the robotcannot autonomously move from one location to another tofind an object. These limitations are also the motivation forour future work.

9. REFERENCES[1] K. Aoyama and H. Shimomura. Real world speech interaction

with a humanoid robot on a layered robot behavior controlarchitecture. In IEEE Int. Conf. Robotics & Automation,pages 3825–3830, 2005.

[2] C. Breazeal, A. Brooks, J. Gray, G. Ho!man, C. Kidd, H. Lee,J. Lieberman, A. Lockerd, and D. Mulanda. Humanoid robotsas cooperative partners for people. Int. Journal of HumanoidRobots, 2004.

[3] J. S. Brunder. Towards a Theory of Instruction. Norton, NewYork, 1966.

[4] J. E. Cahn and S. E. A psychological model of grounding andrepair in dialog. In Proc. Fall 1999 AAAI Symposium onPsychological Models of Communication in CollaborativeSystems, 1999.

[5] H. H. Clark, editor. Arenas of Language Use. University ofChicago Press, 1992.

[6] S. S. Ghidary, Y. Nakata, H. Saito, M. Hattori, andT. Takamori. Multi-modal interaction of human and homerobot in the context of room map generation. AutonomousRobots, pages 169–184, 2002.

[7] A. Haasch, N. Hofemann, J. Fritsch, and G. Sagerer. Amulti-modal object attention system for a mobile robot. InProc. Int. Conf. on Intelligent Robots and Systems, 2005.

[8] L. Itti, C. Koch, and E. Niebur. A model of saliency-basedvisual attention for rapid scene analysis. PAMI,20(11):1254–1259, 1998.

[9] J. M. Iverson, O. Capirci, E. Longobardi, and M. C. Caselli.Gesturing in mother-child interactions. Cognitive Develpment,14(1):57–75, 1999.

[10] M. Kleinehagenbrock, J. Fritsch, and G. Sagerer. Supportingadvanced interaction capabilities on a mobile robot with aflexible control system. In Proc. Int. Conf. on IntelligentRobots and Systems, pages 3649–3655, 2004.

[11] J. P. Lewis. Fast template matching. In Proc. Conf. on VisionInterface, pages 120–123, 1995.

[12] D. McNeill. Hand and Mind: What Gesture Reveal aboutThought. University of Chicago Press, 1992.

[13] P. Prodanov and A. Drygajlo. Decision networks for repairstrategies in speech-based interaction with mobile tour-guiderobots. In IEEE Intern. Conf. Robotics & Automation, pages3052–3057, 2005.

[14] E. Scheglo! and H. Sacks. Opening up closings. Semiotica,pages 289–327, 1973.

[15] D. Traum. A Computational Theory of Grounding in NaturalLanguage Conversation. PhD thesis, University of Rochester,1994.

[16] D. Traum and P. Dillenbourg. Miscommunication inmulti-modal collaboration. In Proc. AAAI Workshop onDetecting, Repairing, And Preventing Human–MachineMiscommunication, 1996.

[17] S. Wrede, M. Hanheide, C. Bauckhage, and G. Sagerer. AnActive Memory as a Model for Information Fusion. In Proc.Int. Conf. on Information Fusion, pages 198–205, 2004.

COGNIRON FP6-IST-002020 Appendix

Appendix page 101 of 101