A breadboard architecture for pervasive context-aware services in smart spaces: middleware...

ORIGINAL ARTICLE

A breadboard architecture for pervasive context-aware servicesin smart spaces: middleware components and prototypeapplications

John Soldatos Æ Nikolaos Dimakis Æ Kostas Stamatis ÆLazaros Polymenakos

Received: 31 January 2006 / Accepted: 20 September 2006 / Published online: 21 November 2006� Springer-Verlag London Limited 2006

Abstract We present an architectural framework

along with a set of middleware elements, facilitating

the integration of perceptual components, sensors,

actuators, and context-modeling scripts, comprising

sophisticated ubiquitous computing applications in

smart spaces. The architecture puts special emphasis

on the integration of perceptual components contrib-

uted by a variety of technology providers, which has

not been adequately addressed in legacy architectures.

Moreover, the introduced architecture allows for

intelligent discovery and management of resources.

Along with the description of this breadboard archi-

tecture, we present its non-functional features and as-

sess its performance. We also outline a rich set of

practical prototype pervasive services that have been

built, based on this architecture. These services

emphasize on providing non-obtrusive human-centric

assistance (e.g., memory aids, meeting recordings,

pertinent information) in the scope of meetings, lec-

tures and presentation, Experiences from building

these services manifest the benefits of the introduced

architecture.

Keywords Smart spaces � Autonomic computing �Ubiquitous computing � Context-awareness �Architecture � Middleware

1 Introduction

Pervasive and ubiquitous computing promise to trans-

form physical spaces into computationally active and

intelligent environments and envision a world, where

computers provide services to humans regardless of

time and users’ location [1]. Ubiquitous and pervasive

services are essentially context-aware, since they

automatically acquire and process information about

their surrounding environment [2]. Thus, context-

aware applications execute service logic not only based

on input provided explicitly by end users, but also

based on implicitly derived information [3]. Context

acquisition can be based on several approaches,

including reading tags to track objects and inferring

context (e.g. [4]), wearable computing systems (e.g.

[5]), as well as using sensors and effectors for natural

interaction in smart spaces (e.g. [6]). Sophisticated

ubiquitous computing applications are likely to lever-

age more than one of these approaches, leading to

hybrid schemes. For instance, wearable computing

applications and tag reading may operate in the scope

of smart spaces. Note that all the above approaches

rely on networked sensing infrastructures, which have

to be as non-obtrusive as possible. Apart from

numerous sensors, smart spaces also include actuators

facilitating the interaction between end-users and the

pervasive computing environment.

No matter which context-acquisition approach is

used, context-awareness relies on hardware sensors

J. Soldatos (&) � N. Dimakis � K. Stamatis �L. PolymenakosAthens Information Technology, 19.5 km Markopoulo Ave.,P.O. Box 68, 19002 Peania, Greecee-mail: [email protected]

N. Dimakise-mail: [email protected]

K. Stamatise-mail: [email protected]

L. Polymenakose-mail: [email protected]

123

Pers Ubiquit Comput (2007) 11:193–212

DOI 10.1007/s00779-006-0102-7

and signal processing algorithms. The latter extract

elementary forms of context (e.g., relating to people

and objects state), which can be combined toward

recognizing more complex contextual states. This

combination relies on middleware components under-

taking the fusing of the output of multiple sensors, as

well as to coordinate the interactions and information

from multiple perceptual components and recognition

algorithms. Additional middleware components are

required to facilitate integration, development and

deployment of complete applications. Integration is a

primary concern, given that pervasive computing

infrastructures are highly distributed and heteroge-

neous. Thus, additional middleware components pro-

viding directory services (i.e., registration and

discovery of components), facilitating interfacing to

sensors, actuators and perceptual components, fusing

information, and overall easing service development

are required.

Middleware components within a pervasive com-

puting environment can be reused across different

applications. Reusable middleware can significantly

ease service implementation, through allowing the

application developer to emphasize on the service lo-

gic, rather than on the integration with the underlying

sensors and context-aware components. Such middle-

ware components are a key prerequisite for the wide

adoption of ubiquitous computing, since they can sig-

nificantly minimize the effort required to develop and

deploy applications [7].

Major pervasive and ubiquitous computing projects

have developed middleware infrastructures facilitating

component integration, as well as development and

deployment of services. As a prominent example, the

Interactive Workspaces project (at Stanford university)

has developed the Interactive Room Operating System

(iROS) [8], which provides a reusable, robust, and

extensible software infrastructure, enabling the

deployment of component-based ubiquitous comput-

ing environments. iROS supports various modalities

and human–computer interfaces, by tying together di-

verse devices, each one having its own operating sys-

tem. The Oxygen project at MIT has produced the

MetaGlue system [9], which constitutes a highly robust

software agent platform. Agents are used to represent

both local resources and interactions with those re-

sources. MetaGlue relies on a custom distributed

communications infrastructure, enabling agents to run

autonomously of individual applications so they are

always available to service multiple applications. Sup-

port for implementing context-awareness is provided

through the GOALS architecture [10], which is the

evolution of the MetaGlue system. The EasyLiving

system developed at Microsoft research [11, 12] has

also produced an architecture enabling coordination of

devices, as well as fusion of contextual information.

Specifically, this architecture employs computer vision

technologies for person-tracking and visual user inter-

action and supports context-awareness based on a

geometric model of the world. It uses device-inde-

pendent communication and data protocol and

accordingly adapts the user interface. Another archi-

tectural model for pervasive computing has been

developed by the Carnegie Mellon University (CMU)

Aura project [13], which targets environments involv-

ing wireless communication, wearable or handheld

computers, and smart spaces. Aura provides software

architectural models that monitor an application and

guide dynamic changes to it. Adaptation takes into

account varying resources, user mobility, changing user

needs and system faults.

All these efforts justify the importance of ubiquitous

computing architectures. At the same time, they

manifest that there is no global unified framework

addressing all needs. Rather, the majority of these

architectures concentrate on one or more application

specific goals. While these goals (e.g., context-compo-

sition, coordination of devices, orchestration of

modalities, and adaptation to context) are common to

the majority of the pervasive computing systems, the

respective solutions emphasize different objectives,

depending on the nature of the target pervasive envi-

ronment. A fundamental limitation of these architec-

tures is that they assume that all context-aware

components (e.g., perceptual components, situation

modeling middleware) are contributed by the same

technology provider, which is not always the case.

Large-scale collaborative projects built demonstrators

from a variety of components that are contributed by

different providers. This is also in-line with the envis-

aged nature of emerging pervasive computing envi-

ronments, which will be composed by a variety of

distributed heterogeneous components from multiple

vendors. Another drawback of these architectures is

that they were built upon legacy technologies and

therefore do not benefit from emerging technologies

(e.g., the semantic web).

In this paper, we introduce an architecture for

ubiquitous, context-aware computing, which can

greatly facilitate the assembly and deployment of

pervasive applications based on components contrib-

uted by different technology providers. We character-

ize this architecture as ‘breadboard’ since it allows

flexible addition and replacement of hardware (e.g.,

sensors, actuators) and software elements. This archi-

tecture comes with a framework implementation of

194 Pers Ubiquit Comput (2007) 11:193–212

123

middleware components enabling the integration of

sensors, devices, actuators, perceptual components,

context modeling elements, as well as other informa-

tion fusion components. Special emphasis is laid on

integrating perceptual components, multi-modal

interfaces and situation recognition middleware. The

vendor-independent integration of other elements

(e.g., sensors, actuators, and devices) leverages existing

standards (e.g., the IEEE 1394 for video sensors) and

research results.

The motivation for implementing systems based on

this architecture was our participation in the Comput-

ers in the Human Interaction Loop (CHIL) project

[14], which is a one of the most prominent European

research initiatives in the areas of pervasive computing

and multi-modal interfaces. CHIL brings together

several research labs and therefore builds services that

leverage numerous perceptual technology components.

CHIL perceptual technologies comprise a rich collec-

tion of 2D-visual components, 3D-visual perceptual

components, acoustic components, audio–visual (i.e.,

multi-modal) components, as well as output perceptual

components like multi-modal speech synthesis and

targeted audio. As a result, CHIL services provide

ground for demonstrating the benefits of the intro-

duced breadboard architecture. Note that CHIL ser-

vices are built within prototype smart spaces, which

serve as test-beds for perceptual, multi-modal, and

middleware development. The authors operate a pro-

totype smart space comprising several sensors and

actuating devices, where several pervasive applications

have been built. Our experiences from building pro-

totype pervasive services provide a first class manifes-

tation of the introduced middleware framework.

While the introduced architecture focuses on the

integration of diverse sensors and perceptual compo-

nents, it also provides most of the features of other

middleware infrastructures for pervasive computing.

These features include directory services, coordination

of devices, autonomic communication, as well as a

variety of non-functional features. In several areas

(e.g., in the area of effective resources discovery) the

introduced architecture achieves significant improve-

ments over conventional approaches. We present all

these features, along with a performance assessment of

the middleware components comprising the architec-

ture. Note that the overall architectural concept takes

into account the authors’ work on more specific aspects

of a pervasive computing architecture (e.g. [15–19]).

The rest of the paper has the following structure:

Overall architecture framework section presents the

overall architectural concept for integrating sensors,

perceptual components, actuating services, information

fusion elements. Sensors and perceptual components

section illustrates how the architecture can flexibly

incorporate sensors and perceptual components, out-

lining also the middleware required for this incorpo-

ration. Our context modeling approach is described in

situation modeling and context-awareness section,

along with the middleware components that implement

it. Intelligent actuating services section, describes the

interfacing to actuators and actuating services. It also

elaborates on our approach for intelligent coordination

and orchestration of devices. Service logic and user

front-end section, describes the implementation of the

service logic executed by a ubiquitous computing

application, while Fault-tolerance and system man-

agement section describes non-functional features (i.e.,

scalability, reliability, and management of the archi-

tecture). Prototype services and applications section

presents prototype applications that leverage the

architecture and illustrates its tangible advantages.

Moreover, it presents some evaluation results targeted

to the approval of the Memory Jog. Performance

measurements section discusses performance issues

based on measurements that have been derived in the

context of a realistic operation of the system. Finally,

conclusions section draws the basic conclusions from

this work.

2 Overall architecture framework

Figure 1 presents the overall architectural framework

for pervasive services in smart spaces along with its

core middleware components. This framework can be

viewed as a composition of three tiers that operate on a

layered fashion in the sense that each tier exploits in-

put from the adjacent ones. The three tiers are as fol-

lows:

• A sensors tier comprising the transparent sensing

infrastructure of the smart space. Sensors collect

signals from the environment. Signals can accord-

ingly be processed to extract context.

• A tier of perceptual components based on signal

processing algorithms. Perceptual components ex-

tract context cues, typically by processing audio and

video signals. Context derived from perceptual

components relates primarily to the identity and

location of people and objects.

• A tier of agents that model and track higher-level

contextual situations, while at the same time

incorporating the service logic of the pervasive

computing services. Context tracking allows for

implementation of non-obtrusive service logic.

Pers Ubiquit Comput (2007) 11:193–212 195

123

Furthermore, implementing the service logic re-

quires access to, and management of, actuating

services. The use of agents in the service layer is

merely an implementation choice that offers several

advantages; other sorts of distributed systems also

could have been used as well. Note that all agents

are implemented based on the JADE [21] platform

and communicate with each other using standard-

ized Foundation for Intelligent Physical Agents

(FIPA) [22] messages/primitives.

Communication and data exchange between these

layers is achieved based on specialized third-party

middleware components, enabling distributed high-

performance access to sensor input, as well as to per-

ceptual components outputs. Specifically, perceptual

components access sensor output through the NIST

Smart Flow middleware [23]. The latter undertakes the

distributed high-performance transfer of streams to

any computer within the smart space, which facilitates

decentralized processing of sensor streams by multiple

consumers. Distributed processing is essential for two

main reasons:

• To allow more than one algorithm to leverage a

particular sensor stream. For example, a video

sequence may be needed by a body tracking

algorithm and a visual personal identification

method.

• To distribute the processing load to more than one

computers. This is particularly important in the case

of computationally demanding real-time multime-

dia processing.

Information exchange between the perceptual

components layer and the agent-based services layer is

achieved based on IBM’s CHILIX library [14]. CHI-

LIX is a middleware component enabling access to

perceptual components outputs in a language and

platform agnostic manner, based on XML over TCP

transfer of information. The information to be trans-

ferred is not a multimedia stream per se, rather the

result of its processing from a perceptual component.

Pervasive services that are built based on this

architecture feature an information flow as follows:

• Visual and acoustic sensors capture audio–visual

streams, which are transferred to perceptual com-

ponents through the NIST Smart Flow System.

Alternatively, sensors that can directly provide

context (e.g., temperature, motion sensors) and

convey this information to the services tier.

• The perceptual components tier processes audio

and video streams, toward identifying and locating

people and objects within the smart space.

• The services layer maintains a high-level context

model, which is assembled based on elementary

contextual information received from either sensors

Fig. 1 High-levelarchitectural framework andmiddleware infrastructure forpervasive computing servicesin smart spaces


123

or perceptual components through CHILIX. The

service logic is triggered upon identification of

particular contextual states, as well as based on user

requests. The service logic execution is likely to

engage actuating services of the smart space.

In such heterogeneous environments, directory ser-

vices are a major concern. Transparent distributed

communication between all these components, presup-

poses mechanisms for locating components and ser-

vices. While numerous directory mechanisms exist (e.g.

[24–28]), we argue that they are not appropriate to deal

with the diversity of a pervasive computing environ-

ment. Technologies such as Universal Plug n’ Play

(UPnP) [24], Service Location Protocol (SLP) [25], and

Universal Description, Discovery, and Integration

(UDDI) [26] provide mechanisms for registering and

discovering resources and services. However, these

mechanisms are not particularly tailored to the range of

information and components that are essential to per-

vasive services. For example, UDDI and SLP are merely

service oriented, while UPnP is device oriented. The

limitations of these technologies are largely due to a lack

of standardized descriptions for perceptual compo-

nents, context models, and actuating services. Moti-

vated by these limitations, we have developed a

directory mechanism leveraging a knowledge base,

which is based on web ontologies. In particular, the

ontologies include all the concepts associated with

hardware, middleware, software, as well as with the

physical objects (people, artifacts) of the smart space.

Our web ontologies are described based on the Web

ontology language (OWL) [29] and accessed through a

variety of distributed access techniques [17–19].

Using the ontology-based directory services mecha-

nisms we can more intelligently answer queries, as is

often required in the scope of context-aware, human-

centric, ubiquitous computing services. The intelligence

lies in the ability to infer information from existing sets

of meta-data according to current context and user

intention. As an example, given the number of different

sensors in a smart space, a service may need to acquire a

reference to the best-view camera for a given situation,

e.g., the camera facing the door for recognizing a person

entering a room. The ability to answering such queries

requires inferring information based on existing meta-

data, based on a reasoning procedure. External rule-

based reasoning systems can access the ontological data

of our OWL-based directory service (e.g. [30]), which

overall results in intelligent directory services. Follow-

ing paragraphs elaborate on the structure, functionality

and implementation details of the main components of

the architectural framework.

3 Sensors and perceptual components

3.1 Sensor virtualization and sensor proxies

Smart spaces applications need to interface with sen-

sors, to acquire and process sensor streams. This

interfacing is based on Application programming

interfaces (APIs) facilitating access to the low-level

capabilities of the sensors. Given the large number of

different sensors and vendors, developers of ubiquitous

computing applications have to deal with a rich set of

diverse APIs (Fig. 2).

To facilitate seamless addition, replacement and

integration of sensors, our architecture provides a set

of vendor-independent APIs that alleviate this diver-

sity. Specifically, an hierarchy of sensor-related classes

and their operations have been defined. This class

hierarchy includes vendor-independent classes for

acoustic and video sensors comprising the sensing

infrastructure of the CHIL smart rooms. Moreover,

classes for other types of sensors have been defined as

well. Vendor-independent operations offer virtualized

access to the capabilities of the underlying sensors.

Any component (e.g., perceptual algorithm, signal

processing algorithm, NIST Smart Flow client) inter-

facing with a certain sensor type exploits the same

single API, regardless of the sensor vendor. As a result,

developers need only to learn and leverage a simple

API to interface with each sensor type. Current APIs’

abstractions cover all the sensors available in the CHIL

smart rooms, which include: active cameras with pan,

tilt and zoom (PTZ cameras), digital IEEE1394

Fig. 2 Sensor virtualization


123

(Firewire) cameras, microphones, microphone clusters,

and NIST Mark III 64 channel microphone arrays [31].

However, the higher-level abstractions are appropriate

for other sensors as well [e.g., Radio-Frequency Iden-

tification Systems (RFIDs), temperature sensors]

(Fig. 3).

Pervasive computing environments tend to be vol-

atile, since components, devices and services can

dynamically join or leave. This dynamism is particu-

larly evident in mobile and nomadic computing

environments, which involve roaming users (e.g. [32]).

In-door environments (such as smart spaces) tend to

be less dynamic, since sensors and devices join or

leave at higher time-scales. The architecture can deal

with this dynamism at the hardware level, through

dynamic discovery of sensors and devices. This is

achieved, thanks to a set of middleware components

(i.e., JADE agents) called sensor proxies. A different

proxy component has been developed for each dis-

tinct sensor class, with a view to managing the sensor.

Accordingly, each sensor instance is associated with

its proxy instance. Proxies register the sensors to the

knowledge base and can be used to convey manage-

ment commands to the sensors (e.g., changing the

frame rate of a camera or the sampling frequency of a

microphone). Proxies support the above-mentioned

virtualized API, providing access to the capabilities of

the sensors. Note that this virtualization requires that

each sensor is wrapped to comply with this API. Once

such a wrapper is implemented, a sensor can be

seamlessly integrated to the overall ubiquitous com-

puting infrastructure.

3.2 Perceptual components APIs

Perceptual components are the primary consumers of

sensor streams. They are implemented as NIST Smart

Flow clients to allow high-performing distributed ac-

cess to sensor output. Perceptual technologies apply

novel algorithms on audio and video streams to extract

several forms of context pertaining to people and ob-

jects within the smart room. CHIL technologies in-

clude person localization and tracking, body detection,

head orientation, face detection and recognition, ges-

ture/posture recognition, head and hand tracking,

pointing gesture recognition, speech recognition

(including far-field), source localization, speech detec-

tion, speaker identification, acoustic and visual emo-

tion recognition, acoustic event classification, activity

recognition, and lips observation. A presentation of

these components is out of the scope of this paper.

Interested readers may access [33–36] for a description

of perceptual components available within our smart

room.

Several of these components are provided by more

than one technology sites within the CHIL consortium,

which creates the need to deal with heterogeneity at

the perceptual components level. To alleviate this

heterogeneity, APIs pertaining to each one of the

component types are defined. Perceptual components

APIs are structured as XML schemas and define input

and output parameters for each component. As soon as

a technology provider exposes this API for its (body

tracker in this case) component, the latter can be

integrated to the overall ubiquitous environment

Fig. 3 Perceptual components virtualization and APIs


123

within minimal customization effort. To the best of our

knowledge this is a first-of-the-kind effort, to stan-

dardize inputs and outputs of perceptive algorithms.

Based on perceptual component APIs, consumers of

the contextual information produced by any perceptual

component can tolerate dynamic switching of tech-

nology providers. Consumers will use the same API

over the CHILIX middleware. In this way, the archi-

tecture acts as a breadboard, since a variety of per-

ceptual components can be integrated in a plug ‘n play

fashion. In addition, perceptual components are

dynamically registered, discovered and managed,

based on a proxy mechanism. Similar to the sensors

case, implementing a proxy for each perceptual com-

ponent provides a uniform way for discovering,

accessing and managing the component.

The seamless integration and operation of percep-

tual components to the overall ubiquitous computing

environment depends not only on an abstract API, but

also on a wide range of environmental parameters and

settings. For example, the performance of image pro-

cessing algorithms (being the cornerstone of video-

based body tracking, face detection, and face identifi-

cation) depends highly on the lighting conditions of the

smart space. Nevertheless, developing components that

are autonomous and adaptive to environment (e.g.,

through adaptive foreground/background segmenta-

tion) falls in the technology provider’s jurisdiction and

is therefore, out of the scope of the architecture.

4 Situation modeling and context-awareness

4.1 Network of situations approach

Perceptual components provide elementary context

cues. These mainly relate to the status of people and

objects within the smart space. Sophisticated context-

aware applications are likely to rely on composite

contextual states, which ask for composing perceptual

components outputs. To this end, the architecture of-

fers a context modeling approach that relies on the

network of situations paradigm [15, 37]. According to

this paradigm, the contextual states of interest are

structured into a (state-diagram) graph (ST, ed), where

the nodes denote the target states and the arcs the

possible transitions between contextual states. Specifi-

cally, edge edij denotes that it is possible to reach state

Sj from state Si.

Contextual states may be arbitrarily complex in

terms of their defining cues. The situation model

is accompanied by a truth table, which depicts the

underlying combination of perceptual component

values (i.e., outputs) that triggers each one of the

composite contextual states. To formally specify how

the situation transitions occur, assume that a ubiqui-

tous computing environment is supported by m com-

ponents having k1, k2,..., km outputs respectively and let

k = max(k1, k2,..., km). Without loss of generality, we

can represent the outputs produced by all perceptual

components at a given time instant (t) using the matrix:

POUTðtÞ ¼ pijðtÞ� �

; ð1Þ

where 1 < i < m and 1 < j < k.

It is assumed that pij(t) = 0 for j > ki, since there are

perceptual components that possess less than k-out-

puts. As a result, Pout (t) contains the measurements

observed by the perceptual components outputs at

time instant t.

For each situation STl (1 < l < n) targeted by a sit-

uation model, we define a matrix Sl comprising the

values of the perceptual components that define Sl., as

follows:

Sl ¼ fsijg; 1\i\m; 1\j\k; ð2Þ

where sij „ 0 if the jth output of the ith perceptual

component contributes in the triggering the state Sl

and sij = 0 otherwise. Toward associating the non-zero

sij-values with the observed outputs pij(t), we perform

an element-wise multiplication of Pout (t), with the

following matrix:

Al ¼ faijg; 1\i\m; 1\j\k; ð3Þ

where aij = 1 if sij „ 0 and aij = 0 otherwise (i.e.,

sij = 0). The result of the element-wise multiplication

will produce a Pl matrix filtering the observed outputs

in a way that only values defining the state STl are

retained:

Pl ¼ aij � pijðtÞ� �

; 1\i\m and 1\j\k: ð4Þ

STl occurs when all the elements of the matrices Pl

and Sl coincide, i.e.,

Sl � Pl ¼ sij � aij � pij

� �¼ Omk; ð5Þ

where O corresponds to the matrix having all its

elements equal to zero. In practice, it is rare to achieve

a simultaneous total agreement between target values

and observed values. Therefore, the triggering of the

situation may be defined as the case when the elements

of the two matrices almost coincide, thus allowing the

perceptual outputs to slightly fluctuate over the target

values:


123

Sl � Pl ¼ sij � aij � pij

� �¼ El ¼ eij

� �; ð6Þ

where either eij fi 0.

In order for situation (STl) to be triggered, condition

(6) has to be fulfilled (for some eij-values), while at the

same time the situation model has to be on a state

(STp) that allows transition to Sl. Therefore,

STp! STlfoccurs whenever Sl�Pl¼El and edpl¼ 1:

ð7Þ

Note that this network of situations approach is

general and applicable not only to perceptual compo-

nents outputs but also to the more general class of

observations, which may also include sensor signals

(e.g., such as a temperature indication or a motion

sensor’s output). In practice, the matrix Sl is likely to

be very sparse, which can greatly simplify equation (6).

It is also noteworthy that Eqs. 1–6 assume numeric

outputs for perceptual components. While this may

sound limiting, it is in general, possible to encode

several other value domains as numbers.

4.2 Sample situation models

Figure 4 depicts a sample situation model for tracking

activities within a meeting. The situation model con-

sists of five states corresponding to the commencement

of a meeting, the start of a presentation during the

meeting, a question on the presentation, the end of a

presentation and the end of a meeting. The arcs in

Fig. 4 denote the possible transitions. For example, a

question can only occur while a presentation is in

progress, since there is no means to reach state S3

unless the model is in situation S2. Table 1 shows how

particular situation states are triggered based on

underlying perceptual components. NIL denotes the

starting state. As an example, the start of the meeting

(i.e., the transition NIL–>S1) occurs when an expected

number of people are speaking very close to the table.

Given a number of perceptual components (i.e.,

TablePeopleCount, WhiteBoardPeopleCount, Speech

Activity Detection, and Acoustic Localization) and

their APIs, Table 1 can be mapped to Eq. 6 in a

straightforward way. Nevertheless, the mapping is in-

general, application specific since the elements of the

matrices E1, E2,...,E5 (see Eq. 6) are likely to be de-

fined based on the problem at hand.

Figure 5, depicts an augmented version of the

meeting tracking situation model. In particular, this

situation model is enhanced to track people entering

(i.e., person entered) and leaving the meeting (i.e.,

person left). Moreover, it tracks situations, where a

person returns to a meeting that is in progress (i.e.,

person returned). Accordingly, Table 2 associates the

extended model with underlying perceptual compo-

nents. Extending situation models to capture/track

additional states, is one direction toward the scalability

of the network of situations solution. The architecture

makes it possible to use more than one situation

models for context tracking.

4.3 Situation model implementation

From an implementation perspective, situation models

are described in XML format, and loaded by the Sit-

uation Modeling Agent (SMA) within the JADE agent

society during the startup sequence. SMA access the

underlying perceptual components through their APIs

(over CHILIX). Accordingly, they keep track of the

current situation state, as well as whether Eq. 7 is ful-

filled for some transition. To this end, they leverage

perceptual component observations along with XML-

based situation model definition. Whenever Eq. 7 is

met, the SMA advances the tracked situation state to

the next one. Note also that the SMA includes data

structures holding information about people, objects

and events that are tracked by the situation model. As

information from perceptual components is fused to-

ward the SMA, these data structures are filled with the

appropriate information.

5 Intelligent actuating services

Actuating services provide the means for the smart

space to communicate with end users, for example by

outputting a voice prompt or displaying a message. TheFig. 4 Situation model tracking activities within a meeting


123

proposed architecture supports dynamic registration,

discovery and invocation of services, based on the al-

ready described proxy mechanism. Just like perceptual

components and sensors, actuating services come with

a proxy that allows them to be registered in the

Knowledge Base and accordingly discovered, invoked,

and managed. Furthermore, the architecture allows for

implementation of ambient services that intelligently

select the optimal actuating mechanism in a given

context. This implementation is performed at the agent

layer. Specifically, for any target intelligent service, a

special agent incorporating ambient service selection

logic needs to be implemented. This (JADE) agent has

the means to intercept incoming requests for actuating

services and to delegate them to the appropriate ser-

vice proxy among a set of actuating services that are

appropriate for the requested service. The service

developer has to extend this agent and provide the

service selection algorithm. The following paragraphs

describe two characteristic examples of intelligent

actuating services that are operational within our smart

room.

Table 1 Mapping perceptualcomponent outputs tosituation transitions

Situation transition Combinations of perceptual components outputs

NIL fi S1 TablePeopleCount = n (i.e., n people in table area)and Speech Activity Detection = 1

S1 fi S2 WhiteBoardPeopleCount = 1 (1 in speaker area),TablePeopleCount = n – 1 (n – 1 in table area)

Acoustic localization = within the whiteboard areaS2 fi S3 Acoustic localization = within the table areaS3 fi S2 Acoustic localization = within the speaker areaS2 fi S4 Face Detection (speaker area and table area)S4 fi S2 WhiteBoardPeopleCount = 1 (1 in speaker area),

TablePeopleCount = n – 1 (n – 1 in table area)Acoustic localization = within the whiteboard area

S4 fi S5 TablePeopleCount = 0 (i.e., 0 people in table area)

Fig. 5 Situation modeltracking activities within ameeting


123

5.1 Intelligent display

The intelligent display service selects the optimal

display device according to the location of the target

person(s), within a smart room. A smart space may

have more than one means (e.g., devices, projectors,

and smartboards) to visually display messages as well

as presentations, posters, etc. The goal of the intel-

ligent display services is to select the device that is

more convenient for the room participants. This is

accomplished as follows: whenever a display service

is requested, a special display agent intercepts this

request and examines it in relation to the current

context. The context of interest includes the location

and orientation of participants, which are tracked by

appropriate perceptual components. Specifically, body

trackers provide information about people location,

while a special video processing component provides

a metric of ‘faceness’ for the target participants [38].

Based on this information, a display selection algo-

rithm has been implemented and incorporated within

an appropriate agent, aiming at choosing the display

that best suits the location and orientation of par-

ticipants within the smart room. The algorithm at-

tempts to provide a satisfactory view for as many

participants as possible.

In obtaining contextual information (number of

participants, locations of participants, orientation of

participants) the display agent accesses the situation-

modeling agent. The latter collects and maintains this

information through body trackers and ‘faceness’

modules, in accordance with the already presented

architectural concepts.

5.2 Intelligent meeting recording: best camera

selection

The architecture offers the capability of storing camera

streams. This is based on a recording service, which can

be instantiated for any of the cameras within the smart

space. Storing camera streams is a key prerequisite to

recording meetings, lectures and presentations within

the smart space. A realistic meeting recording service

is expected to operate like an automated intelligent

cameraman. In particular, an ambient recording should

select the optimal camera view based on the location

and orientation of the participants, as well as on their

activities and role within the group interaction (e.g.,

whether a person is a presenter or speaker) (Fig. 6).

We have implemented such an intelligent meeting

recording service based on a Meeting Recording Agent

that continually monitors the location and orientation

of speakers toward storing the optimal camera stream.

Thus, the meeting recording agent combines informa-

tion from the body tracking, acoustic source localiza-

tion and ‘faceness’ components. This combination is

performed at the SMA, which continually notifies the

Meeting Recording Agent on changes of the orienta-

tion and location of speakers. Accordingly, the

Recording Agent communicates with other JADE

agents controlling the storage services for the various

cameras. A more thorough description of this record-

ing service, as a stand-alone ambient service can be

found in [38].

Table 2 Mapping perceptualcomponent outputs tosituation transitions

Situation transition Combinations of perceptual components outputs

NIL fi S1 BodyTracker = n + 1 (where n is previous number of people in the room)S1 fi S2 FaceID = i (i does not exist as a current ID in the system)S2 fi S4 BodyTracker = n AND TablePeopleCount = NS4 fi S5, S6 fi S5 TablePeopleCount = n – 1 AND WhiteBoardPeopleCount = 1 AND

Acoustic Localization = within the whiteboard areaS5 fi S6 TablePeopleCount = n – 1 AND WhiteBoardPeopleCount = 1 AND

Acoustic Localization = within the table areaS8 fi S9 BodyTracker = 0S4, S5, S6, S7 fi S8 BodyTracker = n – 1 (n is the current number of people in the room)S5, S6, S7 fi S1 BodyTracker = n + 1 (n is the current number of people in the room)

Fig. 6 Elements of the meeting recording service


123

6 Service logic and user front end

The service logic is implemented at the agent society

tier of the architecture. Typically, service logic is trig-

gered either on end users’ demand or wherever a

contextual change occurs. In the former case, end users

need to provide explicit input to the pervasive system,

which is in line with the vast majority of legacy com-

puting applications. In the later case, the context-aware

triggering of the service logic can be based on situation

state transitions, as the latter are modeled in situation

models. Thus, upon the identification of a transition,

some piece of service logic is executed. No matter the

triggering mechanism, the service logic will instigate

requests to available actuating services. These actuat-

ing services will ensure ambient natural context-aware

interaction with the end-users. In the scope of several

pervasive applications however, end-users are likely to

interact with conventional terminal devices (such as

Laptops, PDA, smart phones). Apart from modular-

izing the service logic, the introduced architecture al-

lows for implementation of service specific graphical

user interfaces (GUIs), which may run on different

platforms and terminal devices.

The agent tier of the architecture is a multi-agent

system (depicted in Fig. 7) consisting of the following

agent types:

• Device desktop agent, which implements the user

interface required for accessing the ubiquitous

services. A ‘pluggable’ mechanism (supported

based on the ‘pluggable’ behaviors feature of the

JADE platform) allows the user interface to be

customized to the particular ubiquitous computing

service.

• Device agent, which enables different devices to

communicate with the framework (Fig. 8).

• Personal agent, which constitutes the proxy of the

end user in the agent world. The personal agent

conveys user requests to the agent manager (AM),

which redirects them to other agents that can

handle them appropriately. It maintains the user’s

profile in order to personalize the services to the

end user.

• Basic services agents: these agents incorporate the

service logic of basic services, which are tightly

coupled with the infrastructure of the smart space.

Basic services include the ability to track composite

situations, as well as the control of sensors and

actuators. Tracking of composite situations is per-

formed through the SMA based on the network of

situations context modeling approach. Control of

sensors and actuators is performed through the

Smart Room Agent (SRA). Furthermore a Knowl-

edge Base Agent (KBA), allows the agents of the

framework to dynamically access information on

the state of the components of ubiquitous comput-

ing environment (e.g., sensors, actuators, percep-

tual components), through a Knowledge Base

Server that is supported by the ontology manage-

ment system.

• Agent manager, which allows the system to be

dynamically augmented with additional Service

Agents. Moreover, the AM resolves requests for

agent services, through re-directing them to appro-

priate agents that match the request.

Moreover, the multi-agent system comprises service

agents that implement the non-obtrusive service logic

of the various context-aware services. Each pervasive

computing service is therefore implemented as a ser-

vice agent and accordingly plugged into the framework

based on the JADE ‘pluggable’ behaviors mechanism.

In the scope of the CHIL project, several service

agents corresponding to various ubiquitous computing

services are implemented and integrated into this

framework.

The service logic for a given service is implemented

as a separate Service Agent, which is modular and

‘pluggable’ to the overall architectural framework.

From an implementation perspective, ‘pluggability’

exploits the pluggable behaviors feature of the JADE

environment. Note that ‘pluggability’ supports the

overall concept and goals of the breadboard architec-

ture at the service logic level. Service agents interact

with the SMA, to access contextual information. Con-

text triggering is achieved through conveying state

transitions from the SMA to the Service Agent.

Moreover, the service agent may interact with agents

implementing the user front end for different terminal

devices. To facilitate implementation of Service

Agents (i.e., of the service logic), the architecture

provides template agents that can be directly extended

to incorporate the target service logic. Templates

facilitate the process of access contextual information

and obtaining contextual notifications, allowing the

service developer to emphasize on the service logic

rather than the underlying middleware.

The GUI implementation is incorporated into spe-

cial agents called Device Agents. Different types of

device agents exist for the various terminals (e.g.,

PDADeviceAgent, NotebookDeviceAgent). Device

Agents receive user requests through GUI actions and

convey them to the service logic. Furthermore, they

receive requests from the service logic for GUI inter-

action with the end user (through the user front end).


123

These requests are appropriately filtered from a Per-

sonal Agent (which maintains the user’s profile and

implements personalization). Similar to the Service

Agents’ case, the architecture provides templates for

the Personal Agent and the Device Agents, with a view

to minimizing the service implementation or custom-

ization effort. In this way, reusability at the agent layer

is pushed to the extreme. A more thorough description

of the agent society, including detailed agent descrip-

tions and interactions is contained in [15, 40].

7 Fault-tolerance and system management

The proposed architecture addresses non-functional

requirements relating to the reliability, availability and

ease of management. These sets of non-functional

requirements define softer goals of the target pervasive

services and are not defined as part of use cases and

user requirements. In the sequel, we describe support

of the proposed architecture with respect to fault-tol-

erance, reliability, and system management. The fol-

lowing paragraph presents performance measurements

based on real prototype services that adopt this archi-

tecture.

7.1 Reliability: fault-tolerance

The introduced architecture incorporates recovery

functionalities at the agent layer. This functionality is

applied whenever an agent dies [e.g., due to a hardware

or software (Java Virtual Machine) failure]. The

implementation is based on a controlling agent called

Autonomic Agent Manager (AAM), which has as main

functionality the monitoring of specific agents regard-

ing their status (alive or not) and to act accordingly.

This monitoring is done using a ping-like functionality

(implemented as a JADE behavior), which is initiated

by the AAM to all monitored Agents.

JADE provides functionalities regarding message

exchange among the members of the Agent Society it

supervises. The ping-like behavior initiates an the

transmission of an Agent Communication Language

message (ACLMessage) by the AAM to all monitored

Agents. Upon reception of such a message (sent by the

AAM), the agent replies back with its status. In case of

a significantly delayed response, the AAM considers

that the agent is dead. Upon realizing that the agent

has failed, the AAM attempts to restart the agent on a

given alternative host, which depends on configuration

options. The restarting process distinguishes between

two different methodologies namely stateful and state-

less recovery.

Stateless recovery is initiated to agents, which do not

maintain a state during their execution. Examples of

such agents are all those that convey requests to

actuating services. After the registration of each agent

with AAM, the AAM maintains all the information

needed for re-starting all the agents it supervises. This

information consists of the class file and the name of

agent. Following the restart process, the ping behavior

Fig. 7 Anatomy of the multi-agent system comprising the agenttier

Fig. 8 Delivery of information to multiple terminal devicesthrough the agent layer


123

responsible for listening for ping messages and

answering back with an ‘alive’ response is also started.

Stateful recovery is required for agents that main-

tain state. Characteristic example of such agents is the

SMA, which keeps track of the situation models. In

recovering such agents, we monitor and store their

state to a Relational Database Management System

(RDBMS), thus allowing the state to be retrieved upon

completion of the regeneration sequence. Following

the completion of the restart process (which is similar

to the stateless case), the serialized state is retrieved by

the rejuvenated agent, and becomes the current state.

From an implementation perspective, stateful agents

are stored as serialized objects in the relational data-

base. To this end, we leverage Hibernate [41], which is

a powerful, ultra-high performance object/relational

persistence and query service for high-level systems.

The stateful recovery process is shown in Fig. 9.

7.2 System management

Systems adopting the proposed architecture are in

principle highly heterogeneous and distributed, with no

single centralized control point. Thus, it is fundamen-

tally difficult to build a unified management system for

the whole range of hardware, software, and middle-

ware components comprising the pervasive environ-

ment. To this end, the system management relies on

the disjoint management sub-systems that monitor and

manage individual tiers of the architecture. These sub-

systems are based on tools developed as supplements

to the architecture, as well as on third-party tools.

Third-party tools include the JADE Directory

Facilitator (DF), as well as the NIST Smart Flow

Control Center (SFCC). The DF manages each one of

the agent instance according to sets of unique identi-

fiers. Whenever a new agent is instantiated, it registers

itself to the DF so that all other agents can find it when

and if they need to. This is also shown in Fig. 8, which

depicts that every rejuvenated agent, registers with the

DF. The SFCC provides a management interface used

to instantiate the Smart Flow servers so that perceptual

components access streams in a distributed fashion.

Along with the architecture, we have also developed

the Smart Space Resource Manager (SSRM) [19],

which is a utility enabling monitoring and control of

perceptual components, sensors, and actuating services

available in a smart space (e.g., smart room). Admin-

istrators of the smart space may use the SSRM utility,

to manage a variety of heterogeneous distributed

entities from a single entry point. SSRM has been

implemented as a JADE-based mutli-agent system,

with a twofold objective: (a) to facilitate interopera-

bility between the diverse hardware (i.e., sensing and

actuating devices) and middleware (i.e., perceptual and

actuating components) and (b) to leverage the JADE

access interface to the Knowledge Base Server, along

with the corresponding KBA.

The JADE based proxies of sensors, perceptual

components, devices, and services allow these compo-

nents to be managed by the SSRM. Agent proxies act

as gateways allowing other agents within the architec-

ture to interact with the device. As already outlined,

proxy agents provide a level of abstraction: each proxy

exposes a universal virtualized interface to the agent

framework (i.e., a common interface for sensors,

actuators, perceptual components of the same type).

The building blocks of the SSRM are shown in Fig. 10.

The current implementation of the SSRM can

graphically depict the contents of ontological knowl-

Fig. 9 Stateful agentrecovery based on de-serialization of the agent’sstate


123

edge base through an easy-to-use interface. In par-

ticular, the SSRM dynamically retrieves the regis-

tered elements and presents them in a GUI.

Moreover, it can also issue control commands to

proxy agents of the cameras. Control commands (e.g.,

changing capture frame-rates, changing zoom levels)

on components and devices are handy in cases where

there is a need to adapt the operation of the devices

(e.g., a camera frame rate), to either a particular

context or to the operational requirements of a per-

ceptual processing component (e.g., a face recogni-

tion or visual tracking algorithm). Thus, the SSRM is

a useful tool that can support several processes en-

tailed in the development and deployment cycle of

ubiquitous computing applications. Monitoring of

sensors, actuating devices and perceptual components

is important for developing and debugging ubiquitous

computing services, since it facilitates identification of

component failures and failing factors. Moreover, it

can facilitate deployment through providing an

overall view of the infrastructure and the perceptive

capabilities of a smart space.

8 Prototype services and applications

Based on the presented architectural framework and

its middleware components, we have developed non-

trivial ubiquitous context-aware services within our

prototype smart space. This prototype smart space

consists of a variety of sensors, devices and perceptual

components, and is therefore used as a testbed for

application development in the areas of pervasive

computing and multi-modal interfaces. The floorplan

of this smart space is depicted in Fig. 11.

The middleware described in the scope of the pre-

vious section has greatly facilitated the development of

sophisticated computing services. It is also noteworthy

that the majority of the middleware components have

been proven reusable across different services.

8.1 Memory jog

The goal of the Memory Jog service is to provide non-

obtrusive assistance to humans in scope of meetings,

lectures, presentations, and conferences occurring

within the smart space. It exploits the architecture

along with its associated middleware components to:

• Identify participants and track their locations with-

in the smart room. Identity and location of partic-

ipants can be displayed within different devices.

• Keeping track of meeting progress based on an

agenda, which is known before-hand.

• Recording the meeting, lecture or presentation

based on the best-camera selection mechanism.

The recording is tagged with meta-data from the

situation model of the service, to allow selective

retrieval and post-processing of the recording.

• Provide context-aware assistance through display-

ing relevant information from past meetings (e.g.,

presentations, URLs). Information from past meet-

ings includes video segments recording through the

recording service.

KnowledgeBase

Sensor,Actuator,

Perceptual Component

Low-level driver(e.g. IEEE1394)

CHILIX JNI

JADE ProxyAgent

(sensor,actuator,

perceptualcomponent)

Knowledge Base Agent'KBAgent'

Smart Space ResourceManager(SSRM)

Console & Control Agent

DiscoveryControl

Register/De-Register/Update

StatusControl

Fig. 10 Building blocks of thesmart space resource manager(SSRM)


123

• Assist participants by automatically carrying out

casual meeting tasks, such as opening presentations,

e-mail material to the participants and displaying

notes/messages.

The Memory Jog relies on a variety of perceptual

components including face recognition (e.g., to identify

participants), acoustic localization, visual body track-

ing (e.g., to track participant locations), face detection,

and speech activity detection. It also exploits the situ-

ation model depicted in Fig. 4, to follow higher-level

situations such as agenda tracking, question tracking,

meeting commencement, and meeting finish. This ser-

vice exploits actuating services (targeted audio, text-to-

speech, display services) to output information to

meeting participants. Moreover, information is ren-

dered in GUIs, which run both on PCs and PDAs

(Fig. 12).

The architecture framework kept development ef-

fort for the memory jog service to a minimum, despite

its rich functionality. Specifically, implementing the

Memory Jog service was a matter of:

• Configuring the situation model within the SMA,

i.e., producing the XML description for the Situa-

tion Model.

• Defining the information exchange primitives for

the communication within the agents.

• Implementing a Memory Jog service agent that

incorporated the target service logic. This involved

implementing appropriate message handlers for

messages received from the Situation Model and

Perceptual Components, both through the SMAs.

• Implementing the target GUIs within the respective

device agents. This involved again the task of

implementing message-handling logic for informa-

tion primitives stemming from the Memory Jog

agent.

• Augmenting the recording service to store tagged

video, as well as the GUIs to retrieve and display

the tags.

Note that the following steps pertain to the service

logic and the functional features of the Memory Jog

pervasive service. Tedious tasks such as interfacing

with sensors, perceptual components and actuating

services were handled by the architecture’s middleware

components. Moreover, in implementing, testing,

debugging, and deploying the service, we took advan-

tage of Smart Room Resource Manager.

8.2 WHWIWA: what happened while I was away?

The what happened while I was away (WHWIWA)

service targets intelligent working and living environ-

ments, with the aim to provide people with summaries

of events that occurred during their absence from the

smart space. To this end, the service identifies people

and tracks the moments where they enter and leave the

Fig. 11 Floor plan of thesmart room of the Athensinformation technology


123

smart space. Upon identification of the situation that a

person returns to the smart space, he/she is notified of a

series of contextual state that occurred during his/her

absence. Contextual states are tracked according to the

presented situation modeling approach. Information

can be conveyed to end user through auditory inter-

faces or even a conventional GUI. In its simplest form,

this service is less sophisticated than the Memory Jog,

since the sensors, perceptual components, and actuat-

ing services used for the Memory Jog are significantly

more than those used in the scope of the WHWIWA

service.

The two services can be combined to provide par-

ticipants in meetings or conferences with the option of

obtaining contextual information during short intervals

of absence. This can be supported by the situation

model depicted in Fig. 5. The overall implementation

required a series of implementation steps similar to the

Memory Jog case. Those steps concerned mostly

functional features and FIPA message processing to

realize them, rather than dealing with the low-level

interfacing with sensors, perceptual components, and

actuating services.

While these two service implementations demon-

strate that several components of the architecture can

be reused across services, it does not fully manifest the

importance of the perceptual component APIs. The

value of the APIs lies in that they allow exchange of

perceptual components from different technology

providers. Currently, we have internally tested this

potential for seamless exchange of perceptual compo-

nents through experimenting with different algorithmic

implementations for face identification, body tracking,

and speech activity detection components. Based on

the architecture, perceptual component developers can

work totally independently from service developers, as

long as they respect the APIs for these components.

Following this proof-of-concept validation, we are in

the process of exchanging components with different

technology providers within the CHIL project.

8.3 Evaluation

In evaluating these prototype services, we conducted a

series of six focus groups, involving 25 participants with

diverse backgrounds (students, faculty, technical peo-

ple, management staff). Some participants found indi-

vidual features (e.g., the meeting recording) more

appealing than the whole service. Many participants

expressed concerns about privacy issues, while others

thought that some features are not practical in small

smart rooms (e.g., people identity and location).

Reactions also varied for different groups. Neverthe-

less, all groups found several useful features in both the

Memory Jog and the WHWIWA services. The latest

remarks, along with the technological advances in

context-aware applications and smart spaces, lead us to

believe that smart space applications are likely to move

soon from the research labs to the market. In fact, we

believe that smart space applications will form the

second wave of enterprise-pervasive computing appli-

cations, following the widespread RFID applications

[42–43]. This is also evident from the numerous in-

stances of context-aware services for smart spaces,

which have been developed in the scope of deploy-

ments in smart homes, smart conference rooms and

systems for ambient assisted living (e.g. [43–44]). The

proliferation of such services will be greatly facilitated

by middleware components and libraries such as those

presented in this paper.

Fig. 12 Desktop and PDAversions of the memory jogservice GUI


123

9 Performance measurements

While the presented middleware components facilitate

integration in a distributed multi-vendor environment,

they also incur a slight performance overhead. Wrap-

ping components through uniform interfaces implies

additional overhead in message passing and processing

across the sensing, perceptual processing, situation

identification, and service invocation chain. Given the

real-time nature of various pervasive systems, this

overhead may be essential. To assess this additional

overhead, we conducted a set of performance mea-

surements, while the Memory Jog service was in

operation.

Figure 14 depicts the evolution of the delay for

conveying contextual information from the perceptual

components tier to the agent tier, a sequence, which is

shown in more detail in Fig. 13. Specifically, the delay

is measured over body tracking messages sent from the

body tracker to the SMA. The delay figures/measure-

ments in Fig. 14 include the transfer network delay, the

processing/parsing delay at the CHILIX middleware

layer, as well as the processing delay at the situation-

modeling layer (i.e., the corresponding agent)1. This

aggregate delay has been seen in all cases below 1 s,

which is more than tolerable for the nature of the

applications at hand. Indeed, processing and conveying

body tracking messages to the service logic in less than

1 s, results in a real-time responsiveness in recognizing

situation states and triggering service actions.

The delay appears to be almost uniformly distrib-

uted among three different levels (i.e., 200, 400, and

600 ms), based on an incremental escalation of the

delay. The increments are due to the fact that the body

tracker messages at hand carried information about all

the participants of the memory-jog supported meeting.

Thus, as more and more participants join the meeting,

the body tracker produces and conveys a larger piece

of information. As a result, the delay is grows almost

analogous to the number of people participating in the

meeting. This is also shown in Fig. 15, which demon-

strates that the size of the messages exchanged be-

tween perceptual components and the agent tier

through CHILIX increases as the number of meeting

participants increases. Figure 15 shows also that the

message sizes pertaining to the FIPA communication

ontology increase also as a result of an increasing

number of meeting participants. Note that the

increasing number of participants increases the amount

of information that has to be processed and transmitted

through the overall distributed system. Thus, the

overall delay is inevitably increased.

Figure 16 depicts also the delay within the service

tier (i.e., the JADE layer) of the system. This delay is

again within acceptable levels (less than 0.4 s). In all

cases, absolute processing figures are only indicative,

given that processing time depends heavily on the

processing capacity of the hosting machine. Neverthe-

less, these indicative figures manifest that the archi-

tecture does not constitute a performance bottleneck

for the overall system.

10 Conclusions

Dealing with the heterogeneity and distribution of

software, middleware, and hardware components is a

key challenge in pervasive computing systems. In this

paper, we have presented an architectural framework

and a set of middleware components facilitating inte-

gration of sensors, actuators, perceptual components,

as well as sophisticated context acquisition and mod-

eling. The architecture provides flexibility in managing

and coordinating the various components, along with

their interactions. It operates like a breadboard, en-

abling seamless addition and removal of hardware,

software, and middleware components. To this end it

leverages a knowledge base registry, which is sup-

ported by semantic web technologies and ontological

management systems.

Fig. 13 The processingcomponents in the CHILsystem

1 Agent level processing was supported by a dual-processor(2.4 GHz) Zeon machine.


123

One of the key features of this architecture is that it

incorporates schemes for modeling complex contextual

states. These schemes combine perceptual components

outputs toward recognizing situations, along with

transitions between contextual states. Apart from

sophisticated context modeling, the architecture en-

ables the implementation of intelligent actuating ser-

vices. Prominent examples of such services are an

intelligent display service and a meeting recording

service. The intelligent display service displays infor-

mation in devices based on the orientation of the

audience. Similarly, the meeting recording services

incorporates algorithms that select the best camera for

a given orientation of a speaker within the smart room.

Note that the architecture provides also a number of

reliability and system management features. The latter

facilitate development, debugging and deployment of

pervasive computing applications.

A number of real-life prototype applications have

been developed based on this architecture. The

development of these applications has been carried

out with minimal development effort. The major part

of this effort was allocated to producing the service

logic, since the architecture facilitated the integration.

Message size

020406080

100120140160180200

1 2 3 4

Number of people

Siz

e(b

ytes

)

CHiLiX

Ontology

Fig. 15 Size of messages exchanged in CHILiX and betweenagents (ontology) for varying numbers of meeting participants

Fig. 16 Delay contributed bythe agent layer

Fig. 14 Delay overheadevolution during a meeting


123

Performance measurements demonstrated that the

integration and structural benefits of the architecture

come with a decent performance overhead. Overall, we

have introduced an architecture that offers several un-

ique features and benefits over existing architectures

developed in the scope of previous large scale ubiqui-

tous computing related projects. The main distinguish-

ing features of our architecture lie in its breadboard

nature toward multi-vendor components, as well as in

use of intelligent directory services.

Acknowledgments This work is part of the FP6 CHIL project(FP6-506909), partially funded by the European Commissionunder the Information Society Technology (IST) program. Theauthors acknowledge valuable help and contributions from allpartners of the project, especially of partners participating inWP2, which defines the software architecture of the project.

References

1. Weiser M (1991) The computer for the 21st century. Sci Am265(3):66–75

2. Dey AK (2001) Understanding and using context. PersUbiquitous Comput J 5(1):4–7

3. Dey AK, Salber D, Adowd GD (2001) A conceptualframework and a toolkit for supporting the rapid prototypingof context-aware applications. Hum Comput Interact J16:97–166

4. Want R, Hopper A, Falcao V, Gibbons J (1992) The activebadge location system. ACM Trans Inform Syst 10(1):91–102

5. Smailagic A, Siewiorek D (2002) Application design forwearable and context-aware computers. IEEE PervasiveComput 1(4):20–29

6. Johanson B, Fox A, Winograd T (2002) The interactiveworkspaces project: experiences with ubiquitous computingrooms. IEEE Pervasive Comput Mag 1(2):67–75

7. Yau SS, Karim F, Yu W, Bin W, Gupta SKS (2002) Re-configurable context-sensitive middleware for pervasivecomputing. IEEE Pervasive Comput (joint special issue withIEEE Pers Commun Context-Aware Pervasive Comput)1(3):33–40

8. Ponnekanti SR, Johanson B, Kiciman E, Fox A (2003)Portability, extensibility and robustness in iROS. In: Pro-ceedings of the 1st IEEE international conference on per-vasive computing and communications PERCOM. IEEEComputer Society, Washington, pp 11–19

9. Coen M, Phillips B, Warshawsky N, Weisman L, Peters S,Finin P (1999) Meeting the computational needs of intelli-gent environments: the Metaglue system. In: Proceedings ofthe 1st international workshop on managing interactions insmart environments. Dublin, Ireland, pp 201–212

10. Saif U, Pham H, Paluska JM, Waterman J, Terman C, WardS (2003) A case for goal-oriented programming semantics.In: Workshop on system support for ubiquitous computing(UbiSys’03), 5th international conference on ubiquitouscomputing (Ubicomp 2003). Seattle, 12 October 2003, pp 74–83

11. Brumitt B, Krumm J, Meyers B, Shafer S (2000) Ubiquitouscomputing and the role of geometry. IEEE Pers Commun7(5):41–43

12. Shafer S, Krumm J, Brumitt B, Meyers B, Czerwinski M,Robbins D (1998) The new EasyLiving project at microsoft

research joint DARPA/NIST smart spaces workshop. Gai-therburg, Maryland, 30–31 July 1998, pp 127–130

13. Garlan D, Siewiorek D, Smailagic A, Steenkiste P (2002)Project aura: towards distraction-free pervasive computing.IEEE Pervasive Comput 1(2):22–31

14. The CHIL project, http://www.chil.server.de15. Soldatos J, Pandis I, Stamatis K, Polymenakos L, Crowley JL

(2006) Agent based middleware infrastructure for autono-mous context-aware ubiquitous computing services, Com-puter Communications, Elsevier (in press) DOI 10.1016/j.comcom.2005.11.018

16. Soldatos J, Polymenakos L, Pnevmatikakis A, Talantzis F,Stamatis K, Carras M (2005) Perceptual interfaces and dis-tributed agents supporting ubiquitous computing services. In:Proceedings of the Eurescom Summit 2005. Heidelberg, pp43–50

17. Pandis I, Soldatos J, Paar A, Reuter J, Carras M, Polyme-nakos L (2005) An ontology-based framework for dynamicresource management in ubiquitous computing environ-ments. In: Proceedings of the 2nd international conferenceon embedded software and systems, Xi’an, 16–18 December2005, pp 195–203. ISBN 0-7695-2512-1

18 Pnevmatikakis A, Soldatos J, Talantzis F, Polymenkos L(2006) Robust multimodal audio-visual processing for ad-vanced context awareness in smart spaces. In: Magloniannis I,Karpouzis K, Bramer M (eds) 3rd IFIP conference in artifi-cial intelligence applications and innovations, AIAI 2006.Springer series: IFIP, vol 204. Springer, Berlin HeidelbergNew York

19. Soldatos J, Stamatis K, Azodolmolky S, Pandis I, Polyme-nakos L (2007) Semantic web technologies for ubiquitouscomputing resource management in smart spaces. Int J WebEng Technol (in press)

20. Tesauro G, Chess DM, Walsh WE, Das R, Segal A, WhalleyI, Kephart JO, White SR (2004) A multi-agent systems ap-proach to autonomic computing, AAMAS In: 3rd interna-tional joint conference on autonomous agents and multi-agent systems (AAMAS’04), vol 1. Columbia University,New York, pp 464–471

21. Java Agent Development Environment, http://jadetilabcom/22. FIPA—The Foundation for Intelligent Physical Agents,

http://wwwfipaorg23. Rosenthal L, Stanford VM (2000) NIST smart space: per-

vasive computing initiative. In: Proceedings of the 9th IEEEinternational workshops on enabling technologies: infra-structure for collaborative enterprises WETICE. IEEEComputer Society, Washington, 4–16 June 2000, pp 6–11

24. UPnP Forum (2003) UPnP device architecture 1.0. Availableat: http://www.upnp.org/

25. Guttman E, Perkins C, Veizades J, Day M (1999) RFC 2608:service location protocol, Version 2. Available at: http://www.ietf.org/rfc/

26. UDDI Consortium (2002) UDDI technical white paper.Available at: http://www.uddi.org/pubs/

27. Czerwinski SE, Zhao BY, Hodes TD, Joseph AD, Katz RH(1999) An architecture for a secure service discovery service.In: Proceedings of the 5th annual international conferenceon mobile computing and networking (Seattle, Washington,USA) Mobicom’99. ACM Press, New York, pp 24–35. DOIhttp://doi.acm.org/10.1145/313451.313462

28. Zhu F, Mutka M, Ni L (2003) Splendor: a secure, private,and location-aware service discovery protocol supportingmobile services. In: Proceedings of the 1st IEEE interna-tional conference on pervasive computing and communica-tions PERCOM. IEEE Computer Society, Washington, pp235–242


123

29. W3C Recommendation (2004) OWL web ontology languageoverview. Available at: http://www.w3.org/TR/owl-features/

30. Moller R, Haarslev V (2004) RACER: renamed ABox andconcept expression reasoner, Hamburg. Available at: http://www.sts.tu-harburg.de/~r.f.moeller/racer/

31. Brandstein M, Ward D (2001) Microphone arrays: signalprocessing techniques and applications. Springer-Verlag,Berlin

32. Murphy A, Picco GP, Roman G-C (2001) LIME: a middle-ware for physical and logical mobility. In: Golshani F et al.(eds) Proceedings of the 21st international conference indistributed computing systems(ICDCS-21). Phoenix, AZ, pp524–533

33. Pnevmatikakis A, Polymenakos L (2005) An automatic facedetector and recognition system for video streams. In: Pro-ceedings of the 2nd joint workshop on multimodal interac-tion and related machine learning algorithms. Edinburg

34. Stergiou A, Pnevmatikakis A, Polymenakos L (2005) Audio/visual person identification. In: Proceedings of the 2nd jointworkshop on multimodal interaction and related machinelearning algorithms. Edinburg

35. Talantzis F, Constantinides AG, Polymenakos L (2005)Estimation of direction of arrival using information theory.IEEE Signal Process 12(8):561–564

36. Pnevmatikakis A, Polymenakos LC (2004) Comparison ofeigenface-based feature vectors under different impairments.In: Proceedings of the 17th international conference onpattern recognition (ICPR’04), vol 1. Cambridge, 23–26August 2004, pp 296–299

37. Crowley JL (2003) Context driven observation of humanactivity. Lecture notes in computer science (Ambient Intel-ligence), vol 2875, Springer, Berlin Heidelberg New York, pp101–118

38. Azodolmolky S, Dimakis N, Mylonakis V, Souretis G,Soldatos J, Pnevmatikakis A, Polymenakos L (2005) Mid-dleware for in-door ambient intelligence: the polyomatonsystem. In: Proceedings of the 2nd international conferenceon networking, next generation networking middleware(NGNM 2005). Waterloo, 2–6 May 2005

39. Minar N, Gray M, Roup O, Krikorian R, Maes P (200) Hive:distributed agents for networking things. IEEE Concurr8(2):24–33

40. Soldatos J (2006) Software agents in ubiquitous computing:benefits and the CHIL case study. In: Proceedings of thesoftware agents in information systems and industrial appli-cations (SAISIA 2006) workshop. Karlsruhe

41. Hibernate Relational Persistence for Java and .NET, http://www.hibernate.org

42. Stanford V (2003) Pervasive computing goes the last hun-dred feet with RFID systems. IEEE Pervasive Comput2(2):9–14

43. Stanford V (2002) Pervasive computing goes to work:interfacing to the enterprise. IEEE Pervasive Comput1(3):6–12

44. Stanford V (2002) Pervasive computing: applications—usingpervasive computing to deliver elder care. IEEE Distrib SystOnline 3(3):10–13


123

A breadboard architecture for pervasive context-aware services in smart spaces: middleware...

Documents

Transcript of A breadboard architecture for pervasive context-aware services in smart spaces: middleware...