SOCA (2011) 5:17–35DOI 10.1007/s11761-011-0078-7

ORIGINAL RESEARCH PAPER

Distributing emotional services in Ambient Intelligence throughcognitive agents

Giovanni Acampora · Vincenzo Loia · Autilia Vitiello

Received: 8 May 2010 / Revised: 23 January 2011 / Accepted: 24 January 2011 / Published online: 22 February 2011© Springer-Verlag London Limited 2011

Abstract Ambient Intelligence (AmI) is a pervasive com-puting paradigm whose main aim is to design smart envi-ronments composed of invisible, connected, intelligent andinteractive systems, which are naturally sensitive and respon-sive to the presence of people, providing advanced servicesfor improving the quality of life. Nevertheless, AmI systemsare more than a simple integration among computer technolo-gies; indeed, their design can strongly depend upon psychol-ogy and social sciences aspects describing, analysing andforecasting the human being status during the system’s deci-sion making. This paper introduces a novel methodology forAmI systems designing that exploits a service-oriented archi-tecture whose functionalities are performed by a collectionof so-called cognitive agents. These agents exploit a novelextension of Fuzzy Cognitive Maps benefiting on the theoryof Timed Automata and a formal method for representinghuman moods in order to distribute emotional services ableto enhance users’ comfort and simplify the human/systemsinteractions. As will be shown in experimental results, wherea usability study and a confirmation of expectations test havebeen performed, the proposed approach maximizes the sys-tem’s usability in terms of efficiency, accuracy and emotionalresponse.

Keywords Ambient Intelligence · Cognitive Services ·Fuzzy Cognitive Maps · Timed Automata

G. Acampora (B) · V. Loia · A. VitielloDepartment of Mathematics and Computer Science,University of Salerno, Fisciano, Salerno 84084, Italye-mail: gacampora@unisa.it

V. Loiae-mail: loia@unisa.it

A. Vitielloe-mail: avitiello@unisa.it

1 Introduction

Ambient Intelligence (AmI) is a pioneer computing para-digm whose aim is the integration of computation into livingenvironments in order to enable the people to move aroundand interact with their surroundings more naturally thanthey currently do. Literature shows that different computerapproaches have been exploited in order to try to integrateintelligence in living spaces in a transparent way. In particu-lar, systems’ intelligence could be derived by the applicationof neural or evolutionary techniques that collect data intothe user surrounding to learn the user’s preferences and, asa consequence, define advanced services collections for sup-porting the user during its daily activities. From this point ofview, an AmI system can be viewed as a distributed service-oriented architecture [2], where each service is designed byexploiting computational intelligence methods and computerinternetworking techniques [10]. However, last researchesprove that an AmI system is more than a simple integrationof computational methodologies, and some additional scien-tific backgrounds are necessary to model the emotional andpsychological state of a generic user that interacts with animmersive computing framework. Starting from this consid-eration, our idea is to propose an innovative emotion-awareAmI architecture, based on the integration of methods ofcomputational intelligence and cognitive modelling, whosemain aim is to define a kind of intelligent environment evalu-ating human emotion experiences and providing people withproper emotional services. Emotional services provide theopportune system’s behaviour responding to user’s emotiongeneration [28]. This framework has been designed by inte-grating a multi-agent system (MAS) into a Service-OrientedArchitecture in order to define the so-called cognitive agents,i.e., intelligent agents that distribute emotional servicesand provide people with personalized living scenarios by

123

18 SOCA (2011) 5:17–35

anticipating their behaviour and satisfy their specific require-ments.

Different from other proposed approaches, agents thatpopulate our framework show a time-depending behaviourthat distributes distinct services under the same emotionaland environmental conditions. This choice is particularlyappropriate in AmI where the services distribution maydepend upon temporal information such as the current sea-son or time of day. In order to pursue this aim, cognitiveagents derive their inference capabilities from a novel fuzzycognitive engine that directly manages emotional and tem-poral issues strongly characterizing the immersive comput-ing environments. These innovative inference capabilitiesresult from the joint exploitation of the Thayer’s two-dimen-sional emotion model and Timed Automata-based FuzzyCognitive Maps: the former enables agents to capture andmodel the human emotional state by exploiting a well-definedapproach; the latter are used to infer the most suitable ser-vices collection starting from the analysis of human emo-tions, environmental features and temporal concept. As willbe shown in experiments, where a usability study and a con-firmation of expectations test have been conducted, our pro-posal of cognitive AmI framework improves user’s comfortand satisfaction and enhances the system’s usability in termsof efficiency, accuracy and emotional response. Moreover,because the proposed approach hardly separates temporaldetails from cognitive features of a given system, it stronglysimplifies the system design stage, as will be shown in thecase study.

The paper is organized as follows. In Sect. 2, related workare presented. Sections 3 and 4 introduce the architecture orour proposal of emotion-aware AmI. In the Sect. 5, a fuzzyextension of Thayer’s emotion model is presented. Section 6is devoted to introduce the basic concepts about the TimedAutomata-based Fuzzy Cognitive Maps. Section 5, repre-senting the paper core, shows the formal definition of TimedAutomata-based Fuzzy Cognitive Maps. Finally, Sect. 7 isdevoted to highlight the benefits derived from the TimedAutomata-based Fuzzy Cognitive Maps modelling througha set of experiments together with the related performancesanalysis obtained by means of a usability study and a confir-mation of expectation test.

2 Related works

As above mentioned, one key requirement for AmI frame-work is the automatic adaptation of the behaviour of systemsto users’ activity, needs, preferences and context.

Several research groups are working to develop intelli-gent environments like smart homes [1,13] or other kindof intelligent spaces [11]. In particular, in [16], an archi-tecture of intelligent user services that address the domain

of networked home environments is presented. This workpresents the central achievements of the first 1.5 years in thework package on intelligent user services within the EU-IST funded research project Amigo (Ambient Intelligencefor the Networked Home Environment). This project dealswith the issues of realizing such smart homes that do notonly integrate various household appliances, sensors, andconsumer electronics devices, but also aims at creating ser-vice infrastructures that use context information in order toprovide adapted functionality. These services are defined asintelligent user services in the sense that intelligence refersto adaptive capabilities with regard to being aware of theusage conditions, physical contexts and social situations. [24]presents an agent-based approach into a more general ser-vice-oriented architecture for addressing the requirementsof accessibility content and services in an ambient intelli-gence context. In order to achieve this task, a methodol-ogy for integrating a FIPA-compliant agent platform withthe OSGi service-oriented framework is proposed. Besides,[8] presents the development of an innovating appliance thatincorporates interactive services of leisure and information,offered through a high-quality user interface on the surface ofa mirror. The built prototype offers basic services, like inter-active television, the presentation of personalized weatherdata, news and selections of musical tunes.

However, new researches prove that considering alsouser’s emotional states as context information improves thedistribution of personalized services. For this reason, as willbe described below, our approach considers also user’s moodtogether with environmental features obtaining to build intel-ligent spaces that are more adaptive and personalized withrespect to the aforesaid intelligent environments.

In the current state of art, some frameworks consideringalso user’s emotional behaviour have been proposed. Forexample, in [17], an agent-based architecture to support aubiquitous group decision support system, which is able toexhibit intelligent, and emotional-aware behaviour, and sup-port argumentation, through interaction with individual per-sons or groups, is proposed. Also in [29], the main goal is tocreate a framework for emotion-aware ambient intelligence(AmE). This framework facilitates building applications thattake their user’s emotions into account. It acquires the user’smotivation (identifies the intention of the emotion) and rea-sons a proper service that matches the user’s emotions andother context. The services help the user to perform her/hiseveryday activities, but also train the user to perceive, assessand manage her/his and others’ emotions, that is, to mediateher/his emotional intelligence (i.e., skills to perceive, assessand manage the emotions of one’s self and of others).

Nevertheless, even if, on the one hand, these frame-works reinforce the validity of the innovative emotional-aware approach characterizing our proposal, on the otherhand, it is important to note that they present some limitations

123

SOCA (2011) 5:17–35 19

with respect to our framework. In particular, different fromthe aforesaid systems, our proposal considers emotions ina time-depending way, and so our framework gains moreadaptivity and customization.

As will be described, in order to obtain this time-depending behaviour, a novel cognitive inference engine willbe exploited to directly manage emotional, environmentaland also temporal issues. With regard to computational intel-ligence techniques employing in ambient intelligence, ourapproach is founded in [12] where author asserts: “to man-age the multitude of computational devices and artifacts inpervasive computing spaces, we need a transparent and dis-tributed layer of intelligence....” and according to him, ideas,fuzzy systems and neural networks address this need.

For this reason, our approach exploits computational intel-ligence techniques such as FCMs in the same way as in [23],authors describe a multi-agent framework where agents con-trol sub-parts of the environment using fuzzy rules that linksensors and effectors. In detail, a novel unsupervised onlinereal-time learning algorithm that constructs a fuzzy rulebasederived from very sparse data in a non-stationary environ-ment is implemented. An other example is in [9], where anovel lifelong learning approach for intelligent agents thatare embedded in intelligent environments is described. Theagents aim to realize the vision of Ambient Intelligence inIntelligent Inhabited Environments (IIE) by providing ubiq-uitous computing intelligence in the environment support-ing the activities of the user. An unsupervised, data-driven,fuzzy technique is proposed for extracting fuzzy member-ship functions and rules that represent the user’s partic-ularized behaviours in the environment. The user’s learntbehaviours can then be adapted online in a lifelong modeto satisfy the different user and system objectives. Authorsperformed experiments during a stay of five consecutivedays in the intelligent Dormitory (iDorm). [21] is yet another example. In this paper, authors employ neural net-works in pervasive living spaces to identify patterns of behav-iour and alterations in those patterns to better support caresystems. In particular, this paper presents a system usinga temporal neural-network-driven embedded agent workingwith online, real-time data from a network of unobtrusivelow-level sensors situated in either a simulated environ-ment or a fully fitted real environment such as a wholeflat.

Summarizing, our work tries to improve the current stateof the art of Ambient Intelligence by combining advanta-ges of aforementioned experiences through designing of acollection of cognitive agents whose intelligence is basedupon a new computational intelligence inference engine, i.e.Timed Automata-based Fuzzy Cognitive Maps, that, simulta-neously, exploits emotional, environmental and temporal fea-tures to distribute the most suitable and personalized servicescollection in order to enhance users’ comfort.

ToolsData

FCMs

Timed Automata

Service Oriented Architecture

Sensor Network

Service Oriented ArchitectureService Oriented Architecture

Emotions

Environment

Thayer's ModelTime

ToolsData

FCMs

Timed Automata

Service Oriented Architecture

Sensor Network

Service Oriented ArchitectureService Oriented Architecture

Emotions

Environment

Thayer's ModelTime Cognitive Multi-Agent

System

Cognitive Multi-Agent

System

Fig. 1 A three-layered AmI architecture: the cognitive space

3 Distributing emotional services through cognitiveagents: basic concepts

The goal of this work is to investigate a multi-agent system asa complement to a service-oriented approach towards adapt-able and reconfigurable AmI environments where users areempowered by interacting with an environment that is awareof their presence and emotional conditions. More in detail,as mentioned in the introduction section, our idea is to pro-pose an emotion-aware AmI architecture defining intelligentenvironments able to evaluate human emotions and environ-mental features in order to improve people’s comfort throughthe selection and distribution of appropriate emotional ser-vices that can be selected from a collection of heterogenousSOA systems.

In order to achieve this goal, a three-layered architecture,named Cognitive Space, is suggested (see Fig. 1). The maincomponents of the proposed AmI emotion-aware architec-ture are:

– Sensors Network;– Cognitive MAS;– Service Oriented Architectures.

The Sensors Network layer is the framework module deal-ing with the collection of sensors and actuators capable ofcapturing the user’s emotions and surrounding context, andapplying the appropriate environmental changes in orderto satisfy user’s requirements and needs. The appropriatechanges are computed by jointly exploiting the CognitiveMAS and Service-Oriented Architectures layers.

The Cognitive MAS represents the core of proposed sys-tem. It can be considered as a middleware acting betweenthe Sensors Network layer and the Service-Oriented Archi-tectures one. Its main functionalities are distributed througha collection of autonomous intelligent agents (cognitiveagents) that are designed by following the well-knownNorvig and Russell’s definition [22] and perform thefollowing tasks:

123

20 SOCA (2011) 5:17–35

1. Emotional Services Distribution: each agent detectsemotional and environmental conditions through sensorsbelonging to the first layer (Sensors Network layer) andaccording to them selects the opportune service amongthose provided by the third layer (SOA layer);

2. SOA abstraction: the autonomous agents act as a inter-operability middleware between the sensor network andservice-oriented architectures layers;

The Cognitive MAS achieves the above-mentioned goalsby simultaneous exploiting information such as time, emo-tions and environmental features that are analysed by meansof computing tools such as Timed Automata Fuzzy Cogni-tive Maps and Thayer’s Emotional Model (see Fig. 1). Somesamples of cognitive agent populating this layer are: musicagent, lighting agent, etc.

The Service-Oriented Architectures are a collection ofSOA hosting a set of services capable of modifying the usersurrounding environment in order to improve user’s comfort.Some samples of services are: soft music service, hard musicservice, volume service, etc. These services are opportunelyselected by the corresponding agents that populate the MASlayer.

In this scenario, multi-agent systems offer a computingparadigm based on flexible autonomous actions in dynamicenvironments [27], and, as a consequence, they are appar-ently the most suitable choice for implementing our ambientintelligence frameworks. Indeed, some key features of ambi-ent intelligence such as autonomy, distribution, adaptation,pro-activeness, interoperability and responsiveness are verysimilar to the software agents properties [15]. More in details,in a such environment, agents distribute enhanced services bytaking into account the emotional state of the system’s userand the environmental features (temperature, lighting, etc.,)and provide humans with personalized living background. Inthis scenario, agents anticipate human behaviours in order tosatisfy their specific requirements. As a consequence, users’comfort will be improved, and users/system interactions willbecome simple and intuitive.

Different from other proposed approaches, the agentsthat populate our framework offer time-depending actions,and, as a consequence, they may distribute distinct servicesunder the same emotional and environmental conditions.This choice is particularly useful in ambient intelligence sce-narios where the services distribution may depend upon tem-poral information such as the current season or time of day.For instance, under the same conditions, a lighting agent maydecide to activate two different kinds of services collectionsin winter and summer or, at the same way, a music agent mayactivate distinct services that play different musical styleswith different volume levels in the morning and evening.In order to pursue this aim, agents derive their inferencecapabilities from a novel fuzzy cognitive engine that directly

manages emotional and temporal issues strongly character-izing the immersive computing environments. These innova-tive inference capabilities result from the joint exploitationof the Thayer’s two-dimensional emotion model [25] andTimed Automata-based Fuzzy Cognitive Maps: the formerenables agents to capture and model the human emotionalstate by exploiting a well-defined approach; the latter are usedto infer the most suitable services collection starting fromthe analysis of human emotions, environmental features andtemporal concept. In particular, the Timed Automata-basedFuzzy Cognitive Maps are based on two novel concepts: thecognitive era and the cognitive configuration. A cognitiveera represents the interval of time during which an agentshows the same cognitive configuration or, in other words thesame behaviour. When an agent lives a new cognitive era, itexploits a new behaviour, i.e a new cognitive configurationand, consequently, under the same emotional and environ-mental conditions, it acts in a different way. The cognitiveera and cognitive configuration will be formally introducedin Sect. 6.4, Definition 19. Figure 2 shows the interactionsamong all components of our framework, and, in particular,it underlines our approach independence from the specificSOA implementation.

From an implementation point of view, the agents inter-act with cognitive spaces by means of a sensors networkbased on the Lonworks protocol. Lonworks is Echelons pro-prietary network and encompasses a protocol for buildingautomation. There are many commercially available complexdevices, sensors and actuators for this system. The physicalnetwork installed in our framework is the Lonworks TP/FP10network. The control network is interfaced with computersystems through a gateway to the IP network provided byEchelons iLON 100 web server. This allows the states andvalues of sensors and actuators to be read or altered via astandard TCP/IP communications. Figure 3 shows functionalarchitecture of proposed framework highlighting hardwareand software details related to our proposal.

The following section will introduce our proposal ofagents’ inference engine starting from the definition of thesingle components that realize it. The case study section willbe devoted to present the design of an intelligent agent deal-ing with the distribution of music services and a usability testvalidating the quality of proposed approach.

4 Designing cognitive spaces: emotional and temporalissues

Cognitive spaces are the basic blocks composing our ambi-ent intelligence framework. In order to formally definea cognitive space, it is necessary to individuate a modelfor representing human emotions and an inference engine

123

SOCA (2011) 5:17–35 21

Fig. 2 Cognitive space:user-agents-services interaction

Cognitive Space

Services

Agents

Environment

Emotions

Am

bien

t Int

ellig

ence

Env

ironm

ent

Protocols

music agent

Cognitive Era

Cognitive Con guration

volume service hard music

soft music

users

Sen

sors

Net

wor

k

SOAP

XML-RPC REST

Lega

cy A

gent

Pro

toco

l

Timed Automata based FCMs

Fig. 3 A multi-agentarchitecture for intelligentservices selection

Services Agency

Cognitive Space 1Service Agents

Lighting Agent

HVAC Agent

Music Agent

...

Cognitive Space KService Agents

HVAC Agent

Music Agent

...

Devices MiddlewareLayer

Echelon Lonworks LNS Server

Gateway HardwareLayer

iLon 100 Gateway Server

Ambient DevicesLighting Device

HVAC Device

Music Device

Timed Automata Based Fuzzy Cognitive Reasoning

User Emotions

...

Software Side

Hardware Side

Lighting Agent

User Emotions

123

22 SOCA (2011) 5:17–35

empowering agents with enhanced reasoning capabilities forselecting services in adaptive way.

Our approach uses a modified version of the Thayer’s two-dimensional emotion model together with an extension ofFuzzy Cognitive Maps that, as will be shown hereafter, willenable intelligent agents to select the emotional services thatmeet user’s needs and requirements in a non-deterministicand emotional-based way.

Fuzzy Cognitive Maps (FCM) could be regarded as a com-bination of Fuzzy Logic and Neural Networks. From a graph-ical point of view, an FCM seems to be a signed directed graphwith feedback, consisting of nodes and weighted arcs. Nodesof the graph stand for the concepts that are used to describethe behaviour of the system, and they are connected by signedand weighted arcs representing the causal relationships thatexist between the concepts. Each concept represents a charac-teristic of the system; in general, it stands for events, actions,goals, values and trends of the system that is modelled as anFCM.

In our application, each agent is responsible to distrib-ute services related to a particular environmental aspect byusing an extension of Fuzzy Cognitive Maps, named TimedAutomata-based Fuzzy Cognitive Maps, that are capable ofenhancing agents’ capabilities by providing them with a non-deterministic behaviour improving their intelligence level interms of unpredictability of their actions. More precisely, theapproach based on TAFCM provides agents with a dynamicbehaviour that is capable of adapting agents’ decisions todifferent emotional, environmental and temporal situations.TAFCMs enhance the agents’ behaviour by introducing twonovel concepts: the cognitive era and the cognitive config-uration. By exploiting TAFCMs, the cognitive agents aremodelled by using a timed automata whose states depict theagent’s behaviour, by means an opportune FCM, throughouta limited period of time. In other words, TAFCMs allow anagent to split its life in a sequence of age brackets (cognitiveeras) and, in each age backet, to use the most suitable FCM(cognitive configuration) defining the agents’ decisions inthat bracket.

This approach allows cognitive agents to deal with the col-lection of concepts, better characterizing a cognitive space ina given period of time, together with the relationships amongthem (Fig. 4). In particular, each cognitive agent exploits cog-nitive configuration with the following concepts sets: humanemotions, environmental features and services (Fig. 5).

During its life cycle, each agent analyses the conceptsrelated to the human emotions and environmental features inorder to compute the services’ activation value. Successively,the agent may use a threshold approach to activate serviceswhose activation value is greater than the established limit.

Hereafter, some details related to our extension of Thayer’semotional model and Timed Automata-based Fuzzy Cogni-tive Maps will be provided. Successively, the design of a

Emotional Concepts

Environmental Features Concepts

Services Concepts

TimeC

ognitive Inference

Fig. 4 The agent model for emotion-aware Ambient Intelligence

e2

e3

e1

f1 f2 f3

s3

s4

s5

s2

s1

Services Concepts

Em

otio

nal C

once

pts

Environmental Features Concepts

...

...

...

Fig. 5 A generic cognitive configuration for services selection throughemotional and environmental features analysis

cognitive agent will be provided together with experimentsand usability tests.

5 Thayer’s emotional model: a fuzzy representation

As aforementioned, one of the main goals of this research is tocreate a framework for emotion-aware ambient intelligenceby designing a multi-agent system that distributes advancedservices by detecting the users emotions from living envi-ronment. In order to introduce and deal with emotionalconcepts in our service-oriented framework, the Thayer’stwo-dimensional emotion model is used.

In the last years, a strong interest in emotion study hasbeen manifested by biological, behavioural and social sci-entists. The developed theories are different among themabove all for the different biological and cognitive factorsconsidered to obtain the emotion classification. In partic-ular, Thayer’s model is based on mood consideration as abiopsychological concept. More in detail, Thayer considersmood as an affective condition which is integrally related to

123

SOCA (2011) 5:17–35 23

Valence

Aro

usal

Fig. 6 Thayer’s emotion model

psychophysiological and biochemical components, althoughcognitive processes and external life events have an impor-tant role in its immediate understanding. Thayer sees moodas the interaction of two components: the first one (energeticarousal) is recognizable by subjective sensations of energy,vigour or peppiness, whereas the second one (tense arousal)is associated with feelings of tension, anxiety or fearful-ness. These two central mood systems of energetic and tensearousal, respectively, predispose the individual towards phys-ical activity (calm-energy ranging from energetic to tired) andversus a vigilant preparation for action with the inhibition orrestraint of actual activity (tense-energy ranging from tenseto calm) [7]. In general, calm-energy states can be consid-ered positive whereas tense-energy ones negative in valence.Therefore, as shown in Fig. 6, Thayer’s approach considers atwo-dimensional emotion plane divided into four quadrantswhere the two main dimensions are arousal and valence.Several emotion adjectives are placed over four quadrants.In the model, the amount of arousal and valence is measuredalong the vertical and horizontal axis, respectively. As a con-sequence, the Thayer’s model computes the user emotionalstate (i.e. mood) by taking into account above said dimen-sions.

However, in order to use the Thayer’s two-dimensionalemotional model in our service-oriented approach, it is nec-essary to represent it in a fuzzy way. In particular, in ourscenario, an emotion e is individuated by means of a triple(a, v, l) where a is the arousal level of e, v is the valencelevel of e and, finally, l ∈ [0, 1] represents the so-called acti-vation level of the emotion e. Some examples of emotionsmodelled through the extended Thayer’s model are:

very sad → (−0.6,−0.3, 0.9)

anger → (−0.2,−0.7, 0.5)

happy → (0.5, 0.8, 0.5)

a bit sad → (−0.6,−0.3, 0.2)

The emotions that will be simulated in our system are thoseidentified in the Thayer’s model: agitated, bored, happy,pleasant, still, tired, serene, relaxed, sad, anger, etc. In par-ticular, the value l will be used as input value for the cognitiveconcept related to the emotion e individuated by the a and v

values.

6 Timed Automata-based Fuzzy Cognitive Maps

This section is devoted to introduce the main componentof proposed framework: the Timed Automata-based FuzzyCognitive Maps. TAFCMs are an inference engine used bycognitive agents in order to take appropriate decisions interms of emotion-aware services distribution. In particu-lar, TAFCMs allow an agent to live its life as a biologicalentity characterized by different age brackets where the agentshows the most opportune behaviour to solve the issue forwhich it is designed. In our ambient intelligence scenario,this approach enables agents to show a non-deterministicbehaviour that improves the agents’ intelligence perceptionin terms of unpredictability of the accomplished actions. Byusing this approach, each agent of the proposed system ismodelled through a finite-state machine capable of throwingtime-dependent state transitions which allows the agent todynamically modify its behaviour.

Hereafter, before presenting TAFCMs, a short descriptionof standard Fuzzy Cognitive Maps and Timed Automata isprovided.

6.1 Fuzzy Cognitive Maps

A FCM is a fuzzy signed-oriented graph with feedback thatmodels systems by means of a collection of concepts andcausal relations among concepts. In detail, concepts are rep-resented by nodes in a directed graph, and the graph’s edgesrepresent the casual influences between the concepts. Thevalue of a node reflects the degree to which the conceptis active in the system at a particular time. In general, aFCM functions like a combined approach between associa-tive neural networks [18] and fuzzy systems. All the val-ues in the graph are fuzzy, i.e, concepts take values in therange between [0, 1], and the arcs weights are in the interval[−1, 1]. Between concepts, there are three possible types ofcausal relationships that express the type of influence fromone concept to the others. The weights of the arcs betweenconcept Ci and concept C j could be positive Wi j > 0 whichmeans that an increase in the value of concept Ci leads tothe increase of the value of concept C j , and a decrease inthe value of concept Ci leads to the decrease of the value of

123

24 SOCA (2011) 5:17–35

concept C j . Or there is negative causality Wi j < 0 whichmeans that an increase in the value of concept Ci leads to thedecrease of the value of concept C j and vice versa. Beyondthe graphical representation of the FCM, there is its math-ematical model. It consists of an 1 × n state vector A thatincludes the values of the n concepts and an n × n weightmatrix W which gathers the weights Wi j of the interconnec-tions between the n concepts of the FCM. The matrix Whas n rows and n columns where n equals the total numberof distinct concepts of the FCM and the matrix diagonal iszero since it is assumed that no concept causes itself. Thevalue of each one concept is influenced by the values of theconnected concepts with the appropriate weights and by itsprevious value. So, the value Ai for each concept Ci is com-puted by the following rule:

Ai = f

⎛⎜⎜⎝

n∑j=1j �=i

A j Wi j + Aoldi

⎞⎟⎟⎠ (1)

where Ai is the activation level of concept at time t + 1, A j

is the activation level of the concept C j at time t, Aoldi is the

activation level of concept Ci at time t (it is clear that thevariable t just represents the iteration number between twosuccessive FCM matrix computation), and W ji is the weightof the interconnection between C j and Ci , and f is a thresh-old function, i.e, a function used to reduce unbounded inputsto a strict range. This threshold mapping is a variation offuzzification process in fuzzy logic. The threshold functionsused in the calculation of FCM are, mainly, of two differentkinds:

f (x) =⎧⎨⎩

x if 0 ≤ x ≤ 10 if x < 01 if x > 1

(2)

and

f (x) = 1

1 + e−λx(3)

However, in our proposal, in order to provide agents witha more dynamic behaviour, the cognitive era notion is used.By using the cognitive era, each agent may modify the rela-tionships among its concepts (the cognitive configuration) ina time-dependent way, and, as a consequence, it may dis-tribute different services in different time under the sameemotional and environmental conditions. In order to embedthe cognitive era idea into the FCMs context, it is necessaryto introduce a temporal abstraction capable to deal with thecognitive eras in a direct and formal way. This novel conceptis named T-Time. T-Time can be viewed as a mechanism capa-ble of modifying the FCM structure (in terms of concepts andrelationships among concepts) by moving the system from acognitive era to the successive one.

In order to extend FCMs with the cognitive era and T-Timenotions, the Timed Automata are introduced.

6.2 Timed Automata

A timed automaton is a standard finite-state automatonextended with a finite collection of real-valued clocks provid-ing a straightforward way to represent time-related events,whereas automata-based approaches cannot offer this fea-ture. The transitions of a timed automaton are labelled with aguard (a condition on clocks), an action or symbol on alpha-bet �, and a clock reset (a subset of clocks to be reset). Intu-itively, a timed automaton starts execution with all clocksset to zero. Clocks increase uniformly with time while theautomaton is within a node. A transition can be taken if theclocks fulfil the guard. By taking the transition, all clocks inthe clock reset will be set to zero, while the remaining keeptheir values. Thus, transitions occur instantaneously. Seman-tically, a state of an automaton is a pair of a control node and aclock assignment, i.e. the current setting of the clocks. Tran-sitions in the semantic interpretation are either labelled withan action (if it is an instantaneous switch from the currentnode to another) or a positive real number i.e. a time delay(if the automaton stays within a node letting time pass).

The set of behaviours expressed by an agent modelled bymeans of a timed automaton is defined by a timed language,i.e., a collection of timed words. Both timed concepts aredefined in the following.

Definition 1 A time sequence τ = τ1 τ2 . . . is an infinitesequence of times values τi ∈ R with τi > 0, satisfying thefollowing constraints:

1. Monotonicity: τ increases strictly monotonically; that is,τi < τi+1 for all i ≥ i + 1.

2. Progress: For every t ∈ R, there is some i ≥ 1 such thatτi ≥ t .

Then, a timed word on alphabet � is a pair (σ, τ ) whereσ = σ1σ2 . . . is an infinite word over � and τ is a timesequence. A timed language over � is a set of timed wordson �.

Definition 2 Let X a finite collection of real-valued vari-ables named clocks then the set �(X) of clock constraints δ

is defined inductively by:

δ := x ≤ c|c ≤ x |¬δ|δ1 ∧ δ2

where x is a clock in X and c is a constant in Q, the set ofnonnegative rationals.

A clock interpretation ν for the set X of clocks assigns areal value to each clock; that is, it is a mapping from X toR. A clock interpretation ν for X satisfies a clock constraint

123

SOCA (2011) 5:17–35 25

δ over X if and only if δ evaluates to true using the valuesgiven by ν.

Now, a precise definition of timed transition table, whichdefine the timed automaton behaviour, is given:

Definition 3 A timed transition table A is a tuple 〈�, S, S0,

C, E〉, where:

– � is a finite alphabet,– S is a finite set of states,– S0 ⊆ S is a set of start states,– C is finite set of clocks, and– E ⊆ S × S × � × 2C × �(C) is the collection of transi-

tions. An edge 〈s, s′, a, λ, δ〉 represents a transition fromstate s to state s′ on input symbol a. The set λ ⊆ Crepresents the collection of clocks to be reset with thistransition, and δ is a clock constraint over C .

If (σ, τ ) is a timed word viewed as an input to an autom-aton, it presents the symbol σi at time τi . If each symbol σi

is interpreted to denote an event occurrence, then the corre-sponding component τi is interpreted as the time of occur-rence of σi . Given a timed word (σ, τ ), the timed transitiontable A starts in one of its start states at time 0 with all clocksinitialized to 0. As time advances, the values of all clockschange, reflecting the elapsed time. At time τi ,A state froms to s′ using some transition of the form 〈s, s′, σi , λ, δ〉 read-ing the input σi , if the current values of clocks satisfy δ. Withthis transition, the clocks in δ are reset to 0 and thus startcontinuing time with respect to the time of occurrence ofthis transition. Formally, this timed behaviour is captured byintroducing runs of timed transition tables.

Definition 4 A run r , denoted by (s̄, v̄), of a timed transitiontable 〈�, S, S0, C, E〉 over a timed word (σ, τ ) is an infinitesequence of the form

r : 〈s0, ν0〉 σ1−→τ1

〈s1, ν1〉 σ2−→τ2

〈s2, ν2〉 σ3−→τ3

· · ·

with si ∈ S and νi ∈ [C → R], for all i ≥ 0, satisfying thefollowing requirements:

– Initiation: s0 ∈ S0 and ν0(x) = 0 for all x ∈ C .– Consecution: for all i ≥ 1, there is an edge in E of the

form 〈si−1, si , σi , λi , δi 〉 such that (νi−1 + τi − τi−1) sat-isfies δi and νi equals [λi �→ 0](νi−1 + τi − τi−1).

The timed transition table together with the run conceptare the main notions used in our approach to embed dyna-mism in the standard FCM definition.

6.3 Merging timed automata and Fuzzy Cognitive Maps tomodel cognitive agents’ intelligence

This section is devoted to introduce a novel intelligent agents’inference engine that tries to improve agents’ capabilitiesby merging, in the same methodology, concepts comingfrom computational intelligence area together with systemdynamics. TAFCMs add temporal concepts to standard FCMby exploiting a timed automaton whose possible behavioursdefine all the potential sequences of cognitive eras (and therelated cognitive configurations) that the system could crossduring its life cycle. In particular, TAFCMs improve FCMsby associating each state in a timed automaton with a cog-nitive configuration that describes the behaviour of a systemin a time interval. Therefore, TAFCMs are able to modeldynamic changes in cognitive representation of system and,consequently, perform a more realistic and coherent cogni-tive computation. A TAFCM, as will be formally defined atthe end of this section, is a couple of two components: a timedautomaton that describes the dynamic evolution of a systemand a FCM modelling the cognitive behaviour of system dur-ing first phase of its existence. Once that the automaton com-putation starts over a given timed word, the state transitionswill opportunely modify the initial FCM in order to modelthe system in a better and time-dependent way. These resultswill be achieved by:

– introducing a novel definition of FCM based on the graphstheory;

– modifying the Timed Automata definition introducing theconcepts of cognitive edges, timed cognitive transitiontable, cognitive evolution and cognitive run;

– showing as these definitions are capable of moving a sys-tem among several cognitive eras modifying its behaviourduring the time.

6.4 Timed Automata-based Fuzzy Cognitive Maps

The first step towards the definition of TAFCMs is the rede-fining of the standard FCMs by means of the graph theory.From this point of view, an FCM F can be defined as follows:

F = (V, E)

V = {ci |i > 0}a : V → [0, 1]E = {(ci , c j )|ci , c j ∈ V }w : E → [−1, 1]

(4)

where V is the collection of cognitive concepts; a is a func-tion associating a concept in V with a real activation value in[0, 1]; E is the set of causal relationships between concepts;w is a function associating an edge in E with a real weightin [−1, 1].

123

26 SOCA (2011) 5:17–35

The formal graph view of a FCM only represents a staticvision of our cognitive system. The successive step is to intro-duce a collection of operators able to transform the cogni-tive structure defined in (4). These operators represent thefundamental operations on which constructing the proposedcognitive/dynamic model. They will change the cognitiveconfiguration of a given agent’s FCM, F = (V, E) , by fol-lowing the rules:

– To add concepts;– To add causal relationships;– To remove concepts;– To remove causal relationships;– To magnify/reduce the strength of a causal relationships;– To magnify/reduce the level of system concept.

Definition 5 (Adding a novel cognitive concept - ⊕) Thisoperator modifies in a direct way the set V containing thefuzzy cognitive concepts. Let ct be the concept to add to Vset and let {ck, cl , cm, . . .} ⊆ V be the concepts set influ-encing by ct and let {ch, ci , c j , . . .} ⊆ V be the conceptsset influenced by ck . Then, the behaviour of ⊕ is defined asfollows:

V ′C = VC ∪ {ct }

E ′ = E ∪ ({ct } × {ck, cl , cm, . . .})E ′′ = E ′ ∪ ({ch, ci , c j , . . .} × {ct })

(5)

where F ′ = (V ′C , E ′′) is the FCM resulting from ⊕ applica-

tion.

Definition 6 (Removing a cognitive concept - �) The oper-ator � removes a cognitive concept, named ct , from the con-cepts set V . Let {ck, cl , cm, . . .} ⊆ V be the concepts setinfluenced by ct and let {ch, ci , c j , . . .} ⊆ V the collectionof concepts acting on ct . Then:

V ′C = V ′

C − {ct }E ′ = E − ({ct } × {ck, cl , cm, . . .})E ′′ = E ′ − ({ch, ci , c j , . . .} × {ct })

(6)

where F ′ = (V ′C , E ′′) is the FCM resulting from � applica-

tion.

In our scenario, the two operators of adding and removinga cognitive concept can be exploited to add or remove, forinstance, an emotional service in order to modify the currentcognitive configuration and, as a consequence, to adapt theagents’ behaviour to new user’s requirements and needs.

Definition 7 (Additive modification of cognitive concept - †)This operator modifies the value of a concept ct value in addi-tive way. Let r ∈ R be a real number, then, if r ≥ 0:

V ′C =

{V − {ct } ∪ {(ct + r)} if (ct + r) ≤ 1V − {ct } ∪ {1} if (ct + r) > 1

(7)

or, if r < 0:

V ′C =

{V − {ct } ∪ {(ct + r)} if (ct + r) ≥ 0V − {ct } ∪ {0} if (ct + r) ≤ 0

(8)

and F ′ = (V ′C , E) is the FCM resulting from † application.

Definition 8 (Multiplicative modification of cognitiveconcept- ‡)

This operator changes the value of a concept ct valuethrough a multiplicative constant. Let r ∈ R

+ be a real num-ber, then,

V ′C =

{V − {ct } ∪ {(ct · r)} if 0 ≤ (ct · r) ≤ 1V − {ct } ∪ {0} if (ct · r) > 1

(9)

and F ′ = (V ′C , E) is the FCM resulting from ‡ application.

Definition 9 (Adding a novel cognitive causal relation-ship - �) This operator is used to add a novel causal rela-tionship to a FCM F . Let (ci , c j ) be causal relationship withci , c j ∈ V and (ci , c j ) �∈ E and let wi j ∈ [−1, 1] be theweight of relationship (ci , c j ), then:

E ′ = E ∪ {(ci , c j )}w′ : E ′ → [−1, 1] with w′/E = w and w′((ci , c j )) = wi j

(10)

F ′ = (VC , E ′) and are, respectively, the FCM and weightfunction resulting from the application of � operator.

Definition 10 (Removing a cognitive causal relationship-�)This operator removes a given causal relationship from thecollection of causal relationships E . Let (ci , c j ) be the rela-tionship to remove , with ci , c j ∈ V, (ci , c j ) ∈ E andw((ci , c j )) = wi j ∈ [−1, 1], then:

E ′ = E − {(ci , c j )}V ′ = V − {ci , c j }w′ : E ′ → [−1, 1]

(11)

where the behaviour of function named w′ is the same of w

because E ′ ⊂ E . Then, F ′ = (V, E ′) is the FCM resultingfrom � application, and w′ is the weight function of F ′.

In our scenario, the two operators of adding and removinga novel cognitive causal relationship can be applied to addor remove a relationship which associates an emotional orenvironmental concept with emotional service concepts inorder to change agents’ behaviour in a more adequate onewith respect to living user context.

Definition 11 (Additive modification of causal relationship-�) The operator named � modifies the value of a givencausal relationship by adding it to a real value α. This task isaccomplished by opportunely redefining the weight assign-ment function w. Let (ci , c j ) be the causal relationship to

123

SOCA (2011) 5:17–35 27

modify in additive way, then w′ : E → [−1, 1] is the modi-fied function defined as follows:

w′((cl , cm))

=

⎧⎪⎪⎨⎪⎪⎩

w((cl , cm)) if (cl , cm) �= (ci , c j )

w((cl , cm))+α if (cl , cm)=(ci , c j ) and −1 ≤ w((ci , c j ))+α≤1−1 if (cl , cm)=(ci , c j ) and w((cl , cm))+α <−11 if (cl , cm)=(ci , c j ) and w((cl , cm))+α > 1

(12)

Then, F ′ = (V, E) is the obtained cognitive map and w′ isits weight function.

Definition 12 (Multiplicative modification of causal relationship-�) This operator changes the value of a given causal rela-tionship by multiplying it for a real value β. Analogouslyto the previous operator, this task is performed by redefin-ing the weight assignment function w. Let (ci , c j ) be thecausal relationship to modify in multiplicative way, thenw′ : E → [−1, 1] is the modified function defined asfollows:

w′((cl , cm))

=

⎧⎪⎪⎨⎪⎪⎩

w((cl ,cm )) if (cl , cm) �= (ci , c j )

w((cl , cm)) · β if (cl , cm)=(ci , c j ) and −1 ≤ w((ci , c j )) · β ≤ 1−1 if (cl , cm)=(ci , c j ) and w((cl , cm)) · β <−11 if (cl , cm)=(ci , c j ) and w((cl , cm)) · β > 1

(13)

Then, F ′ = (V, E) is the obtained cognitive map and w′ isits weight function.

In our approach, the two operators of additive and mul-tiplicative modification of causal can be used to magnify orreduce the strength of a casual relationship between an emo-tional or environmental concept and an emotional serviceconcept in order to support, for instance, environmental con-ditions changing in the time.

Definition 13 (Cognitive identity �) The operator � repre-sents a simple function identity that transforms a FCM F initself.

This last operator does not have practical usefulness but itsdefinition it is necessary to introduce the definition of TAF-CM in a more simple way.

Once defined the operators changing FCM configuration,it is possible to define the cognitive operator set:

Cop = {⊕,�, †, ‡,�,�,�,�,�,�}The Cop allows us to redefine the Timed Automata concept

in order to introduce a novel kind of transition edges capa-ble of changing the cognitive configuration of the modelledsystem. In particular, the standard transitions set of timedautomata is replaced with the following edges set:

EC ⊆ S × S × � × 2C × �(C) × Cop (14)

and, the novel definition of timed automata based on cogni-tive edges idea is:

Definition 14 A timed cognitive transition table At is a tuple〈�, S, S0, C, EC 〉, where:

– � is a finite alphabet,– S is a finite set of states,– S0 ⊆ S is a set of start states,– C is finite set of clocks– EC ⊆ S × S × � × 2C × �(C) × Cop is the collection

of cognitive transitions. An edge 〈s, s′, a, λ, δ, ◦〉, with◦ ∈ Cop produces the same effect of a standard transition〈s, s′, a, λ, δ〉, but it individuates the task defined by theoperator ◦ ∈ Cop. The set λ ⊆ C represents the collectionof clocks to be reset with this transition, and δ is a clockconstraint over C .

At this point, it is possible to give a formal definitionof a TAFCM and the properties characterizing its dynamicbehaviour.

Definition 15 A TAFCM T A is an ordered pair composedby an initial cognitive configuration named F0 together witha timed cognitive transition table TM which represents themathematical entity acting as melting point between cogni-tivism and dynamism in system modelling. Formally:

TA = (F0, TM ) (15)

The TAFCM properties that define the dynamic behaviourof an agent are: cognitive evolution and cognitive run.

The cognitive evolution is a mapping among the statesS contained in TM and the collection of possible cognitiveconfigurations obtained starting from F0. More in detail, thecognitive evolution is a mathematical succession, generatedin an inductive way, which maps each state in S with a oneor more cognitive configurations obtained by sequentiallyapplying over F0 the cognitive operators in S × S × � ×2C × �(C) × Cop. Then:

Definition 16 (Cognitive evolution) Let TA = (F0, TM )

be a TAFCM defined over a timed cognitive transitiontable 〈�, S, S0, C, EC 〉 with S = {s0, s1, . . . , s|S|−1} thefinite set of automaton states; let F∗ be the collection ofall FCMs defined by means of expression (4) over a set{cl ∈ R : l = 1 . . . k} of initial cognitive concepts; let� = {◦1, ◦2, . . . , ◦|�|} be a subset of operators in Cop usedto define the edges in EC . Then, the cognitive evolution �

over a state s0 ∈ S0 is:

� : N → S × F∗

defined inductively, as follows:The base case (i = 0). Let s0 ∈ S0 be an initial state of〈�, S, S0, C, EC 〉, then:

�(0) = (s0, F0)

123

28 SOCA (2011) 5:17–35

The inductive step. Let �(i − 1), with i > 1, be the cog-nitive pair defined as:

�(i − 1) = (si−1, Fi−1)

where si−1 ∈ S and Fi−1 ∈ F∗, then:

�(i) = (si , Fi )

with si ∈ S, Fi =◦i (Fi−1), ◦i ∈ � and 〈si−1, si , a, λ, δ, ◦i 〉∈ EC .

The image of function �, I� , can be finite or infinite.This depends upon the topology of graph modelling thecomponent TM of the TAFCM. Indeed, if the topology ofTM contains cycles, then the edge 〈si−1, si , a, λ, δ, ◦i 〉 ∈EC can be crossed more times and, consequently, variousFCM can be associated with a same state sh ∈ S. i.e.,sh ≡ st ≡ st+c1 ≡ st+c2 ≡ . . . ≡ st+ck ≡ . . . and{(st+c1, Ft+c1), (st+c2 , Ft+c2), . . . , (st+ck , Ft+ck ), . . .}⊆I� .

More intuitively, the expression (17) shows the sequenceof pairs composing a cognitive evolution over s0 ∈ S0

together with the fuzzy cognitive transformations obtainedby exploiting the ◦i operators.

�(0) : s0 ∈ S0 → F0

↓ ◦0

�(1) : s1 ∈ S → F1

↓ ◦1

�(2) : s2 ∈ S → F2

↓ ◦2...

...

↓ ◦ j−1

�( j) : s j ∈ S → F j

↓ ◦ j...

...

(16)

Obviously, the cognitive evolution only represents a map-ping between the states of timed automaton TA and the collec-tion of cognitive configurations computable starting from F0

by applying different sequence of operators in �; no dynamicaspects are considered in the cognitive evolution definition,and therefore, we introduce the idea of cognitive run extend-ing the initial idea of the run of standard timed transitiontable.

Definition 17 Let � be a cognitive evolution, then a cog-nitive run rc, denoted by (s̄, ν̄), of a timed transition table〈�, S, S0, C, EC 〉 over a timed word (σ, τ ) and a collectionof cognitive operators � ⊆ Cop, is an infinite sequence ofthe form

rc : 〈s0, ν0〉 σ1,◦1−−−→τ1

〈s1, ν1〉 σ2,◦2−−−→τ2

〈s2, ν2〉 σ3,◦3−−−→τ3

· · ·

with si ∈ S, νi ∈ [C → R], for all i ≥ 0, and ◦i ∈ Cop, forall i ≥ 1, satisfying the following requirements:

– Initiation: s0 ∈ S0 and ν0(x) = 0 for all x ∈ C .– Consecution: for all i ≥ 1, there is an edge in E of the

form 〈si−1, si , σi , λi , δi 〉 such that (νi−1 +τi −τi−1) sat-isfies δi and νi equals [λi �→ 0](νi−1 + τi − τi−1).

– Atomicity: The operators ◦i ∈ Cop are atomic operationsand their computation time is equals to 0, i.e., they donot modify the duration of permanence in the automatonstate si , (τi − τi−1).

– Evolution: each state si of a pair 〈si , νi 〉 in rc is mappedon a FCM Fi as described by the cognitive evolution �.

If TA = (F0, TM ) is a TAFCM that models a given sys-tem, then the set of cognitive run rc defined over the timedlanguage L , generated by TM , completely describes the col-lection dynamic behaviours of the system, whereas the cog-nitive run rc defined over a single word wi ∈ L defines aprecise dynamic behaviour of the system, i.e., wi defines theT-Time.

Definition 18 (T-Time) If TA = (F0, TM ) is a TAFCM andTM is a timed automaton recognizing the timed languageL = {w1, w2, w3, . . . , wi , . . .} and wi is a timed word andrc is a cognitive run defined over wi , then wi is a T-Time ofthe system.

Starting from the T-Time definition, a formal descriptionof cognitive era and cognitive configuration is given.

Definition 19 (Cognitive era and cognitive configuration) Ifrc is a cognitive run defined over the T-Time wi = (σ, τ ) ∈L:

rc : 〈s0, ν0〉 σ1,◦1−−−→τ1

〈s1, ν1〉 σ2,◦2−−−→τ2

〈s2, ν2〉 σ3,◦3−−−→τ3

· · ·then time interval between the instant τi and τi+1 is the i thcognitive era of system and the FCM Fi that depicts the sys-tem during the same interval is defined as the i th cognitiveconfiguration.

Both the cognitive evolution and the cognitive run arepotentially based on the infinite concept. In fact, the cogni-tive evolution can exploit an infinite application of cogni-tive operators in � to compute the mappings between thestate si and the FCM Fi , whereas the cognitive run uses atimed word, defined as a infinite sequence of ordered pairs,to describe the cognitive/dynamic behaviour of the system.Consequently, in order to simulate the behaviour of a TAF-CM during the first n cognitive eras, the nth-order cognitiveevolution and cognitive run are introduced.

Definition 20 (nth-order cognitive evolution) If � is a cog-nitive evolution, then the set

�n = {�(i) = (si , Fi ) : i = 0 . . . n − 1}which contains the first n-ordered pairs computed by �

through Definition 16 is the nth-order cognitive evolution.

123

SOCA (2011) 5:17–35 29

Definition 21 (nth-order cognitive run) Let rc be a cognitiverun defined over a cognitive evolution �: then

rc : 〈s0, ν0〉 σ1,◦1−−−→τ1

〈s1, ν1〉 σ2,◦2−−−→τ2

〈s2, ν2〉 σ3,◦3−−−→τ3

· · ·then the nth-order cognitive run rn

c is the sequence of first nelements of rc:

rnc : 〈s0, ν0〉 σ1,◦1−−−→

τ1〈s1, ν1〉 σ2,◦2−−−→

τ2. . .

σn−1,◦n−1−−−−−−→τn−1

〈sn−1, νn−1〉

where the mapping between the automaton states si and theFCM Fi is computed by the nth-order cognitive evolutionrelated to �.

Once that TFCMs have formally defined, a more preciselydefinition of cognitive agent, cognitive space and emotion-aware AmI is provided. In particular, a cognitive agent a isdefined as a TAFCM Ta = (F0, TM ) where the initial cogni-tive maps contain concepts related to emotions, environmen-tal features and services. A cognitive space C Sj , composedp cognitive agents, is defined as:

C Sj = {(F0, TM )1, (F0, TM )2, . . . , (F0, TM )p}As a consequence, an emotion-aware AmI system A, com-posed by q cognitive spaces, is depicted as:

A = {C S1, C S2, . . . , C Sq}

7 Case study and experiments

This section is devoted to validate proposed approach throughthe design of a cognitive agent aimed to the distribution ofmusical services. Moreover, usability and confirmation ofexpectations tests have been defined and performed in orderto quantify the benefits offered by our services in terms ofusers’ comfort and satisfaction. In this case study, the cogni-tive concepts and weights have been opportunely chosen byusing expert knowledge.

7.1 A case study: the music agent

In order to better explicate and validate our proposal, thissection shows the behaviour of an agent, the so-called musicagent, that is responsible for computing the music service’sactivation value by considering some of emotional and envi-ronmental concepts. As previously described, the behaviourof each agent can be defined by means of a ordered pairT = (TM , F0). In particular, the music agent exploits a FCMF0 characterized by the following features:

– seven emotional concepts representing seven moods (sad,angry, nervous, happy, bored, relaxed, sleepy) chosenamong emotional states of the Thayer two-dimensionalemotion model;

E1

E2

E3

E4

E5

E6

E7

I1 I2

S1

S3

S2

S4

0.3

0.40.6

0.4

0.50.8

0.7

0.3

-0.9

-0.6 0.5

0.80.9

0.70.2

0.70.7

Fig. 7 The Fuzzy cognitive map F0 related to the music agent

– two environmental concepts representing the outdoortemperature and luminosity (outdoor temperature andoutdoor light);

– four musical service concepts where a service managesthe music volume (volume service), whereas the remain-ing ones manage the music style to play (soft music, hardmusic, mid-music).

The relationships among these concepts represent theinfluence values of the emotional and environmental fea-tures with respect to the services’ activation value. The Fig. 7shows the FCM F0 related to the music agent. The agent eval-uates the environmental features (I1 and I2) and the currentuser’s mood (Ei with i = 1, . . . , 7) and decides which ser-vices (Si with i = 1, . . . , 4) have to be activated in order tosatisfy user’s musical requirements. In particular, the musicagent decides to turn up or down the music volume and selectsthe most opportune music style among available ones.

In order to model the dynamic component of the agent’sbehaviour, our approach uses a timed automaton named TM .Thanks to this component, agents are capable of dynamicallychanging their behaviour by updating their cognitive behav-iour by means of the application of the operators belongingto the set Cop. More in detail, the music agent exploits theautomaton TM in order to adapt the relationships strengthbetween environmental features and services to a periodof year marked by particular weather patterns and daylighthours and to a particular time of day.

123

30 SOCA (2011) 5:17–35

Fig. 8 Music agent timed automaton

Table 1 F0’s Concepts

Concepts Concept labels

Sad mood E1

Angry mood E2

Nervous mood E3

Happy mood E4

Bored mood E5

Relaxed mood E6

Sleepy mood E7

Outside temperature I1

Outside light I2

Volume service S1

Soft music service S2

Hard music service S3

Mid-music service S4

Figure 8 shows the timed automaton TM . Each automa-ton state represents the behaviour of the music agent dur-ing a well-defined time period. In particular, the state set{S11, S12, S13, S14} represents the agent’s behaviour duringthe winter season; the state set {S21, S22, S23, S24} representsthe agent’s behaviour during the spring season; the state set{S31, S32, S33, S34} represents the agent’s behaviour duringthe summer season; the state set {S41, S42, S43, S44} repre-sents the agent’s behaviour during the autumn season. In eachset, the states Si1, Si2, Si3, Si4, with i = 1, . . . , 4, corre-spond to different time of day (morning, afternoon, evening,night). The transitions among the states apply a collection ofcognitive operators that modify the weight of concepts rela-tionships. This behaviour updating will adapt agent’s actionsto user’s needs in different time. (Table 1)

The dynamic music agent behaviour can be representedby means of the two properties of TAFCMs described in theSect. 6,i.e, the cognitive evolution and the cognitive run. Inparticular, considering the automaton TM and the timed word(a, 8), (b, 13), (c, 18), it is possible to define a 4th-order cog-nitive run and a 4th-order cognitive evolution of the automa-ton which specifically individuate the music agent behaviourduring the time of a day in the winter season. In detail, the4th-order cognitive evolution is represented by the followingsequence of pairs composed by an automaton state and a cog-nitive configuration obtained by exploiting the (�, · · · ,�)

operators starting from F0:

�(0) : s0 ≡ s11 → F0

↓ (�, · · · ,�)0

�(1) : s1 ≡ s12 → F1

↓ (�, · · · ,�)1

�(2) : s2 ≡ s13 → F2

↓ (�, · · · ,�)2

�(3) : s3 ≡ s14 → F3

(17)

Figures 7 and 9 show, respectively, the Fuzzy CognitiveMap F0 associated with the initial state (S11) and the cog-nitive map F1 obtained by applying the operators related tothe automaton transition S11 → S12.

Instead, the 4th-order cognitive run is as follows:

rc : 〈s11, [0, 0]〉 a,(�,··· ,�)−−−−−−→8

〈s12, [8, 8]〉b,(�,··· ,�)−−−−−−→

13〈s13, [13, 13]〉 c,(�,··· ,�)−−−−−−→

18〈s14, [18, 18]〉

(18)

where the clock interpretation is represented by [x, y] and xand y are the two clocks of timed automaton TM .

The Fig. 10 shows the agent’s behaviour related to softmusic, hard music and mid-music services in the winter

123

SOCA (2011) 5:17–35 31

E1

E2

E3

E4

E5

E6

E7

I1 I2

S1

S3

S2

S4

0.35

0.450.6

0.45

0.550.8

0.7

0.3

-0.9

-0.6 0.5

0.80.9

0.70.2

0.70.7

Fig. 9 The Fuzzy cognitive map F1 related to the music agent

morning (S11). The agent’s behaviour is evaluated by takinginto account the outdoor light and several user’s moods suchas sad (Fig. 10a), bored (Fig. 10b), angry (Fig. 10c), happy

(Fig. 10d). Each surface shows the activation level of softmusic, hard music and mid-music services. Figure 10 provesthat the music agent is able to provide different behavioursunder distinct emotional and environmental conditions in thesame time period, i.e., a winter morning.

Moreover, the Fig. 11a, b shows the surfaces related to thesoft music service, Fig. 11c, d shows the surfaces related tothe hard music service, and Fig. 11e, f shows the surfacesrelated to the mid-music service. Each figure shows the acti-vation level of the corresponding music service in differenttime of day (S11 ≡ morning, S12 ≡ a f ternoon, S13 ≡evening, S14 ≡ night). Figure 11 proves that the musicagent is able to adapt its behaviour to different time periods,and, in particular, these figures show how the music agentdistributes different services in different time of day underthe same emotional and environmental conditions.

7.2 Usability test

A usability test has been performed in order to evaluate theproposed cognitive approach in terms of user’s satisfaction.This test has been designed by providing a realistic envi-ronment, the Cognitive Assisted Living Testbed, based onthe control network protocol known as Echelon Lonworks.Consisting of a single cognitive space located in a livingroom, the testbed framework provides the ideal environment

Fig. 10 Surfaces related to music service showing music agent’s behaviour in the automaton state S11 (winter morning)

123

32 SOCA (2011) 5:17–35

Fig. 11 Surfaces related to hard music service, soft music service and mid-music service in the four time periods of a day of winter season

to evaluate smart home services like the automatic settingsof lighting, temperature, music, etc., under realistic circum-stances. In particular, the cognitive space is composed bythree different agents collection: light agents, temperatureagents and music agents. At present, the framework is notequipped with sensors able to individuate the user’s emo-tional state. For this reason, the user communicates his moodto the test framework by exploiting a mobile device as shownin Fig. 12. In detail, the mobile device implements a graphical

user interface based on the three-dimensional Thayer’s modelintroduced in the Sect. 5.

The usability test has been performed by inviting 30 users,chosen among researchers and students of University ofSalerno, to live in our Cognitive Assisted Living Testbed. Themain aim of this test is to evaluate agents’ behaviour duringa temporal window in order to verify the dynamical aspectsof agents’ decisions. In particular, this test chooses the dayas the temporal window size and it splits this window in four

123

SOCA (2011) 5:17–35 33

Fig. 12 Graphical interface based on the fuzzy extension of the Tha-yers emotion model

sequential time period named, respectively, morning, after-noon, evening and night. In this way, the user may appreciatethe non-deterministic behaviour of a cognitive agent when itselects services in different time of day. During the usabilitytest, the users interacted with the framework, and they tookadvantage of distributed services. Successively, users ratedservices distribution by means of a predefined questionary,and they proposed new ideas for eventual improvements.

Once a user starts its interactions with the framework andservices collections are distributed, he sets its satisfactionlevel by selecting a vote belonging in the real range [1, 10](the higher the better) for the three following usability aspects(Fig. 13):

– Appropriateness of Response which measures the effec-tiveness of the system’s behaviour in response to user’semotional states (set by user) and environmental features(measured by the system). This aspect is directly relatedto the system’s design and particularly to modelling ofthe TAFCM;

– Realism of System’s Response which measures the lagbetween a user’s request and the corresponding (correct)system’s reply. This value is affected by the automaton’stransition functions (which typically have a negligiblecomputational load);

– System Usefulness which measures the overall useful-ness of the system with regard to the improving the life’squality.

Table 2 shows, for each aforementioned aspect, the mini-mum, maximum and average scores related to the four differ-

Fig. 13 Graphical user interface for the usability test

ent day periods. These results show that proposed approachsatisfies user’s requirements in a more than sufficient way.

7.3 Confirmation of expectations test

In order to strength the validation of our ambient intelligencesystem, a confirmation of expectations test has been accom-plished. Expectation–confirmation theory (ECT) is widelyused in the consumer behaviour literature to study consumersatisfaction, post-purchase behaviour (e.g., repurchase, com-plaining) and service marketing in general [6,26]. The pro-cess by which users reach repurchase intentions in an ECTframework is as follows [20]:

1. Users form an initial expectation of a specific serviceprior to purchase or use it;

2. They accept and use that service and after a period ofinitial consumption, they form perceptions about its per-formance;

3. They assess its perceived performance vis–vis their orig-inal expectation and determine the extent to which theirexpectation is confirmed;

4. They form a satisfaction, or affect, based on their confir-mation level and expectation on which that confirmationwas based.

In our study, the expectation level of test participants ismeasured by means of a seven-point semantic differentialscale [14,19]. The semantic differential scale (SDS) is a scal-ing tool that has been used frequently for measuring socialattitudes, particularly in the fields of linguistics and social

123

34 SOCA (2011) 5:17–35

Table 2 Results of usability testTest parts Appropriateness of response Realism of response System usefulness

Min. Avg. Max. Min. Avg. Max. Min. Avg. Max.

Morning test 5 7.5 9 5 6.8 8 7 7.8 9

Afternoon test 5 8 9 4 7 8 7 8 9

Evening test 5 7.5 8 4 6.7 8 7 8.1 9

Night test 4 7.2 8 5 7.1 8 6 7.5 9

Table 3 Results related to the confirmation of expectations test

Min. Avg. Max.

Confirmation of expectations 2 5.5 7

psychology [5]. In order to construct a seven-point seman-tic differential scale, users required to rate their expecta-tion–confirmation in respect to two adjectives with oppositemeanings. In a more detail, the scale allows users to expresstheir idea about the issue: is the tested framework’s behaviourworse or better than expected one?

In order to apply the seven-point semantic differentialscale in our ambient intelligence scenario, the following pairof opposite adjectives has been chosen: unfulfilled and ful-filled.

Once the users rated the seven-point semantic differentialscale form, the expectations–confirmation test is completedas shown in Table 3. This table shows the minimum, max-imum and average scores for confirmation of expectationsrelated to test participants: a low value means that “partic-ipants’ expectations were too high and so the frameworkis worse than they thought”, vice versa, a high value indi-cates that “participants’ expectations were too low and sothe framework is better than they thought”. An average valueequals to 5.5 shows that results obtained for this testing ses-sion have been positive and encouraging in terms of users’expectations.

Finally, the usability study confirms that for the most partof users, the cognitive living environment improves theirlife’s quality by maximizing the human comfort and fulf-ilment.

8 Conclusions and future works

AmI environments are systems obtained as fusion of threedifferent methodologies: ubiquitous computing, ubiquitousnetworking and intelligent user-friendly interface. Our pro-posal has shown an innovative emotion-aware AmI archi-tecture, based on the integration of methods of distributed

artificial intelligence and cognitive modelling, whose mainaim is to define a kind of intelligent environment evaluat-ing human emotion experiences and providing people withproper emotional services. Emotional services provide theopportune system’s behaviour responding to user’s emo-tion generation [28]. This architecture has been designedby exploiting a multi-agent system (MAS) approach anddefining the so-called cognitive agents, i.e., intelligent agentsthat distribute emotional services and provide people withpersonalized living scenarios by anticipating their behav-iour and satisfy their specific requirements. These innovativecapabilities resulted from the joint exploitation of the Tha-yer’s two-dimensional emotion model and Timed Automata-based Fuzzy Cognitive Maps: the former enables agents tocapture and model the human emotional state by exploit-ing a well-defined approach; the latter are used to infer themost suitable services collection starting from the analysis ofhuman emotions, environmental features and temporal con-cept. As has been shown in experimental results, where ausability study and a confirmation of expectations test havebeen performed, the proposed approach maximizes the sys-tem’s usability in terms of efficiency, accuracy and emotionalresponse.

However, in future, our idea is to improve experimentalenvironment by equipping it with sensors able to individ-uate the user’s emotional state and, as a consequence, byavoiding that the user communicates his mood by meansof a mobile device. In particular, in order to model humaninformation, our idea is to exploit the Human To MarkupLanguage (H2ML) [3,4], i.e a language for modelling aphysical/emotional description of humans by means of acollection of low-level morphological features (as closed oropened eye, voice level, mouth shape, body temperature, etc).This information will be captured by using a collection of sen-sors such as motion sensors, thermo cameras, voice sensors,etc., and successively, they will be coded in H2ML programsthat will be read by our Cognitive MAS in order to capturethe real user status and provide the opportune services collec-tion. This environment’s enrichment will enable to perform amore realistic test case, but, however, our first experimentalresults provide a good validation of proposed approach.

123

SOCA (2011) 5:17–35 35

References

1. Aarts EHL, Eggen B (eds) (2002) Ambient Intelligence in Hom-eLab. Neroc

2. Acampora G, Loia V (2005) Fuzzy control interoperability andscalability for adaptive domotic framework. IEEE Trans Ind Inf1(2):97–111

3. Acampora G, Loia V, Nappi M, Ricciardi S (2005) Hybrid com-putational intelligence for ambient intelligent environments. In:Szczepaniak PS, Kacprzyk J, Niewiadomski A (eds) Advancesin web intelligence, lecture notes in computer science, vol 3528.Springer, Berlin/Heidelberg, pp 26–31

4. Acampora G, Loia V, Nappi M, Ricciardi S (2006) A semanticview for flexible communication models between humans, sensorsand actuators. In: Systems, man and cybernetics, 2006. SMC ’06.IEEE international conference on, vol 5, pp 3640–3646

5. Al-Hindawe J (1996) Considerations when constructing a semanticdifferential scale. La Trobe Papers in Linguistics 9(7)

6. Anderson EW, Sullivan MW (1993) The antecedents and conse-quences of customer satisfaction for firms. Mark Sci 12(2):125–143

7. Buck R (1990) Review: mood and emotion: a comparison of fivecontemporary views. Psychol Inq (4):330–336

8. Ceccaroni L, Verdaguer X (2004) Agent-oriented, multimedia,interactive services in home automation. In: Proceedings of the sec-ond European workshop on multi-agent systems (EUMAS 2004)

9. Doctor F, Hagras H, Callaghan V (2005) An intelligent fuzzy agentapproach for realising ambient intelligence in intelligent inhabitedenvironments. IEEE Trans Syst Man Cybernet Part A Syst Hum35(1):55–65

10. Gaeta M, Capuano N, Gaeta A, Orciuoli F, Pappacena L, RitrovatoP (2004) A service oriented virtual organisation for elearning. In:Learning grid of excellence working group

11. Gal CL, Martin J, Lux A, Crowley JL (2001) Smart office: designof an intelligent environment. IEEE Intelligent Systems 16: 60–66.doi:10.1109/5254.941359

12. Hagras H (2007) Embedding computational intelligence in perva-sive spaces. IEEE Pervasive Comput 6:85–89

13. Kientz JA, Patel SN, Jones B, Price E, Mynatt ED, Abowd GD(2008) The georgia tech aware home. In: CHI ’08: CHI ’08extended abstracts on Human factors in computing systems. ACM,New York, pp 3675–3680, doi:10.1145/1358628.1358911

14. Linda G, Oliver RL (1979) Multiple brand analysis of expectationand disconfirmation effects on satisfaction. In: Proceedings 87thannual convention of the American psychological association

15. Maes P (1994) Agents that reduce work and information overload.Commun ACM 37(7):30–40

16. Magerkurth C, Etter R, Janse M, Kela J, Kocsis O, Ramparany F(2006) An intelligent user service architecture for networked homeenvironments. In: Proceedings of the 2nd international conferenceon intelligent environments, pp 361–370

17. Marreiros G, Santos R, Ramos C, Neves J, Novais P, Machado J,Bulas-Cruz J (2007) Ambient Intelligence in emotion based ubiq-uitous decision making

18. Mateou NH, Andreou AS (2008) A framework for developingintelligent decision support systems using evolutionary Fuzzy Cog-nitive Maps. J Intell Fuzzy Syst 19(2):151–170

19. Oliver RL (1977) Effect of expectation and disconfirmation onpostexposure product evaluations: An alternative interpretation.J Appl Psychol 62:480–486

20. Oliver RL (1980) A cognitive model for the antecedents and con-sequences of satisfaction. J Mark Res 17:460–469

21. Rivera-Illingworth F, Hagras H, Callaghan V (2006) Automateddiscovery of human activities inside pervasive living spaces. In:Pervasive computing and applications, 2006 1st international sym-posium on, pp 77–82

22. Russell SJ, Norvig P (2003) Artificial intelligence: a modernapproach. Pearson Education, NJ

23. Rutishauser U, Joller J, Douglas R (2005) Control and learning ofambience by an intelligent building. IEEE Trans Syst Man Cyber-net Part A Syst Hum 35(1):121–132

24. Spanoudakis N, Moraitis P (2008) An ambient intelligence appli-cation integrating agent and service-oriented technologies. In:Research and Development in Intelligent Systems XXIV, SpringerLondon, chap 9, pp 393–398

25. Thayer RB (1989) The biopsychology of mood and arousal.Oxford University Press, Oxford

26. Tse DK, Wilton PC (1988) Models of consumer satisfaction: anextension. J Mark Res 25:204–212

27. Wooldridge MJ (2000) Reasoning about rational agents. Intelligentrobots and autonomous agents series. The MIT Press, Cambridge

28. Zhou J, Kallio P (2005) Ambient emotion Intelligence: frombusiness-awareness to emotion-awareness. In: Proceeding of 17thinternational conference on systems research, informatics andcybernetics

29. Zhou J, Yu C, Riekki J, Karkkainen E (2007) Ame framework:a model for emotion-aware ambient intelligence. In: The secondinternational conference on affective computing and intelligentinteraction (ACII2007)

123