Ambient intelligence in manufacturing I. Maurtuaa, M.A. Péreza, L. Susperregia, C. Tubioa, A. Ibargurena

a Fundación Tekniker, Av. Otaola 20, 20600 Eibar, Spain Abstract There are several references to projects linked to the AmI concept in technical publications. These are mostly centred on applications linked to everyday situations concerning individuals: at home, in the street, in the car, in public places and relating to leisure. The AMILAB research group at TEKNIKER is analysing the technological, human and social needs and implications for AmI in the manufacturing field. For this aim we have set up a laboratory to spread the model of AmI to the area of manufacturing. This paper presents that Lab, the agent based architecture we have implemented as well as the application of machine learning techniques to identify the activities a worker is doing. Keywords: Ambient Intelligence, context, agents, manufacturing 1. Introduction

Ambient Intelligence (AmI) [1] is a vision of the future of the Information Society that has attracted the attention of numerous research groups. AmI stems from the convergence of three key technologies: Ubiquitous Computing, Ubiquitous Communication, and Intelligent User Friendly Interfaces. All of them collaborate to create an intelligent environment that identify people, get adapted to them and with which interact in a natural way.

Ambient systems need to address some key issues: • Context awareness: have a predictive

behaviour based on knowledge of the environment.

• Natural Interaction: relates, in a natural manner, to the users by means of multi-modal interfaces, movements and gestures, images, etc.

• Adaptation: adaptation to users and context in an autonomous way.

• Integration and ubiquity: technology will

become invisible, embedded in our natural surroundings, present whenever we need it, offering services regardless of where the user is located, of the position from where the user demands the services and the artefacts available at that particular moment.

• New services: produce new services in fields such as entertainment, security, health, housework, the work environment, access to information, computing, communications, etc., to improve the quality of life by creating adequate atmospheres and functions.

There are several references to projects linked to the AmI concept in technical publications. These are mostly centred on applications linked to everyday situations concerning individuals: at home, in the street, in the car, in public places and relating to leisure. The opportunities for its application in the Industrial and Productive environment are frequently cited, although it is mainly the office environment that is mentioned.

The key technologies in the future of

manufacturing identified in two of the most important studies relating to manufacture [2,3] are clearly linked to the vision of AmI: flexible manufacturing and control systems, decision support systems, improved human-machine interfaces, equipment, re-configurable and adaptable processes and systems, distributed computation, etc.

Potentially, the adoption of this vision could affect all stages in the manufacturing process: the design of the plant or the product, engineering, the organisation and management of production, process control, quality control, maintenance, logistics and the management of the product throughout its life cycle, including its re-cycling. The AMILAB research group at TEKNIKER is analysing the technological, human and social needs and implications for AmI in manufacturing. In the following sections we present some of the real industrial applications Tekniker is developing. From the technological point of view we describe the most important achievements: (1) an AmI environment in the manufacturing field, (2) context aware agents able to offer contextual information and to anticipate user needs, and (3) a system that can identify the activities a worker is doing. 2. AmILAB

Tekniker has set up a laboratory to spread the model of AmI to the area of manufacturing. The main research objectives of the laboratory are:

• To support complex tasks with a minimum of human-machine interaction.

• To enable mobile professionals to keep their attention focused on the interaction with the work environment.

• To investigate the user acceptance of wearables, as well as different methods for user interaction.

• To identify processes suited to wearables in industry.

The laboratory – see figure 1– reproduces an industrial environment equipped with machines (high-speed milling machine and 6 axes robot) and various devices and applications:

• Interface technologies: voice recognition systems, head mounted displays and data gloves.

• Location and tracking system based on RFID tags and sensor networks.

• Wearable and portable computing.

• Fingerprint identification system. • Instrumentalised gloves • Traditional applications/functions in a

manufacturing environment, such as monitoring and operating the machine, production follow-up, maintenance. These applications have been re-designed and implemented in the form of Agents on the JADE platform [4], incorporating the concept of intelligent interfaces, context sensitivity and automatic learning

Fig.1. AmILAB Laboratory

3. Agent based architecture

One of the aims is to create applications that make

use of context information to provide users the information they wish, when and how they need it. The definition of “context” refers to any relevant fact in the environment.

To reach this objective a multi-agent system, called AMICO, has been developed. It creates an integrated AmI environment in manufacturing. AMICO is able to support and follow users along the Laboratory, offering them the information needed at anytime in the most suitable device available. The activity has been focused on several user profiles: the machine operator, the maintenance operator and quality manager.

The main functions provided by AMICO are: • To show contextual information taking into

account three criteria: user profile, the most suitable device, and the user location.

• To allow the access to machine functionalities according to the user context.

• To learn and adapt to the user. Figure 2 shows the components of the AMICO

prototype and the communication schema among the agents. To sum up, different agents co-habit in different devices (PDA, Smartphone, Xybernaut, NC,...), these are:

Broker agent: maintains a model of the context for the rest of agents, services and devices and decides which information and to which device should be delivered. This Broker agent acts as a middleware with external applications that are not implemented as agents, such as real time machine information retrieval, centralised voice recognition system, etc. Tracker: is in charge of tracking and reporting the position of users and devices in the laboratory. An RFID system provides geographical information of all users and devices in the laboratory. Mobile Agent: is the “user interface” agent. It offers the user context aware information, i.e. that information need for the user according to his preferences, the activities he is doing or the interfacing devices he can access to. Register: registers all the devices in the agent platform and manages the main characteristics of them. Input Agent: is a data input agent called UbiKey (Ubiquitous Keyboard and Mouse). This mobile Agent can be called anytime by users, the Broker agent decides which keyboard equipped device is near the user and sends Ubikey to it. From that moment on the user can use that device’s keyboard or mouse as if they were physically attached to his wearable computer in order to input data to other Agents running in it.

Identification and Tracking system

SERVER

RS232

RFID Reader

TRACKER

DF Services management

AMS Agents management

Context management User profiles Resources management

BROKER

Tracking

REGISTER

Devices Registry

MOBILE

Interface Adaptation

Legacy Applications Machine Control, Technical Assistance Service, Production Control, Database, etc.

Fig.2. The AMICO agent architecture

NC agent:–see figure 3- allows the different

agents operating on the platform to access certain NC functionalities through the standard mechanisms defined by the FIPA, in other words, through ACL messages. It offers the following services:

• Access to the NC internal variables: spindle speeds, feed, running program being, positions

of the machine axes, etc. • Execution of certain instructions: execution of a

program, sending programs, etc The agent is executed on the Fagor 8070 NC, which offers the functionalities of a PC and which is part of the JADE platform. The NC Agent calls up the different functions of the API supplied by the NC manufacturer to interface with it.

Fig. 3. The NC agent

The Java Agent DEvelopment Framework,

JADE1, is used to develop and run the system. A JADE-LEAP (“JADE powered by LEAP”) version is used which provides a runtime environment for enabling FIPA agents to execute on lightweight devices such as cell phones running Java. This free platform (under LGPL license) supports multi-agent implementation following standards defined by FIPA, Foundation for Intelligent Physical Agents. Furthermore, it offers a reduced version compatible with mobile devices that we use on Symbian (smartphone), besides the versions for Windows (NC and Xybernaut) and Pocket PC (PDA).

JADE includes two agents that facilitate agent management: “Agent Management System” (AMS) and “Directory Facilitator” (DF). The first one provides information of all agents registered in the platform and the DF offers a yellow page service with the services registered in the platform. There is a central server or “main-container” where the Broker, Tracker and Mobile agents are initialised. Besides each active device in the laboratory defines its particular container, where mobile agents might migrate when required.

Communication, coordination and cooperation are 1 http://jade.tilab.com/

key points for a multiagent system. All information exchange among agents is done using FIPA ACL standard and JADE provided mechanisms.

Regarding Ontologies, the ones used for this multi-agent system are the mobility Ontology implemented by JADE and the Device Ontology defined by FIPA.

4. Context detection

The definition of “context” refers to any relevant fact in the environment:

• The user position and related parameters: indoor or outdoor, temperature, humidity, date and time, background noise, user’s physiological status, activity, etc.

• Devices at hand: telephone, PDA, printer, etc. operating system and communication and interaction capabilities, display size, memory, battery status, etc. Accessible networks, pricing and bandwidth.

• User’s preferences: voice versus text messages, favourite devices, font size, ...

• Available services and people.

In short, the context could be any data available when interacting with the system, being specially relevant in a mobility environment.

One of the factors that offer a richer source of information is the activity carried out by a person, which in the case of a productive environment acquires a particular relevance.

Once the activity developed by a worker is known, the possibilities offered by an AmI environment are unlimited: making the interaction with information systems easier, offering the necessary information each time, monitoring the observance of security measures, controlling the observance of the manufacturing procedures, etc. 4.1.Instrumentalised data gloves

There are different alternatives for the characterisation of a person’s activities, from the use of sensors to the inference from the interaction with the systems. However, the alternatives are neither equally valid, nor even acceptable in some situations. One of the research areas we are working on in our laboratory is the identification of the activities a worker develops based on the information given by gloves he wears during his activity.

Using instrumentalised gloves to identify gestures

and facilitate the human interaction with deaf-mute people has been the objective of several researchers. Waleed [5] tried to recognise the 95 signs of the Australian Sign Language (Auslan), the language used by the Australian Deaf community. Fels and Hinton [7] have developed a system that allows controlling a real time speech synthesiser using data gloves and neural networks. More recently, within the wearIT@work2 project, researchers from ETHZ and UMIT have used several in-body sensors to identify activities during the assembly of car’s front-lights in Skoda production facilities [6] 4.2.Methodology

Several Machine Learning paradigms have been used in order to test their behaviour with this specific problem. The three methods selected were: one based on instances such as the K-NN [8], the widely used C4.5 [9] for creating decision trees and finally the probabilistic method known as Naive-Bayes [8].

Genetic Algorithms [10,11] were additionally used for some specific optimisation tasks. These algorithms allow to obtain almost optimal solutions with problems where the search space is too wide. In this way, it is possible to get a good solution within a reasonable time.

Several experiments were set up using the glove, starting with learning from data coming from a single person, and later, carrying out more complex tests in order to arrive at a learning procedure based on data from several people.

The objective of the experiments is simple: try to ascertain what the object being used by the person wearing the glove is – see Fig.4.

Fig.4. An operator wearing data gloves at work

2 http://www.wearitatwork.com

In order to achieve this, it was decided that

predictions were to be made on the basis of 10 classes: the motionless hand, the hand in transition (picking or leaving an object), and handling of 8 different objects used often in the workshop context: a hammer, screwdriver, PDA, wrench, pliers, tester, drill and a bolt that was screwed manually.The data sent by the glove was read while different persons used the objects. 5 readings with each object were done with each person: 3 using the object and another 2 doing transitions (starting with the motionless hand, picking up and then using the object, and finally setting down the object and the hand at rest).

In this test, the algorithms used were the already mentioned 1-NN (K-NN, with K=1), the Naive-Bayes and the C4.5, which represent different families of classifiers. The validation was done with the 10-fold cross-validation method.

Once this first experiment was done, and the results seen, we started a second stage where we tried to make the predictions based on data from different individuals; from now called interpersonal learning.

The idea of being able to predict which object a person is using based on the data of other individuals is very interesting. In fact, physical characteristics of human hands and the different ways of handling objects introduce an extra degree of complexity.This is the concept that we found behind what we call interpersonal learning or leave-one-person-out: trying to predict the objects without having previous data of the person wearing the glove.

In order to simulate this idea, we took readings from 5 persons. We used the readings of one of them for the test and we trained our model with the data of the other persons. In this way, during the training, there was no access to the cases used in the test. We carried on this process for the data of each one of these 5 persons. 4.3.Results

In the first experiment it was observed that, once an individual's data are gathered, it is possible to create a robust and reliable classifier with a 99% recognition rate that predicts the different objects that this person may be using - see Fig. 5.

We have created an application capable of predicting the different objects on-line, once the training data has been gathered. Every 15ms, data are read from the glove and predictions are made by using a C4.5 classifier. In order to stabilise these predictions

we use a buffer that records the last 30 readings and makes the predictions on this data within a reliability limit set by the user.

859095

100105

4 5 6 7 8 9 10Number of classes

Rec

ogni

tion

rate

(%)

1-NN Naive-Bayes C4.5F

ig. 5. Recognition rates in unipersonal training

The case of interpersonal learning is completely different. Using the C4.5 algorithm we were only able to achieve a poor 25% success rate. The recognition rate achieved is presented in Table 1:

Person 1 2 3 4 5 Mean Rate 10,7

8 42,23 25,31 22,3

1 24,60

25,05

Table. 1. Recognition rate in interpersonal learning

It is clear that this inadequate rate makes it

impossible to use the algorithm in real systems. In order to increase the success rate in interpersonal learning, we decided to make a selection and weighting of variables by using a Genetic Algorithm:

• Selection of Variables: to completely eliminate the variables that have a negative influence on the predictions.

• Weighting of Variables: to give the different variables their due importance. In order to do so, we applied changes to some of them for “weighting” the data they contained and decreasing its importance. We carried this out by raising all the values of the variable to an integer value. In our case we used the following formula:

YX (1)

where X is the value of the variable and Y an integer value between 1 and 6

The use of the Genetic Algorithm for selection and weighting of variables has achieved an increase in this rate to 35% - see Table 2 -, but even these success rates are far from being usable within any application.

Person 1 2 3 4 5 Mean Rate 16,4

9 53,89 44,92 41,4

8 39,1

1 39,18

Table. 2. Recognition rate with interpersonal

training using variable selection and weighting 5. Conclusions and future work

In the area of manufacturing, AmI is not only going to affect the way in which processes develop, but will also provide new ways of working and doing business. The development of new products and services and the shift in the focus of attention of the worker from the machine to their immediate working environment will be the immediate consequences of the adoption of AmI vision. In no case this is proposed as a new manufacturing paradigm but, whatever models are followed, these will have to take into account these changes.

Not only several technological challenges such as miniaturisation, inter-operability and energy management have to be addressed by research teams across the world in the present decade, but also a strong focus on the social and organisational aspects of AmI has to be taken to overcome barriers to its realisation.

On the other hand, in the manufacturing field there is a great amount of already existing legacy systems that should be integrated in a Manufacturing AmI environment for which there is a need of integration mechanisms

Regarding human aspects, cost, risk of intrusion, potential employee resistance, legal and ethical restrictions, and risks associated with high system complexity can be main drawbacks in the widespread adoption of AmI.

The AMICO prototype offers a new work environment providing high quality information and content to any user, anywhere, at any time, and on any device.

The AmILAB group aims to continue the research in different topics:

• To seek to achieve a more intelligent contextual information management.

• To create new agents that will be added to the AMICO platform.

• To integrate Machine learning techniques in order to adapt to the user and the context.

• To extend the AmI Laboratory by introducing new interaction technologies.

Relating to activities’ identification, the challenge

is trying to increase the recognition rate of what we call interpersonal learning. We are creating more complex classifiers that will be capable of solving the evident problem derived from the diverse physical characteristics of human hands and the different ways of handling objects.

The laboratory created and operating in Tekniker is a good test bed where research on technology and human factors is done in a holistic way. Acknowledgements This research is possible thanks to the Basque and Spanish Government research programs. The wearIT@work3 project is funded by the European Community FP6. Fundacion Tekniker is partner of the EU-Funded FP6 Innovative Production Machines and Systems (I*PROMS)4 Network of Excellence. References [1] K. Ducatel, M. Bogdanowicz, F. Scapolo, Leijten J., and

J.C. Burgelma. Istag: Scenarios for ambient intelligence in 2010. ISTAG 2001 Final Report, 2001.

[2] Expert Group on Competitive & Sustainable Production and Related service Industries in Europe in the Period to 2020, EU Commission, “Sustainable Production: Challenges & objectives for EU Research Policy”, 2001

[3] Visionary Manufacturing Challenges for 2020. Committee on Visionary Manufacturing Challenges, National Research Council. NATIONAL ACADEMY PRESS,Washington, D.C. 1998

[4] JADE. Java agent development framework, 2004. URL: http://jade.cselt.it/index.html.

[5] Mohammed Waleed Kadous. “Machine Recognition of Auslan Signs Using Power Gloves: Towards Large-Lexicon Recognition of Sign Languages”, 1996

[6] T.Stiefmeier, Paul Lucowicz, “Showcase platform architectural design and specification – A technical document”. Internal deliverable wearIT Project, 2005

[7] S. S. Fels and G. Hinton. Glove-talk: A neural network interface between a data-glove and a speech synthesiser. IEEE Trans. on Neural Networks, 4(1):2{8, 1993.

[8] T. Mitchell. “Machine Learning”, McGraw-Hill, 1997 [9] J. R. Quinlan. “C4.5: Programs for Machine Learning”.

Morgan Kaufmann, 1993 [10] D. Goldberg. “Genetic Algorithm in search,

optimisation and machine learning”. Addison-Wesley, 1989

[11] T. Bäck. “Evolutionary Algorithms in Theory and Practice”. Oxford University Press, 1996

3 http://www.wearitatwork.com 4 http://www.iproms.org