New challenges in wireless sensor networks:fault tolerance and real time

J. V. Capella, R. Ors, A. Bonastre, J. J. SerranoTechnical University of Valencia

Camino de Vera s/n, 46022 Valencia, Spain{ jcapella, rors, bonastre, jserrano}@disca.upv.es

Abstract- The wireless communications field is incontinuous growth and revolution. The current trendtoward the use of distributed wireless systems is due tothese systems have demonstrated to be a good option dueto its flexibility and power.In this line, nowadays there is an enormous demand ofwireless connectivity for any industrial controlapplications, included critical systems.In this paper we want to discuss possibilities andnecessary developments to set up wireless communicationfor industrial applications in general and critical systemsin particular. Furthermore, an abstract definition of alayered architecture for the development of intelligentand distributed control systems based on wireless sensornetworks is proposed.

I. INTRODUCTION

One definition of distributed system can be a systemconsisting of several physically dispersed nodes, each ofthem independent of the rest, but collaborating for thedevelopment of a common task. The collaboration impliesthe need of a communication network among the nodes, andthe stating of a common communication protocol. The maindifference between a distributed system and a networked oneconsists on transparency. From the user's point of view, allthe nodes and network are abstracted in a single virtualcentralised system. In this manner, the distribution of nodesis transparent to the user, relying on the system the nodesdistribution solution.A distributed system based on a wireless network offer

several advantages when implementing industrial controlsystems [1], such as scalability, modularity, power andportability but also an significant increase in flexibility andmobility due mainly to the elimination of cables.

Several problems arise when dealing with this approach.First of all, an automatic programming of each node must beperformed by the system, distributing the variables and theprogram between possibly different nodes. No distribution ofinputs and outputs is possible, because the node physicallyconnected to a system variable must handle it, but the subsetof nodes related to every variable should know its currentvalue. Also the complexity of using a programming languageto the general public, as well as the necessity to know thecomplex functions that manages the protocols to control thenetwork access, without forgetting the coherence problemsdue to the parallelism, etc.

In these systems the control response should take place ina deterministic amount of time. A too late action could beeven worst than a wrong one or no action at all. In this sense,

Real-Time capabilities of the control system must beconsidered [2].

Fault tolerance is the other main issue to study. It isnecessary to know how the distributed wireless system willact if one node fall down, specially in systems with lowmaintenance possibility.The proposed architecture takes into account these

questions. In this manner, several nodes collaborate to fulfilthe control system task. Each node reads its input variablesfrom the system, by means of a A/D converter if themagnitude is reflected in analog values, or by means ofdigital inputs, and takes its output lines as actuator over thesystem (again, they can be analog or digital variables). Everynode executes its own subset of the whole program, readinginputs and writing outputs and, if necessary, distributing itsmodified variables through the wireless network. It can alsotake into account global variables, modified or obtained atother nodes. The user interface resides at the supervisionnode. This node allows the user to interact with the controlsystem, design and verify the control procedure and, finally,performs the distribution of the control program among thecontrol nodes.The paper has been structured in six sections. The

following paragraphs present the advantages of the wirelesssensor networks, the abstract architecture definition, therequired capabilities of the different layer implementations,the interfaces between layers and several guidelines to adaptthe implementation of the studied systems. The paper endswith the conclusions and future work.

II. WIRELESS SENSOR NETWORKS IN CONTROL SYSTEMS

In automation a major trend consists on increase thenumber of electrical and electronic systems. Many of thesesystems have been introduced to offer innovations to thecustomer by an increase in functionality, comfort, and safety.Another reason is the partial replacement of mechanicalcomponents by electrical components or their integration intomechatronic systems. On a more technical level some othermajor trends are: increasing number of monitoring anddiagnostic tasks, extended automation and increasing numberof control tasks, increasing number of variants, increasingdegree of decentralization, and increasing demand forcommunication between several control units as well asbetween sensor/actuator units and control units, and otherpossible patterns.

Using conventional wired technologies only, these trendscause growing difficulties. Some of the biggest problems arethe increasing number of wires and complexity of harness,and therefore the increasing number of connectors, augment

0-7803-9484-4/05/$20.00 ©2005 IEEE 1 385

of malfunctions due to connector failures, high cost in case ofdesign modifications, and high cost for assembly and service.A well known technical solutions to overcome the

problems of increased complexity of wiring consist in theintroduction of bus systems for networking. Today mostvehicle manufacturers are using bus systems in their designof a control architecture. And the other step that still offersbetter characteristics, consists on the use of wirelessnetworks, like this paper explains. This type of networks canbe used in control systems and technical tasks [3], althoughcertain properties of this kind of networks, such as boundedresponse time or fault tolerance, should be studied and solvedin the architecture, if the system to control requires some ofthose characteristics.An additional trend is the increasing data processing

capability and performance of the field level components, forexample sensors and actuators. Together with newtechnologies for wireless communication this may drivefuture developments with mutual influences on field devicesand controllers as well as communication systems. Thesetrends may result in a new generation of a systemarchitecture with a more decentralized and distributedstructure of intelligent devices.

In addition, a wireless network offers a bigger reliabilityand dependability due to the following reasons:

- Wiring in problematic areas (difficult access, adverseenvironmental conditions, etc.) is not necessary.

- Connectors are not used (elimination of a possiblesource of faults).

- Wireless technology is robust in front of mechanicalvibrations and is resistant to the corrosion, because thistechnology does not use cables.

III. PROPOSED ARCHITECTURE

The proposed architecture defines the services offered byeach layer, and specifies the interface between layers. It alsodefines the required characteristics of the technologies andprotocols at any layers, and the capabilities that shouldimplement.

This generic architecture distinguishes four levels in adistributed wireless control system. Upper layer consists of adigital control layer, where high level decisions are taken.This level should work with concepts (i.e. the temperature ishigh) rather than numbers (the temperature is 40 °C).The second layer deals with numeric values, performs

analog operations and implement analog control loops. It isalso in carry of the conversion from numerical variablesvalues to evaluated expressions.

The wireless communications system appears in the thirdlayer. The network infrastructure must perform theinformation share between nodes, offering to the upper layersa data transmission service.

Finally, every node should implement a hardware interfacewith the process to control. It is implemented by means ofA/D D/A converters prior to its transmission through thenetwork.

Each of this layers is commented below in deep. Everylayer must communicate with its upper and lower layerfollowing a specified interface, as show the fourth section.The system is controlled by means of a set of analog inputs

and outputs. Hardware level converts these analogmagnitudes into discrete (digitalized) data. These data can betransferred by means of the network to every node in thisform (raw mode) or being evaluated by the analog levelbefore its transmission. If the latter occurs, no analog data,but variable concepts expressions is transmitted. This featureincrease the power of the architecture by reducing thebandwidth required (only elaborated and new concepts andtransmitted, avoiding the transmission of repetitive or non-relevant data).

In this sense, network may transmit raw data or processeddata (conceptual values), so both formats should beconsidered.Analog level is in charge of the numerical data

interpretation to fit a conceptual value of each variable.These concepts must be the base of digital control execution,given place to high abstraction level procedures.

Detailed description of these levels and its suitableprocedures is described in the following sections.

A.. Digital Layer

This level implements the discrete control functions,system supervision as well as diagnosis and fault tolerance,making intelligent the whole control system.The IA techniques to implement this level should complete

the following requirements:

- Possibility to guarantee the design correction.- Possibility to be executed in a distributed way.- Possibility to delimit the response time.- Abstraction of the distributed system, formed of diverse

nodes, so that from the designer's point of view itbehaves as a centralized system. This is the distributedsystems methodology.

- Possibility to be executed in generic nodes, possiblyvery different between them.

- Possibility that even non-expert users could design thecontrol system, by means of simple programming thatkeeps in mind the possible non sequential execution ofthe instructions. It is sought the designer only providethe basic system operation lines, and be this last whodecides what rules are applicable and when.

An example of suitable technique for this level are theRuled Based Expert Systems, concretely the Rule Nets (RN)[4]. RN have been applied obtaining excellent experimentalresults [5], providing a suitable base for the distributedcontrol systems implementation by several reasons:

- Simplicity and power: RN allows the complex controlsystems specification in an easy and intuitive way,without limiting for it the complexity of the controlfunctions to implement.

1386

- Analysis easiness: Given the similarity of the RN andthe Petri Nets it has been possible the definition ofseveral purification techniques on the RN that allow theuser to guarantee the correct control system behavior. Asexample of these techniques we can mention frombehavior simulation systems until purification methodsbased on the expansion tree.

- Compactness: RN topology can be represented in acompact way by means of binary matrices (antecedents,consequent and unmarked) and its state by means ofbinary vectors. This representation is very appropriateas much for its execution in microcontroller basedsystems as for its distribution through digital nets. TheRN execution by means of elementary logical operationsis of special interest.

B. Analog Layer

The discrete and continuous control techniques are notincompatible, but complementary, when dealing withcomplex systems. Combination of both techniques offersnew possibilities, such as the implementation of hierarchicalcontrol systems. In this scheme, this level performscontinuous control loops, diagnostics tasks and analogoperations.When dealing with a continuous control system

implementation it was needed to decide the most accuratecharacteristics, such as:

- To be able to control any system almost with the samebenefits that other control system already existent.

- Possibility that even non-expert users could design thecontrol system, and even allowing the capacity for theself-learning of the system.

- Possibility to be executed in a distributed way.- Easiness when being transmitted through the net for their

execution in generic nodes (possibly very differentbetween them).

Afterwards an exhaustive study of different continuouscontrol techniques, the conclusion that Neural Networks[6](NN) meets all the previous conditions has been reached.Indeed, NN not only completes the first condition since theycan deal successfully with any control system, but, farbeyond, they are able to control systems where no othertechniques can be applied.

It also fulfil perfectly the second condition due to theirlearning characteristics [7], making possible that even non-expert users in the control systems design can be capable, ina centralised way, to define the desired system behaviour.NN are capable to learn in an automatic way thought theanalysis of a group of samples that reflect the answersexpected in real situations of execution. Even more, it ispossible in the design phase the simulation of the systemoperation in front of hypothetical situations with the purposeof checking that it fulfils the desired specifications.

Since the NN are formed by perfectly detachable units(neurons), the distribution of these neurons in different nodesis not very difficult, so the results obtained by each neuron

must be spread thought the interconnection network to beused as an input for any other neuron needing it.

Finally, it is possible to characterise a neural network as agroup of neurons. All neurons present a common structure,and they are easily adapted in function of a parameters set.Therefore, it is possible to locate several generic neurons inthe nodes and make a particularization of them in function ofthe desired control system by means of the transmission ofthese parameters.

C. Wireless Communications Layer

The goal of this layer is to provide the informationexchange among remote nodes, so that they can carry out itsactivities.As example of suitable networks can be indicated ZigBee

[8], Bluetooth [9][10] or WiFi (802.11) [11] because of itsparticular features.

ZigBee allows multiple kinds of traffic with differentcharacteristics, including periodic data, intermittent data, andrepetitive low latency data.The medium access control (MAC) of ZigBee provides

network association and disassociation, has an optionalsuperframe structure with beacons for time synchronization,and a guaranteed time slot mechanism for high prioritycommunications. The channel access method is carrier sensemultiple access with collision avoidance (CSMA-CA).ZigBee defines the network, security, and applicationframework profile layers for an IEEE 802.15.4 based system.ZigBee's network layer supports three networkingtopologies; star, mesh, and cluster tree. Star networks arecommon and provide for very long battery life operation.Mesh, or peer-to-peer, networks enable high levels ofreliability and scalability by providing more than one paththrough the network. Cluster-tree networks utilize a hybridstar/mesh topology that combines the benefits of both forhigh levels of reliability and support for battery-powerednodes.

ZigBee permits data and control messages to be passedfrom one node to other node via multiple paths. This featureextends the range of the network and improves datareliability. This peer-to-peer capability may be used to buildlarge, geographically dispersed networks where smallernetworks are linked together to form a 'cluster tree' network.ZigBee provides a security toolbox to ensure reliable andsecure networks. Access control lists, packet freshness timersand 128-bit encryption protect data transmission and ZigBeewireless networks.

Another appropriate networks for this level are Bluetooth,WiFi, or others [12] mainly for their biggest covering area.

Over the selected wireless network technology will benecessary a protocol that provides the followingcharacteristics.

C. 1 Remote identification ofthe connected devicesThe communications protocol will be able to detect what

system devices are really in operation, without necessity ofcarrying out the access to the net verification in anindividualised way.

1387

In all studied application protocols, the system designershould know all the existent nodes in the network andintroduce this information in all the devices that require it.For the communication problems detection in a node, it isnecessary to carry out explicit diagnoses (communicationtests) on the same ones.

C.2 Net virtualisationFrom the user's point of view, the whole network should

be contemplated as a centralised system in the program tool,with all its inputs and outputs. It is transparent to the user thefact that these inputs and outputs do not reside in one centralnode, rather they are distributed among the nodes. This isgotten thanks to the virtualization of each device in a seriesof inputs and outputs, the nodes identification to the centralnode in a transparent way to the user and the nodesprogramming interface. Also, this feature make possible todesign general purpose nodes, generic and from differentproviders whose integration does not present problems.

C. 3 CentralizedprogrammingOnce the user has all the system inputs and outputs in the

program tool, he can proceed to the programming of thebehaviour wanted in the same one, without necessity ofcontemplating the details corresponding to the distribution.The programming in the central node is carried out from anidentical way to the corresponding to a centralised system. Inthis manner is possible the off-line verification of the systemoperation, that is to say, without necessity of executing theprogram on the network, by means of simulation. Thisanalysis guarantee a correct behaviour of the system. It isalso possible to calculate the system states diagram, and overthem verify the absence of not desired states. Once checkedthe system, the protocol will take charge of its distributionamong the nodes.

Several studied protocols allow the transmission of datablocks that could contain program code. However, this codeshould be characteristic of the device implementation. Noneof this protocols contemplates the possibility of systembehaviour simulation, neither the properties analysis in orderto guarantee the properly system operation

C. 4 Real Time supportIt should be possible to delimit the response time. In case

of using Rule Nets, their mathematic form allows theexecution time evaluation. In this sense, and taking intoaccount that this execution consists on simple binaryoperations (AND & OR) between bit matrices (usuallyimplemented by bytes), a worst case is defined, thus responsetime can be bounded.On the other hand, the protocol over the selected network

technology will delimit in some way the response time.

C. 5 Fault toleranceThe protocol will allow the implementation of several

fault-tolerance techniques. Some of them can need a centralnode as a system watch dog, and the adoption of severalactions to deal with failure events. In this case, the centralnode would send a WHO message periodically, which would

be replied by means of an ALIVE message. If the ALIVEmessage does not arrive, and after several tries, thesupervision node will assume a failure in correspondingnode.When a node fail is detected, several actions can be carried

out. The user will be informed anyway, but it is also possibleto restart the system, or even to start a degraded workingmode.

In certain cases should be possible to carry out votingmechanisms among the conclusions of several nodes, also, incase of some nodes falls down, another node could follow thereasoning line and, therefore, it would minimise the effect ofthis failure. Of course, the node fall implies the impossibilityof managing the inputs and outputs physically connected tothe same one, but it does not impede the continuation of thereasoning based on the last well-known state of thisvariables, until the guardian node mechanism detects thefailure and take the opportune measures.

IV. INTERFACES BETWEEN LEVELS

Following this architecture, every level must communicatewith its upper and lower layer following a specified interface.Fig. 1 shows the different interfaces at each level.The interfaces among the different levels are the following

ones.The digital level works with variables. These can be

transmitted among digital levels of different nodes. Also, thenumerical value of variables can be necessary for theanalogical level.The analogical layer works with discrete values that can be

transmitted by the network to other analogical levels of othernodes that require them.

These values come from the inferior level (inputs) or to gotoward this (outputs).

Wireless network may transmit raw data or processed data(conceptual values), so both formats should be supported.The level A/D D/A convert the physical magnitudes in

discreted values that will use the superior levels.

Fig. 1. Interfaces between architecture levels.

1388

In this manner, it should be able to carry out the necessaryconversion functions to translate the logical values of thevariables in digital values that will be programmed in theanalogical digital converters to activate the outputs, as wellas the conversion of analogical magnitudes to digital valuesthat will be discreted to represent the different input variablesstates. The requirements of this process are minimum,consisting exclusively on a comparisons series. It should alsobe able to obtain and to write values with the purpose ofallowing the calibration.

V. ARCHITECTURE ADAPTATION

To apply the proposed architecture to a concrete system,the designer should choose the technique or technology moreappropriate for the implementation of each level. Thiselection will be carried out according to the concreteapplication to which will dedicate the system and theappropriate integration of this technique with the rest oflevels. The designer will also keep in mind that the techniquein question will fulfil the previously exposed requirements inthe presentation of each layer.

In the following paragraphs, some guidelines are shown,so that the designer can have some valid criteria when applythe proposed architecture to a concrete system and to choosewith success the techniques to use in each level of the sameone. Of this decision will depend the benefits and

characteristic of the whole system implemented.So, a concrete solution is better if it optimises the

following parameters [13]:

- Distribution efficiency and flexibility: so that thetechnique to use allows to improve the efficiency, thefault tolerance, the traceability, etc.

- Low overload: the data admit an efficient representationfor the execution and their transmission through thenetwork.

- Storage and execution efficiency: the technique can beexecuted in low cost microcontrollers as well as itrequires little memory in the nodes.

- Possibility that even non-expert users could design thecontrol system, by means of simple programming thatkeeps in mind the possible non sequential instructionsexecution. It is sought the designer only provide thebasic lines of the system operation, and be this last whodecides what rules are applicable and when.

The election of the wireless network technology is veryimportant, since the whole system characteristics will dependin great measure of its features and behavior. Also thedesigner will choose it in agreement with the systemcharacteristics and operation manner. In this way, in a closedcontrol system where the covering area is of the order of a

Temperaturesensor

r A/C Control

Boiling control

V Meteorologicalstation

Temperature/Humidity sensor

Fig. 2. An example of application.

1389

Voicecommands

Reserved

dozen of meters the ZigBee wireless network can be adopted.If the covering area should be bigger Bluetooth or even WiFi(802.11) can be used. Designer will keep in mind thetransmission characteristics of the selected wirelesstechnology to integrate it appropriately in the system designand fulfil the requirements of the final system [14].

It is fundamental to keep in mind that the global systemoperation will depend on all and each one of the levels, andtherefore of the appropriate adaptation and interaction amongthe technologies that implement them.

There is a wide field of potential applications for theproposed architecture. An example of distributed system thatfollows the guidelines explained in this paper can be seen inthe fig. 2, where the integral control of a house is carried out.

The system structure would be the following one: upperlayer implemented by means of Rule Nets, deals with thediscrete control, carries out the supervision of the lowerlayers and must perform the adequate actions in case offailure. Middle layer implemented by means of NeuralNetworks would keep into account the continuous controlloops (heating/cooling, brightness, etc.) and advancedfeatures (voice, fingerprints and shape recognition) includinghardware supervision. Finally, lower layer consists on agroup of sensors and actuators that carry on the actionsdefined by upper layers.

Physically, the system would consist on a set ofmicroprocessor based nodes interconnected by means ofZigBee.

VI. CONCLUSIONS

A vision of the new challenges and trends in the wirelesssensor networks field has been presented. In this line, ageneric approach for distributed systems based on wirelesssensor networks that take in account this trends has beendescribed. Following this conceptual architecture, thedesigner can choose to implement each level with the mostappropriate technique according to the final control system toimplement and following the interfaces and guidelinesexpressed in the present work after an exhaustive study. Witheverything it is pursued to get a flexible and genericarchitecture for the distributed wireless systemsimplementation that facilitates in great way the designerstask.

The required communications protocols for the networklayer are being studied and developed, according to theexplained features.

Additionally, a study that considers more wirelesstechnologies and control techniques for the proposedarchitecture is being carried out. For each possiblecombination the adaptation, compatibility, performance, realtime and fault tolerance will be considered, following theguidelines enunciated in the paper.

[2] H. Kopetz, Real-time systems: Design principles fordistributed embedded applications, Kluwer Academic, 1997.

[3] A. Willig, M. Kubisch, C. Hoene, A. Wolisz, "Measurementsof a wireless link in an industrial environment using an IEEE802.11-compliant physical layer," IEEE Trans. IndustrialElectronics, vol. 49, no. 6, Dec. 2002, pp. 1265 - 1282.

[4] A. Bonastre, Industrial Local Area Network: A newarchitecture for control distributed systems, Ph. D. onComputer Science, Technical University of Valencia, 2001

[5] J.V. Capella, A. Bonastre, R. Ors, "Application of DistributedExpert Systems to the advanced control of smart houses," inProc. 5th IFAC Int. Conference on Fieldbus Systems and theirApplications, Aveiro, Portugal, 2003.

[6] P.D. Wasserman, Neural computing theory and practice, NewYork: Van Nostrand Reinhold, 1989.

[7] M. Anthony and P. Bartlett, Neural Network Learning:Theoretical Foundations, Cambridge University Press, 1999.

[8] IEEE 802.15.4 standard, 2003.[9] Bluetooth specification version 1.2, 2003[10] J. Bray, C.F. Sturman, BLUETOOTH: Connect without cables,

Prentice Hall, 2002.[11] IEEE 802.11d standard, 2001.[12] A. Brand, H. Aghvami. Multiple access protocols for mobile

communications GPRS, UMTS, and beyond, John Wiley &Sons, Chichester, 2002.

[13] A. Bonastre, J.V. Capella, R. Ors, "A new Hierarchical andIntelligent Control Architecture based on Distributed NeuralNetworks and Expert Systems applied to vehicle automation,"in Proc. IEEE Int. Conf on Control and Automation, Xiamen,China, 2002.

[14] G. J. Pottie, W. J. Kaiser, "Wireless Integrated NetworkSensors", Communications of the ACM, vol. 43, no. 5, May2000, pp. 51 - 58.

REFERENCES

[1] I.F. Akyildiz, W. Su, Y. Sankarasubramaniam, E. Cayirci, "ASurvey on Sensor Networks," IEEE CommunicationsMagazine, Aug. 2002, pp. 102 - 114.

1390