Intrusion-Tolerant Protection for Critical Infrastructures

Intrusion-Tolerant Protection forCritical Infrastructures

Alysson Neves BessaniPaulo Sousa

Miguel CorreiaNuno Ferreira Neves

Paulo Verissimo

DI–FCUL TR–2007–8

April 2007

Departamento de InformaticaFaculdade de Ciencias da Universidade de Lisboa

Campo Grande, 1749–016 LisboaPortugal

Technical reports are available athttp://www.di.fc.ul.pt/tech-reports . The filesare stored in PDF, with the report number as filename. Alternatively, reports are available bypost from the above address.

Intrusion-Tolerant Protection forCritical Infrastructures

Alysson Neves Bessani Paulo Sousa Miguel CorreiaNuno Ferreira Neves Paulo Verissimo

LASIGE, Faculdade de Ciencias da Universidade de Lisboa – Portugal∗

Abstract

Today’s critical infrastructures like the Power Grid areessentially physical processes controlled by computers con-nected by networks. They are usually as vulnerable as anyother interconnected computer system, but their failure hasa high socio-economic impact. The paper describes a newconstruct for the protection of these infrastructures, basedon distributed algorithms and mechanisms implemented be-tween a set of devices called CIS. CIS collectively ensurethat incoming/outgoing traffic satisfies the security policy ofan organization in the face of accidents and attacks. How-ever, they are not simple firewalls but distributed protec-tion devices based on a sophisticated access control model.Likewise, they seek perpetual unattended correct operation,so they are designed with intrusion-tolerant capabilities andhardened with proactive recovery. The paper discusses therationale behind the use of CIS to improve the resilience ofcritical infrastructures and presents a design using logicalreplication based on virtual machines.

1 Introduction

Critical infrastructures like the power, water and gas dis-tribution networks have a fundamental role in modern life.These infrastructures are essentially physical/mechanicalprocesses controlled electronically. The control systems,usually called SCADA (Supervisory Control and Data Ac-quisition) or PCS (Process Control System), are com-posed by computers interconnected by computer networks[25, 40].

In recent years these systems evolved in several aspectsthat greatly increased their exposure to cyber-attacks com-ing from the Internet. Firstly, the computers, networks andprotocols in those control systems are no longer proprietary

∗This work was partially supported by the EC through project IST-2004-27513 (CRUTIAL) and NoE IST-4-026764-NOE (RESIST), and bythe FCT through project POSI/EIA/60334/2004 (RITAS) and the Large-Scale Informatic Systems Laboratory (LaSIGE).

but standard PCs and networks (e.g., wired and wirelessEthernet), and the protocols are often encapsulated on topof TCP/IP. Secondly, these networks are usually connectedto the Internet indirectly through the corporate network orto other networks using modems and data links. Thirdly,several infrastructures are being interconnected creating acomplexity that is hard to manage [38].

Therefore these infrastructures have a level of vulnera-bility similar to other systems connected to the Internet, butthe socio-economic impact of their failure can be huge. Thisscenario, reinforced by several recent incidents [42, 8], isgenerating a great concern about the security of these in-frastructures, especially at government level.

A reference architecture was recently proposed to protectcritical infrastructures, in the context of the CRUTIAL1 EU-IST project [40]. The whole infrastructure architecture ismodeled as a WAN-of-LANs. This topology allows simplesolutions to hard problems such as legacy control subnet-works, and interconnection of critical and non-critical traf-fic. Typically, a critical information infrastructure is formedby facilities, like power transformation substations or cor-porate offices, modeled as collections of LANs and inter-connected by a wider-area network, modeled as a WAN, inthe WAN-of-LANs model.

This architecture allows defining realms with differentlevels of trustworthiness. In this paper we are interested inthe problem of protecting realms from one another, i.e., aLAN from another LAN or from the WAN. However, giventhe ease of defining LANs in today’ IP architectures (e.g.,through Virtual switched LANs), there is virtually no re-striction to the level of granularity of protection domains,which can go down to a single host. In consequence, ourmodel and architecture allow us to deal both with outsiderthreats (protecting a facility from the Internet) and insiderthreats (protecting a critical host from other hosts in thesame physical facility, by locating them in different LANs).

Protection of LANs from the WAN or other LANs is

1Critical UTility InfrastructurAL Resilience: http://crutial.cesiricerca.it .

1

made by a device calledCRUTIAL Information Switch(CIS). A CIS provides two basic services: theProtectionService (PS)and theCommunication Service (CS). The PSensures that the incoming and outgoing traffic in/out of theLAN satisfies the security policy of the infrastructure. TheCS supports secure communication between CIS and, ulti-mately, between LANs. The CS provides secure channels,reliable and atomic multicast primitives, and probabilisticgossip-based information diffusion between CIS. Althoughthe CIS supports these two services, this paper presentsonly the protection service, not the communication service.Therefore, from now on “the CIS” means “the CISProtec-tion Service”. Figure 1 ilustrates the use of CIS protectingseveral LANs of a critical infrastructure.

� � ��

� � � � � � � � � � � � � �

� � � � � � � � � �

� � �

� � �

� � � � � � � � � � � � �

�

�

� � � �

� � � �

� � � �

� � � �

� �

� � � �

� � �

� � �

� � ��

� � � � � � � �

� � �

� � � � �

� �

� � � � � � � �

� � � � � � � � � �

� � � � � � � � � ��

� � � � � � � � � � � � � �

� � � � � � � � � � � � � �

Figure 1. WAN-of-LANs connected by CIS.

A CIS can not be a simple firewall since that would putthe infrastructure at most at the level of security of currentInternet systems, which is not acceptable since intrusions inthose systems are constantly being reported [19]. Instead, aCIS is a distributed protection device based on a sophisti-cated access control model. The main characteristics of theCIS are the following. Firstly, it has similarities to adis-tributed firewall[4], since CIS can be deployed not only onthe network border but inside the networks to better protectcritical equipment. Secondly, the CIS uses arich accesscontrol modelthat takes into account the involvement ofdifferent organizations and allows the access control rulesto depend on context information [1]. Thirdly, the CIS isintrusion-tolerant[17, 41], i.e., it operates correctly evenif there are intrusions in some of its components and with-stands a high degree of such hostility from the environment,seeking unattended perpetual operation [35].

The paper is essentially about the last topic, the design ofan intrusion-tolerant CIS. The CIS is replicated in a set ofmachines. If there are intrusions in some of the replicas, theCIS masks these faults and follows its specification. Thismechanism is hardened by proactively recovering replicas

periodically [30], in order to avoid that an attacker succes-sively corrupts replicas until it controls enough to corruptthe behavior of the CIS. This paper extends previous workonproactive recoveryby using alsoreactive recoveryto re-cover replicas that are detected to be compromised.

Several intrusion-tolerant services have been proposedin the literature, either based on Byzantine quorum systems(BQS) [26, 44] or state machine replication (SMR) [33, 10,2]. However, the CIS design presents two very interestingchallenges that make it essentially different from those ser-vices. The first is that a firewall-like component has to betransparent to protocols that pass through it, so it can notmodify the protocols themselves to obtain intrusion toler-ance. This also means that recipient nodes will ignore anyinternal CIS intrusion-tolerance mechanisms, and as suchthey can not protect themselves from messages forwardedby faulty replicas not satisfying the security policy. Thepaper shows that these two challenges are not trivial andpresents a solution that is based neither on BQS nor onSMR.

As mentioned before, the initial motivation for develop-ing the CIS was the protection of critical infrastructures.However, the CIS is sufficiently generic to allow its use inother scenarios with a high level of protection requirements.

The paper has the following main contributions: (1)it presents the design of an intrusion-tolerant firewall-likecomponent adequate for protecting networked infrastruc-tures and services from outsider and insider accidental ormalicious threats, such as critical information infrastruc-tures; (2) it proposes a novelproactive-reactive recoveryscheme and algorithmthat not only proactively recoversreplicas but also forces recovery of replicas detected to befaulty; and (3) to our knowledge it presents the first imple-mentation of an intrusion-tolerant service in a single ma-chine, using logical replication based on virtual machines.The evaluation of this prototype has shown that the limitedresources of a single machine constrain the performance ofthe service, but lead to an economic solution. However,more hardware-lavish (and more performant) solutions arenot precluded, as it will be shown.

2 The CIS Protection Service

The CIS works mainly like a firewall. It captures packetsthat pass through it, checks if they satisfy the security policybeing enforced, and forwards the approved packets, discard-ing those that do not satisfy the policy. However, severalother characteristics of the CIS make it a unique protectiondevice. These characteristics are presented in this section.Distributed firewall. CIS can be used in a redundant way,enforcing the same policies in different points of the net-work. An extreme case in the SCADA/PCS side is to havea CIS in each gateway interconnecting each substation net-

2

work, and a CIS specifically protecting each critical com-ponent of the SCADA/PCS network. The concept is akinto using firewalls to protect hosts instead of only networkborders [4], and is specially useful for critical informationinfrastructures given their complexity and criticality, withmany routes into the control network that can not be eas-ily closed (e.g., Internet, dial-up modems, VPNs, wirelessaccess points) [8].

Application-level firewall. Critical infrastructures havemany legacy components that were designed without secu-rity in mind, thus do not employ security mechanisms likeaccess control and cryptography [15]. Since these securitymechanisms are not part of the SCADA/PCS protocols andsystems, whichmust still be protected, protection must bedeployed in some point between the infrastructure and thehosts that access it. The CIS has to inspect and evaluate themessages considering application-level semantics because,as already said, the application (infrastructure) itself doesnot verify it.

Rich access control model. Besides the capacity to in-spect application-level messages, the CIS needs to support arich access control policy that takes into account the multi-organizational nature of the critical infrastructures as wellas their different operational states. Taking the Power Sys-tem as an example, there are several companies involved ingeneration, transmission and distribution of energy, as wellas regulatory agencies, and several of these parties can ex-ecute operations in the power grid. Moreover, almost allPower System operation is based on a classical state modelof the grid [16]. In each state of this model, specific actionsmust be taken (e.g., actions defined in a defense plan, toavoid or recover from a power outage) and many of theseactions are not allowed in other states (e.g., a generator cannot be separated automatically when the Grid is in its nor-mal state). These two complex facets of access control incritical infrastructures require more elaborated models thanbasic mandatory, discretionary or role-based access control.To deal with this, in the architecture of CRUTIAL we adopta more elaborated model, OrBAC (Organization Based Ac-cess Control) [1]. It allows the specification of security poli-cies containing permissions, prohibitions, obligations andrecommendations, taking into account the role of the sub-ject, who is part of an organization, the action it wants toexecute, the target object of this action, and the context inwhich it is executed. An example: “In context ‘emergency’,operators from company C can execute maintenance opera-tions on device D.”

Intrusion-tolerant firewall. As discussed in the introduc-tion, the level of security of current systems connected tothe Internet is not adequate for the infrastructures we areconcerned with, given their criticality. To improve the se-curity and dependability of the CIS, it is designed to beintrusion-tolerant [41]: it is replicated inn machines and

follows its specification as long as at mostf of these ma-chines are attacked and have their behavior corrupted. Toenhance the resilience of the CIS, we employ proactive re-covery to periodically rejuvenate the replicas, guaranteeingperpetual correct operation as long as no more thanf repli-cas become faulty in a given recovery period [30]. Thesemechanisms can be integrated to a basic, non-replicated,design of the CIS in an incremental way. We can first builda non-replicated CIS, that works much like a firewall, andthen add proactive recovery to it. After, a replicated ver-sion is built using the protocols described in this paper. Thereplicated version can be built using virtual machines in thesame host or different physical machines. These two de-signs can be later extended to consider additional replicasand maintain the system liveness even when some replicasare recovering. All these different designs are discussed inmore detail in Section 3.7.

In this paper, we address the problem of designing anintrusion-tolerant distributed firewall. Other complex ques-tions related with the CRUTIAL security architecture, likepolicy dissemination and consistency between different CISin the same security domain, was left as future work.

3 CIS Intrusion Tolerance

In this section we describe the design of the intrusion-tolerant CIS, starting with the design rationale and with thedefinition of the algorithms it executes, then we presentsome considerations on how to support statefull firewalls,and finally we describe different ways of implementing aCIS offering different levels of guarantees in terms of criti-cality and overhead.

3.1 Design Rationale

To understand therationale of the designof theintrusion-tolerant CIS, consider the problem of implement-ing a replicated firewall between a non-trusted WAN and thetrusted LAN that we want to protect. Further assume thatwe wish to ensure that only the correct messages (accordingto the deployed policy) go from the WAN side, through theCIS, to thestation computers2 in the LAN. A first problemis that the traffic has to be received by alln replicas, insteadof only 1 (as in a normal firewall), so that they can performtheir active replication mechanisms. A second problem isthat up to f replicas can be faulty and behave maliciously,both towards other replicas and towards the station comput-ers.

Our solution to the first problem is to use a device (e.g.,an Ethernet hub) to broadcast the traffic to all replicas.

2Station computers in SCADA/PCS networked systems are the front-ends of control devices.

3

These verify whether the messages comply with the OrBACpolicy and do a vote, approving the messages if and only ifat leastf +1 different replicas vote in favor. A message ap-proved by the CIS is then forwarded to its destination by adistinguished replica, theleader, so there is no unnecessarytraffic multiplication inside the LAN. The attentive readerwill identify three problems, addressed later in the paper:dealing with omissions in the broadcast; ensuring that thereis always one and only one leader; and that it is correct.

The existence of faulty replicas is usually addressed withmasking protocols of the Byzantine type, which extract thecorrect result from then replicas, despitef maliciouslyfaulty: only messages approved byf + 1 replicas shouldgo through (one of which must be correct since at mostfcan be faulty). Since the result must be sent to the stationcomputers, either it is consolidated at the source, or at thedestination.

The simplest and most usual approach is to implement afront-end in the destination host that accepts a message if:(1) f + 1 different replicas send it; or (2) the message hasa certificate showing thatf + 1 replicas approve it [5]; or(3) the message has a signature generated byf +1 replicasusing a threshold cryptography scheme [14]. This wouldimply modifying the hosts’s software. However, modify-ing the software of the SCADA/PCS system can be com-plicated, and the traffic inside the protected LAN would bemultiplied by n in certain cases (every replica would sendthe message to the LAN), so this solution is undesirable.

So we should turn ourselves to consolidating at thesource, and sendingonly one, but correct, forwarded mes-sage, in a way similar to an active replication scheme ofDelta-4 [11]. However, what is innovative here is thatsource-consolidation mechanisms should be transparent tothe (standard) station computers. Moreover, a faulty replica(leader or not) has access to the LAN (contrarily to the pro-posal of [11] where only the fail-silent adapters had accessto the LAN) so it can send incorrect traffic to the stationcomputers, which typically can not distinguish faulty fromcorrect replicas. This makes consolidation at the source ahard problem.

The solution to the second problem lies on using IPSEC,a set of standard protocols that are expected to be gener-alized in SCADA/PCS systems, according to best practicerecommendations from expert organizations and govern-ments [29]. Henceforth, we assume that the IPSEC Au-thentication Header (AH) protocol [23] runs both in thestation computers and in the CIS replicas. The basic ideais that station computers will only accept messages witha valid IPSEC/AH Message Authentication Code(MAC),which can only be produced if the message is approved byf + 1 different replicas. However, IPSEC/AH MACs aregenerated using a shared key3 K and a hash function, so it

3We assume that IPSEC/AH is used with manual key manage-

is not possible to use threshold cryptography. As the at-tentive reader will note, the shared key storage becomes avulnerability point that can not be overlooked in a high re-silience design, therefore,there must be some secure com-ponent that stores the shared key and produces MACs formessages approved by f+1 replicas.

As a final facet of our design, in order to ensure perpetualcorrectness we employ proactive recovery, which avoids re-source exhaustion due to accidental or malicious corruptionof components [35]. The idea is simple: components areperiodically rejuvenated to remove the effects of maliciousattacks/faults. If the rejuvenation is performed sufficientlyoften, then an attacker is unable to corrupt enough resourcesto break the system. Therefore, using proactive recoverywe can increase the resilience of the replicated CIS: an un-bounded number of intrusions may occur during its lifetime,as long as no more thanf occur between rejuvenations.Given that proactive recovery can only be implemented withsynchrony assumptions [36, 35], a final catch is thattheremust be some local secure and timely component responsi-ble for triggering and executing periodic recoveries.

The requirements in the previous two paragraphs, forsecure and/or timely components in an otherwise asyn-chronous and Byzantine-on-failure environment, call for ahybrid architecture that includesa distributed secure andtimely wormhole[39]. This wormhole can be deployed as aset of local trustworthy components (one per replica) con-nected by a separate control channel. Figure 2 depicts theintrusion-tolerant CIS architecture.

Figure 2. Intrusion-tolerant CIS architecture.

Each CIS replica is deployed in a different operating sys-tem (e.g., Linux, FreeBSD, Windows XP), and the operat-ing systems are configured to use different passwords anddifferent internal firewalls (e.g., iptables, ipf, Windows fire-wall). The wormhole (represented by the small W boxes) issynchronous and secure enough that it can trigger the proac-

ment [24].

4

tive recovery on time and execute a simple voting protocolthat produces a MAC for a message if at leastf +1 replicasapproved it (if the wormhole is secure, no malicious replicacan force it to sign an unapproved message). Moreover,the wormhole also simplifies the election of a new leader,as discussed later. A final feature of the wormhole has todo with recovery. As already said in the introduction, inthis paper we extend the proactive recovery model to sup-port alsoreactive recovery: when the wormhole recognizesa replica as faulty, it recovers the replica as soon as possi-ble, ensuring nevertheless that there is always an amount ofreplicas to sustain system’s correct operation.

A second traffic replication device (see figure, right handside) is used precisely for the replicas to receive whateverthe others send to the LAN. When a (correct) replica be-lieves that another replica is faulty (e.g., if it sends non-approved messages to the LAN, or it is the leader and doesnot forward approved messages), this replica is suspected tobe faulty. When a quorum of replicas suspect some replica,it is recovered.

The CIS does not provide exactly-once semantics, i.e.,messages can be lost when the traffic is high and the recep-tion buffers of the replicas become full, or when the leaderis recovered. This is not different from firewalls, exceptthat the CIS operation is more complex so the throughputis expected to be lower. However, it is important to noticethat almost all wide area control protocols, and speciallythe ones designed for Power Systems like ICCP [21] andIEC 61850 [22], are designed on top of TCP, which ensuresreliable communication even if the network (in this case theCIS) loses messages.

3.2 System Model

The system is composed byn CIS replicasCIS ={CIS1, ...,CISn}. These replicas are deployed in the inter-section between the WAN and the LAN in such a way thatall data crossing the boundaries of one of these networksmust pass through the CIS. The hybrid system model en-compasses two parts [39]: thepayloadand thewormhole.Payload. Asynchronous systemwith n≥ 2 f +1 replicas inwhich at mostf can be subject toByzantine failuresin agiven recovery period. If a replica does not fail betweentwo recoveries it is said to becorrect, otherwise it is saidto be faulty. Every CIS replica has a local clock which isassumed only to make progress. These clocks are not syn-chronized. We assumefault independencefor the replicas,i.e., the probability of a replica be compromised is indepen-dent of another replica failure. This assumption can be sub-stantiated in practice using several kinds of diversity [28].Some concerns about this issue are presented in Section 4.Finally, we assume also that the station computers can not

be compromised4.Wormhole. Synchronous subsystem W= {W1, ...,Wn} inwhich at mostfc≤ f local wormholes canfail by crash. Weassume that when a local wormholeWi crashes, the corre-sponding payload replicaCISi crashes together. The controlchannel used by the wormhole is isolated and synchronous,offering reliable multicast semantics. Each local wormholeand the station computers share a symmetric keyK, and thestation computers only accept messages authenticated usingthis key.Networks. Besides the CIS replicas model, we have to de-fine the assumptions underlying LAN and WAN commu-nication. We consider that the messages that arrive at CISreplicas both from the WAN and the LAN haveunreliablefair multicastsemantics, an adaptation of the fair links ab-straction to multicast: if a message is multicasted infinitelymany times it will be received by all its receivers infinitelymany times. The two primitives offered by this service are:U-multicast(G,m), to multicast a messagem to the groupG, andU-receive(G,m), to receivem that was multicast toG. This is substantiated in practice by the traffic replicationdevices. We assume thatall communication between repli-cas and other machines from the WAN and the LAN arebased in these primitives. For the LAN, we assume also thatcommunication is source-authenticated and the sender of amessagem is denoted bysender(m). Later we will showhow these assumptions were substantiated in our prototype.Cryptography. The MACs used in IPSEC/AH are as-sumed to inherit thecollision resistanceproperty from thehash functions in which they are based [29], i.e., that it isinfeasible to find two messages that for a keyK have thesame MAC.

3.3 Service Properties

First of all, let us define the concept of legal message:a message is said to belegal if it is in accordance with thecurrent deployed policyP. A message not in accordancewith P is said to beillegal. Moreover, a message is saidto beprocessedby its destination machine if its content isdelivered to the application layer (e.g., the SCADA system).The basic properties offered by the CIS are the following:

• Validity: A legal message received by at least one cor-rect CIS replica that is not recovered for long enough,is forwarded to its destination;

• Integrity: An illegal message is never processed by itsdestination machine.

4It is the trusted network that we aim to protect. Moreover, it does notmake sense to protect something that can be compromised and attack thefirewall.

5

Notice that these two properties are sufficiently weak tobe satisfied by a system with unreliable fair multicast com-munication and strong enough to ensure that only legal mes-sages will be processed at LAN hosts. The notion of a cor-rect CIS replica not being recovered for “long enough” isformalized in Appendix A. It captures the idea that if onlyone correct replica receives the message and is recoveredimmediately after, the message is lost and is not forwarded.Notice that this type of property is a consequence of usingproactive and reactive recovery. Any system enhanced withthis type of recovery mechanisms will need to do somethinguseful in the interval within recoveries, or else the systemwill be always recovering without executing any of its func-tionalities.

The two properties described above are satisfied in a sys-tem with n CIS replicas if the maximum number of faultyreplicas in a given recovery period ist ≤ f .

3.4 Message Processing Algorithm

This section presents the main algorithm executed by theCIS to process messages incoming from the WAN to theprotected LAN. The same algorithm is used to handle mes-sages coming from the opposite direction (possibly with amore relaxed policy).

The policy verification is made in apolicy engineac-cessed through the functionPolEngverify.Wormhole interface. The interactions between a replicaCISi and its local wormholeWi are made through a welldefined interface, offering the following services:

• W sign(m): returns a MAC of messagem obtainedwith the shared keyK, if and only if at leastf +1 CISreplicas callW sign(m) in their local wormholes;

• W verify(mσ ): returnstrue if σ is a MAC of m pro-duced withK, andfalseotherwise;

• W leader(): returns the id of the current leader replica;

• W suspectsilent( j) andW suspectmalicious( j): no-tify the wormhole, respectively, that the replicaCISj

appears to be silent or that the replicaCISj executedsome malicious action. Iff + 1 replicas suspect ofCISj , it is recovered. This recovery can be done imme-diately, without availability concerns, in the presenceof at leastf +1 malicious suspicions, given that in thiscase at least one correct replica detected that replicaCISj executed some malicious action. However, if thenumber of malicious suspicions is less thanf +1 , i.e.,if some of the suspicions are of the silent type, thenreplicaCISj may be correct. Therefore, the recoveryneeds to be coordinated with the periodic proactive re-coveries in order to guarantee that a minimum number

of correct replicas is always available. This mecha-nism will be explained in Section 3.5.2. Moreover, ifthe replica to be recovered is the leader, a new one iselected by the wormhole.

Algorithm 1 CIS payload (replicaCISi).{Parameters}integer Tvote{Expected time to vote a message}integer Ot {Omission threshold for the leader}integer Tforward {Expected time to forward a message}{Variables}integer leader= 0 {Current leader}integer leader omissions= 0 {Leader omissions counter}setVoting= ∅ {Messages being voted}setPending= ∅ {Not yet forwarded messages}setTooEarly= ∅ {Messages forwarded before their arrival}upon U-receive(WAN,m)

1: if PolEngverify(m) then2: Voting← Voting∪{m}3: σ ←W sign(m)4: Voting← Voting\{m}5: if mσ ∈ TooEarlythen6: TooEarly← TooEarly\{mσ}7: else8: Pending← Pending∪{mσ}9: if leader= i then U-multicast(LAN,mσ ) end if

10: end if11: end ifupon U-receive(LAN,mσ )12: if mσ ∈ Pendingthen13: Pending← Pending\{mσ}14: else ifW verify(mσ ) then15: TooEarly← TooEarly∪{mσ}16: else17: W suspectmalicious(sender(mσ ))18: end iffor eachTvote time units thatVoting 6= ∅19: U-multicast(WAN,Voting)for eachTforward time units thatmσ ∈ Pending

20: U-multicast(WAN,m)21: leader omissions← leader omissions+122: if leader omissions> Ot then W suspectsilent(leader)end ifupon leader 6= W leader()23: leader←W leader()24: leader omissions← 025: if leader= i then26: for all mσ ∈ Pendingdo27: U-multicast(LAN,mσ )28: end for29: end if

The Algorithm. The CIS replicas execute Algorithm 1.There are three parameters:Tvote, the expected time re-quired to receive, vote and sign a legal message;Ot, the

6

omission threshold, i.e., the maximum number of omissionsof a leader before it is suspected; andTforward, the expectedtime required for the leader to forward a message to the sta-tion computer. Additionally, the algorithm uses five vari-ables:leader, the id of the current leader;leader omissions,a counter that stores how many times the current leader didnot forward some message;Voting, the set of messages be-ing voted; Pending, the set of messages received and ap-proved by the replica that were already signed by the worm-hole but not forwarded by the leader to the station computer;and TooEarly, the set of correctly signed messages for-warded by the leader but not yet received (from the WAN)by the replica.

The algorithm begins when a replica receives a messagecoming from the WAN (lines 1-11). If the received messageis legal, all correct replicas will approve it (line 1) and signit using the wormholeW sign service (line 3). While themessage is being voted and signed by the wormhole, it isstored in theVotingset (lines 2 and 4). Then, the messageis inserted in thePendingset (line 8) and it stays in this setuntil it is received in the LAN (lines 12-13). Finally, if thereplica is the leader, it forwards it to the LAN (line 9).

The algorithm contains several controls to deal with mes-sage losses, replica failures and abnormal message order-ing. The first of these controls deals with message omis-sions on the WAN: when a replica receives and approvesa messagem, it stores it in theVoting set (line 2) beforerequesting the wormhole to sign it. The message is re-moved from the set when the message is signed (line 4).For eachTvote time units thatVoting is not empty, its con-tent is multicasted to the network (line 19). This ensuresthat the message being voted will eventually be received byother correct CIS replicas (due to the fairness assumption).The most important problem that the replicas must deal withis the failure of the leader. The leader can fail maliciously,forwarding illegal messages to the LAN, or in benign way,crashing, staying silent or even being too slow. In the firstcase, the illegal message diffusion will be perceived by cor-rect replicas that will verify that this message is not pend-ing (line 12) and was not correctly signed (line 14). Sincewe assume that the LAN traffic is source-authenticated, thewormhole will suspect the leader and recover it (line 17).There is also a counter to deal with leader crashes and omis-sions (lines 20-22). For eachTforward that a message stays inthePendingset, it is retransmitted in the WAN and an omis-sion counter is increased (lines 20-21). When this counter isgreater thanOt, the leader is suspected (line 22). With thismechanism, if the leader crashes (or stays silent), it eventu-ally will omit more thanOt messages, be suspected by allcorrect replicas and recovered. It is possible that the leaderforwards a correctly signed message to the LAN withoutsome replicas having received it from the WAN. In orderto deal with these “early messages” on the LAN, we use

theTooEarlyset. When a not pending legal message is re-ceived in the LAN, it will be stored in this set (lines 14-15)and stay there until it is received from the WAN (lines 5-6). This control is used to ensure that an already forwardedmessage will not stay pending forever in some replica.

When a new leader is elected by the wormhole, all repli-cas will discover it and the new leader will forward all mes-sages in itsPendingset to the LAN (lines 23-29).

The proof of correctness of this algorithm is presented inAppendix A. The algorithm is composed of five code blocksand in our proofs we assume that each of these blocks isexecuted by a different thread. Moreover, we assume theexistence of synchronization mechanisms that manage theconcurrent access of the various threads to the shared vari-ables. We chose to not include such mechanisms explicitlyin the algorithm in order to ease its readability. These con-siderations apply also to the remaining algorithms presentedin this paper.

3.5 Wormhole

In this section we present the algorithms executed in-side the CIS wormhole. These algorithms are executed asthreads in a real-time environment with apreemptive sched-uler where static priorities are defined from 1 to 3 (priority1 being the highest). In these algorithms we do not con-sider explicitly the clock skew and drift, since we assumethat these deviations are compensated in the protocol para-meters.

We start by describing the interactive services that aredirectly accessed by the replica payload (i.e., by Algorithm1), then explain the proactive-reactive recovery service.

3.5.1 Interactive Services

The interactive services are presented in Algorithm 2. Theshared keyK (see Section 3.2) is used as a parameter insome services. Moreover, the services provided by eachlocal wormholeWi use four variables:current leaderstoresthe id of the current leader;V is a vector of sets in thateach setV[σ ] stores the ids of the replicas that are currentlyapproving the message with MACσ ; and the setsSuspectmandSuspects contain the processes that suspect replicai ofbeing, respectively, malicious or silent.

W sign(m) calculates the MACσ of m using the sharedkey K and then sends this MAC in a VOTE message to alllocal wormholes (lines 1-2). When such a message is re-ceived, the id of its sender is inserted in setV[σ ] (line 5).After the reception off +1 messages containingσ (line 3),this MAC is returned to the client (line 4). It ensures thatthe message was approved by at least one correct replica, soit is legal.

7

Algorithm 2 Wormhole interactive services (local w.Wi).{Parameters}key K {Service Symmetric key – used for authentication}{Variables}integer current leader= n {Current leader wormhole}set∀σ : V[σ ] = ∅ {Processes approving message with hashh}setSuspectm = ∅ {Processes suspecting me of being malicious}setSuspects = ∅ {Processes suspecting me of being silent}{All interactive service threads with priority 3}serviceW sign(m)

1: σ ←MAC(m,K)2: R-multicast(W,〈VOTE, i,σ〉)3: wait until |V[σ ]| ≥ f +14: return σ

upon R-receive(W,〈VOTE, j,σ〉)5: V[σ ]←V[σ ]∪{ j}

serviceW verify(mσ )6: return MAC(m,K) = σ

serviceW leader()7: return current leader

serviceW suspectsilent( j)8: send( j,〈SUSPECT-SILENT〉)

serviceW suspectmalicious( j)9: send( j,〈SUSPECT-MALICIOUS〉)

upon receive( j,〈SUSPECT-SILENT〉)10: Suspects← Suspects∪{ j}upon receive( j,〈SUSPECT-MALICIOUS〉)11: Suspectm← Suspectm∪{ j}

W verify(mσ ) receives as input a messagem allegedlysigned withK. Returnstrue if the MAC for m obtainedusingK corresponds toσ , andfalseotherwise (line 6).

W leader() returns the id of the current leader (line 7).W suspectsilent( j) and W suspectmalicious( j)

send, respectively, a SUSPECT-SILENT orSUSPECT-MALICIOUS message toWj (lines 8-9).When a local wormholeWi receives such a message fromWj , j is inserted in one of theSuspectsets according to thetype of the message (lines 10-11). The content of these setsmay trigger a recovery procedure, as explained in the nextsection.

3.5.2 The Proactive-Reactive Recovery Service

Proactive-reactive recoveryinitiates recoveries both period-ically (time-triggered) and whenever something bad is de-tected or suspected (event-triggered). Periodic recoveriesR1, ...,Ri , ...,Rn are done in sequence, so no more than oneCIS replica is restarting at the same time. However, theinterval between these recoveries is not tight. Instead we al-locate f intervals for recovery between periodic recoveriessuch that they can be used by event-triggered recoveries.

Algorithm 3 presents an implementation of this service,which uses two main parameters:TP and TD. TP definesthe maximum time interval between consecutive triggeringof therecoverprocedure; andTD defines the worst case ex-ecution time of the a recovery. In Appendix B we provethat under these assumptions the system operates correctlyperpetually.

The approach is based on real-time scheduling with anaperiodic server taskto model aperiodic tasks [37]. Theidea is to consider the recovery as the resource and ensurethat no more than one correct replica will be restarting si-multaneously. This condition is important to ensure that thesystem always stays available. Two types of real-time tasksare utilized by the proposed mechanism:

• taskRi : represents the recovery of nodeCISi . All thesetasks have worst case execution timeTD and periodTP;

• taskA: is the aperiodic server task, which can handleat most f recoveries every time it is activated. Thistask has worst case execution timef TD and period( f +1)TD.

TaskRi is executed at local wormholeWi , while taskAis executed in all wormholes, but only the ones with thepayload suspected of being faulty are recovered. The timeneeded for executing oneA and oneRi is called therecoveryslot i and is denoted byTslot. Every sloti has f recoverysubslotsbelonging to the A task, each one denoted bySip,plus aRi . Figure 3 illustrates how time-triggered periodicand event-triggered aperiodic recoveries are combined.

� � ��

� � � �

� � � � � � � � � �

� � ��

� � � �

� � � � � � � � � �

� � ��

� � � �

� � � � � � � � � �

� � � � � �

� � �

� � � � � � � � �

� � �

Figure 3. Recovery schedule.

Notice that a reactive recovery only needs to bescheduled according to the described mechanism ifthe replica CISi to recover is not assuredly mali-cious, i.e., if less than f + 1 replicas have calledW suspectmalicious(i) (but at least f + 1 repli-cas have called eitherW suspectmalicious(i) orW suspectsilent(i)). If the wormholeWi knows withcertainty that replicaCISi is faulty, i.e., if a minimum off + 1 replicas have calledW suspectmalicious(i), CISi

can be recovered without extra care, since it is accountedas one of thef faulty replicas.

The proactive-reactive recovery service is presented inAlgorithm 3. This algorithm uses five variables:tlast storesthe instant when the last periodic recovery was triggered

8

by local wormholeWi ; theCrashedset stores the id of thecrashed replicas (at mostfc); current leader, Suspectm, andSuspects, correspond to the variables with the same name inAlgorithm 2.

Theproactiverecovery() procedure is triggered by eachlocal wormholeWi at boot time (lines 1-7). It starts by call-ing a routine that (i.) synchronizes the clocks of the localwormholes with the goal of creating a virtual global clock,and (ii.) blocks until all local wormholes call it and can startat the same time (line 1). The primitiveglobal clock() re-turns the current value of the global clock. After the initialsynchronization, the variabletlast is initialized (line 2) ina way that local wormholes trigger periodic recoveries ac-cording to their id order, and the first periodic recovery trig-gered by every local wormhole is finished withinTP fromthe execution of line 1. After this initialization, the pro-cedure enters an infinite loop where a periodic recovery istriggered withinTP from the last triggering (lines 3-7).

Reactive recoveries can be triggered in two ways:(1) if the local wormholeWi receives at leastf + 1SUSPECT-MALICIOUS messages, then recovery is ini-tiated immediately (line 8); (2) otherwise, iff + 1SUSPECT-MALICIOUS or SUSPECT-SILENT messagesarrive, then replicai is at best silent. In both cases, thef +1bound ensures that at least one correct CIS detected a prob-lem with replicai. In the silence scenario, recovery does nothave to be started immediately because the replica mightonly be slow. Instead, the aperiodic task finds the closestslot where the replica can be recovered without endangeringthe availability of the replicated CIS. The idea is to allocateone of the (reactive) recovery subslots depicted in Figure 3.This is done through the functionallocatesubslot() (line9 – explained later). Notice that if the calculated subslotSjp is located in the slot where the replica will be proac-tively recovered, i.e., ifj = i, then the replica does not needto be reactively recovered (line 10). If this is not the case,thenWi waits for the allocated subslot and then recovers thecorresponding replica (lines 11-12). Notice that the expres-sion global clock() mod TP returns the time elapsed sincethe beginning of the current period, i.e., the position of thecurrent global time instant in terms of the time diagram pre-sented in Figure 3.

The recover() procedure (lines 14-22) starts by check-ing if the current leader is going to be recovered. If thisis the case, a new leader is elected and this information issent to all local wormholes (lines 15-16). The leader elec-tion is done through a deterministic local algorithm (i.e.,no distributed communication is needed) that calculates thereplica most recently rejuvenated through a periodic recov-ery (lines 23-27). After leader election, the payload oper-ating system (OS) is shutdown (line 18). Then, the OS andCIS codes are restored (e.g., from a read-only medium suchas a Read-Only-Memory or a write-protected hard disk),

and finally the OS is booted from the clean code, bring-ing the replica to a correct state (lines 19-20). We assumethat the local CIS replica code is automatically started oncethe OS finishes the booting process. The last two lines of therecover() procedure re-initialize some variables because theCIS replica is now assuredly correct (lines 21-22).

The protocol deals also with the crash of at mostfc wormholes (and, consequently, the corresponding CISreplicas). These events are detected by a perfect failure de-tector that can be easily implemented in a crash-prone syn-chronous system. When a leader is detected to be crashed,its id is added to theCrashedset and a new leader is elected(lines 29-30). Notice that the leader election procedure doesnot elect leader a crashed replica (lines 24-25).

Subslot management is based on a simple state machinereplication scheme (lines 34-36). A subslot is allocated byWi by invoking the functionallocatesubslot(), which sendsan ALLOC message using total order multicast (line 34) toall local wormholes and waits until this message is received.At this point the functionlocal allocatesubslot(i) is calledand the next available subslot is allocated to the replica (line36). Total order multicast ensures that all local wormholesallocate the same subslots in the same order (line 37). Theexact algorithm of thelocal allocatesubslot(i) function isnot presented in order to ease the readability of the overallalgorithm. The goal oflocal allocatesubslot(i) is to man-age the various recovery subslots and to assign them to thereplicas that request to be recovered.

Notice that the CIS may become unavailable during arecovery if f replicas (different from the one that is beingproactively recovered) are faulty [34]. In order to avoid thisunavailability problem, the CIS should tolerate the “pseudo-crash” provoked by a recovery. Therefore, given that CISreplicas recover in sequence, one at a time (formally provedin Appendix B), we needn≥ 2 f + 2 replicas in order toguarantee availability during recoveries. This lower-boundresults from applying the reasoning presented in [34] toCIS: if one assumes that at mostk replicas recover at thesame time,n≥ 2 f +k+1 replicas will be needed to ensureCIS availability.

3.6 Supporting Statefull Firewalls

The CIS design applies to stateless firewalls, which islargely the most common type of firewall used. How-ever, some applications require statefull security policies,in which traffic is approved or denied taking into consider-ation the messages approved/denied in the past (e.g., somedata message is only forwarded if some connection messagewas sent before). As already said, several critical infrastruc-tures applications are not secure-aware and must rely on ex-ternal protection mechanisms (like the CIS) for its security.Such applications need statefull firewalls in order to enforce

9

Algorithm 3 Wormhole proactive-reactive recovery service (local wormholeWi).

{Parameters}integer TP {Periodic recovery period}integer TD {Recovery duration time}{Constants}integer Tslot , ( f +1)TD {Slot duration time}{Variables}integer tlast = 0 {instant of the last periodic recovery start}setCrashed= ∅ {Crashed replicas (will not be recovered)}integer current leader= n {Current leader wormhole}setSuspectm = ∅ {Processes suspecting me of being malicious}setSuspects = ∅ {Processes suspecting me of being silent}{Periodic recovery thread with priority 1}procedureproactiverecovery()

1: synchronizeglobal clock()2: tlast← global clock()−TP +((i−1)Tslot+ f TD)3: loop4: wait until global clock() = tlast+TP5: tlast = global clock()6: recover()7: end loop{Threads with priority 2}upon |Suspectm| ≥ f +1

8: recover()upon (|Suspectm|< f +1)∧ (|Suspects+Suspectm| ≥ f +1)

9: 〈 j, p〉 ← allocatesubslot()10: if j 6= i then11: wait until global clock() modTP = ( j−1)Tslot+(p−1)TD12: recover()13: end if

procedure recover()14: if current leader= i then15: elect new leader()16: R-multicast(W,〈NEW-LEADER, i,new leader〉)17: end if18: shutdownOS()19: restorecode()20: boot OS()21: Suspectm←∅22: Suspects←∅procedureelect new leader()23: new leader← last periodic recoverednode()24: while new leader∈ Crasheddo25: new leader← ((new leader−2) modn)+126: end while27: current leader← new leader

upon R-receive(W,〈NEW-LEADER, l ,new leader〉)28: if l = current leaderthen current leader← new leaderend ifupon current leaderis faulty

29: Crashed← Crashed∪{current leader}30: elect new leader()procedure last periodic recoverednode()31: tround← global clock() modTP32: j ← (b tround

Tslotc modn)

33: if j = 0 then return n else return j end ifprocedureallocatesubslot()34: TO-multicast(W,〈ALLOC, i〉)35: wait until TO-receive(W,〈ALLOC, i〉)36: return local allocatesubslot(i)upon TO-receive(W,〈ALLOC, j〉)∧ j 6= i

37: local allocatesubslot( j)

powerful security policies.

The CIS policy engine can provide statefull semanticsas long as one ensures that all messages are verified in thesame order in all CIS replicas. In other words, we needan agreement protocol to ensure that all messages are eval-uated in total order. This protocol could require eitherfmore replicas and some timing assumptions (e.g., [10]) orthe same number of replicas and a distributed secure andtimely wormhole [12] (which is already part of the designbut does not implement the required services). Moreover,if the policy engine is statefull, its state needs to be trans-ferred from other replicas after each recovery. State transferis needed because the recovering replica will not receive themessages that pass through the firewall during the recoveryprocess. Additionally, the recovering replica may have beencompromised before the recovery, and it may be necessaryto transfer a clean state from remote replicas. In [10], itis presented an efficient mechanism to perform checkpointsand state transfer, specially tailored for replication mecha-nisms subject to Byzantine faults.

3.7 Deployment Space

The intrusion-tolerant CIS presented in the previous sec-tions can be implemented and deployed in several ways, de-pending on the criticality of the LAN being protected andthe tolerance to unexpected delays of the services deployedin this LAN. Figure 4 presents the main four design options.

In this figure it is possible to identify two main dimen-sions in the design space. If the LAN protected by the CISprovides a very critical service, the implementation mustbe based on the use of different physical replicas for eachCIS replica, allowing tolerance to physical and softwarefaults. However, since price is always a major concern, amuch more cost effective solution can be attained by resort-ing to virtualization. The various replicas are deployed inthe same host, using virtual machines (VM) to isolate thedifferent runtime environments, preventing intrusions frompropagating from one replica to the others. If the serviceprotected is part of an application with hard real-time guar-antees, without tolerance to unexpected delays, there must

10

Criticality

Real−TimeRequirements

Low

High

Softor No

Hard

FPR MPR

VMFR VMMR

Physical Replicas

Virtual Machine

2f+1 2f+k+1

Figure 4. Deployment space for the intrusion-tolerant CIS.

be at least 2f + k+ 1 replicas to ensure safe operation andavailability even in case off faults ank recovering replicas.On the other hand, if occasional delays are not a problem,2 f + 1 replicas suffice to maintain correct operation of thesystem. The four main options are the following:

• VMFR (Virtual Machines with Few Replicas): 2 f + 1replicas are deployed as virtual machines running inthe same host. The system does not tolerate physicalfaults or bugs in the virtual machine monitor, but offerssome level of protection if a virtual machine is subjectto an intrusion;

• VMMR (Virtual Machines with Many Replicas): Thereare 2f +k+1 logical replicas running in different vir-tual machines. The tolerance to malicious behavior isthe same of VMFR but now the system continues toverify and forward messages even withf faulty andkrecovering replicas;

• FPR (Few Physical Replicas): We expect this to be themost common form of deployment, as depicted in Fig-ure 2. There is some tolerance to accidental hardwareand software faults as well as intrusions;

• MPR (Many Physical Replicas): This is similar to FPRbut withk more replicas, to allow availability even withfaulty replicas.

These deployments can be used in different points of thecritical infrastructure to protect different applications [18].For example, a MPR deployment can be used to protect acritical grid defense system (e.g., Automatic Grid Separa-tion System – that performs automatic grid separations inemergency states) while a VMMR implementation can beadequate to protect a substation data historian network.

4 CIS Prototype

We have chosen to implement the VMFR and VMMRdesigns for two main reasons: first, our group already ex-perimented a wormhole construction with different physi-cal replicas [13] but to the best of our knowledge there areno previous experiments with this kind of system deployedin virtual machines to obtain intrusion tolerance; second, atypical critical infrastructure has hundreds of station com-puters and other hosts to be protected, and a solution basedon virtual machines can reduce the deployment cost5.

Our implementation is based on the XEN virtual ma-chine monitor [3] with the Linux operating system. Thearchitecture is presented in Figure 5. A XEN system hasmultiple layers, the lowest and most privileged of which isXEN itself. XEN may host multiple guest operating sys-tems, every one executed within an isolated VM or, in XEN

terminology, adomain. Domains are scheduled by XEN tomake effective use of the available physical resources (e.g.,CPUs). Each guest OS manages its own applications. Thefirst domain,dom0, is created automatically when the sys-tem boots and has special management privileges. Domaindom0builds other domains (dom1, dom2, dom3,...) andmanages their virtual devices. It also performs administra-tive tasks such as suspending and resuming other VMs, andit can be configured to execute with higher priority than theremaining VMs.

� � � � � �

� ��

� �

� � �

� � � � � �

� ��

� �

� �

� � � � � �

� ��

� �

� � �

� � �

�

� � �

� � �

� � � � � � � � � � � � � � � � � � � � � � � � � � � � � �

� � � � � � � � � � � � � � � � � � � � � � � � � �

� � � � � � � � � � � � � � � � � �

Figure 5. VM-based CIS architecture.

As depicted in Figure 5, a centralized version of thewormhole (W) was deployed in domaindom0. This version(implemented in C) emulates then different local worm-holes, and each of these wormholes receives requests and

5For example, the Italian Power System contains about 1000 stationcomputers, thus using three machines and two hubs to protect every oneof these computers would require 3000 and 2000 extra machines and hubs,respectively.

11

sends replies to their corresponding replica. The controlchannel is emulated using shared memory. In the payloadside, each replicai runs on a different domaindomi. Thereplica payload is composed of the policy engine, the CISprotection service (PSi), and a Java Virtual Machine (JVM),given that both policy engine and protection service are im-plemented in Java. In the current prototype, the policy en-gine is very simple, since it just checks if certain strings arepresent in the received packets.

Given that the various payloads and wormhole run indifferent VMs, the communication between them is donethrough a virtual bridge. This bridge emulates a private pipebetween each replica and its corresponding local worm-hole. The two remaining bridges emulate the WAN andthe LAN networks. These two bridges are configured tooperate in broadcast mode, and the LAN bridge authenti-cates the source address of the messages sent by replicas(i.e., replicas can only send messages using their own id)6.This way, the authentication assumption in Section 3.2 issubstantiated.

To simplify the design and to avoid changes in the kernel,the CIS prototype operates at the UDP level, instead of IPlevel as most firewalls do. Therefore, there was no needto implement packet interception because packets are sentdirectly to the CIS. Moreover, authentication is not done atthe IP level (as when using IPSEC/AH), but in alternativethe wormhole calculates the HMAC7 of the payload UDPpacket, and then the two are concatenated. Notice that thistype of authentication implies the same type of overhead ofIP authentication.

The periodic recovery resets the state and restores thecode of a replica. While this is useful to restore the cor-rectness of the replica, it would be interesting if we wereable to introduce diversity in the recovery process. For in-stance, each recovery could randomize the address space ofthe replica (e.g., using PAX8) in order to minimize the use-fulness of the knowledge obtained in the past to increase thechances of future attacks. The XEN and PAX communitiesare currently making efforts to release a joint distribution,and we plan to integrate this mechanism in the prototypewhen it is available.

Domaindom0 executes with higher priority than the re-maining VMs, but since it is based on a normal Linux OS, itprovides no real-time guarantees. Currently, only modifiedversions of Linux and NetBSD can be run ondom0. How-ever, XEN is continuously being improved and we expectthat in the near future one can use a real-time OS.

6In a physical switch, a similar effect can be attained by using thefailopen mode to force broadcast, and the switch’s access control lists toprevent replicas from spoofing their MAC addresses, which can be used asidentifiers.

7HMAC is a standard for calculating MACs, and in the prototype weused the SHA-1 hash function.

8Available athttp://pax.grsecurity.net/ .

5 Experimental Evaluation

We set up a VM-based CIS and connected it to two 100Mbps switches representing the WAN and the LAN. A PCemulated the station computer and, in the WAN side, twoPCs were deployed: agood sender trying to transmit le-gal traffic to the station computer, and amalicioussendercausing a denial of service attack. The CIS executed on adual-processor 2.4 GHz Pentium Xeon PC with 2 GB RAMand XEN 3.0; the good sender a 500 MHz Pentium 3 PCwith 256 MB RAM; the malicious sender a 2.8 GHz Pen-tium 4 PC with 2 GB RAM; the station computer a 1.8 GHzCeleron PC with 512 MB RAM. The malicious sender runLinux 2.6.11, and the others run Linux 2.6.18. The JVMwas Sun JDK 1.5.09.

During the experiments, the malicious sender is con-stantly injecting illegal traffic at a certain rate. The IPERF

tool9 was employed to perform this task. The good sendertransmits legal packets (of 1470 bytes) at a different rate.In the throughput and loss rate experiments, legal packetsare sent at a rate of 124 packets/second and the obtainedresults correspond to the worst case values (lowest through-put and highest loss rate) that were observed. Notice thatthis is a very high traffic for a SCADA system, since it rep-resents, for instance, 124 MMS commands being sent to adevice per second. In the latency experiment, a legal packetis transmitted after the arrival of an acknowledgement of theprevious packet. The displayed results correspond to the av-erage and standard deviation values of 1000 transmissions,after excluding the 5% most distant from the average.

Figure 6 presents the latency, throughput and loss rateof several CIS configurations, when distinct levels of illegaltraffic were being injected in the network. The configura-tions correspond basically to a system with different num-ber of replicas, resulting from various values off andk (seeSections 3.4 and 3.5). Notice that the results for the con-figurations stop at distinct maximum levels of illegal traffic.This happens because after certain traffic levels, the config-urations become unstable, leading to high latencies and/orloss rates.

As expected, the results show that latency and mes-sage loss increases, while throughput decreases, when theamount of illegal traffic directed from the WAN to the LANbecomes larger. There are no pre-established acceptablelimits for these parameters, except for the latency that inpower systems is typically satisfactory if it remains lowerthan 200 milliseconds [18]. In the figure it is possible toobserve that latency stayed way below this limit for veryreasonable levels of traffic.

Comparing the maximum amount of illegal traffic thatthe several configurations can deal with, one can see that

9Available athttp://dast.nlanr.net/Projects/Iperf/ .

12

0 2 4 6 8

10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40 42

0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75

late

ncy

(ms)

illegal traffic (Mbps)

unreplicatedn=2, f=0 and k=1n=3, f=1 and k=0n=4, f=1 and k=1

(a) Latency

0

10

20

30

40

50

60

70

80

90

100

110

120

130

140

150

160

0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95

thro

ughp

ut (

pack

ets/

sec)



(b) Minimum Throughput

0 0.5

1 1.5

2 2.5

3 3.5

4 4.5

5 5.5

6 6.5

7 7.5

8 8.5

9 9.5 10

0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95

loss

rat

e (%

)



(c) Maximum Loss Rate

Figure 6. Experimental evaluation of the CIS prototype.

this threshold is directly proportional to the number of repli-casn. This happens because all replicas share thesamephysical machine and, consequently, the amount of avail-able resources has to be divided. In particular, our exper-iments allowed us to conclude that the amount of memoryallocated for each VM is a critical factor in how much il-legal traffic the CIS supports. For a setup with 4 replicas,resisting 20 Mbps of illegal traffic is a good achievement,specially if we take into account that it is very easy to limitthis traffic at the WAN gateway.

In a setup with more resources, such as one withn phys-ical replicas, we expect that the overall behavior of the CISwill be closer to the non-replicated case in the figure (witha minor degradation due to the distributed implementationof W sign). This means that the physically distributed CISwould maintain acceptable operation even in the presenceof illegal traffic of up to 70 Mbps, which is a value close tothe network capacity, 100 Mbps. This result is even moreinteresting if we consider that the CIS was implemented atapplication level and not inside the kernel.

6 Related Work

Work on intrusion-tolerant serviceshas been fundamen-tally done considering two replication models: Byzantinequorum systems (e.g., [26, 44]) and state machine repli-cation (e.g., [10, 2]). The main difference between theseapproaches is on the way they maintain state consistencyamong the replicas: quorum systems are based on intersec-tions between different quorums (sets of servers) while statemachine replication relies on total order multicast protocolsto ensure that all replicas evolve in the same way. Neither ofthese approaches is used in the CIS algorithms. The repli-cated CIS forwards a message if it is accepted byf +1 repli-cas. Due to this particularity, CIS requires significantly lessreplicas (n≥ 2 f +1) than the usualn≥ 3 f +1 bound.

There was also some previous research on using proac-tive recovery to increase system resilience [44, 10, 35]. Tothe best of our knowledge, we are the first to combine reac-tive and proactive recovery in a single approach.

Some intrusion-tolerant services have been proposed in

13

the past: file/storage systems [17, 10, 27], certification au-thorities [32, 44], and DNS [9]. The only other researchthat we are aware of about intrusion-tolerant firewalls is theprivacy firewall described by Yin et al. [43]. The privacyfirewall was proposed to enforce confidentiality in Byzan-tine services. Since this objective differs from ours, the CISdesign requires much less replicas (n≥ 2 f + 1 instead ofn≥ ( f +1)2) and does not need expensive threshold signa-tures. Most important, the privacy firewall requires mod-ification on both sides of the firewall, to be able to verifyapproved messages, i.e., to verify if incoming messages arecorrectly signed. In addition, the privacy firewall does notrecover replicas. Moreover, from the perspective of the ser-vice provided, the CIS is closer to normal firewalls than tothe privacy firewall.

Regarding thedependability of critical infrastructures,several works talk about how to integrate security in thesesystems. Most of this research either analyzes current secu-rity incidents (e.g., [8]) or focus on the use of traditional se-curity mechanisms (like firewalls) to protect SCADA/PCSsystems (e.g., [6]). However, people have argued that thecritical infrastructures requirements are different from theones in corporate networks, and, more importantly, it hasbeen shown that most firewalls available today are unableto deal with these requirements [7].

There is one project that investigates an advanced pro-tection and communication infrastructure for critical in-frastructures, GRIDSTAT and its successor TCIP (Trustwor-thy Cyber Infrastructure for the Power Grid). This projectadvocates the use of a publish-subscribe infrastructure toprovide secure and QoS-enabled communications betweendifferent control centers in a power grid. Much of the workbeing developed in these projects is orthogonal to the workpresented in this paper and to the CRUTIAL project in gen-eral, since it is focused on QoS, real-time communication,and trust management [20].

Regarding the use of virtualisation technology to con-struct intrusion-tolerant services, the VM-FIT architec-ture proposed in [31] provides basic support for intrusion-tolerant replication of services. Similarly to our VM-basedCIS prototype, VM-FIT uses virtualisation to provide atrusted component (i.e., a wormhole) on each machine.However, the goal of the VM-FIT wormhole is to interceptclient-service interaction and to distribute requests to thereplica group. Overall, the goal of the VM-FIT architectureis conceptually different from the goal of CIS. VM-FIT is ageneric architecture that allows to transform any determin-istic centralized service in an intrusion-tolerant replicatedone, whereas CIS is a specific intrusion-tolerant replicatedservice.

7 Conclusions and Future Work

This paper presented the CIS protection service, imple-mented by highly resilient distributed devices, aimed at pro-tecting critical information infrastructures. The CIS hasthree distinguishing features: it is intrusion-tolerant, it seeksunattended perpetual operation and it supports a rich accesscontrol model that takes into account application semantics.The design of this protection device is based on well-knowtechniques – replication and proactive recovery – combinedand extended in innovative ways, to solve the non-trivialproblem of ensuring transparent intrusion-tolerant opera-tion for standard protocols and components.

Additional results are the combination of reactive andproactive recovery in a single approach, without losingavailability, and the first implementation of an intrusion-tolerant service in a single machine, using logical replica-tion based on virtual machines. Preliminary experimentswere conducted on the VM-based CIS prototype, in order toevaluate its behavior in face of attacks. The results exhibitedan overall good performance in terms of latency, throughputand packet loss rate. Moreover, they helped us to realize thatreplication based on VMs must be used judiciously, since itimplies less resources for each individual replica.

As future work, we expect to advance our understand-ing of the design and implementation of intrusion-tolerantfirewalls, through the evolution of the VM-based prototypepresented in this paper, and the implementation of a distrib-uted version of CIS with physical replicas. Additionally,we will use the protection device presented in this paper asa building block of the CRUTIAL protection framework,which must take into account several other issues such aspolicy dissemination and consistency across different secu-rity domains.

Acknowledgments

We warmly thank Hans P. Reiser and Mario Calha fortheir valuable comments on improving this paper.

References

[1] A. Abou El Kalam, R. E. Baida, P. Balbiani, S. Benfer-hat, F. Cuppens, Y. Deswarte, A. Miege, C. Saurel, andG. Trouessin. Organization Based Access Control. InProc.of 4th IEEE Int. Workshop on Policies for Distributed Sys-tems and Networks – Policy’03, June 2003.

[2] Y. Amir, C. Danilov, D. Dolev, J. Kirsch, J. Lane, C. Nita-Rotaru, J. Olsen, and D. Zage. Scaling byzantine fault-tolerant replication to wide area networks. InProc. Int. Conf.on Dependable Systems and Networks, pages 105–114, June2006.

14

[3] P. Barham, B. Dragovic, K. Fraiser, S. Hand, T. Harris,A. Ho, R. Neugebaurer, I. Pratt, and A. Warfield. Xen andthe art of virtualization. InProc. of the 19th ACM Symp. onOperating Systems Principles - SOSP’03, Oct. 2003.

[4] S. M. Bellovin. Distributed firewalls.;login:, Nov. 1999.[5] G. Bracha. An asynchronousb(n−1)/3c-resilient consen-

sus protocol. InProceedings of the 3rd ACM Symposiumon Principles of Distributed Computing - PODC’84, pages154–162, Aug. 1984.

[6] British Columbia Institute of Technology. NISCC goodpractice guide on firewall deployment for SCADA andprocess control networks. Technical report, National In-frastructure Security Co-ordination Centre, London, UK,Feb. 2005.

[7] E. Byres, D. Hoffman, and N. Kube. The special needs ofSCADA/PCN firewalls: Architectures and test results. InProc. of the 10th IEEE Int. Conf. on Emerging Technologiesand Factory Automation, 2005.

[8] E. Byres and J. Lowe. The myths and facts behind cybersecurity risks for industrial control systems. InProc. of VDEKongress, 2004.

[9] C. Cachin and A. Samar. Secure distributed DNS. InPro-ceedings of the International Conference on DependableSystems and Networks (DSN’04), pages 423–432, 2004.

[10] M. Castro and B. Liskov. Practical Byzantine fault-toleranceand proactive recovery.ACM TOCS, 20(4):398–461, 2002.

[11] M. Cherec, D. Powell, P. Reynier, J.-L. Richier, and J. Vo-iron. Active replication in Delta-4. InProc. 22th Symp.on Fault-Tolerant Computing - FTCS’92, pages 28–37, July1992.

[12] M. Correia, N. F. Neves, and P. Verissimo. How to toleratehalf less one Byzantine nodes in practical distributed sys-tems. InProceedings of the 23rd IEEE Symposium on Reli-able Distributed Systems - SRDS 2004, pages 174–183, Oct.2004.

[13] M. Correia, P. Verissimo, and N. F. Neves. The design of aCOTS real-time distributed security kernel. InProceedingsof the Fourth European Dependable Computing Conference,Toulouse, France, Oct. 2002.

[14] Y. Desmedt and Y. Frankel. Shared generation of authen-ticators and signatures (extended abstract). InProceed-ings of the 12th Annual International Cryptology Confer-ence on Advances in Cryptology - CRYPTO’92, pages 457–469, Aug. 1992.

[15] D. Dzung, M. Naedele, T. P. V. Hoff, and M. Crevatin. Se-curity for industrial communication systems.Proc. of theIEEE, 93(6):1152–1177, 2005.

[16] L. H. Fink and K. Carlsen. Operating under stress and strain.IEEE Spectrum, Mar. 1978.

[17] J. Fraga and D. Powell. A fault- and intrusion-tolerant filesystem. InProceedings of the 3rd International Conferenceon Computer Security, pages 203–218, 1985.

[18] F. Garrone (editor). CRUTIAL: Analysis of new control ap-plications, Jan. 2007. Deliverable D2, EC project IST-2004-27513, http://crutial.cesiricerca.it.

[19] L. A. Gordon, M. P. Loeb, W. Lucyshyn, and R. Richardson.2006 CSI/FBI computer crime and security survey. Com-puter Security Institute, 2006.

[20] C. H. Hauser, D. E. Bakken, I. Dionysiou, K. H. Gjer-mundrød, V. S. Irava, J. Helkey, and A. Bose. Security, trustand QoS in next-generation control and communication forlarge power systems.International Journal of Critical In-frastructures, 2007. to appear.

[21] International Electrotechnical Commission. Inter controlcenter protocol (ICCP). IEC 60870-6 TASE.2, 1997.

[22] International Electrotechnical Commission. Communica-tion networks and systems in substations. IEC 61850, 2003.

[23] S. Kent. IP Authentication Header. RFC 4302 (ProposedStandard), Dec. 2005.

[24] S. Kent and K. Seo. Security Architecture for the InternetProtocol. RFC 4301 (Proposed Standard), Dec. 2005.

[25] V. Madani and D. Novosel. Getting a grip on the grid.IEEESpectrum, 42(12):42–47, Dec. 2005.

[26] D. Malkhi and M. Reiter. Byzantine quorum systems.Dis-tributed Computing, 11(4):203–213, Oct. 1998.

[27] M. A. Marsh and F. B. Schneider. CODEX: A robust andsecure secret distribution system.IEEE Transactions on De-pendable and Secure Computing, 1(1):34–47, Jan. 2004.

[28] R. R. Obelheiro, A. N. Bessani, L. C. Lung, and M. Cor-reia. How practical are intrusion-tolerant distributed sys-tems? DI-FCUL TR 06–15, Dep. of Informatics, Univ. ofLisbon, 2006.

[29] R. Oppliger. Security at the internet layer.IEEE Computer,31(9):43–47, Sept. 1998.

[30] R. Ostrovsky and M. Yung. How to withstand mobile virusattacks (extended abstract). InProc. 10th ACM Symp. onPrinciples of Distributed Computing, pages 51–59, 1991.

[31] H. P. Reiser and R. Kapitza. VM-FIT: Supporting intrusiontolerance with virtualisation technology. InProceedingsof the Workshop on Recent Advances on Intrusion-TolerantSystems, March 2007.

[32] M. K. Reiter, M. K. Franklin, J. B. Lacy, and R. N. Wright.The omega key management service. InProceedings of theACM Conference on Computer and Communications Secu-rity, pages 38–47, 1996.

[33] F. B. Schneider. Implementing fault-tolerant services usingthe state machine aproach: A tutorial.ACM Computing Sur-veys, 22(4):299–319, Dec. 1990.

[34] P. Sousa. Proactive resilience. InSixth European Depend-able Computing Conference (EDCC-6) Supplemental Vol-ume, pages 27–32, Oct. 2006.

[35] P. Sousa, N. F. Neves, A. Lopes, and P. Verissimo. On the re-silience of intrusion-tolerant distributed systems. DI/FCULTR 06–14, Dep. of Informatics, Univ. of Lisbon, Sept 2006.

[36] P. Sousa, N. F. Neves, and P. Verissimo. How resilient aredistributed f fault/intrusion-tolerant systems? InProc. ofInt. Conf. on Dependable Systems and Networks (DSN’05),pages 98–107, June 2005.

[37] B. Sprunt, L. Sha, and J. Lehoczky. Aperiodic task schedul-ing for hard-real-time systems.Real-Time Systems, 1(1),1989.

[38] M. van Eeten, E. Roe, P. Schulman, and M. de Bruijne. Theenemy within: System complexity and organizational sur-prises. In M. Dunn and V. Mauer, editors,Int. CIIP Hand-book 2006, volume II, pages 89–110. CSS, ETH Zurich,2006.

15

[39] P. Verissimo. Travelling through wormholes: a new look atdistributed systems models.SIGACT News, 37(1), 2006,http://www.navigators.di.fc.ul.pt/docs/abstracts/ver06travel.html.

[40] P. Verissimo, N. F. Neves, and M. Correia. CRUTIAL: Theblueprint of a reference critical information infrastructure ar-chitecture. InProc. of CRITIS’06 1st Int. Workshop on Crit-ical Information Infrastructures Security, Aug. 2006.

[41] P. Verissimo, N. F. Neves, and M. P. Correia. Intrusion-tolerant architectures: Concepts and design. InArchitectingDependable Systems, volume 2677 ofLNCS. 2003.

[42] C. Wilson. Terrorist capabilities for cyber-attack. InM. Dunn and V. Mauer, editors,Int. CIIP Handbook 2006,volume II, pages 69–88. CSS, ETH Zurich, 2006.

[43] J. Yin, J.-P. Martin, A. Venkataramani, L. Alvisi, andM. Dahlin. Separating agreement form execution for Byzan-tine fault tolerant services. InProc. 19th ACM Symp. onOperating Systems Principles - SOSP’03, pages 253–267,2003.

[44] L. Zhou, F. Schneider, and R. Van Rennesse. COCA: Asecure distributed online certification authority.ACM TOCS,20(4):329–368, Nov. 2002.

A Correctness of the Payload Protocol

This appendix proves that the CIS message processingprotocol satisfies the conditions defined in Section 3.3, as-suming the correctness of the wormhole protocols, whichwe prove in Appendix B. The proof starts with a lemmashowing that the retransmission mechanism defined for theprotocol ensures that a received message eventually will besigned.

Lemma 1 In Algorithm 1, if a legal message is receivedby one correct CIS replica that is not recovered for longenough, it will eventually be signed.

Proof (sketch):When a legal messagem is received by acorrect CIS replica, it will be verified by the policy engine(line 1), stored in theVotingset (line 2) and then passed tothe wormhole for signing (line 3). Knowing that theW signservice of the wormhole only returns at some replica if itis called by at leastf + 1 replicas, we will show that thiscall eventually returns. By the algorithm, if a replica staysmore thanTvote blocked at line 3, the retransmission of line19 will be triggered periodically (at eachTvote time units)until Voting= ∅ (which happens only if the replica executesline 4, after the wormhole signs the message). Since weassume that the unreliable multicast is fair (see Section 3.2),if a replica retransmitsmmany times, eventually all correctreplicas will receive it. Given thatn≥ 2 f + 1, there willbe alwaysf +1 correct replicas that will receivemand callW sign(m). When this happens, the wormhole will producethe signature and return it to the payload. �

Now we present theorems for the Validity and Integrityproperties.

Theorem 1 (Validity) Algorithm 1 ensures that a legalmessage received by at least one correct CIS replica thatis not recovered for long enough, is forwarded to its desti-nation.

Proof (sketch):By Lemma 1, we know that a messagem re-ceived by some correct replica will be signed. Since a mes-sage is signed only iff +1 processes approved it, we knowthat at least one correct replica that signedm will store iton thePendingset. The message will stay in this set untilit is received in the LAN, i.e. it was forwarded (lines 12-14). While stored in this setmwill be retransmitted period-ically (line 19). AfterOt retransmissions, the current leaderreplica will be suspected (line 22). Due to the fair broad-cast assumption, all other correct replicas eventually willreceivem and approve, sign and store it in theirPendingsets. Consequently, if the message was not forwarded bythe leaderl , eventually f + 1 replicas will suspect it, andthe wormhole will change the leader replica. This proce-dure of leader change can happen at mostf times until a

16

correct replica becomes the leader, forwardsm, and storesin its Pendingbuffer (lines 23-29).

The notion of a replica not being recovered for longenough should now be clear. Suppose only one correctreplicaCISi receives the message. In that case,CISi cannot be recovered until other correct replicas, that also arenot recovered for long enough, receive it. Notice, however,that the normal is for all correct replicas to receive the samemessages, since the traffic replication device multicasts themessages to all replicas. �

Theorem 2 (Integrity) Algorithm 1 ensures that an illegalmessage is never processed by its destination machine.

Proof (sketch):Recall that(i.) in our system model, the des-tination machine can not be corrupted and it only processesmessages signed with keyK; and(ii.) the keyK is stored inthe wormhole, and a messagem is only signed with this keyif at least f + 1 CIS replicas callW sign(m). Given thesetwo facts, it is almost direct to see that illegal messages (ap-proved by at mostf processes) can not be signed (due to(ii.)) and, would not be processed by their destination ma-chines even if forwarded by a malicious leader (due to(i.)).Consequently, an illegal message will never be processedby its destination machine. �

B Correctness of the Wormhole Protocols

In this appendix we prove the correctness of the worm-hole protocols.

Theorem 3 (Signature) Algorithm 2 ensures that theW sign(m) service returns a MAC produced with the sharedkey K for message m, if and only if at least f+1 CIS repli-cas called it in their local wormholes.

Proof (sketch):When the serviceW sign(m) is called at alocal wormhole, it blocks until receiving a VOTE messagewith the correct MACσ (produced with the shared keyKfor messagem) from at leastf +1 local wormholes (line 3).A VOTE message with the correct MACσ is sent by a localwormhole if and only ifW sign(m) is called (lines 1 and 2),since we assume collision-resistant MACs. Consequently,the W sign(m) service returns a MAC produced with theshared keyK for messagem, if and only if at leastf + 1CIS replicas called it in their local wormholes. �

Theorem 4 (Leader Change) Algorithms 2 and 3 ensurethat a leader i is changed if at least f+1 CIS replicas calleither W suspectsilent(i) or W suspectmalicious(i).

Proof (sketch):Consider that replicai is the leader and thatat leastf +1 CIS replicas call eitherW suspectsilent(i) orW suspectmalicious(i). When these services are called, a

SUSPECT-SILENT or SUSPECT-MALICIOUS message issent to replicai. Therefore, if the services are called by atleast f +1 replicas, then at leastf +1 suspect messages aresent with different sender ids (Algorithm 2, lines 8 and 9).The rest of the proof depends on the specific suspect servicebeing called.

1. If at least f + 1 replicas callW suspectmalicious(i),then at leastf +1 different ids will be added to the setSuspectm of replicai (Algorithm 2, line 11). This im-mediately triggers the recovery of replicai (Algorithm3, line 8). Given that replicai is the current leader, anew leader is elected (Algorithm 3, lines 14-15).

2. If at least f + 1 replicas call eitherW suspectmalicious(i) or W suspectsilent(i),then at leastf + 1 different ids will be added to theunion of the setsSuspectm and Suspects of replicai (Algorithm 2, lines 10-11). This triggers theallocation of a slot and a subslot to perform a sporadicrecovery (Algorithm 3, line 9). If the allocated slotcorresponds to the one of replicai, then it means thatreplica i will be recovered through periodic proactiverecovery. Otherwise, replicai will be recovered in theallocated slot. �

Theorem 5 (Number of Recovering Replicas) Algo-rithms 1, 2 and 3 ensure that at most one correct replicais recovering at any time.

Proof (sketch):Consider a replicai. A recovery can only betriggered due to one of the following three reasons:

1. periodic proactive recovery (Algorithm 3, line 6);

2. reactive recovery because replicai is compromised(Algorithm 3, line 8);

3. reactive recovery because replicai is suspected to besilent (Algorithm 3, line 12).

We have to show that if replicai is recovered due to rea-sons 1 or 3, then it is guaranteed that no other correct replicais recovering at the same time. If replicai is recovered dueto reason 2 then it is compromised, and thus nothing has tobe proved.

If replica i is recovered due to a periodic proactive recov-ery (reason 1), then the proactive recovery algorithm (Al-gorithm 3, lines 1–7) guarantees that replicai always startrecoveriesTslot time units after the previous periodic recov-ery. Given thatTslot ≥ TD and that it is assumed thanTD

is the maximum recovery duration time, then it is ensuredthat when a replicai starts a periodic recover, the previ-ous periodic recovery has already finished. Moreover, thereactive recovery algorithm (Algorithm 3, lines 9–13) guar-antees that replicai always start recoveries at leastTD time

17

units after the previous reactive recovery. Therefore, it isensured that when a replicai starts a proactive recovery, theprevious proactive and reactive recoveries have already fin-ished.

If replica i is recovered due to a reactive recovery be-cause it is suspected to be silent (reason 3), then the reac-tive recovery algorithm (Algorithm 3, lines 9–13) guaran-tees that replicai always start recoveries at leastTD timeunits after the previous periodic recovery. Given thatTD

is the maximum recovery duration time, then it is ensuredthat when a replicai starts a reactive recovery of this nature,the previous periodic recovery has already finished. More-over, the reactive recovery algorithm guarantees that replicai always start recoveries at leastTD time units after the pre-vious reactive recovery. Therefore, it is ensured that whena replicai starts a reactive recovery because it is suspectedto be silent, the previous proactive and reactive recoverieshave already finished. �

Theorem 6 The CIS protection service is exhaustion-safe(i.e., at most f replicas are failed at the same time) if it ispossible to lower-bound the time needed for an attacker toproduce f+1 replica failures by a known constant Texhmin,and TP +TD < Texhmin.

Proof (sketch):In addition toTP andTD, the Proactive Re-silience Model (PRM) [35] defines a third parameterTπ ,but it only has meaning if the recovery is a distributed pro-cedure (with a set of nodes recovering at the same time)and therefore it does not apply here given that CIS replicasrecover one at a time. As shown in [35], if it is possibleto lower-bound the time needed for an attacker to producef +1 replica failures by a known constantTexhmin, then it isguaranteed that no more thanf replicas will be faulty at thesame time as long asTP +TD < Texhmin. �

18

Intrusion-Tolerant Protection for Critical Infrastructures

Documents

Transcript of Intrusion-Tolerant Protection for Critical Infrastructures