An Arbitrary Tree-Structured Replica Control Protocol

Jean Paul Bahsoun, Robert BasmadjianUniversite de Toulouse

Institut de Recherche en Informatique deToulouse ( I.R.I.T )

{bahsoun,basmadji}@irit.fr

Rachid GuerraouiEcole Polytechnique Federale de Lausanne

Distributed Programming Laboratory( LPD )

rachid.guerraoui@epfl.ch

Abstract

Traditional replication protocols that arrange logicallythe replicas into a tree structure have reasonable availabil-ity, low communication costs but induce high system load.

We propose in this paper the arbitrary protocol: a tree-based replica control protocol that can be configured basedon the frequencies of read and write operations in orderto provide lower system load than existing tree replicationprotocols, yet with comparable cost and availability. Ourprotocol enables the shifting from one configuration into an-other by just modifying the structure of the tree. There is noneed to implement a new protocol whenever the frequenciesof read and write operations change.

At the heart of our protocol lies the new idea of logi-cal and physical levels in a tree. In short, read operationsare carried out on any physical node of every physical levelof the tree whereas the write operation is performed on allphysical nodes of a single physical level of the tree.

We discuss optimal configurations, proving in particulara new lower bound, of independent interest, for the case ofa binary tree.

1. Introduction

In large distributed systems, data is replicated in orderto achieve fault-tolerance as well as improve system per-formance. However, underlying synchronization protocols,also known as replica control protocols, are required in or-der to maintain data consistency among the replicas. Theseprotocols basically implement two operations: read andwrite. When one-copy equivalence is to be ensured, whichis the case we consider in this paper, the write operationmust store a new value of the data whereas the read opera-tion must return the most recent value written.

Given the importance of the topic, several replicationprotocols have been described in the literature. They dif-fer according to various parameters such as their commu-

nication costs, their ability to tolerate replica failures (alsotermed as their availability), as well as the load they imposeon the system. For a system of n replicas, the well knownReadOneWriteAll (ROWA) [3] protocol has a read cost of 1and a write cost of n. That is, a client needs only to contacta single replica to read a value, whereas it needs to contactall (n) replicas to write a value. Read operations are highlyfault-tolerant and induce a load of 1

n whereas the availabil-ity of write operations is penalized due to the fact that thecrash of a single replica prevents the write operations fromterminating successfully; they induce a system load of 1.

The Majority Quorum protocol [13] has both read andwrite communication costs of n+1

2 for an odd-sized num-ber of replicas n and imposes a system load of at least 0.5.It tolerates replica failures for read and write operations atthe expense of increased read costs with respect to those ofROWA. Both ROWA and Majority Quorum protocols havea communication cost of O(n): the cost thus increases lin-early with the number of replicas in the system.

By imposing a logical structure on the replicas of thesystem, it is possible to reduce the communication costs fur-ther. The

√n protocol (finite projective plane FPP) [9], the

Grid protocol [4], the Tree Quorum protocol [2], the HQCprotocol [8] and the Paths system [10] arrange logically thereplicas of the system within a specific structure while, justlike the Majority Quorum protocol, still making use of aquorum system. The load of these protocols was studiedin [10] using linear programming. It was proven that thebest optimal load of a quorum system of n replicas is 1√

n

if the smallest quorum of the system is of size√

n. It wasalso established that the load of the system becomes higheras much as the size of the smallest quorum of the systembecomes less than

√n.

The logical organization of n replicas of the system intocertain structures such as a finite projective plane or a gridmakes the smallest quorum become of size

√n resulting in

the best optimal load of 1√n

. The tree structures providemuch smaller quorums of size log(n) but at the expense ofvery high system loads.

The 28th International Conference on Distributed Computing Systems

1063-6927/08 $25.00 © 2008 IEEEDOI 10.1109/ICDCS.2008.51

500

The 28th International Conference on Distributed Computing Systems

1063-6927/08 $25.00 © 2008 IEEEDOI 10.1109/ICDCS.2008.51

502

The motivation of this paper is to ask whether it is possi-ble for a tree-based replica control protocol to induce a lowsystem load with low communication costs and acceptableavailability.

In a tree protocol, every node of the tree represents areplica of the system: the height of the tree is typically de-noted by h whereas n denotes the number of replicas. Oneof the first tree-based protocols [1] has a read cost that varies

from 1 to (d+1)h and a write cost of (d+1)h+1−1d where each

node of the tree has 2d+1 descendants (for a non negativevalue of d). In [7], the the idea of [1] was generalized toderive a protocol with a read cost that ranges from 1 to Sh

whereas the write cost is of O(log(n)) where every node ofthe tree has S = 3 descendants. In [5], the proposed protocolrequires a much smaller read costs than in [7] where such aprotocol has a read cost that ranges from 1 to S

h2 whereas

the write operation has a cost of O(log(n)) where everynode of the tree has S = 3 descendants. All these protocolshave a read cost of 1 in the best case. However such a costis achieved by accessing only the root of the tree: thereforethe read operations of [1] and [7] having a cost of 1 induce asystem load of 1 whereas those of [5] induce a system loadof 0.5. Also, all these protocols are vulnerable to the rootcrashing; in this case, no more write operations can be ac-complished and these write operations impose a system loadof 1 because the root is a member of every write quorum.

The drawback of not completing any write operationwhen the root of the tree crashes was solved in [2]. A read orwrite operation is performed by selecting a path that startsfrom the root of the tree and ends up at one of the leaves.If such a path cannot be constructed due to some failuresthen any inaccessible replica is replaced by paths that startfrom all of its descendants and end up at their correspondingleaves. This protocol has thus a read or write cost that variesfrom log n to n+1

2 . For efficiency purposes, the authors con-sidered the case of a binary tree. It was shown in [10] thatthe optimal system load of [2] is 2

h+2 when performing reador write operations. However, this optimal load is achievedwith a communication cost much higher than log n. A com-munication cost (or a quorum size) of log n can be achievedby taking a path that starts from the root of the tree and endsup at one of the leaves. Such a quorum size of log n inducesan optimal system load of 1 since every such quorum hasthe root of the tree as its member and any strategy of pick-ing quorums of this size picks the root.

In Hierarchical Quorum Consensus (HQC) [8], a nodedoes not necessarily correspond to a replica. The protocolis based on the Quorum Consensus approach and organizesthe replicas of the system into a logical, multilevel hierarchywhere only the leaf nodes of the tree represent replicas ofthe system. For efficiency purposes, the case of a ternarytree is considered and is proved that the size of the quorumsof such a system is n0.63 if the size of a quorum at each

level is 2. It was shown in [10] that the optimal system loadof HQC is n−0.37which is higher than the best load of 1√

n.

We propose in this paper a protocol, which we call thearbitrary protocol, and which somehow generalizes the pro-tocol of [8]. As in the latter, we also organize logically thereplicas into a tree structure and every node of this tree canbe either logical or physical: a physical node correspondsto a replica of the system whereas a logical node does not.At the heart of our protocol lies the new notions of physicallevel which contains at least one physical node and logicallevel which has all of its nodes logical. Basically, the readoperation is carried out on any single physical node at ev-ery physical level of the tree whereas the write operation isperformed on all physical nodes of a single physical level ofthe tree. The structure of this tree can be configured basedon the frequencies of read and write operations. If the writeoperations dominate in the system, then we add as muchphysical levels to the tree as possible. On the other hand,if the system is mostly-read, then all the replicas of the sys-tem are arranged into one physical level. We propose analgorithm that configures the tree structure by taking intoaccount both read and write frequencies. We show that thewrite operations need only induce a system load of 1√

nwith

a communication costs of√

n, which is lower than exist-ing tree replication protocols. Yet we preserve comparablewrite availability. On the other hand, we show that the readoperations need only induce a cost of

√n which is lower

than the existing tree replication protocols with comparablesystem load and availability. We prove a new lower boundof the system load of the binary tree structure of [2] andshow that the write operations of our protocol induce a loadof 1

log(n+1) which is lower than the one proved in [10] of2

log(n+1)+1 . Hence our approach provides a “spectrum” al-gorithm which can be tuned to suit the read-write ratio inthe system.

The rest of the paper is organized as follows. We startby introducing few notations and give our representation ofa tree structure using these notations. Then we detail howread and write operations of our protocol are performed andgive for each operation its corresponding communicationcost, availability and system load. We discuss the trade-offsin constructing the tree structure based on the frequenciesof read and write operations.

2. Preliminaries

2.1. Definitions, notations and propositions

Most of the definitions, propositions and notations of thissection are inspired by [10].

DEFINITION 2.1 A Set system S = { S1, S2, . . . , Sm }is a collection of subsets Si ⊆ U of a finite universe U. A

501503

Quorum system is a set system S that has the intersectionproperty: S∩R �= ∅ for all S,R ∈ S. The sets of the systemare called quorums.

DEFINITION 2.2 A coterie is a quorum system S that hasthe minimality property: there are no S,R ∈ S; S ⊂ R .

DEFINITION 2.3 A bi-coterie is a system S that hasa separate set of read quorums R = { R1, R2,. . . ,Rm }where Ri ⊆ U and a separate set of write quo-rums W = { W1, W2, . . . ,Wm } where Wi ⊆ U , such thatevery read quorum intersects every write quorum,i.e. R ∩ W �= ∅ ∀R ∈ R and ∀W ∈ W.

DEFINITION 2.4 Consider a distributed systemS = { S1, S2, . . . , Sm } over a finite universe U. Wesay that w ∈ [0, 1]m is a strategy for S if it is a probability

distribution over the subsets Sj ∈ S, i.e.m∑

j=1

wj = 1.

DEFINITION 2.5 Let a strategy w be given for a systemS = { S1, S2, . . . , Sm } over a finite universe U of size n.For a replica of the system i ∈ U , the load induced by w on

i is lw(i) =∑

i∈Sj

wj . The load induced by a strategy w on a

system S is:

Lw(S) = max lw(i) ∀i ∈ U.

The system load on a system S is given by:

L(S) = minLw(S) ∀w ,

where the minimum is taken over all strategies w.

NOTATION 2.1 Let n denote the number of replicas of thesystem and let h denote the height of the tree.

NOTATION 2.2 For a vector y ∈ [0, 1]n and a set S ⊆ U ,

let y(S) =∑

i∈S

yi.

NOTATION 2.3 For a set of write quorums W, let m(W)denote the number of write quorums of W. For a set of readquorums R, let m(R) denote the number of read quorumsof R.

PROPOSITION 2.1 Let a system S be given, and let w bea strategy for S with an induced load of Lw(S)= L. ThenL is the optimal load iff there exists: y ∈ [0, 1]n such thaty(U) = 1 and y(S)≥ L ∀S ∈ S .

2.2. System model

A distributed system consists of a set of distinct sites thatcommunicate by exchanging messages through communi-cation links. A site consists of a processing unit and a stor-age device and has a unique SID whereas a communicationlink is a bidirectional communication medium between twosites. Sites can fail by stopping, as a result of the crash of acritical subsystem, or they can fail by performing arbitraryor malicious actions. We do not deal with such Byzantinefailures [11] in this paper. We assume that sites fail inde-pendently with the same probability q. Moreover they arefail-stop and failures are transient and detectable. Links canfail either by not delivering messages at all to their desireddestination, or by dropping or excessively delaying someof them. We also consider a special case of site and linkfailures that lead to the partitioning of the system whereonly sites in the same partition can communicate with eachother. No assumption is made on the underlying topologyof the network. We assume that sites of the system can belogically organized into a tree structure. Hence, we con-sider basic tree terminologies such as root, children, parent,leaves, height, node and level. Each client uses a centralizedconcurrency control scheme to synchronize accesses to thereplicas. We assume that users interact with sites by meansof transactions which are partially ordered set of read andwrite operations. Moreover, we assume that transactionsare executed atomically, i.e. a transaction either commitsor aborts at all participating sites. If a transaction containswrite operations, a 2-phase-commit protocol at the end ofthe transaction is executed among all sites. Finally, we con-sider timestamps that consist of a version number and anSID which are used for read and write operations.

3. Our protocol

3.1. Notations and tree representation

Given a distributed system of n replicas, we organizethem logically into a tree structure of height h where anynon-leaf node of this tree can have any number of descen-dants. More precisely, let S(i,k) denote the ith node of thekth level of the tree where the orientation is taken from leftto right and top to bottom respectively such that i ∈ [1,mk]and k ∈ [0, h]. Let mk denote the total number of nodes at

level k such that mk =mk−1∑

i=1

m(i, k − 1) ∀k; k ∈ [1, h] and

m0 = 1. Let m(i, k) denote the number of descendants ofa non-leaf node S(i,k), where i and k have the same defini-tions as above, such that m(i, h) = 0 ∀i; i ∈ [1,mh].

A node of the tree is said to be logical if it doesnot represent a replica; otherwise the node is said

502504

to be physical. Let S log(i,k) and Sphy(i,k) denote alogical and a physical node of the tree respectively.Let mlog(i,k) and mphy(i,k) denote respectively the num-ber of logical and physical descendants of a non-leafnode S(i,k) such that m(i,k) = mlog(i,k) + mphy(i,k) , wherem(i,h) = mlog(i,h) = mphy(i,h) = 0 ∀i; i ∈ [1,mh]. Thenthe total number of physical nodes at level k is denoted

by: mphy k =mk−1∑

i=1

mphy(i, k − 1) ∀k; k ∈ [1, h] and mphy 0 = 1

if the root of the tree is a physical node. On the otherhand, the total number of logical nodes at level k is denoted

by: mlog k =mk−1∑

i=1

mlog (i, k − 1) ∀k; k ∈ [1, h] and mlog 0 = 1

if the root of the tree is a logical node. Hence we havemk = mphy k + mlog k ∀k; k ∈ [0, h] such that m0 = 1 because

the root of the tree is either a physical or a logical node.A level of the tree is called physical if it contains at least

one physical node. Let Kphy denote the set of all physicallevels of the tree where |Kphy| denotes the number of physi-cal levels such that 1 ≤ |Kphy| ≤ h + 1. Let Kphy[u] indicatethe level number of the physical level at index u of Kphy

such that u ∈ [0,|Kphy| – 1]. We assume that the physicallevels of Kphy are sorted in ascending order of their levelnumbers. On the other hand, we call a level of the treelogical if all of its nodes are logical, i.e. mk = mlog k andmphy k = ∅. Let Klog denote the set of all logical levels of thetree and let |Klog| denote the number of such logical levelswhere 0 ≤ |Klog| ≤ h and |Klog| + |Kphy| = 1 + h .

ASSUMPTION 3.1 We assume that:mphy 0 < mphy 1 ≤ mphy 2 ≤ . . . mphy h such that the total number

of replicas of the system is given by: n =∑

k∈Kphy

mphy k

Figure 1. A tree whose purple circles repre-sent logical nodes and the blue ones repre-sent physical nodes. Such a tree has one log-ical level (0) and two physical levels (1 and 2).

3.2. The operations

Our system is a bicoterie where the set of read quorumsR and the set of write quorums W are constructed basedon definition 2.3. Furthermore, we assign separate strate-gies of picking read and write quorums using definition2.4. More precisely, let wread denote a strategy for pick-

ing read quorums of R such that wread =m(R)∑

j=1

wread j and

let wwrite =m(W)∑

j=1

wwrite j denote a strategy for picking write

quorums of W where wread j, wwrite j, m(R), and m(W)are defined in next sections. The availability computa-tions are carried out by taking the assumption that ev-ery replica is independently available with a probabilityp = 1 – q greater than 1

2 : it was proven in [12] that if thereplicas are fail-prone, with an individual availability prob-ability less than 1

2 , then the best strategy is not to repli-cate and to pick a single centralized king. In the rest ofthis section, we use d to denote the minimal number ofphysical nodes of the physical levels of the tree such thatd = min {mphy k ∀k; k ∈ Kphy}, and use e to denote the max-imal number of physical nodes of the physical levels ofthe tree such that e = max {mphy k ∀k; k ∈ Kphy}. Next, wedemonstrate how we construct the read and write quorumsand give the communication cost, availability and systemload of read and write operations of our protocol.

3.2.1 The read operation

A read operation takes place by accessing all the mem-bers of a read quorum Rj ∈ R and retrieves the value ofdata whose timestamp has the highest version number andthe lowest site identifier. A read quorum Rj is constructedby taking as its members any physical node of every physi-cal level of the tree:i.e. Rj = {Sphy(i,k) ∀k; k ∈ Kphy such that ∃i; i ∈ [1,mk] }.

FACT 3.2.1 Let R = { R1, R2,. . . ,Rj } be the set of readquorums such that every read quorum Rj is constructed inthe same manner as explained in the previous paragraph.

Then m(R) =∏

k∈Kphy

mphy k .

In order to compute the load of the system induced bythis read operation, we take a strategy wread that picks eachread quorum Rj with a probability wread j = 1

m(R) where

j ∈ [1,m(R)].The read operation of our protocol has a communica-

tion cost of RDcost = 1 + h – |Klog|, an availability of

RDavailability (p) =∏

k∈Kphy

(1 − (1 − p)mphy k ) and imposes

503505

an optimal system load of LRD = 1d . It is important to note

that as much as d increases, as much as the load imposed bythe read operations on the system and communication costdiminish and the availability increases.

3.2.2 The write operation

A write operation, after obtaining the highest versionnumber of data and incrementing it by one, accesses all themembers of a write quorum Wj ∈ W in order to updatetheir data with a new value and timestamp. A write quorumWj is constructed by taking as its members all the physicalnodes of any single physical level of the tree:i.e. Wj = {Sphy(i,k) ∃k; k ∈ Kphy such that ∀i; i ∈ [1,mk] }.

FACT 3.2.2 Let W = { W1, W2,. . . ,Wj } be the set of readquorums such that every read quorum Wj is constructed inthe same manner as explained in the previous paragraph.Then m(W) = 1 + h – |Klog|.

In order to compute the load of the system induced bythis write operation, we take a strategy wwrite that pickseach write quorum of our system Wj with a probabilitywwrite j = 1

m(W) where j ∈ [1,m(W)].The write operation of our protocol thus has a min-

imum cost of d, a maximum cost of e and an average

cost of WRcost =∑

k∈Kphy

mphy k * wwrite j. Hence such a

strategy wwrite of picking write quorums induces a com-munication cost of n

1+h−|Klog| . This operation has an

availability of WRavailability (p) = 1 – WRfail (p) where

WRfail (p) =∏

k∈Kphy

(1 − pmphy k ) and it imposes an optimal

system load of LWR = 11+h−|Klog| . It is important to note

that as much as the number of physical levels of the treeincreases, as much as the load imposed by the write oper-ation on the system and communication cost diminish andthe availability increases.

3.2.3 Discussion

The proofs of the optimality of system load induced bythe operations of our protocol are provided in the appendixof this paper. The optimal system loads are computed by as-suming that all replicas of the system are functioning prop-erly. Therefore, the load of the system imposed by the op-erations becomes higher as the replicas of the system startto fail one after another. In order to compute the expectedload knowing that replicas are available with a probabilityp, we use the following two equations:EQUATION 3.2ELRD = RDavailability(p) ∗ (LRD−1) + 1ELWR = WRavailability(p) ∗ LWR + WRfail(p) ∗ 1.

Therefore, we can notice from these two equations thatas much as the availability of the operations is high, as muchas the expected load is close to the computed system loadinduced by the operations. We call a system that has thischaracteristic stable.

Next, we demonstrate that our system is a bicoterie i.e.any read quorum has a non empty intersection with anywrite quorum. The read and write quorums of our proto-col are of the following form:Read Quorum = any replica of every physical levelWrite Quorum = all replicas of any physical level

The proof is by induction on the number of physical lev-els of the tree:Basis: Trivial for a tree of one physical level because all thereplicas of the system are found at the same level.Induction hypothesis: Assume that it holds for a tree of hphysical levels: |Kphy| = h .Induction Step: Consider a tree of h + 1 physical levels.Since every read quorum already had an intersection withevery write quorum of h physical levels (induction hypoth-esis step) then it holds true because the fact of adding onenew physical level does not prevent the read quorums tohave a non empty intersection with any write quorum ofh physical levels. On the other hand, since the read quo-rums contain a replica from the new physical level and thenew write quorum contains all the replicas of this same newphysical level, then any read quorum has a non-empty in-tersection with this write quorum. Hence, by induction,our protocol guarantees non-empty intersection of read andwrite quorums.

3.3. The tree organization and the trade-offs

Given a distributed system of n replicas, we organizethem logically into a tree structure as explained in section3.1 . However, this tree structure must be configured in sucha way that it takes into account the frequencies of read andwrite operations of the system. For instance, as much as thenumber of physical levels of the tree increases, as much asthe communication cost and the system load of the write op-erations diminish and the availability of the write operationsbecomes better. However, the fact of adding more physicallevels to the tree results in increasing the communicationcost and the system load of the read operations and in dete-riorating their availability. Therefore, such a configurationis suitable for systems where write operations dominates theread ones. On the other hand, decreasing the number ofphysical levels of the tree to the minimal (only one physicallevel) diminishes the communication cost and the systemload of the read operations and improves their availability.Therefore, such a configuration is appropriate for systemswhose operations are mostly-read and such a configuration

504506

behaves like ROWA. Finally, if both read and write opera-tions happen in proportional frequencies, then the tree mustbe configured in such a way that both operations provideacceptable results. For a distributed system of n replicaswhere every replica is available independently with a prob-ability p such that n > 32 and p > 0.65, the following or-ganization of replicas always gives us satisfactory resultsfor both read and write operations: mlog 0 = 1, mphy 1 = 4,mphy 2 = 4, mphy 3 = 4, mphy 4 = 4, mphy 5 = 4, mphy 6 = 4 andmphy 7 = 4 whereas the remaining (n – 28) replicas of the

system can be added to the succeeding physical levels ofthe tree in such a way that they obey the assumption 3.1.Next we give an algorithm to construct the arbitrary treewhen n > 64:

Algorithm 1 An algorithm to construct the arbitrary tree1- Construct the arbitrary tree of a logical root node by fix-ing |Kphy| =

√n and h = |Kphy| .

2- Arrange 4 replicas at the 1st seven physical levels of thetree.3- Arrange n−28√

n−7replicas at every remaining physical level

of the tree by obeying the assumption 3.1.

Therefore, when using Algorithm 1 to configure the tree,our protocol behaves in the following manner: the writeoperation has a minimum cost of 4, a maximum cost ofn−28√

n−7and an average cost of

√n. It has a failure probabil-

ity of WRfail (p) = (1 − p4)7 ∗ (1 − p( n−28√

n−7 ))(√

n−7)

and imposes an optimal system load of 1√n

.On the other hand, the read operation has acommunication cost of

√n, an availability of

RDavailability(p) = (1 − (1 − p)4)7 ∗ (1 − (1 − p)(n−28√

n−7 ))x

where x =√

n− 7, and imposes an optimal system load of0.25. In order to study the behavior of the availability ofthe operations for very large number of replicas n such that0.5 < p <1, we compute:lim

n→∞WRavailability(p) = 1 − limn→∞WRfail (p) =

1−(1−p4)7 and limn→∞RDavailability(p) = (1−(1−p)4)7.

We can notice from these two equations that when p > 0.8,both operations have an availability of ≈ 1.

Finally, if we assume that our protocol is applied directlyto a tree structure where every node of the tree has thesame number of descendants and represents a replica of thesystem, then the write operations of our protocol imposean optimal load of 1

log(n+1) and these write operationsare highly available (always greater than the availabilityprobability p) and have an average cost of n

log(n+1) whereasthe read operations of our protocol impose the highestload of 1 to the system and these operations are poorlyavailable (always less than the availability probability p)but has a communication cost of log (n + 1). Furthermore,

mk mphy k mlog k

m0 = 1 mphy 0 = 0 mlog 0 = 1m1 = 3 mphy 1 = 3 mlog 1 = 0m2 = 9 mphy 2 = 5 mlog 2 = 4

Table 1. The total, physical and logical nodesof every level of the tree of Figure 1.

it was proven in [10] that the optimal system load imposedby read or write operations of [2] is 2

log(n+1)+1 which ishigher than the system load imposed by the write operationof our protocol. Therefore, we provide a new lower boundon the load of the tree structure of [2] imposed by writeoperations of our protocol.

3.4. An example

In this section, we give an example of a distributed sys-tem of 8 replicas which are organized logically into a treestructure as illustrated in Figure 1. We can notice that, sucha tree has a height of 2, one logical level at zero, and twophysical levels (1 and 2). Table 1 gives the correspondingnumbers of total, physical and logical nodes of every levelof the tree using the notations of section 3.1. Then we have:

• n = mphy 1 + mphy 2 = 3 + 5 = 8 such that this tree organi-zation obeys to the assumption 3.1

• Kphy = {1,2}, |Kphy| = 2 and Klog = {0}, |Klog| = 1

• m(R) = mphy 1*mphy 2 = 3*5 = 15 and m(W) = 2

Using the definitions of sections 3.2.1 and 3.2.2 we obtain:

• RDcost = 2, RDavailability (0.7) = 0.97 and LRD = 13

• WRcost = 4, WRavailability (0.7) = 0.45 and LWR = 12

• ELRD = 0.35 and ELWR = 0.775

In the rest of this paper, we represent such an arbitrary treein the following manner: 1–3–5 where the numbers 3 and 5denote the number of physical nodes at each physical levelof the tree and “1” denotes a logical root node.

4. Configurations

In this section, we compare the communication costs aswell as the system and expected system loads of read andwrite operations by setting up six different configurations.In the first one, which we call “BINARY”, we take the caseof a binary tree of [2] such that the computations of the loadinduced by read and write operations are based on the re-sults of section 6.3 of [10]. The communication costs and

505507

availability of the operations are based on the results of sec-tion 4 of [2] where the communication costs are computedby giving f a value of 2

2+h where h is the height of the bi-nary tree and f indicates the fraction of quorums that includethe root of a tree of level l+1. In the second case, we ap-ply the read and write operations of our protocol directly tothe binary tree structure of [2] without any modifications ofthis structure and we call such a configuration “UNMOD-IFIED”. The third case is taken by creating a completelyarbitrary tree of a logical root node by means of Algorithm1 and apply the read and write operations of our protocolwhere we call such a configuration “ARBITRARY”. The“HQC” is studied by taking the tree structure of [8] wherethe computations of the load induced by read and write op-erations are based on the results of section 6.4 of [10] andthe communication costs as well as availability of the op-erations are based on the results of sections 4 and 5 of [8].In the fifth case, we configure the tree in such a way thatthe root is a logical node and all the replicas of the systemare placed in one and only one physical level. By applyingread and write operations of our protocol, it behaves likeROWA and we call such a configuration “MOSTLY-READ”.Finally, the last configuration is determined by constructingthe tree of a logical root node in such a way that for an odd-sized number of replicas, |Kphy| = n−1

2 and h = |Kphy| suchthat every physical level consists of two replicas and we callsuch a configuration “MOSTLY-WRITE”. It is important tonote that, these last two configurations are special cases ofthe 3rd configuration. The communication costs, availabil-ities and system loads of read and write operations of theconfigurations 2, 3, 5, and 6 are based on the definitions ofsections 3.2.1 and 3.2.2.

The configurations are carried out by considering a dis-tributed system composed of 15, 31, 63, 127 and 255 repli-cas. The followings are the representations of the arbitrarytree of the third configuration for the above-mentioned num-ber of replicas respectively:1-3-3-3-3-3, 1-3-4-4-4-4-4-4-4, 1-4-4-4-4-4-5-5-10-10-13, 1-4-4-4-4-4-4-4-24-25-25-25and 1-4-4-4-4-4-4-4-25-25-25-25-25-25-25-25-27.

4.1. The communication costs

Figure 2 illustrates the communication costs of read andwrite operations of our configurations. We can notice that,of the six configurations, “MOSTLY-READ” has the low-est communication cost of 1 for read operations and theworst cost of n for write operations. These costs are dueto the way n replicas are arranged logically into one andonly one physical level. For an odd-sized number of repli-cas n, “MOSTLY-WRITE” has the highest costs of n−1

2 forread operations and the lowest cost of two for write oper-ations. This is because “MOSTLY-WRITE” maximizes thenumber of physical levels of the tree by adding at each phys-

Figure 2. The communication costs of readand write operations of our configurations

ical level two replicas. However, these two configurationsare suitable for specific environments where the read andwrite operations do not happen in proportional frequencies.Hence in the rest of this section, we compare the commu-nication costs of read and write operations of the first fourconfigurations: “BINARY”, “HQC”, “UNMODIFIED”, and“ARBITRARY”.

Of these four configurations, “BINARY” has the high-

est costs of 2h∗(1+h)h

h∗(2+h)h−1 – 2h for both read and write opera-

tions where h denotes the height of the binary tree. On theother hand, “ARBITRARY” has the lowest communicationcosts for write operations whereas for read operations, ithas lower costs than “BINARY” and “HQC”, where the lat-ter has a read and write costs of n0.63, and has comparablecosts with respect to “UNMODIFIED”. Finally, this latterhas the least communication costs of log(n + 1) for readoperations and has write costs of n

log(n+1) which is com-parable with respect to “ARBITRARY” when n < 200 andcomparable to those of “HQC” for n > 200.

4.2. The (expected) system loads

In this section, we present the (expected) system loadsof read and write operations of our configurations. The ex-pected system loads are computed using equations 3.2.

4.2.1 The read operation

The (expected) system loads of read operations are illus-trated in Figure 3. We can notice that, of the six configu-rations, “MOSTLY-READ” has the lowest system load of 1

nand we can observe that such a system load is stable and di-minishes as the number of replicas increases. On the otherhand, “MOSTLY-WRITE” has a system load of 1

2 for anynumber of replicas and such a system load is instable and

506508

reaches easily to 1 due to the fact that at every physical levelthere are two replicas. In the rest of this section, we com-pare the system and expected system loads imposed by readoperations of the first four configurations.

Figure 3. The (expected) system loads of readoperations of our configurations

We can observe that “UNMODIFIED” has the highest sys-tem and expected system loads of 1 for any number of repli-cas among the six configurations. This is due to the factthat the root is a member of every possible read quorumand hence every read operation must access it. On the otherhand, we can notice in Figure 3 that “BINARY”, “HQC” and“ARBITRARY” have quite stable system loads which is dueto the high availability of their read operations.

Among the first four configurations, “HQC” has the leastsystem loads of n−0.37 and the least expected system loadswhen n > 15. On the other hand, “BINARY” and “ARBI-TRARY” have quite similar (expected) system loads andthat they are comparable to those of “HQC”. Moreover,“ARBITRARY” has always a system load of 1

4 when n > 32,whereas “BINARY” has always a system load of 2

log(n+1)+1 .

4.2.2 The write operation

The (expected) system loads of write operations are illus-trated in Figure 4. We can notice that, among the six config-urations, “MOSTLY-READ” has the highest (expected) sys-tem loads of 1 due to the fact that all n replicas of the sys-tem participate in every write operation. On the other hand,“MOSTLY-WRITE” has the least system load of 2

n−1 for anodd-sized number of replicas n and such a system load isstable and diminishes as the number of replicas of the sys-tem increases. In the rest of this section, we compare thesystem and expected system loads imposed by write opera-tions of the first four configurations .We can observe that among the first four configurations,“BINARY” has the highest system load of 2

log(n+1)+1 and

the highest expected system load for any number of repli-cas. “ARBITRARY” has the least system load for any num-ber of replicas. The load imposed by the write operations of

Figure 4. The (expected) system loads ofwrite operations of our configurations

our protocol is always 1√n

whenever the tree is constructedusing Algorithm 1. Furthermore, this configuration has thesmallest expected system load for small number of replicasn and has comparable expected system loads with respectto “HQC” as the number of replicas n becomes larger dueto the fact that the availability of the write operations of“HQC” is better than that of “ARBITRARY” when p < 0.8.Additionally, the write operations of “HQC” impose a sys-tem load of n−0.37 and have the best expected system loadfor large number of replicas n. Finally, “UNMODIFIED”has the second lowest system load of 1

log(n+1) and has com-parable expected system loads with respect to the config-urations “BINARY”, “HQC” and “ARBITRARY”. Howeveras we have seen in the previous section such a configura-tion has the worst system load for read operations. The ex-pected system loads imposed by read and write operationsof “ARBITRARY” become similar to the computed systemloads, as the availability probability p of the replicas be-comes higher than 0.8.

5. Conclusion

In a large replicated system, multiple copies of data mustbe kept synchronized by means of replica control protocols.Such protocols induce trade-offs between the communica-tion costs and the system load. Whenever a low communi-cation costs is desired for both read and write operations, ahigh load penalty must be paid by the system and vice versa.Tree-based replica control protocols can for instance pro-vide low communication costs of O(log(n)) for both readand write operations but at the expense of very high systemloads.

507509

In this paper, we proposed a new tree-based replicationprotocol: the arbitrary protocol. This can be configuredappropriately based on the frequencies of read and writeoperations of the system. “MOSTLY-READ” is best suitedfor systems where read operations are carried out exces-sively, whereas “MOSTLY-WRITE” is appropriate for sys-tems where write operations dominate. We identified a con-figuration which has the best combined read and write com-munication costs of 2 ∗√n and has the least system load of1√n

imposed by write operations compared to the existingtree replica control protocols and its read operations alwaysinduce a system load of 1

4 . Our proposal enables the shift-ing from one configuration into another by just modifyingthe structure of the tree. There is no need to implement anew protocol whenever the frequencies of read and writeoperations of the system change.

Acknowledgement: We would like to thank the depart-ment of Mathematics of Paul Sabatier University and pre-cisely Mr. Dominique Bakri and Miss Guitta Sabiini fortheir kind remarks on mathematical expressions.

References

[1] D. Agrawal and A. E. Abbadi. The tree quorum protocol: Anefficient approach for managing replicated data. Proceed-ings of the 16th VLDB Conference, pages 243–254., 1990.

[2] D. Agrawal and A. E. Abbadi. An efficient and fault-tolerantsolution for distributed mutual exclusion. ACM Transactionson Computer Systems, pages 1–20, 1991.

[3] P. Bernstein and N. Goodman. An algorithm for concurrencycontrol and recovery in replicated distributed databases.ACM Transactions on Distributed Systems, 9 (4), 1984.

[4] S. Cheung, M. Ammar, and A. Ahamad. The grid proto-col: A high performance scheme for maintaining replicateddata. Proc. IEEE Sixth Int’l Conf. on Data Engineering,pages 438–445, 1990.

[5] S. Choi, H. Youn, and J. Choi. Symmetric tree replica-tion protocol for efficient distributed storage system. ICCS,pages 474–484, 2003.

[6] H. Garcia-Molina and D. Barbara. How to assign votes in adistributed system. J ACM 32, pages 841–860, 1985.

[7] H. Koch. An efficient replication protocol exploiting logicaltree structures. The 23rd Annual International Symposiumon Fault-Tolerant Computing, pages 382–391, 1993.

[8] A. Kumar. Hierarchical quorum consensus: A new algo-rithm for managing replicated data. IEEE Transactions onComputers, pages 996–1004, 1991.

[9] M. Maekawa. A√

n algorithm for mutual exclusion in de-centralized systems. ACM Transactions on Computer Sys-tems, pages 145–159, 1985.

[10] M. Naor and A. Wool. The load, capacity, and availabilityof quorum systems. SIAM J. Comput, pages 423–447, 1998.

[11] M. Pease, R. Shostak, and L. Lamport. Reaching agreementin the presence of faults. J ACM, pages 228–234, 1979.

[12] D. Peleg and A. Wool. The availability of quorum systems.Inform. and Comput, pages 210–223, 1995.

[13] R. Thomas. A majority consensus approach to concurrencycontrol for multiple copy databases. ACM Transactions onDatabase Systems, 4 (2):180–207, 1979.

6. Appendix: proofs of system load optimality

In this section, we provide the proofs of optimal systemloads induced by read and write operations of our protocol.

6.1. The load of read operations

In this section, we prove that the optimal system loadimposed by read operations of our protocol of n replicasarranged logically into a tree structure is: LRD = 1

d whered = min {mphy k ∀k; k ∈ Kphy}.

6.1.1 Proof of the upper bound: LRD ≤ 1d

Let R = { R1, R2,. . . ,Rj } be the set of readquorums such that every read quorum Rj is con-structed in the same manner as explained in section

3.2.1. Then by Fact 3.2.1, we have m(R) =∏

k∈Kphy

mphy k.

Let f(u) =|Kphy|−1∏

x=u

mphy Kphy[x] ×u−1∏

y=0 ; y �=u

mphy Kphy[y] and

let g(u) = mphy Kphy[u]. Then every replica Sphy(i,Kphy[u])of the system is a member of mq read quorums wheremq is given by: mq = f(u)

g(u) ∀u;u ∈ [0, |Kphy| − 1] and∀i; i ∈ [1,m Kphy[u]].

Let wread =m(R)∑

j=1

wread j be a strategy of picking read

quorums Rj ∈ R as in definition 2.4 such that everywread j = 1

m(R)∀j; j ∈ [1,m(R)]. It is clear that wread is a

strategy sincem(R)∑

j=1

wread j = 1. By definition 2.5, the load

induced by this strategy wread on every replica of the sys-tem is: lwread (Sphy(i,Kphy[u])) = mq × wread j = f(u)

g(u)×m(R)

∀u;u ∈ [0, |Kphy| − 1] and ∀i; i ∈ [1,m Kphy[u]].

However, f(u) =∏

k∈Kphy

mphy k ∀u;u ∈ [0, |Kphy| − 1],

then the load induced on every replica of the system is:

lwread (Sphy(i,Kphy[u])) =1

g(u)

∀u; u ∈ [0, |Kphy| − 1] and ∀i; i ∈ [1,m Kphy[u]].Hence by definition 2.5, the load induced by this strategy

508510

wread on the set of read quorums R is:

Lwread(R) = max lwread (Sphy(i,Kphy[u]))

∀u; u ∈ [0, |Kphy| − 1] and ∀i; i ∈ [1,m Kphy[u]].Let d = min{ mphy Kphy[u] } ∀u; u ∈ [0, |Kphy| − 1] thenLwread(R) = 1

d and hence LRD ≤ 1d . Keep in mind that

Lwread(R) is the load induced by a strategy wread on the set ofread quorums R. Next we have to prove that such a strategyis the optimal one that induces the optimal system load ofLRD when performing read operations of our protocol.

6.1.2 Proof of the lower bound: LRD ≥ 1d

Given a tree of height h, let t denote the first physical levelof this tree where the orientation is taken from top to bot-tom. Let mphy t denote the number of physical nodes at levelt. Then by assumption 3.1, we have mphy t = d where d de-notes the minimal number of physical nodes of the physicallevels of the tree. Let us suppose that there exists a strategywread that induces a system load of Lwread(R) = 1

d when per-forming read operations of our protocol. Now we have toshow that such a load is the optimal one. By using notation2.2, then ∀i; i ∈ [1,mt] where mt denotes the total num-ber of nodes at level t, let yj = 1

d for every replica Sphy(i,t)and yj = 0 for the replicas other than those at level t. Thenclearly y(U) = 1 where U is the universe of the system andy(Rj) = 1

d for every read quorum Rj ∈ R. Then by propo-sition 2.1 such a load is optimal and therefore LRD ≥ 1

d .Hence we deduce from the lower and upper bounds thatLRD = 1

d .

6.2. The load of write operations

In this section, we prove that the optimal system load im-posed by write operations of our protocol on a distributedsystem of n replicas organized logically into a tree struc-ture is: LWR = 1

1+h−|Klog| where h is the height and

|Klog| is the number of logical levels of the tree such that0 ≤ |Klog| ≤ h.

6.2.1 Proof of the upper bound: LWR ≤ 11+h−|Klog|

Let W = { W1, W2,. . . ,Wj } be the set of write quo-rums such that every write quorum Wj is constructed asexplained in section 3.2.2. Then by Fact 3.2.2, we havem(W) = 1 + h – |Klog| where h is the height of the tree and|Klog| is the number of logical levels. Furthermore, everyreplica Sphy(i,Kphy[u]) of the system is a member of one andonly one write quorum Wj where u ∈ [0, |Kphy| − 1] andi ∈ [1,m Kphy[u]].

Let wwrite =m(W )∑

j=1

wwrite j be a strategy of picking write quo-

rums as in definition 2.4 such that every wwrite j = 1m(W ) . It

is clear that wwrite is a strategy sincem(W )∑

j=1

wwrite j = 1 . Then

by definition 2.5, the load induced by this strategy wwrite onevery replica of the system is :

lwwrite (Sphy(i,Kphy[u])) =1 ∗ 1

m(W )

∀u;u ∈ [0, |Kphy| − 1] and ∀i; i ∈ [1,m Kphy[u]]. Hence bydefinition 2.5, the load induced by this strategy wwrite on theset of write quorums W is:

Lwwrite(W) = max lwwrite (Sphy(i,Kphy[u]))

∀u;u ∈ [0, |Kphy| − 1] and ∀i; i ∈ [1,m Kphy[u]].Then from the above equation we can saythat: Lwwrite(W) = 1

1+h−|Klog| .Therefore, we can conclude from the proof thatLWR ≤ 1

1+h−|Klog| .

6.2.2 Proof of the lower bound: LWR ≥ 11+h−|Klog|

Given a tree of height h that has |Klog| logical levels,then we have ( 1 + h – |Klog| ) write quorums such that ev-ery replica r of the universe U belongs to one and only onewrite quorum. Let us suppose that there exists a strategywwrite that induces a system load of Lwwrite(W) = 1

1+h−|Klog|when performing write operations of our protocol. The nextstep is to prove that such a strategy is the optimal one thatinduces the optimal load. As we have mentioned above, thesystem consists of ( 1 + h – |Klog| ) write quorums. Byusing notation 2.2, then ∀u;u ∈ [0, |Kphy| − 1] we pickone replica Sphy(i,Kphy[u]) from every physical level of thetree and assign a value of 1

1+h−|Klog| to its yj and a value

of zero to the yj’ s of all other replicas Sphy(i,Kphy[u]) otherthan the picked one. Then clearly y(U) = 1 where U is theuniverse and y(Wj) = 1

1+h−|Klog| for every write quorumWj . Then by proposition 2.1, such a load is optimal andtherefore LWR ≥ 1

1+h−|Klog| .Hence we deduce from the lower and upper bounds thatLWR = 1

1+h−|Klog| where h is the height and |Klog|is the number of logical levels of the tree such that0 ≤ |Klog| ≤ h.

509511