Sierra: a power-proportional, distributed storage system

Microsoft Research Ltd. Technical Report MSR-TR-2009-153, November 2009

Sierra: a power-proportional, distributed storage system

Eno Thereska, Austin Donnelly, Dushyanth NarayananMicrosoft Research, Cambridge UK

Abstract

We present the design, implementation, and evaluation ofSierra: a power-proportional, distributed storage system.I/O workloads in data centers show significant diurnalvariation, with peak and trough periods. Sierra powersdown storage servers during the troughs. The challengeis to ensure that data is available for reads and writes atall times, including power-down periods. Consistencyand fault-tolerance of the data, as well as good perfor-mance, must also be maintained. Sierra achieves all thesethrough a set of techniques including power-aware lay-out, predictive gear scheduling, and a replicated short-term versioned store. Replaying live server traces froma large e-mail service (Hotmail) shows power savings ofat least 23%, and analysis of load from a small enterpriseshows that power savings of up to 60% are possible.

1 Introduction

Server power consumption is a major problem for bothsmall and large data centers, since it contributes sub-stantially to an organization’s carbon footprint and powerbills. Barroso and Holzle have argued forpower propor-tionality [3]: the power used should be proportional tothe system load at any time, rather than to the peak loadthat the system is provisioned for. A power-proportionalsystem could exploit temporal variations in load, such asthe diurnal peaks and troughs (e.g., the “Pacific Oceantrough” [6]) seen in many user-facing services.

Storage is a key component for many large, scalableand highly available services such as web email (Hot-mail or Google Mail), Amazon’s EC2, and WindowsAzure [1]. Because storage is not power proportional, itlimits the power proportionality of the whole data center.

Some power-proportionality is achievable at the hard-ware level, e.g., using dynamic voltage scaling (DVS) forCPUs. However non-CPU components, especially disks,are not power-proportional.

We believe that at data center scale, the most promis-ing approach is to power down entire servers rather savepower in individual components. This is the position wetake in this paper. It is challenging to power servers downand yet maintain high service availability. Although mostservices have clear troughs, the troughs do not go to zero.Furthermore, the system must still provide good avail-ability and performance for users during troughs. Tomaintain service availability, all the load must first be mi-grated from the servers to be powered down, and consol-idated on the active servers. Storage presents a challengehere again. While computational state can be migratedand consolidated, e.g., using virtualization techniques,itis not possible to migrate terabytes of on-disk state perserver daily.

In this paper we present Sierra, a power-proportionaldistributed storage system. Sierra is a replicated objectstore that allows storage servers to be put into low powerstates (standby or powered down) when I/O load is low.It does this without making the storage unavailable orsacrificing consistency, performance, and fault tolerance.This paper addresses these challenges entirely from thepoint of view of distributed storage. Megascale data cen-ters often co-locate computation and storage on the sameservers; thus a complete solution would integrate Sierrawith existing techniques for CPU consolidation.

Sierra exploits the redundancy that is already presentin large-scale distributed storage, typically in the formof three-way replication across servers. The design ofSierra is based on the idea of running the system in alower “gear” — a smaller number of active replicas perobject — when load is low. This allows servers host-ing inactive replicas to be powered down. In a three-wayreplicated system Sierra allows up to2

3of the storage

servers to be in standby. Sierra avoids sending clientrequests to the powered-down servers, and ensures thatthere are always active servers that can serve accesses toevery object. This ensures that all objects are availableeven when a majority of the servers are powered down.

1


30%

40%

50%

60%

70%

80%

90%

100%

Norm

alized d

isk I/O

rate

0%

10%

20%

30%

09/07 09/08 09/09 09/10 09/11 09/12 09/13

Norm

alized d

isk I/O

rate

Date (in 2009)

(a) Hotmail

30%

40%

50%

60%

70%

80%

90%

100%

Norm

alized d

isk I/O

rate

0%

10%

20%

30%

09/07 09/08 09/09 09/10 09/11 09/12 09/13

Norm

alized d

isk I/O

rate

Date (in 2009)

(b) Messenger

�� 2/23 2/24 2/25 2/26 2/27 2/28 3/1 3/2

Date (in 2007)

(c) Cambridge file servers

Figure 1: One week of I/O load for two large and one small service

There are several challenges in ensuring that gear-ing down does not compromise the availability, fault-tolerance, or performance of the storage system. First,at least one replica of each object must be on an activeserver even in the lowest gear. Second, the system mustallow updates when in low gear, and these updates mustbe kept persistent, consistent, and have the same repli-cation for fault tolerance as data written in high gear.Third, there should be enough active servers to handlethe load at any given time, and the load should be bal-anced across the active servers. Finally, gearing downshould not impede recovery actions taken when a serverfails either transiently or permanently.

Sierra achieves these goals through a combination oftechniques. A power-aware layout scheme ensures thatall objects are kept available even when a significantfraction of servers is powered down. A predictive gearscheduler exploits observed diurnal patterns to sched-ule servers for power-up and power-down. Read avail-ability is provided through a pro-active primary migra-tion protocol. Write availability is provided by using ashort-term versioned store. These techniques also ensurethat the system maintains read/write consistency, toler-ates any two transient or permanent server failures, andcan efficiently re-replicate the contents of a failed server.

This paper makes three contributions. First, using realload traces from both large and small services as evi-dence, we show that there are significant diurnal troughsin I/O load, which can be exploited for power savings.Second, we describe the design of a power-proportionaldistributed storage system that exploits these troughswithout compromising consistency, availability, load bal-ancing, or fault tolerance. Third, we present an evalua-tion of a prototype system running on a hardware testbed,using I/O traces from production servers of the Hotmailservice, and achieving power savings of 23%.

2 Evidence and motivation

The design of Sierra is motivated by the observation thatmany workloads in both small and large data centers have

30%

40%

50%

60%

70%

80%

90%

100%

Fraction of hours in week

Hotmail

Messenger

Cambridge

0%

10%

20%

30%

0% 20% 40% 60% 80% 100%

Fraction of hours in week

Utilization relative to peak hour

Cambridge

Figure 2: CDF of utilization for all three services

periodic I/O load patterns with large peak-to-trough ra-tios. CPU load also shows these characteristics, and vir-tual machine migration and voltage scaling can be usedto save CPU power during the troughs. There are noanalogous solutions for disk, and hence we need a newapproach to storage power proportionality.

Figure 1(a) and 1(b) show the aggregated I/O load fortwo large online services, for 1 week. The load is ag-gregated over tens of thousands of servers for Hotmail(Windows Live Mail), and thousands of servers for Win-dows Live Messenger. The graphs show the total num-ber of disk bytes transferred, aggregated across all theback-end storage in the service at one-hour intervals. Theload is normalized to the peak value for each service. Weobserve clear periodic patterns with significant peak-to-trough ratios. In general, this load correlates with the ex-pected diurnal variation for user-facing services. Thus,there seems to be substantial scope for operating withfewer resources during the troughs without significantlyimpacting request latency. We observed that an alterna-tive, consolidation of resources across services, did noteliminate the troughs. Much of the load is correlated intime and furthermore, Hotmail needs an order of magni-tude more servers than the other services.

Figure 1(c) shows the variation in I/O traffic for a dif-ferent environment: a small/medium enterprise with a

2


small number of on-site servers. Specifically we showthe I/O aggregated across 6 RAID volumes on two fileservers at MSR Cambridge; the graph is derived fromthe publicly available MSR Cambridge I/O traces [12, 7].Here, the periodicity is less obvious; however there areclear peak and troughs, indicating a significant potentialfor power savings during the troughs.

Figure 2 shows the CDF of time spent at differentutilization levels, where utilization is normalized to thepeak seen in each trace. Substantial periods of time arespent with utilization much lower than the peak. Hencethere is significant scope for exploiting these periods forpower savings, even in a system that is optimally provi-sioned, i.e., with no power wasted at peak load.

3 Design and implementation

The base system model for Sierra is that of a repli-cated, cluster-based object store similar to GFS [5], theWindows Azure blob store [1], FAB [10], or Ursa Mi-nor [2]. The base system is designed to provide scala-bility, load balancing, high availability, fault tolerance,and read/write consistency. We first briefly describe thebasic architecture of Sierra, which is largely based oncurrent best practices. We then outline the challenges inmaking such a system power-proportional without losingthe other properties, and present in detail the techniquesSierra uses to address these challenges.

3.1 Basic architecture

Figure 3 shows the basic architecture of Sierra. Sierraprovides read/write access to objects in units ofchunks.The chunk size is a system parameter: a typical valueis 64 MB. Each chunk is replicated on multiplechunkservers; the default replication factor is 3. For any givenchunk, one of the replicas is designated as the primaryat any given time, and the others are secondaries. Atany time a chunk server will be the primary for someof the chunks stored on it and a secondary for the oth-ers. All client read and write requests are sent to theprimary, which determines request ordering and ensuresread/write consistency. Since we are designing a general-purpose storage system for use by a variety of applica-tions, Sierra supports overwriting existing data in addi-tion to appends of new data. Client reads and writes canbe for arbitrary byte ranges within a chunk.

Read requests are sent by the primary to the localreplica; write requests are sent to all replicas and ac-knowledged to the client when they all complete. Loadbalancing in Sierra is done by spreading a large num-ber of chunks uniformly over a smaller number of chunkservers, and also by choosing primaries for each chunkrandomly from the available replicas of that chunk.

Metadata service (MDS)

Chunk servers

chunk ID, primary

,...

read (chunk ID, offset, size,...)

write (chunk ID, offset, size, data,...)

Client

lookup(o

bject ID, offset)

load s

tatistics

gear schedules

Gear scheduling

service

Figure 3: Sierra architecture

Sierra has a centralized metadata service (MDS),which functions as a naming service. It is implementedas an in-memory, deterministic state machine which canbe replicated for high availability using state machinereplication techniques [11]. It maps each object to itsconstituent chunks, and each chunk to its current pri-mary. The MDS is not on the data path between clientsand chunk servers; its state is updated when chunks arecreated or deleted but not when they are read or written.

The MDS also tracks chunk server availability, reas-signs primaries as necessary, and initiates recovery ac-tions when a server fails permanently. Availability istracked through periodic heartbeats from chunk serversto the MDS. In response to each heartbeat, the MDSsends the chunk server a lease for the set of chunks that itis currently the primary for, and the locations of the sec-ondaries for those chunks. Leases are set to expire beforethe MDS times out the heartbeat, and servers send freshheartbeats before their leases expire. The MDS reassignsprimaries on demand for chunks whose primaries havelost their lease. A chunk server with an expired leasewill return an error to a client trying to access data on it;after a timeout period the client fetches and caches thenew chunk metadata from the MDS.

3.2 Challenges

This baseline design provides good consistency, avail-ability, load balancing and fault tolerance when all ormost chunk servers are available. However, there areseveral challenges in saving power and maintaining theseproperties when significant numbers of servers are peri-odically powered down and up.

The first challenge in achieving power savings islay-out: the assignment of chunks to chunk servers. Sec-tion 3.3 shows how a “naive random” approach, althoughsimple and good for load balancing, does not give a goodtradeoff between power savings and availability. Thisis surprising because this is the most commonly usedapproach today. To address the problem we devised apower-aware layout, described in Section 3.3. This al-

3


A

B

C D

EF

(a) Naive random

AB

C

D

FE

Replica group 1

Replica group 2

(b) Naive grouping

AB

D

F

E

C

Gear

group 0

Gear

group 1

(c) Power-aware grouping

Figure 4: Different layouts for 6 chunks, 4 servers, and 2-way replication.

lows anr-way replicated system to be in any gearg suchthat only g

rof the servers need to be active to keepg

replicas of each object on an active server.The second challenge is to correctly predict the num-

ber of servers required at any time to sustain the systemload, so that excess servers can be turned off. This isdone by thegear scheduler component of Sierra. Thiscomponent tracks and predicts the load on chunk serverson a coarse time granularity (hours to days). Based onload predictions, it then sends “gear schedules” to thechunk servers specifying when they should shut downand start up. We have found that a simple predictor basedon the hour of day works well on the workloads we haveevaluated. Section 3.4 describes the load metrics and pre-diction algorithm used by the gear scheduler.

A third challenge is maintaining availability and loadbalancing during gear transitions, i.e., server power-upsand power-downs. While we use standard techniques todeal with server failures, we do not treat server power-ups and power-downs identically to failures. The reasonis that, unlike failures, we expect a significant fraction ofthe servers to change their power state at least once overthe period of a day. Section 3.5 describes how Sierrapro-actively migrates primaries away from a server thatis about to be powered down. Hence, at every gear level,Sierra ensures that every chunk primary is on an activeserver, and that the primaries are uniformly distributedacross the currently active servers.

While primary migration gives availability and load-balancing for reads, we also want to support writes tochunks during low-gear periods, when all replicas of thechunks are not available. Further, we want writes dur-ing low-gear periods to maintain the replication factorr;to maintain read/write consistency (i.e., every read seesthe result of the last committed write); and to ensurethat writes are eventually applied to all chunk replicaswhen they become available. Sierra achieves this using areplicated short-term versioned store, described in Sec-tion 3.6. This is a service that can use spare disk band-

width and capacity on existing chunk servers, and/or asmall number of dedicated servers. This mechanism alsolets Sierra chunk servers support overwriting of chunkdata, even when one or more replicas have failed. Hencechunk data is kept available for both reading and writingas long as at least one replica is on an active server.

As a final challenge, in addition to existing failuremodes such as chunk server failures, Sierra must han-dle new failure modes, e.g., failure of an active replicawhen other replicas are powered down. Section 3.7 de-scribes fault tolerance in Sierra, focusing on the novelfailure modes and solutions in Sierra.

3.3 Power-aware layout

The layout defines the way in which chunks are assignedto chunk servers at chunk creation time. Our goal is alayout that allowsr−g

rof the servers to be powered down

(wherer is the replication factor and0 ≤ g ≤ r) whilekeepingg replicas of each chunk available. We refer tothis as putting the system in thegth gear. Thus, a 3-wayreplicated system could be in gears 3, 2, 1, or 0. Gear0, while possible in theory, is unlikely to be useful inpractice, due to the high latency penalty of waiting for aserver to start up to service a request.

One simple approach is thenaive random layout: eachnew chunk is assigned replicas on three servers chosenat random. However, we discovered that this severelylimits the scope for power savings. This layout makesit hard to power down more thanr − 1 servers in theentire system. Figure 4(a) shows a simple example with4 servers, 6 chunks, and 2-way replication. Since everychunk is 2-way replicated, we would like to be able to put2 of 4 servers into standby at low load and yet keep onereplica of each chunk available. With the layout shown,however, it is not possible to put more than one serverinto standby (there is at least one object for which bothreplicas would be unavailable if we did so). As the sizeof the system increases, the independent random layout

4


Power-down RebuildNaive random r − g N

Naive grouping N r−g

r1

Power-aware grouping N r−g

rNr

Table 1: Number of servers that can be powered down ingearg, and the write parallelism for data rebuild.N isthe total number of servers andr is the replication level.

makes it increasingly improbable that more thanr − g

servers can be powered down while still keepingg activereplicas of every chunk. Additionally, with this layout,finding a maximal set of servers that can be turned offwithout losing availability is likely to be computationallyhard, since it is a special case of the NP-complete setcovering problem.

An alternative approach is to put servers intoreplicagroups, each of sizer (Figure 4(b)). A chunk is then as-signed to one replica group rather than tor independentlychosen servers. Now we can switch offr − g servers ineach replica group and still haveg replicas of each objectavailable. However, naive grouping of servers reducesthe rebuild parallelism. When a server suffers a perma-nent failure, with naive grouping its entire contents mustbe rebuilt (re-replicated) on a single new server, and thisserver becomes the bottleneck for the rebuild process.With a typical commodity disk with a write bandwidthof 80 MB/s and 1 TB of data, this would take 3.6 hours.With the naive random approach on the other hand, eachchunk stored on the failed server can be independentlyrebuilt on any of the other servers in the system. Thisgives a high degree of rebuild parallelism, and hence ahigher rebuild rate.

Sierra uses a generalized notion of grouping thatachieves both power savings and high rebuild parallelismby usingpower-aware grouping. Each server is assignedto exactly one ofr gear groups. Each new chunk is as-signed exactly one replica from each gear group; selec-tion of the server within each gear group is done uni-formly at random. Now the system can be put into anygearg by turning off r − g gear groups. If a server insome gear groupG fails, then its data can be rebuilt inparallel on all remaining servers inG. Thus the rebuildparallelism isN

rwhereN is the total number of servers.

Table 1 summarizes the three approaches. Note that allthree layouts are equivalent with respect to load balanc-ing of client requests, since all three allow chunk replicasand primaries to be spread uniformly over servers.

The above analysis also applies to cross-rack replica-tion. For each of the three layouts, we can further con-strain the replica choice such that different replicas arein different fault domains, typically different racks in thedata center. However, the naive random policy will still

2 2 2 2

3 3 3 3

1 1 1 1

1 1 1 1

(a) Rack-aligned

2 3 1 2

3 1 2 3

1 2 3 1

1 2 3 1

(b) Rotated

Figure 5: Two ways of configuring gear groups

only be able to turn offr − g racks, rather thanr−g

rof

all servers. Similarly the naive grouping policy can onlyrebuild over one rack rather than1

rof the racks.

Sierra uses power-aware grouping with cross-rackreplication. It supports two variants: rack-aligned androtated (Figure 5). While both allow the same number ofservers to be turned off in a given gear, the location of theservers is different. In the rack-aligned case, all serversin a given rack are in the same power state; this could al-low some additional power savings, for example, by turn-ing off rack-wide equipment such as switches. However,the rotated layout might be useful if it is more importantto distribute the powered-up servers (and hence the ther-mal load) evenly across racks. For concreteness, in therest of the paper will assume a gear-grouped, cross-rack,and rack-aligned layout.

3.4 Gear scheduler

The main aim of gear shifting is to exploit the 24-hourcycle in load. It does not make sense to gear-shift at atime scale of minutes or seconds, since a server can takeseveral minutes to start up and several seconds to comeout of standby. Hence, in Sierra we gear-shift on a timescale of hours: the aim is to shift gears a few times a dayto capture the broad diurnal patterns in load.

The load metric used by the gear scheduler uses therate of reads and of writes, both aggregated over all thechunk servers; the write rate is weighted by a factorr,since each write is replicatedr times. It also considersseparately the random-access I/O rate measured in IOPS,and the streaming I/O rate measured in MB/s. Given theknown performance per chunk server in terms of IOPSand MB/s, the load can then be computed in units of thenumber of chunk servers required to sustain the load.

Lnonseq =TotalIOPSread

ServerIOPSread

+ r ·TotalIOPSwrite

ServerIOPSwrite

Lseq =TotalMBPSread

ServerMBPSread

+ r ·TotalMBPSwrite

ServerMBPSwrite

L = max(Lnonseq, Lseq)

5


The load is measured at 1 sec intervals on each chunkserver, and aggregated once an hour. By default we usethe peak (i.e., maximum) load observed in the hour asthe load for that hour: since I/O load is often bursty,using the mean value can significantly degrade perfor-mance during bursts. The gear scheduler then predictsthe load for each hour of the following day, by averagingthe past load samples for that hour in previous days.

To compute the gearg for a given hour, the gear sched-uler measures whether the predicted load for that hourexceeds1

r, 2

r, etc. of the total number of chunk servers

N . It then chooses the lowest gear which leaves enoughservers active to sustain the load:

g =

⌈

L

Nr

⌉

The Sierra gear scheduler is a centralized componentthat periodically aggregates load measurements from allthe chunk servers, and computes the gear schedules forthe following day. For any given gearg, the servers in thefirst r−g gear groups are scheduled to be powered down.The gear schedules are then pushed to the chunk servers,and each server follows its own gear schedule. Over sev-eral days, the gear scheduler rotates the ordering of thegear groups, so that in the long term all servers spendan equal amount of time powered up. This allows allservers to do background maintenance tasks, e.g., scrub-bing, during idle periods.

3.5 Primary migration protocol

Sierra balances load by spreading primaries uniformlyacross active servers, and hence across active geargroups. This means that when servers in some gear groupG are powered down, any chunk primaries on thoseservers would become unavailable, and clients wouldnot be able to access those chunks. Since Sierra han-dles server failures using a heartbeat mechanism (Sec-tion 3.7), we could simply treat the power-down eventsas failures; the MDS will reassign the primaries when itdetects a failure. However, this will result in data beingunavailable until the “failed” server’s leases expire.

To avoid this, Sierra chunk servers pro-actively mi-grate all chunk primaries onto other replicas before pow-ering down, using the following migration protocol. If aserverS wishes to power down, it executes the followingprotocol for each chunkC for whichS is a primary:

1. S updates its in-memory state to mark itself as asecondary for chunkC. An error will be returnedon future client requests. Client requests currentlyin flight will complete normally.

2. S signals the MDS withreleased primary(C).3. The MDS randomly picks another replicaS′ of the

chunk C as its primary and modifies its internal

state to reflect this.4. The MDS signalsS′ with become primary(C).5. S′ initializes any required in-memory state and

starts servicing client requests as the primary forC.The window of unavailability for chunkC is now onenetwork round trip plus the time required to update MDSstate and initialize the new primary onS′. If a clientaccessesC during this short window it will retry the op-eration, converting this temporary unavailability into ahigher latency.

When the chunk serverS has no more primaries oroutstanding requests, it sends a final “standby” messageto the MDS and goes into standby. The MDS then sends“gear shift” messages to all the peers ofS (i.e., serverswhich share one or more replicated chunk withS) to in-form them thatS is no longer active. This is an opti-mization that avoids peers ofS timing out on requests toS when accessing it as a secondary. When a chunk serverS wakes up from standby it resumes sending heartbeatsto the MDS. When the MDS receives a heartbeat froma server that was previously in standby, it rebalances theload by moving some primaries from other servers toS.This is done by sendingS a list of chunk IDs to acquireprimary ownership for, and the current primary for each.S then contacts each of the current primaries, which theninitiate a primary migration protocol similar to the above.

In Sierra, chunks are collected into chunk groups toreduce the number of MDS operations involved in pri-mary migration. All chunks in a chunk group are guar-anteed to be replicated on the same servers, and havethe same primary at any given time. Migration of pri-maries requires one MDS operation per chunk grouprather than per chunk. Chunks are assigned randomly tochunk groups on creation. The number of chunk groupsis a system parameter that trades off MDS load for fine-grained load balancing across servers. Sierra currentlyuses 64N chunk groups whereN is the total number ofchunk servers.

3.6 Replicated short-term versioned store

We want writes to be stored persistently and consistentlyeven when chunk replicas are powered down or other-wise unavailable. The replicated short-term versionedstore is used by Sierra primaries to ensure these prop-erties. The basic design of this store is similar to thatused in our previous work in the context of RAID ar-rays [7, 8], but has now evolved as a distributed systemcomponent. Here we give a high-level description of itsbasic properties and (network) optimizations in the dis-tributed setting.

When one or more secondaries is unavailable (pow-ered down for example), a Sierra primary enters “loggingmode”. In this mode it sends writes to the short-term ver-

6


Client

Secondaries

Primary

Log client

chunk files

Loggers

Chunk files

Non

logging

mode

writes

All

writesReads

Logging

mode

writes

log-structured, versioned storage

Reclaims

Figure 6: Data paths in logging and non-logging modes.The dotted box represents a single chunk server acting asa primary.

sioned store instead of the secondaries. When all secon-daries are available again, the primary startsreclaimingthe writes: data is read from the versioned store, writtento all replicas, and then deleted from the versioned store.

Although the short-term store can share disk resourceswith the existing storage, i.e., the chunk servers, it islogically a separate service. Versions are essential inthe short-term store to ensure that the state visible toclients is consistent and recoverable. The chunk serverson the other hand are designed to run on standard filesystems such as NTFS or ext3, which do not support ver-sioning. Using a separate implementation for the short-term store also allows us to optimize it for the write-dominated workload and the relative short lifetime ofstored data items. Specifically, the short-term store usesa log-structured disk layout which is known to work wellin this scenario.

The short-term store has two components: alog clientthat is associated with each primary, andloggers whichsend writes to a local on-disk log file. When in loggingmode, the log client sends each write tor loggers in r dif-ferent racks, maintaining the same fault-tolerance prop-erties as the chunk data. Log clients track the locationand version of logged data in memory; this state can bereconstructed from the loggers after a failure. Figure 6shows the logging and reclaim data paths from the pointof view of a single chunk server primary.

When sending writes to the versioned store, the pri-mary also writes them to the local replica. This allowsreads to be served from the local replica and reduces theload on the versioned store. It also avoids rewriting thiscached data to the primary during reclaim. The metadatathat tracks these locally cached writes is kept in memory,is small, but is not recoverable. Thus, this optimization

C C C L

C C C L

C C C L

C C C L

(a) Dedicated

C C C C

C C C C

C C C C

C C C C

L L L L

L L L L

L L L L

L L L L

(b) Co-located

Figure 7: Two ways of configuring loggers (L) and chunkservers (C)

only helps primaries that have not failed or migrated. Inall cases correctness is maintained, with data being readfrom the loggers, if necessary, to service a client read.

Loggers can be run on dedicated servers or co-locatedwith chunk servers. Figure 7 shows examples of a dedi-cated and a co-located configuration. The dedicated con-figuration has the advantage that it minimizes contentionbetween the chunk server workload and the logger work-load, specifically allowing the loggers to service mostlywrites, for which they are optimized. The dedicated con-figuration does require additional resources; however, weexpect that these additional resources will be small, e.g.,one dedicated logger per 20 chunk servers in a rack.

For each logging mode write, the log client can chooseany r available loggers that are on different racks. Inour previous work using the versioned store, these werechosen primarily by disk load. However, in a scalabledistributed system such as Sierra, it is important also tominimize the network overheads of using the versionedstore, and especially to minimize the use of scarce cross-rack network bandwidth. Hence for every log write, thelog client for a chunk groupG sorts the loggers in itslogger view in the following order:

1. Loggers on the same server as a replica ofG,2. Loggers in the same rack as a replica ofG,3. Loggers in other racks.

Within each of these groups, loggers are sorted by diskload. For each log write the client greedily chooses thefirst r loggers that are in different racks. For log readsand reclaims only one logger is needed, the one closestto the primary (either co-located or on the same rack).

3.7 Fault tolerance and recovery

Sierra uses standard techniques (heartbeats and primaryreassignment) to maintain read availability during tran-sient errors; additionally it uses the short-term versionedstore to maintain write availability. Here we describehow we handle new failure modes resulting from gear-shifting.

Failures when in low gear: Chunk server failures

7


might occur when in low gear, when some of the chunkservers are already in standby. When the MDS detectsfailure of a chunk serverS, it wakes up all the servers thatshare any chunks withS. Since wakeup from standbytypically takes a few seconds, and even powering up amachine can be done in minutes, this does not signifi-cantly increase the window of vulnerability for a secondand third failure. However, when the system is alreadyin the lowest gear (gear 1), failure of a server can causethe last active replica of a chunk to become unavailablewhile other replicas are being woken up. This will re-sult in a large latency penalty for any client accessing thechunk during this window.

The Sierra gear scheduler takes theminimum gearlevel gmin as a policy input. The value of this parameterdepends on the desired tradeoff between power savingsand the risk of temporary unavailability on failure. Weexpect that for a 3-way replicated system,gmin will typ-ically be 1 (for higher power savings) or 2 (for higheravailability). Gear 0 is problematic because any accessto a chunk will see a large latency penalty even in theabsence of failures.gmin = 3 will not save any power.

Logger failures: Logger servers can also fail. Whena server fails, this log data becomes unavailable; how-ever, two other replicas of each log record are still avail-able on other servers. Thus, logged data has the samelevel of fault tolerance as unlogged data. One option tomaintain this fault tolerance is to re-replicate data withinthe logging service on failure. However, since the datawill eventually be reclaimed back to the chunk replicas,this results in wasted work. Instead, Sierra primaries re-claim at high priority any at-risk data, i.e., logged datawith fewer than three available replicas. At the end ofthe reclaim, the data will be no longer on the loggers butthree-way replicated on chunk servers.

Permanent failures in low gear: On a permanent fail-ure, the MDS initiates the rebuild of data stored on thefailed server; this requires peers of the failed server tobe powered up to participate in the rebuild. Sierra pow-ers up peers of a serverS whenever a transient failure issuspected onS. Hence the time to power up the peersis overlapped with the detection of permanent failure. Inany case, the time to transfer the data to new replicasdominates the total recovery time. Hence, waking upmachines in standby does not significantly increase thewindow of vulnerability to a second permanent failure.

Replica divergence: In a primary/backup replicationsystem such as Sierra, it is possible for a server failureduring a write request to result in replica divergence, withsome replicas having applied the write and others not.Note that this problem is not specific to Sierra but to allsystems which apply updates concurrently to replicas. InSierra primaries react to update failures by re-sending theupdate to the versioned store. The versioned store avoids

replica divergence by using explicit versions. If the pri-mary fails, then the client will retry the request, and thenew primary will send the update to the versioned store.However, if both the primary and the client fail while awrite request is in flight, then replica divergence is pos-sible. If this scenario is a concern, thenchain replica-tion [13] could be used, where updates are applied seri-ally to the replicas rather than in parallel. Chain repli-cation prevents replica divergence at the cost of higherupdate latencies. We have not currently implementedchain replication; however, adding it to the system onlyrequires small changes and is orthogonal to the gearingand power-saving aspects of Sierra.

3.8 Implementation status

The evaluation in the following section is based on ourSierra prototype, which is implemented entirely at userlevel, with the MDS and each chunk server each runningas a user-level process, and a client-side library that ex-ports objectread(), write(), delete() and create() calls.The core Sierra implementation is 10 KLOC of C code,with an additional 8 KLOC for the logger and log clientimplementations. Although the MDS is implementedas a deterministic state machine we have not currentlyimplemented MDS replication; however, standard tech-niques exist for state machine replication and we are con-fident that the MDS could be replicated if required.

4 Evaluation

In Section 2 we saw that large data center services suchas Hotmail as well as small data center services suchas the Cambridge file servers, have substantial poten-tial for power savings. Realizing this potential usingSierra requires sufficient, predictable troughs in the I/Oload. Additionally, we would like the baseline systemto have good, scalable performance, to maintain perfor-mance while in low gear and while transitioning betweengears, and to have efficient rebuild of data on server fail-ure. In this section we evaluate these different aspects ofSierra using real workloads as well as microbenchmarks.

First, Section 4.1 evaluates the accuracy of our loadprediction algorithm as well as the expected power sav-ings, from the three workloads described in Section 2:Hotmail, Messenger, and Cambridge. This analysis isbased on coarse-grained measurements of load aggre-gated over the entire services for one week.

The rest of the section then evaluates the Sierra pro-totype running on a cluster testbed. We use I/O requesttraces from a small sample of Hotmail servers to mea-sure the power savings and performance on real hard-ware. Using microbenchmarks, we then show the scal-

8


0

0.5

1

1.5

2

2.5

3

1 2 3 4 5 6

No

rma

lis

ed

RM

S e

rro

r

Number of training days

CambridgeHotmail

Messenger

Figure 8: Prediction accuracy

ing of the base system’s read/write performance and datarebuild rate as function of layout.

4.1 Load trace analysis

We applied the simple “hour of day” load prediction al-gorithm (see Section 3.4) to the aggregated load mea-surements described in Section 2; the load metric usedwas mean bytes transferred per hour, since that is all wehave available from this data. For each of the three work-loads, we have 7 days of data. We train the predictor onthe firstn days of the data and test it on the remaining7 − n days. The error metric is the root-mean-square(RMS) error, normalized by the mean load during thetest period to give a scale-free measure of error. Figure 8shows how the error changes asn is increased.

We see that for Hotmail and Messenger, the error islow even after a single day of training and does notchange significantly afterward. For Cambridge this error,while initially high, drops as more training data is used.These load errors translate into gear selection errors asfollows (using 6 days for training and 1 for testing): forHotmail all gears are correct; for Messenger, 90% of thetime the gears were correct; for Cambridge, 75% of thetime the gears were correct.

The left half of Figure 9 estimates the correspond-ing power consumed, defined as the average fraction ofservers that cannot be switched off using Sierra. Thenumbers assume correct gear selection as defined by theactual load (i.e., an “oracle” predictor). The power sav-ings look promising, but we cannot deduce how wellworkloads would perform in lower gears. In the nextsection, we evaluate power savings and performance fora live run of Hotmail I/O traces on real hardware.

4.2 Hotmail I/O traces

The next few sections show results obtained using I/Otraces from 8 Hotmail back-end servers over a 48-hourperiod, starting at midnight (PDT) on Monday August 42008. Note that these I/O traces are from a different timeperiod than the Hotmail load traces shown so far. TheI/O traces are taken at the block device level, i.e., be-low the main memory buffer cache but above the storagehardware. During our experiments we disable caching,prefetching and write-backs, thus enabling accurate tracereplay. During trace collection, each I/O to a block de-vice results in a trace record containing the timestamp,the device number, the type of request (read or write),the logical block position accessed, and the number ofblocks read or written.

The I/O traces include accesses both to data files (e-mail messages), which form the bulk of the storage ca-pacity used, and metadata databases (user profiles, searchindexes, etc.). Data files can be directly stored as objectsin Sierra, whereas the metadata would be best stored us-ing distributed tables. Since Sierra is a blob store andnot a distributed table service, we ignore accesses to themetadata, which would have to be hosted elsewhere.

We map the traces to Sierra usingvirtual disks. Eachblock device in the trace maps to a virtual disk, whichcorresponds to a unique object ID in Sierra. The vir-tual disk object is then stored in Sierra as a set of chunkscorresponding to logical extents within the disk. Thus,the trace replay mechanism converts an access of<blockdevice, logical block number, size in blocks> to <objectID, offset in bytes, size in bytes>.

We measure the power savings and performance ofSierra based on these traces. As we only have 2 days oftraces it is not meaningful to train a model for gear pre-diction. Hence, we use the “oracle” gear selection policyfor these I/O traces.

4.3 Testbed setup and provisioning

Our experimental testbed consists of 31 identical serversin 3 racks in one data center. Each rack has a CiscoCatalyst 3750E as a Top-of-Rack (ToR) switch providing1 Gbps ports for the servers, and a 10 Gbps fiber uplink toa Cisco Nexus 5000. The testbed is assigned 10 serversin each rack, plus an extra server in one of the racks onwhich we run the MDS. Each server has two four-core2.5 Ghz Intel Xeon processors, 16 GB of RAM, a 1 TBsystem disk and a 1 TB disk that holds the Sierra chunkfiles and log files. Although the machines have plentifulRAM, we do not use it for caching in our experiments,to match the traces available, which are taken below themain memory buffer cache. Each server runs WindowsServer 2008 Enterprise Edition, SP1.

9


min= 1

Messenger

min= 2

Hotmail

= 1 (chunksrv)

= 2 (chunksrv)��

�� Cambridge

gmin

Messenger

gmin

Hotmail

gmin= 1 (chunksrv)

gmin= 2 (chunksrv)

�� !� "!� #� � �� $�� %& '� ��!�( Figure 9: Estimated power consumed for three servicesand actual power consumed for the Hotmail I/O trace

A note on terminology: throughout the paper we usebytes, not bits, e.g., MB, not Mb, and standard powers-of-two notation, e.g., 1 KB is 1024 bytes (not 1000).

For meaningful experimental results it is importantto correctly provision the system for the workload, i.e.,chose the correct number of chunk servers. Overpro-visioning the system would increase the baseline sys-tem’s power consumption unnecessarily, and thereby in-flate the relative power savings of Sierra. Underprovi-sioning the system would also be meaningless becausethe system could not sustain the peak load even with allservers powered up.

In addition to performance, we must also match theavailability and capacity requirements of the workload.For availability, we place each replica in a separate faultdomain (in our case, in a separate rack). For capacity, weare limited by the total storage capacity of our servers,which is not sufficient to hold the entirety of the virtualdisks in the trace. However, it is sufficient to store thespecific chunks that are accessed during any of our ex-perimental runs, if we use a chunk size of 1 MB. Hence,for each experiment, we pre-create exactly the chunksthat are accessed during the experiment, using a chunksize of 1 MB. In practice a larger chunk size, e.g., 64 MBis more common [5].

To calculate the number of chunk servers neededto support a workload, we use the load metricL de-scribed in Section 3.4. It converts four workload met-rics — streaming read and write bandwidth (in MB/s)and random-access read and write IOs per second (IOPS)— to a single metric in units of chunk servers. Theserver metrics (e.g.,ServerIOPSwrite), are obtainedby benchmarking the servers in the system (Table 4 inSection 4.6 shows the results.) We then use the maxi-mum value ofL over the trace, rounded up to the nearestmultiple of 3 for 3-way replication.

The peak load in our traces happens just after mid-night during what we believe is a 2-hour period of main-

0

10

20

30

40

50

60

06:00 12:00 18:00 00:00 06:00 12:00 18:00 00:00

Lo

ad

level (#

serv

ers

)

Time (4-5 August 2008)

Steady state

Up-shift

Down-shift

Gear levelPeak load

Figure 10: Gear schedule for Hotmail I/O trace

tenance background activity. We do not provision for thispeak, but for the peak excluding this maintenance win-dow. We do keep the maintenance period in the highestgear though. If we provisioned for the background peak,or ran the 2-hour period in gear 1 (intuitively the back-ground jobs need to complete but are unlikely to havetight performance requirements) we would save morepower.

For the Hotmail traces this methodology resulted in15 chunk servers to meet the performance requirement(5 in each rack). The trace replay clients are load bal-anced on 9 of the remaining machines (3 in each rack).For all the performance experiments, we compared twoconfigurations. TheSierra configuration had the above 5chunk servers and 1 dedicated logger per rack. TheBase-line configuration was provisioned with the same totalresources, i.e., 6 chunk servers per rack and no loggers.In all cases we pre-create the necessary system state, in-cluding logger state, by first replaying the relevant pre-ceding portions of the trace (up to 8 hours in one case).

4.4 Power savings

This section presents the results of replaying the Hotmailtraces on the Sierra testbed. We first show the power sav-ings achieved using the gear scheduler’s decisions. Wethen present the performance results from live runs.

Figure 10 shows the load metric for each hour of theHotmail I/O trace, and the gear chosen by the oracle pre-dictor for each hour. The right half of Figure 9 showsshows the actual power consumed as a percentage of thebaseline system, which has the same total number ofservers (18). We show the results for two policies: thedefault policy (gmin = 1) where the system is requiredto keep a minimum of one active replica per object, and a“high availability” policy (gmin = 2) where two replicasare always kept active per object, to avoid any temporaryunavailability if a server fails.

10


��

��

�� !"� #$%& � !"�'��((a) Mean response time

)**)+*,**,+*-**-+*+**./0123 415 406

789:;<=: ><:??8*+*@**@+* ABCDEF GBDBC HI GJKLB MNOP GJKLBQ10R

(b) 99th percentile response time

Figure 11: Performance comparison for Hotmail I/O trace

We make two observations. First, the actual powersavings from this real run match the analytical expecta-tions for Hotmail. Second, even with agmin = 2 policythe power savings are significant. These numbers includethe power consumption of the loggers. Since our testbedis small, and we need one logger per rack at minimum,the loggers’ contribution is actually larger than in a realsystem. The last two bars in Figure 9 show the powerconsumed by the 15 chunk servers alone, as a percent-age of a baseline system with 15 chunk servers. We seethat the power savings are very close to that predictedfrom the Hotmail load traces (forgmin = 1). In a largersystem setup we expect a much lower ratio of loggers tochunk servers than the 1:5 seen here (this expectation isconfirmed in the next section), hence the relative powercost of the loggers will decrease.

4.5 Performance and efficiency

For the performance experiments, we selected three tracesegments that represent the worst case for three aspectsof Sierra. Thesteady-state experiment chooses, of all the1-hour periods spent in the lowest gear, the one with thehighest peak load. The aim is to show the performanceimpact of putting the system in a low gear. TheUp-shiftexperiment chooses the transition into the highest gearhaving the largest amount of logged data. The aim is toshow the performance impact of reclaiming logged datato the chunk servers. The run is from 10 minutes be-fore the transition until 1 hour after the transition. TheDown-shift experiment chooses the down-transition hav-ing the highest load in the minute immediately follow-ing the transition. The aim here is to show the effect onclients of the primary migration protocol and any result-ing client retries. The run is from 10 minutes before thetransition until 10 minutes after the transition. Figure 10shows the times in the trace corresponding to the threeexperiments.

Total data logged in 48 hrs 166 GBTime in top gear 14 hrsRequired reclaim rate 3.4 MB/sData reclaimed in up-shift 22 GBAchieved reclaim rate 6.3 MB/sLogger avg. queue size (steady state) 0.09Logger avg. queue size (reclaim) 2.3

Table 2: Reclaim statistics

Request response times: Figure 11 shows the per-formance of the baseline and Sierra configurations dur-ing the three experiments; we show both the mean re-sponse time and the 99th percentile response time. Wemake several observations. First, given that these are theworst three scenarios, the performance penalty for Sierrais small. Second, the steady-state and down-shift exper-iment results show that our provisioning methodology isreasonable; the performance in the lower gear (Sierra) iscomparable to the performance in gear 3 (Baseline). Forboth experiments the main reason performance slightlydegrades is that our provisioning method only consid-ers first-order performance metrics (IOPS and stream-ing bandwidth). In reality, workloads have second-orderproperties too (e.g., spatial and temporal locality) thatour method does not capture. Third, the up-shift experi-ment sees the worst performance degradation of all three,since the foreground workload in the highest gear inter-feres at the disk with the reclaim process. We look nextin depth at the reclaim rate and possibilities for improv-ing the performance during up-shift even further.

Reclaim rate: A key requirement is that the reclaimrate be sufficient so that the amount of logged data doesnot increase over the course of a day, since the loggersare not provisioned or optimized for long-term storage.Hence, we also estimated the required reclaim rate (tophalf of Table 2), by measuring the number of uniquebytes logged just before each up-shift in the 48-hour pe-

11


Number of migrations 102Total migration time 28 msNumber of retries 77

Table 3: Down-shift statistics

riod, and summing these values. This gives an upperbound on the amount of data that needs to be reclaimedwhen in high gear. Another key requirement is that thereclaim process should not interfere with the foregroundworkload. We saw in the previous experiment that theinterference can lead to some performance degradation.

The bottom part of Table 2 shows the measurementsfrom the up-shift run. We easily meet the first require-ment: the reclaim rate should be 3.4 MB/s, and weachieve 6.3 MB/s. Note that these reclaim rates indicatethe network is unlikely to be a bottleneck. We can meetthe second requirement and thus decrease performancedegradation further if we throttled the reclaim processto 3.4 MB/s. We believe the right way to slow this pro-cess down is to have support for native background (low-priority) I/O in Sierra, where the reclaim I/Os would getbackground priority. We are in the process of implement-ing this support.

Since the chunk servers are currently the bottleneckfor reclaim, we can increase the number of chunk serversper logger until the loggers or the network become abottleneck. In our experiments we found that the log-gers had an average disk queue size of only 2.3 whilereclaiming, indicating a low level of load to achieve al-most twice the target reclaim rate. The logger queue sizein the steady-state experiment was only 0.09. The net-work usage is similarly low. Hence we believe that in alarger system we can support a substantially higher ratioof chunk servers to loggers than in our small testbed.

Primary migration : We measured the time it takesto migrate primaries when down-shifting. This processshould be quick, so that few client requests have to retry.Measurements from the down-shift experiment indicatethat it took a total of 28 milliseconds to migrate 102chunk group primaries, as shown in Table 3. Withinthis period, each primary would have a smaller win-dow of unavailability corresponding to its own migra-tion. This resulted in 77 client requests retrying once(out of 254,536 total requests in the 20 minute interval).

Metadata state size: We measured the amount ofmetadata for the longest experiment, the up-shift one.The metadata service had about 100 MB of state in-memory. It contained information on 320 chunk groupsand 4.6 million chunks. Hence, the MDS has on average23 bytes of data per chunk or 320 KB of data per chunkgroup, i.e., a very small overhead.

Writes ReadsBandwidth (MB/s) 82/82/82 82/82/82IOPS 144/179/224 129/137/147

Table 4: Single-server performance. The min/avg/maxmetric is shown for 5 runs.

Writes ReadsBandwidth (MB/s) 96 (246) 348(738)IOPS 465(537) 1152(1233)

Table 5: Peak performance. In brackets is theideal per-formance as nine times the performance of a single serverfor reads, and a third of that for writes.

4.6 Microbenchmarks

The goal of this section is to measure using microbench-marks, the scalability of Sierra’s read/write performanceas well as the impact of layout on rebuild rates.

Single-server performance: This experiment estab-lishes a baseline single-server performance. First, wemeasure single client streaming read and write band-width from a single server in MB/s using 64 KB readsand writes to a 2.4 GB file. Second, we measure random-access read and write performance in IOPS (I/Os per sec-ond) by sending 100,000 IOs to the server. The clientkeeps 64 requests outstanding at all times. Table 4 showsthe results. Write performance is more variable than readperformance due to inherent properties of NTFS.

Multi-server performance: This experiment showsthe peak performance of our system when multipleservers and clients are accessing data. Rack 1 is usedfor chunk servers. Rack 2 is used for the clients. Themetadata service is placed on a machine in rack 1. 9clients (each identical in setup to the single one above)make read and write requests to 9 chunk servers. Table 5shows the results in terms of aggregate server perfor-mance. Variance is measured across clients. For all writeexperiments it is negligible. For the streaming read ex-periment the minimum client performance was 37 MB/sand the highest was 41 MB/s. For the random-accessread experiment the minimum client performance was109 IOPS and the highest was 137 IOPS.

Several observations can be made. First, in all cases,reads are faster than writes (by about 3x) because 3-wayreplication reduces the writes’ “goodput” to one-third.Second, the servers’ disks become the bottleneck for therandom-access workloads. All numbers in those casesare close to the ideal. Third, we only get around a third ofthe expected bandwidth with streaming reads and writes.This is because of a well-known problem: lack of perfor-mance isolation between concurrent streams. Streamingaccesses from one client interleave with accesses from

12


100

150

200

250

300

Rebuild rate (MB/s)

64 MB chunks

1 MB chunks

0

50

100

0 2 4 6 8

Rebuild rate (MB/s)

Number of write servers

Figure 12: Rebuild rate

other clients and the combined accesses to the disk arenot sequential anymore. A technique like Argon [14]would solve the problem, but we have not yet imple-mented it.

Recovery and layout: Choosing the correct layout isimportant for power savings, but also for recovery dur-ing permanent failures. Our scheme accommodates bothneeds. This next experiment shows recovery rates whenwe kill one server holding approximately 200 GB of data.We vary the number of write servers one can rebuild on.This serves to show the difference between naive group-ing and power-aware grouping, first discussed in Sec-tion 3.3. With naive grouping, there would only be onewrite server, and the rebuild rate for the failed server’sdata correspondingly low. Figure 12 shows the results.As expected, the rebuild rate increases linearly with thenumber of servers.

We used a chunk size of 1 MB throughout the paperdue to the need to accommodate the Hotmail I/O trace’scapacity requirement on our testbed. Chunk sizes couldbe larger in practice, and we re-evaluated rebuild rateswith a chunk size of 64 MB. As seen from Figure 12,the large chunk size improves the rebuild rate substan-tially. Although we are rebuilding 200 GB in both cases,larger chunks mean fewer per-chunk fixed costs associ-ated with copying a chunk (e.g., fewer file creates, whichare expensive). Larger chunks also reduce the perfor-mance interference mentioned earlier, between multiplechunks being rebuilt in parallel on the same server. Inour experiments, we are able to increase rebuild rate lin-early with the number of write nodes, at both 1 MB and64 MB chunk sizes. With a sufficient number of writenodes, the network would become the bottleneck; due tothe small scale of our testbed, the chunk servers remainthe bottleneck.

5 Related Work

Section 3 described the non-power related previous workon which Sierra builds on: scalable distributed storagesystems [1, 2, 5, 10] and our own previous work on short-term versioned stores [7, 8]. We also note that a largeamount of work exists in saving power by turning offservers that are stateless or that have in-memory state thatcan be migrated. This allows consolidation of the CPUworkload during troughs and is complementary to Sierra.

Here we contrast Sierra with previous work on savingenergy in server storage. Sierra is designed for clustersof commodity servers, where “failure is a common case”and the system must keep data available despite serverfailures. Previous work is mostly aimed at saving powerwithin RAID arrays attached to individual servers, anddoes not address the challenges of maintaining availabil-ity in a distributed system. A second key difference isthat in Sierra I/O requests do not block waiting for a com-ponent to wake up from a low power state. This “spin-up wait” is a problem for many schemes based on enter-ing low power states when idle. Powering entire serversdown would extend this wait from seconds to minutes.

Write off-loading [7] exploits long periods of low,write-only load on RAID arrays on enterprise servers.During these periods, all the disks in the array are spundown, and the write load is consolidated onto versionedlogs on a small number of active volumes, and reclaimedin the background when the original volume is spun up.However reads that go to a spun-down volume cannot beserviced until all the disks spin up again.

Popular Data Concentration (PDC) [9] exploits spatialrather than temporal workload properties, by periodicallymigrating hot data onto a small number of disks, andspinning down the cold disks. However, when the “cold”data is eventually accessed, the relevant disk must bespun up. PDC intentionally unbalances the load acrossdisks, and a subset of the disks must now deliver the per-formance previously delivered by all of them. Hence,this is only applicable for disk arrays that are overprovi-sioned for performance, i.e., capacity-constrained ones.

Hibernator [16] adapts to varying loads by changingthe rotational speed of disks in a RAID array. Data is keptavailable except while changing speeds. However this re-lies on multi-speed disks, which are not widely availabletoday and they seem unlikely to enter widespread use.

Power-aware RAID (PARAID) [15] applies the notionof “gears” to RAID arrays. Data is simultaneously storedon multiple RAID arrays of different sizes, overlaid onthe same set of disks. A RAID array with fewer disksis used when load is low, with the remaining disks beingspun down. Data is kept available at the price of keepingone copy of it in each overlaid RAID array. This wastesboth capacity and write bandwidth.

13


6 Future work and conclusion

In the short term, there is room for fine tuning theimplementation of our protocols and improving perfor-mance (e.g., especially for handling performance insu-lation among concurrent streams). In the medium-termwe plan to investigate the potential for more fine-grainedgearing, i.e., turn individual servers on and off in re-sponse to load rather than entire gear groups. We alsowant to investigate the tradeoffs involved when erasurecodes are used for redundancy instead of replication.

In addition, we believe our work on Sierra opens upthree broad future research directions. First, it is worthexamining ways of “filling the troughs” with more work,rather than powering servers down. For very large datacenters the server cost is higher than the power cost [6]and improving server utilization could be more cost effi-cient than saving power. Troughs could be filled by run-ning additional background tasks unrelated to the currentservices, e.g., by offering a generic utility computing ser-vice. Additionally, existing services could be modified sothat their background maintenance tasks are more care-fully scheduled.

Second, work is needed to align methods methods forconsolidating computational tasks (e.g., virtualization)with the I/O load consolidation that Sierra offers. Re-moving both CPU and I/O load from a server allows theentire server to be powered down, rather than individ-ual components. Ideally the system should also preservelocality while shifting gears, i.e., the co-location of com-putation with the data it computes on.

Third, more work is needed to achieve power-proportionality for optimistic concurrency control sys-tems such as Amazon’s Dynamo [4], which uses “sloppyquorums” to achieve a highly available, “eventually con-sistent” system. Similarly it would be interesting to ex-tend the ideas from Sierra (which has a fail-stop failuremodel) to Byzantine fault-tolerant systems.

In conclusion, Sierra is, to the best of our knowl-edge, the first power-proportional distributed storage sys-tem. Achieving power-proportionality required main-taining similar consistency, fault-tolerance and availabil-ity as a default storage system, all while ensuring goodperformance. Live runs of a subset of a large-scale ser-vice, Hotmail, confirmed that achievable power savingsmatch expected ones.

7 Acknowledgements

We thank Bruce Worthington and Swaroop Kavalanekarfor the Hotmail I/O traces, Tom Harpel and Steve Lee forthe Hotmail and Messenger load traces, Dennis Crain,Mac Manson and the MSR cluster folks for the hardware

testbed and the IT facilities in Microsoft Research Cam-bridge for the Cambridge traces.

References

[1] Windows Azure Platform. http://www.microsoft.com/azure, Oct. 2009.

[2] M. Abd-El-Malek, W. V. Courtright II, C. Cranor, G. R.Ganger, J. Hendricks, A. J. Klosterman, M. Mesnier,M. Prasad, B. Salmon, R. R. Sambasivan, S. Sinnamo-hideen, J. D. Strunk, E. Thereska, M. Wachs, and J. J.Wylie. Ursa Minor: versatile cluster-based storage. InProc. USENIX Conference on File and Storage Technolo-gies (FAST), Dec. 2005.

[3] L. A. Barroso and U. Holzle. The case for energy-proportional computing.IEEE Computer, 40(12):33–37,2007.

[4] G. DeCandia, D. Hastorun, M. Jampani, G. Kakula-pati, A. Lakshman, A. Pilchin, S. Sivasubramanian,P. Vosshall, and W. Vogels. Dynamo: Amazon’s highlyavailable key-value store. InProc. ACM Symposium onOperating Systems Principles (SOSP), Oct. 2007.

[5] S. Ghemawat, H. Gobioff, and S.-T. Leung. The Googlefile system. InACM Symposium on Operating SystemPrinciples, Lake George, NY, Oct. 2003.

[6] J. Hamilton. Resource consumption shaping.http://perspectives.mvdirona.com/2008/12/17/ResourceConsumptionShaping.aspx,Jan. 2008.

[7] D. Narayanan, A. Donnelly, and A. Rowstron. Write off-loading: Practical power management for enterprise stor-age. InProc. USENIX Conference on File and StorageTechnologies (FAST), San Jose, CA, Feb. 2008.

[8] D. Narayanan, A. Donnelly, E. Thereska, S. Elnikety, andA. Rowstron. Everest: Scaling down peak loads throughI/O off-loading. Proc. Symposium on Operating SystemsDesign and Implementation (OSDI), 2008.

[9] E. Pinheiro and R. Bianchini. Energy conservation tech-niques for disk array-based servers. InProc. Annual Inter-national Conference on Supercomputing (ICS’04), June2004.

[10] Y. Saito, S. Frølund, A. Veitch, A. Merchant, andS. Spence. FAB: building distributed enterprise disk ar-rays from commodity components. InProc. InternationalConference on Architectural Support for ProgrammingLanguages and Operating Systems (ASPLOS), Oct. 2004.

[11] Schneider. Implementing fault-tolerant services using thestate machine approach: A tutorial.CSURV: ComputingSurveys, 22, 1990.

[12] SNIA. IOTTA repository. http://iotta.snia.org/, Jan. 2009.

[13] R. van Renesse and F. B. Schneider. Chain replicationfor supporting high throughput and availability. InOSDI,pages 91–104, 2004.

14


[14] M. Wachs, M. Abd-El-Malek, E. Thereska, and G. R.Ganger. Argon: performance insulation for shared stor-age servers. InFAST, 2007.

[15] C. Weddle, M. Oldham, J. Qian, A.-I. A. Wang, P. Rei-her, and G. Kuenning. PARAID: The gear-shifting power-aware RAID. InProc. USENIX Conference on File andStorage Technologies (FAST’07), Feb. 2007.

[16] Q. Zhu, Z. Chen, L. Tan, Y. Zhou, K. Keeton, andJ. Wilkes. Hibernator: Helping disk arrays sleep throughthe winter. InProc. ACM Symposium on Operating Sys-tems Principles (SOSP), Brighton, United Kingdom, Oct.2005.

15

Sierra: a power-proportional, distributed storage system

Documents

Transcript of Sierra: a power-proportional, distributed storage system