An Architecture to Stimulate Behavioral Development of Academic Cloud UsersI

Gabor Kecskemetia,b,∗, Simon Ostermanna, Radu Prodana

aDistributed and Parallel Systems group of the Institute of Computer Science at the University of Innsbruck, Technikerstraße 21a,Innsbruck 6020, Austria

bLaboratory of Parallel and Distributed Systems of the Institute of Computer Science and Control of the Hungarian Academy of Sciences,Kende u. 13-17, Budapest 1111, Hungary

Abstract

Academic cloud infrastructures are constructed and maintained so they minimally constrain their users. Since theyare free and do not limit usage patterns, academics developed such behavior that jeopardizes fair and flexible resourceprovisioning. For efficiency, related work either explicitly limits user access to resources, or introduce automatic rationingtechniques. Surprisingly, the root cause (i.e., the user behavior) is disregarded by these approaches. This article comparesacademic cloud user behavior to its commercial equivalent. We deduce, that academics should behave like commercialcloud users to relieve resource provisioning. To encourage commercial like behavior, we propose an architectural extensionto existing academic infrastructure clouds. First, every user’s energy consumption and efficiency is monitored. Then,energy efficiency based leader boards are used to ignite competition between academics and reveal their worst practices.Leader boards are not sufficient to completely change user behavior. Thus, we introduce engaging options that encourageacademics to delay resource requests and prefer resources more suitable for the infrastructure’s internal provisioning.Finally, we evaluate our extensions via a simulation using real life academic resource request traces. We show a potentialresource utilization reduction (by the factor of at most 2.6) while maintaining the unlimited nature of academic clouds.

Keywords: Cloud Computing, Pricing, Infrastructure as a Service, Energy Awareness, Academic Clouds

1. Introduction

Academic computing infrastructures are built and main-tained in order to support scientific users in their researchendeavors. Introducing limitations on the hardware usagein any ways would defeat the very reason for the existenceof these infrastructures. However, the more limitless a sys-tem is the more responsibility it requires from the scientificusers. For example, they must learn to eliminate their im-pact on other user’s workings. Therefore, maintainers ofsuch systems traditionally make the compromise of intro-ducing such limitations for the users that stop uninten-tional obstructions on the work of other users [1]. Mean-while, for future systems, computer science tries to reducethe amount of limitations and their impact on the scientificusers.

Infrastructure as a service (IaaS) cloud computing sys-tems [2] are amongst the most recent developments inthis field. These systems offer on demand resource ac-cess with such flexibility in software configurations [3] thatthe users could even utilize highly customized operating

IThe research leading to these results has received funding fromthe Austrian Science Fund projects TRP 237-N23 and ICT COSTAction IC1305.

∗Corresponding author. Tel: +36 1 279 6065Email addresses: gabor@dps.uibk.ac.at (Gabor Kecskemeti),

simon@dps.uibk.ac.at (Simon Ostermann), radu@dps.uibk.ac.at(Radu Prodan)

systems and support environments for their tasks. Thisflexibility is achieved through the application of virtual-ized data centers. Although, the cloud computing concepthas been proposed by commercial companies (e.g., Ama-zon1, Rackspace2), academic solutions (like Eucalyptus [4],Nimbus [5] or OpenNebula [6]) started to arise first by im-itating the behavior of the commercial solutions then byadvancing towards specific academic needs.

Pricing is one of the essential aspects of commercial IaaSsystems [7] that academic solutions did not copy. Thusacademic providers who apply such academic solutions willappear as offering unlimited resources for free to academicusers. This promise is tempting for the users as it lifts oneof their last remaining limitations. Unfortunately, this set-ting leads to an unprecedented demand of resources that isoften latent (e.g., users maintaining demand for resourcessimilarly to pilot jobs in grids [8]).

Academic providers have to fulfill these demands withthe limited physical resources they are operating on. Tomeet the demands with the infrastructure’s real capabili-ties they usually apply two solutions: (i) access rationing,(ii) under provisioning (N to 1 mapping of virtual to physi-cal resources). Both approaches were utilized in academicinfrastructures even before the cloud era, but they both

1http://aws.amazon.com/ec22http://www.rackspace.com/

Preprint submitted to Sustainable Computing: Informatics and Systems April 17, 2014

have serious downsides for academic uses. First, accessrationing directly intrudes the freedom of researchers ac-cess to the infrastructure [9]. For example, when a creditsystem is applied for rationing, then the research of userswith no credits could be postponed for indefinite time pe-riods (i.e., until they acquire some new credits for compu-tation). Second, providers with under provisioning poli-cies promise resources that are heavily shared amongstusers [10], therefore these shared resources could vary inperformance over time (the unintentional effects of otherswho introduce background load to the shared resource).

Instead of the previously applied solutions, we proposeto direct users towards self-rationing. We derived therationing problem from the missing pricing in academicclouds and argue that it is possible to construct academicsystems that feature similar behavior to commercial clouds(where the rationing is imposed by the cost of further leas-ing resources) but still promises unlimited resources andunprecedented software configurability. We achieve thisbehavior with an architecture that exposes energy effi-ciency metrics to the users. First, our architecture pro-vides the foundations for various leader boards where aca-demic users can compete with each other on how energyefficient they use the acquired computing resources. Sec-ond, to ignite the rivalry on the leader boards, we recom-mend providers to allow the specification of energy relatedconstraints on resource requests. Finally, we introduce theconcept of engaging options (electronic representations ofunderused capacities) that allow users to attract others forparticular resources. These options offer a chance to usersto utilize resources from hosts that are already used andthus these hosts could operate more energy efficiently.

The proposed architecture is built on three fundamentalassumptions: (i) availability of energy readings, (ii) ap-plication of energy aware virtual machine placement, and(iii) leader board publicity. First, we require the energyreadings because we propose to publicize the accountableuser consumption either directly or on a transformed waythrough leader boards. Second, users should be able toinfluence their leader board position, thus particular re-source requests should have deterministic energy behav-ior. This behavior should be guaranteed by an energyefficient virtual machine placement policy at the provider.Finally, the expected effects of the leader boards and en-gaging options are really dependent on their publicity, thusit is expected that they are soon adopted by a significantpercentage of the academic community. For example, theadoption could be forced by the providers by automaticallyenlisting their users in local leader boards.

To test the feasibility of our architecture and its posi-tive effects on the academic cloud communities, we haveanalyzed the behavior of typical academic users. Firstwe have classified the users by behavior and identifiedthe ways users could be transformed to behave more self-constraining while still performing their tasks. Second,we simulated the possible behavior of the academic usersfound in the Grid Workload Archives. Based on our sim-

ulations, we have concluded that there is a high chanceof increasing energy efficiency and reducing resource de-mand on the provider side while still performing all usertasks. Our findings show, that the effect of our architec-ture could decrease the energy footprint of the provider’scomputing infrastructure by a factor of 2.6 at most. Wehave also revealed that a few users could particularly re-duce the demands of the infrastructure, thus we introducedthe concept of the hall of shame (for the list of least ef-ficient users). This list should be presented alongside theleader boards in order to put immediate tension on themisbehaving users by their fellows.

The rest of the paper is structured as follows. First,we provide an overview on related research topics and theresearch issues in Section 2. Next, in Section 3, we revealour architecture that could answer the identified researchissues. Afterwards, in Sections 4 and 5, we discuss theinner workings of the architecture, first starting with thecharacteristics of leader boards, their contents and theirrelations with the users. Then, we continue with engagingoptions and we show their life cycle from the time they areissued to the time they are used or become invalid. Later,in Section 6, we analyze the effects of the architecture ina simulated environment. Finally, Section 7 concludes ourfindings and summarizes the architecture’s properties.

2. Related work

Cloud computing is interesting for the scientific com-munity from the beginning of the transformation of Ama-zon Web Services towards the Amazon Elastic ComputeCloud. Early evaluations investigated how scientists canbenefit from this new technological infrastructure com-pared to Grids [11], especially with respect to storage [12]and computation [13] costs.

Maximizing the revenue from infrastructure operationis an important objective of Cloud providers, analyzed in[14] based on SLA relationships. This work does not takeinto account the possible energy savings that may furtherhelp in reducing the costs.

In [15], it is shown that saving power can increase therevenue of Cloud providers with only slight impact in theoverall performance. Academic clouds are not adaptingsuch power optimizations promptly, because the personsresponsible for the resource usage are not responsible forthe incurred electricity costs.

Rigid allocation mechanisms such as accounting basedlimitations can restrict many cloud and grid computing usecases and [9] reduce the overall scientific productiveness.Contrary to credit or accounting based limitations, ourapproach motivates the users to optimize their resourceusage through reduced power consumption.

Existing resource management and scheduling systemssuch as SLURM [16] and Maui [17] incorporate fairsharemechanisms enforcing user quotas or data access withinownership domains which constrain scientists in using such

2

systems. These approaches however do not want to edu-cate their users on expected user behavior. Therefore, evenif academic clouds apply these resource management sys-tems, users could still behave on a way that these sched-ulers could not resolve with their fairshare mechanisms.For example, incoming virtual machine and job requestscould be so unnecessary large and long running that thesealgorithms cannot manage them energy efficiently on theirown without external assistance (e.g., someone/somethingsuggesting users for more energy efficient request timingand size that could still fulfill user tasks).

The work in [18] demonstrates that user awareness ofpower consumption results in possible savings. In this pa-per, we also aim at increasing user awareness but pro-pose to do so by using leader boards. The leader boardis a heavily utilized concept in computing to increaseuser involvement and awareness regarding particular top-ics (through user ranking and competition). Furthermore,several leader boards not only offer rankings for individ-uals, but also groups. Group rankings within a leaderboard system can be used to start additional competi-tion between countries, research institutes or other self-forming user groups (e.g., multinational research groups).Leader boards became widely known through projects likeSETI@home [19] or the TOP500 supercomputer ranking[20]. Similarly to these projects we also use the leaderboard concept, where we introduce a new ranking basedon energy consumption and efficiency.

[21] evaluated and quantified the impact of user feedbackon power savings as 5% - 15%. To allow feedback on thepower consumption, there is a need for a model that mapsthe physical power consumption to the virtual machines.Such models are provided in [22, 23] and show sufficient ac-curacy to be employed in our proposed architecture. Mea-suring the individual power consumption of VMs executedon a physical machine results in a higher power consump-tion per VM if the machine is under utilized and the staticidle power is shared [24, 25]. We apply this knowledge inour approach so we can present academics valuable infor-mation on how they can improve their energy efficiencymeasures.

There is also an interest in saving power in Cloud in-frastructures by turning off unused virtual machines tooptimize resource use [26, 27]. This case is relevant forcommercial providers, as users are not willing to pay forthe idle times of their virtual machines. Optimized VMplacement can save up to 55% of energy using the ap-proach in [28]. Combining the savings possible throughawareness of the energy consumption and optimized VMplacement can significantly reduce the power consumptionand further increase the productivity. Unfortunately, thesetechniques are only achieving these significant savings ifrational users are utilizing the infrastructure, which is notthe case with current academic users. Thus in order tofully utilize the effects of these techniques this paper aimsat transforming academic users to behave more rational(i.e. like the price constrained commercial cloud users).

Engaging options proposed in Section 5 show similarcharacteristics as referral programs studied in [29]. Byhelping the Cloud provider to better utilize its virtualizedhardware, the energy consumption accounted to the usersaccounts can be reduced. Research shows that such pro-grams can achieve up to 16% better results [29]. Referralsare often more personal than normal advertisements orSLAs and, therefore, show a high impact on user behavioras showed in [30]. Similar impact is possible with our ap-proach because engaging options from coworkers are moretrustworthy than SLAs from Cloud providers.

3. The architecture

3.1. Behavioral differences amongst cloud users

In response to the unprecedented latent demand fromusers, state of the art research focuses on changing theoperation of academic cloud providers so they no longerappear to provide unlimited resources to their users. Un-fortunately, this approach does not provide the cure forthe root cause: the misbehavior of the users. This misbe-havior is caused by the provider’s promise about unlimitedresources, and can be characterized as follows:

• Unnaturally and unnecessarily long infrastructureleases: E.g., academics frequently maintain their vir-tual machines even if they don’t use their resourcesjust to avoid the often long times they have to waitfor the resource and its preparation for their particu-lar need.

• Academic users also tend to prefer resources with thehighest performance without giving too much consid-eration on other properties of the acquired resources(e.g., availability, energy efficiency, effect on otherusers).

In contrast, users of commercial cloud systems behavemore rationally, since the prices imposed by commercialclouds ensure that users will not maintain economicallyunsustainable virtual infrastructures. These are the rele-vant characteristics of commercial users:

• They delay the instantiation of their virtual machines.For example, until these machines are an absolute ne-cessity for the further progression of user tasks.

• They try to ensure continuous use of the acquiredVMs (doing as much work as possible during the timethe VM is available).

• They terminate VMs early on (considering the billingperiods – e.g., on Amazon there is no use to termi-nate a VM before one hour). Thus they immediatelyterminate a VM that has no further tasks or it is notexpected to have a task for it in the foreseeable future.

3

Table 1: The effects of the various pricing models in commercial cloudsNote: this table is discussed throughout Sections 3–5, thus some of its definitions are put into context later on.

User characteristicsPricing models offered by commercial providers

Standard Reserved SpotDelay instantiation Need triggered Limit tasks to reserved VMs Price & need triggeredEnsure continuous use Use the VM regularly Prolonged use is desired Burst VM use until abortionEarly VM termination Terminate if not needed If unjustified, sell reservation

and switch modelTerminate if not needed

Performance compromise Instantiate a smaller VM Bound to a resource type Abrupt VM abortion

• They make a compromise between price and perfor-mance and allow increased task makespans. For ex-ample, a smaller priced instance could still be capableto perform the necessary tasks within the billing pe-riod of the provider, thus if the tasks are not timecritical, they could take a little longer. Or they wouldpostpone the execution of a new task until an exist-ing VM could run them (assuming that serializing thetasks is more cost effective than having a new VM forthe new task).

Commercial providers (like Amazon) establish such usercharacteristics through the following three pricing models:(i) Standard where the resource usage is paid with a con-stant hourly price, (ii) Reserved where a upfront paymentreduces the hourly prices and (iii) Spot where the com-promise of a possible VM abortion potentially lowers theprices. In Table 1, we show how the aforementioned usercharacteristics are achieved by these three pricing mod-els. The table also reveals the rational user actions onecan observe when a particular pricing model is applied.Fortunately, all these user actions are possible in currentacademic clouds (i.e., these actions can be accomplishedthrough the usually available IaaS interfaces). Therefore,if academics would have the incentive to take these actionsthen they would bear similar characteristics as commer-cial users. And commercial like user characteristics wouldallow academics clouds to maintain sufficient balance be-tween their users and resources.

3.2. IaaS extensions to support behavioral change

To encourage such user behavior, we propose to moti-vate the academics through presenting them the energyimpact of their operations. We propose such architecturalextensions to academic cloud environments that not onlycollect and present the energy consumption data to aca-demic users but also provide them information on how toincrease their efficiency. Figure 1 presents these extensionsto an existing IaaS software stack (shown in the bottomright corner) and shows their relations to academic users.As the extensions are aimed at academics, other actors likesystem administrators are not shown. In fact, these exten-sions do not even change the ways the users would interactwith the existing cloud environment, they just add an op-tional functionality that – if used widely – could lead users

Mimics Spot pricing model

Mimics Standard & Reserved

pricing model

IaaS

Clo

ud P

rovi

der

Power meter

Physical Machine 1

VM xVM xVM x

Power meter

Physical Machine 2

VM xVM xVM x

Power meter

Physical Machine N

VM xVM xVM x

...

Virtual Machine Manager

IaaS Frontend

Web UI CLIVMAPI

AccountingAPI

Leader boards

Academic User

Beha

vior

alte

ring

IaaS

ext

ensi

ons

VM Accounting

Engaging Option

Manager

Figure 1: Overall view of the proposed architecture

towards self-rationing. So the users still utilize their usualcommand line interfaces or web portals, but by having ac-cess to our extensions they are expected to change theirbehavior towards these interfaces. These extensions arebuilt on two cornerstones: (i) leader boards (for dissemi-nating and comparing how energy efficient are the users)and (ii) engaging options (to ensure that VM requests arearriving in groups more suitable for those physical ma-chines that will host the VMs). The following paragraphsdiscuss how these cornerstones are related to user behav-ior and show their basic properties. Later these two arefurther elaborated in the following two sections.

First, similarly to the account statements of commercialproviders, with our extensions, academic cloud providerscollect the energy consumption accounted to particular ac-tions of their users. Our architecture ensures that theypublish this data through an Accounting API. Throughthis API, trusted third parties are allowed to query theenergy consumption accounted to particular virtual ma-chines (e.g., a user can query the consumption of its ownVMs). Leader boards are special users of this API whocollect data regarding every registered user. They aggre-gate energy consumption data for users. Depending on the

4

intended user base, each leader board can use their ownmethod for aggregation. At the end, to each user theyassign a score that is comparable with other scores in thesame leader board. Every board presents a ranking listfor its users (the more energy efficient a user is the higherhis/her ranking is). This list is our major motivational in-strument as users often compare themselves to their peersand try to improve their ranking.

Next to the list, leader boards present users with tech-niques that could increase their score. Recommended tech-niques reveal how to accomplish similar behavior to theusers of commercial clouds. Table 1 also acts as a sum-mary for the recommended techniques. So, for example,academics are expected to terminate their unused virtualmachines. The details of the resulting behavior and scor-ing are discussed in Section 4. Unfortunately, there aresome user behavior (e.g., price and need triggered VM in-stantiation or the use of dedicated VMs for repetitive andprolonged tasks) that the accounting API and the leaderboards cannot impose on academics.

To guide users towards these behavioral patterns we alsointroduce the concept of engaging options. These optionsare such electronic documents that represent inefficienciesin the system. If one user is assigned to a physical machinethat’s power efficiency could be increased then he/she re-ceives some engaging options. The received options thencan be exchanged with other users. Amongst the tech-niques listed on the leader boards, users will be noted thatif they would share/wait for such options, then they wouldhave a chance to increase their scores. Thus users will actsimilarly as those in commercial clouds: they will wait fora suitable engaging option to appear before instantiatingtheir VMs (just like commercial users would behave forspot pricing). Thus, our architecture extends current IaaSsystems with the management of the entire life-cycle ofengaging options (from their issuing to their dissolution).Figure 1 refers to this extension as the Engaging OptionManager. We discuss its properties and behavior in Sec-tion 5.

4. Towards the behavior of static pricing models

Leader boards are known to have an attractive influenceon most scientists, as can be seen from TOP500 supercom-puting list [20] or the various volunteer computing solu-tions like [19]. Especially when ranked in groups [31], rank-ing can have high influence on all members of the group,thus they will all try to achieve higher scores. Therefore,we propose to set up leader boards where users can com-pete by using cloud resources in a energy efficient way. Wedefine leader boards as such entities that are independentfrom cloud providers but still able to present the account-ing data (as scores) to their users so they start to challengeeach other. In this section, we first analyze the relevantcharacteristics of leader boards. Then we show a way tomotivate academic users through leader boards so theystart to resemble their commercial counterparts.

Power meter

Physical Machine N

VM xVM xVM x

Power meter

Physical Machine 2

VM xVM xVM x

Power meter

Physical Machine 1

VM xVM xVM x

Virtual Machine Manager

IaaS Frontend

WebUI

CLIVMAPI

AccountingAPI

Leader boardsAcademic User

Accounting system

VM Energy Model

Extensible Scoring

Mim

ics

Stan

dard

& R

eser

ved

pric

ing

mod

els

Exte

nded

IaaS

Clo

ud P

rovi

der

Figure 2: Leader boards and our accounting extensions

4.1. An extensible scoring system

The scoring system and its presentation on the leaderboard represents a core asset. With improper solutionsthe academics will not be motivated to submit to the be-havioral patterns of commercial users. The major successof similar leader boards is based on the ability that userscan heavily influence their scores and thus can ignite com-petition amongst each other. The cornerstone of the scorebased motivation is that the leader board should presentthe way it transforms the accounting data to the actualscores. Since users are not expected to know the internalworkings of the cloud systems, the leader board should alsopresent the behavioral patterns that positively affect userscores. Users also receive a breakdown on their scores sothey can know how their particular activities affect theirscore. With all this information the users will have a bet-ter chance to influence their scores and they will performbetter in their competitions.

Motivation can be further increased with the forma-tion of user groups within a leader board [31]. In gen-eral, groups allow group members to compete with eachother. If groups also receive an overall score based on theirmember’s scores, then group members are encouraged topursue higher scores together by enabling the competi-tion of groups based on overall group scores. To reachhigher overall scores, enthusiastic users will try to con-vince more resistant users to revise their resource usagepatterns. Since group interaction is essential to strengthenthe overall motivation in the system, our leader boardsautomatically form groups based on user affiliation andinterest (e.g., groups are created for departments or com-puter scientists in general). These groups help building upthe initial momentum towards widespread user behavioralchanges, but users are not restricted to them. New groupscan be formed by enthusiasts also.

Since the scoring is such an important motivational fac-tor for both individuals and groups, our IaaS extensionsprovide a foundation for scoring systems. As seen in Fig-ure 2, this foundation is built around two vital elements:

5

(i) an accounting API, (ii) an extensible scoring system.Our accounting API is responsible to offer details (likeenergy, storage usage) about current and past virtual ma-chines in the academic cloud. While, the extensible scor-ing system offers ways to transform and aggregate thedata from accounting to stable and motivating scores pre-sentable through leader boards.

4.1.1. Accounting

The accounting API is the major information sourcefor the scoring system. Behind the API an accountingdatabase is set up on which the API offers a simplifiedview. This database contains static and dynamic datafrom various sources. The static data entries represent thegeneral properties of the infrastructure that are not of-fered by other information systems (like the energy char-acteristics of the available physical machines in the infras-tructure). While the dynamic entries represent virtualmachine related data recordings (e.g., energy consumed,storage used, network utilization, physical machine allo-cation). Most of the data aggregated in the accountingdatabase is already available or could be collected withsome means from the IaaS provider. For example, whenthe IaaS runs its virtual machine placement algorithm, itcan notify interested parties about the VM placement deci-sions. Alternatively, the virtual machine mappings can bedetermined by agents deployed on the physical machinesof the IaaS. Although these agents could report not onlythe placement information but other VM related informa-tion, their impact on energy metering results is undesir-able. Thus our extension offers hooks to placement algo-rithms and only falls back to the agents when these hooksare unusable by the original IaaS system. Unfortunately,the energy consumption details are unlikely to be availableon the level of the virtual machines. Thus this subsectionis focusing on how these details are collected and calcu-lated.

In order to collect the necessary data for VM energyaccounting, we first shortly review the life cycle of a vir-tual machine. There are four major phases in the VM’slife: (i) prelude (from the request for a VM until the usercan actually do some processing with the VM), (ii) run-time (while the VM is capable to do the tasks of theuser), (iii) migrating (while the VM is down because itis moving between physical machines) and (iv) post run-time (from the user termination request, until there are nofurther tasks performed by the IaaS regarding the VM).All phases, except runtime, represent operations done bythe IaaS because of the VM. To determine the energy con-sumed because of the existence of the VM, one shouldaccount for all the IaaS components that are involved inthe VM’s life cycle (e.g., IaaS frontend, VM Manager, orvirtual appliance storage subsystem). Thus overall VMenergy consumption could be described as follows:

E(VM) = EPL(VM)+ER(VM)+EM (VM)+EPR(VM)(1)

Where E(VM ) defines the total energy consumed by theVM , which is a composition of the energy consumed dur-ing the lifetime of the VM. Thus, EPL(VM ), ER(VM ),EM (VM ) and EPR(VM ) represents the VM’s consump-tion during the prelude, runtime, migration and post run-time phases respectively. This is the function that our VMEnergy Model component in Figure 2 estimates. The restof this subsection is focusing on the behavior of this com-ponent and how it estimates the energy consumption of aparticular VM.

There has been significant research on the runtime be-havior of VMs and their energy characteristics [23, 25]. So,these energy models can be utilized to determine ER(VM ).Unfortunately, these works do not consider the energy con-sumption of the other phases in the VM’s life cycle. Thereason they are not considered is because the other en-ergy consumption values are often not comparable to theruntime consumption of the VM (e.g., if the VM runs forseveral days, then the few minute long VM instantiationand termination does not increase E(VM ) significantly).On the other hand, in this article, we are trying to ensurethat users run their VMs for as short time as possible. Thisbehavior results in a smaller gap between the runtime con-sumption and the other consumption values. Therefore,we argue that these values should also be represented inthe accounting.

Per VM consumption of the IaaS. Since EPL(VM ) andEPR(VM ) are mostly dependent on the IaaS behavior, wepropose to check the consumption of those machines thatare not hosting virtual machines. These machines are thereto support the instantiation, the termination and otherVM management tasks. Although it is not an easy task toseparate consumption dedicated to virtual machines, werecommend to share the energy consumption of these ma-chines amongst the VMs that they handled in the periodwhen the consumption was recorded. If there are no sig-nificant differences between the user VM requests (e.g., nosignificant size difference on their appliances, or no unusualSLA requirements present that need extra operations onthe IaaS side), then we can assume this share is equal.If there are significant differences then, we recommend tofirst take into account the appliance size as the base of theshare:

EPL(VM ) + EPR(VM ) =EIaaS · size(VM )∑vm∈VMS size(vm)

(2)

In this equation, we mark the overall energy consumptionof the non hosting machines as EIaaS . VMS defines theset of all VMs that coincide with the VM in question.We refer to the size of the virtual appliance of a virtualmachine as size(VM ). The more virtual machines an IaaShandles in a given time period, the more constant this partof the energy consumption becomes.

Next, we also need to define the energy consumption ofVM migration. For the sake of simplicity, we decomposethis consumption in two parts: (i) NM (VM ) the number

6

of migrations that take place for a given VM, (ii) Em(VM )the estimated consumption of a single migration. We as-sume the cost of a single migration for a particular VM isconstant, and thus we will use the following equation forthe overall cost of migration:

EM (VM ) = NM (VM ) · Em(VM ) (3)

In most cases, when there is no migration during the VM’slifetime, this consumption can be neglected. On the otherhand, the longer the runtime of the VM the more likely itwill be accounted for migration energy also.

From the individual user point of view, it is impossi-ble to influence the consumption of the IaaS, thus userscould see it as a constant consumption that is accountedto them. In case of infrastructure initiated migrations, theuser has also no chance to directly reduce the consumptionEM (VM ). But the migration related consumption can bemitigated if the users terminate their VMs as soon as pos-sible, reducing the chances to be automatically migratedto some other host. This shows us that with this energymetric exposed to the academics, they would be temptedto terminate their VMs when the migration energy cost isstill marginal (this behavior is similar to commercial userswho time VM terminations according to billing periods).Finally, users only have direct influence on their runtimeconsumption. Therefore the following paragraphs are fo-cusing on what influences their VM’s runtime consumptionand how they can change it.

Runtime VM consumption. Finally, we shortly discuss themodel of runtime energy consumption. Virtual machineconsumption is derived from the physical machine’s con-sumption based on two components: (i) Eidle(VM ) theidle or static consumption, (ii) Euse(VM ) the usage re-lated consumption:

ER(VM ) = Eidle(VM ) + Euse(VM ) (4)

The first component is the result of the physical machine’sbase power draw that is consumed even without any par-ticular load on the machine. The second component is de-pendent on the actual use of the resources of the physicalmachine. In the followings we discuss how these compo-nents are considered in our accounting database.

Albeit, related works (e.g., [23, 25]) vary on how theymap the idle consumption of the host to a particular VM,they all agree that this component is heavily dependenton the number and kind of virtual machines that sharethe physical host. Therefore, the virtual machine place-ment heavily impacts how the idle component of the VM isevaluated: by opting for multi-tenancy or by choosing ma-chines with smaller idle consumption academics could re-duce the idle component of their VMs. In the first case, theusers willingly share resources and thus reduce the mini-mum size of the infrastructure that could serve them (di-rectly increases flexibility). In the second case, academics

could put economical considerations before their perfor-mance requirements, thus making a performance compro-mise similar to commercial cloud users (see Table 1 aboutuser applied bounds to specific resources or smaller pricedresources because of pricing advantages).

To support the usage related consumption calculations,we collect the instantaneous power draw of the physicalhosts alongside with the usage patterns of the various VMson the host. We collect the basic and widely available in-formation on when a particular VM arrives to the host andwhen it leaves from it (e.g., via termination or migration).We collect the VM placement information via either al-ready existing IaaS interfaces, or if they are not availablewe offer interface extensions on the applied VM Manager.Unfortunately, some IaaS toolkits do not provide detailedVM level usage information necessary to evaluate exist-ing runtime energy models. Therefore, we inject utiliza-tion collector agents into the physical machines that couldhost the VMs. This approach notably raises the idle con-sumption of the machine (that will be accounted to theVMs indirectly), however it provides the fine grained VMconsumption metrics needed. So it is important to selectthe agents that can provide the necessary input for theruntime energy models with the smallest possible energyimpact.

4.1.2. Scoring

Requirements. Scoring is the quintessential part of leaderboards. To determine the features and properties of thenecessary scoring system for our extensions, we first iden-tified the minimum requirements that this scoring sys-tem should adhere: (i) it should highlight the behav-ioral differences between academics and commercial cloudusers, (ii) it should provide higher scores to academicswho behave more closely to commercial users (see the ex-pected behavior in Table 1), (iii) scores should be inde-pendent from the time the users joined the leader board,(iv) good user behavior should always provide high scoresindependent from the use of the underlying infrastructure,(v) users should be able to compare their scores to theirpast selves (whether they improve or worsen comparedto the expected behavior), (vi) one’s contribution to agroup score should not diminish because of diminishingresource use, and (vii) scores should be independent fromhow many virtual machines for how long the users use.

Next, we investigate the expected scoring behavior inthe scope of the user characteristics presented in Table 1:

Need triggered If the VM is started too early, the VM’sutilization is minimal before the first tasks arrive.Scoring should punish this with smaller scores. Conse-quently, users only instantiate their VMs when thereare no external dependencies that could defer theircomputational tasks planned for the VM.

Use the VM regularly If a VM has usage gaps itshould have smaller score than a fully utilized onein the same situation.

7

Terminate if not needed The VM’s score will decreaseif it is not executing tasks for a considerable amount oftime, thus users are inclined to stop their VMs whenthere are no tasks in the foreseeable future for theVM.

Instantiate a smaller VM The achievable score shouldbe smaller if a not so energy efficient physical machinehosts the VM. Also, the score should be smaller if theuser would request a VM with such resources thathe/she cannot utilize fully later on. Thus users wouldlimit the resource requests to the bare minimum thatis sufficient for their tasks so they can maximize theuse of the acquired resources.

Limit tasks to reserved VMs The score should bebetter for those who almost continuously use efficientresources than those who terminate and request VMswith little time left in between. There are two ma-jor user scenarios for this behavior: (i) guaranteedhigh efficiency VMs for longer tasks, (ii) reducing theimpact of IaaS energy share.

In the first scenario we assume that the user alreadyhas a more energy efficient VM than the VMs he/shecan have in the close future. In such situation, theuser could wait until his/her existing VM finishes itstasks and then assign new tasks for it. This behav-ior would render a higher score because of the otherpossible VM’s less efficient operation.

In the second scenario, the user has such a short taskthat it is not beneficial to create a dedicated VM forthe task (e.g., the energy accounted for the VM wouldbe dominated by the IaaS’s operations – see Equa-tions 1 and 2). Instead it is more beneficial to waituntil an already existing VM becomes available.

Prolonged use is desired When the user expects torun tasks for an extensive time period, and these tasksmostly fill this time period (i.e., there are no big gapsin between), then acquiring the most energy efficientVM possible for these tasks should result in betterscores than asking for VMs on demand.

If unjustified sell the reservation and switch modelWhen the users badly estimate the amount (or length)of tasks in the previous scenario, they should imme-diately terminate the acquired VMs and reconsiderthe VM usage scenario. Otherwise they should endup with smaller scores.

Bound to the resource type If the user acquired oneof the most energy efficient VMs possible for pro-longed use, then it is beneficial to mostly utilize thisVM for its computations, unless there is some ur-gency. The scores will be higher for the user even ifhe/she delays some of his/her computations becauseif those computations would run on other VMs they

would receive less points caused by their inferior en-ergy efficiency.

Burst VM use until abortion VMs are not abortedautomatically in academic clouds, but otherwise thiscase can be derived from to the behavior titled “UseVM regularly”.

The behavioral patterns “Price & need triggered” and“Abrupt VM abortion” are not possible to reach with scor-ing only, thus we propose the use of engaging options (de-tailed in Section 5) to achieve this behavior.

The remainder of this section shows the basic proper-ties of our extensible scoring system (seen in Figure 2).This scoring will already aim at supporting the expectedbehavioral changes of academics. However, in Section 4.2,we show how various leader boards could already extendthis basic functionality for advanced leader board setups.

The score of an individual virtual machine. Based on therequirements above, we aim at first defining a score for anindividual virtual machine. Then we will provide a wayto aggregate these scores so they allow the comparison ofusers.

For virtual machine level scoring, we borrow the conceptof energy conversion efficiency (η) from physics. This con-cept shows the relation of the energy invested in to a trans-formation process and the actual useful energy after theprocess. In our application, we would like to convert aca-demics to behave more like commercial users. Thus, theenergy invested will be represented with the actual energyconsumed by a virtual machine (E(VM ) – see Equation 1for details) of an academic user. While the useful energywill be represented by the energy that would have beenconsumed by a virtual machine (E∗(VM )) utilized by aprice constrained rational user. In both cases, we assumethat the same tasks are executed in both virtual machines.Also, we assume that the price constrained (or commerciallike user) would utilize a fully equivalent hypothetical in-frastructure (in terms of physical machines, actual runningother virtual machines, and other IaaS components). Thusthe only difference between the academic infrastructureand the hypothetical one is that the hypothetical wouldoffer functionalities that are needed to accomplish com-mercial pricing models. This definition renders our base –or virtual machine level – score (S(VM )) as the following:

S(VM ) =E∗(VM )

E(VM )(5)

The calculation of the rational user’s energy consumptionshould be set up according to the needs of the particu-lar leader board. When a new leader board is set up, theway this estimate is made should be given by the leaderboard’s owner. Although, we offer a basic definition withthe leader boards, this definition cannot take into consid-eration all circumstances in the particular situation (e.g.,it is not known in advance what information is going to be

8

available through the accounting API as collecting some ofthe data might be tedious work for the actual provider).Our scoring is extensible as it allows the leader board own-ers to set up their own base score (and only utilizing thescore aggregating facilities of the leader boards). There-fore either the following definition has to be adopted forthe use of the new leader board (with an option to chooseonly the energy model for the VM), or a brand new for-mula has to be developed. We define the rational user’senergy consumption as the minimum consumption possi-ble of a virtual machine with the same tasks on the samephysical host (assuming that the physical machine’s everyother resource is fully utilized).

For every user, his/her own virtual machines’ individualscores are presented. This means that a general leaderboard has an overall view (to compare users with eachother), and a user specific one, that reveals the competitionbetween the virtual machines of a particular user. On thisuser specific leader board (which is only offered to the userwho owns the VMs), the best and the worst scored virtualmachines are prominently highlighted. To allow the usersto learn from their past bad practices, this leader boardgives a reasoning for the worst scores, e.g.: (i) VM was ona not so efficient machine, (ii) the VM was left runningon a machine that was not fully utilized by other users– shows the need for multi-tenancy –, (iii) the VM wasplaced on a machine with further resources (i.e., the VMwas not occupying all the remaining resources of a physicalmachine) but other users did not arrive to utilize themearly enough (the VM was terminated before other userscome).

User level scoring. Although, this virtual machine levelscore fulfills most of our requirements already, it is not theindividual virtual machines that we would like to enterinto a competition but their users. Thus there is a needfor an overall score. The overall score – S(U) – should keepthe properties of our virtual machine level scoring, whileit should present the overall conversion of an academic.For this score, we propose to weight virtual machine levelscores with the execution time – t(VM ) – of the particularvirtual machines:

S(U) =

∑∀VM of U S(VM ) · t(VM )∑

∀VM of U t(VM )(6)

4.2. Infrastructure independent scoring

Our overall score is already offering a great detail forthe user, however it suffers from issues with comparabil-ity when it would be applied across cloud infrastructures.Therefore, in the following section we detail a second levelof scoring (building on top of our extensible scoring seenin Figure 2). This second level builds on our overall score,but improves its comparability by transforming the scoreto a new one that is more suitable for a particular situa-tion. Below we list the various cases when there is a needto adjust scoring.

4.2.1. Provider leader boards

If an academic cloud provider frequently faces resourceprovisioning issues (e.g., unexpected jumps in demand, or-phaned VMs, VMs with little or no utilization of extensiveperiods of time), then the adoption of our extensions couldoffer remedy without investments in newer hardware. Insuch case leader boards are maintained internally by thecloud provider (one provider offers a single leader board forall its users) as shown in Figure 3a. In this leader boardthe users are automatically registered. Then, the providershould prominently advertise its existence so users will notmiss the fact that they are ranked. This advertisement al-lows the early adoption of the system and ensures thatscoring conscious users build up a critical mass. As a re-sult, users at the provider start to have a chance to com-pete with each other and become more and more poweraware. As an advantage, this not only helps the provider tobetter serve its user community but also increases user sat-isfaction (as seemingly more resources are available whenmore users adopt our self-rationing scheme). This ap-proach however needs a substantial amount of users whoare willing to join the competition early. Therefore, theseprovider level leader boards are unusable when there areonly a few users served by the provider.

The rest of the leader boards are external to theproviders (they have no influence neither on their scoringnor on their maintenance).

4.2.2. Collaborative leader boards

If there are some enthusiasts who would like to com-pete despite they are served by different cloud providers,then they should set up a collaborative leader board ontheir own (see Figure 3b). However, the different prop-erties of the various clouds (e.g., different machines withdifferent power consumption profiles) would not allow di-rect comparison of the participant’s scores. Therefore, acollaborative leader board should apply a relative scoringsystem, e.g.: (i) scores could be calculated based on ranksat the different clouds if there are provider leader boards,or (ii) users should be ranked based on how close theyare to the maximally achievable energy efficiency at theirprovider. These kind of leader boards even allow the inclu-sion of providers with few users because the participants onthe collaborative leader boards are enthusiasts and knownto be more active than regular users would be. Unfortu-nately, these leader boards are of limited use because ofthe expertise needed to set up such a leader board.

4.2.3. Federation wide leader boards

In case a cloud federation aims at reducing its over-all energy footprint it could operate a federation levelleader board (as revealed in Figure 3c). These boardscould present the absolute scores similar to provider leaderboards (absolute scores will not hide the differences be-tween the energy efficient behavior of the various providersin the federation). As a result, they could influence users

9

IaaS Provider

Provider Leaderboard

(a) A leader board for a single cloud provider

Collaborative Leader board

IaaS Provider 1

IaaS Provider2

IaaS ProviderN...

User 1 User 2 User 3 User M

...

(b) A collaborative leader board for several users

Cloud federation

Federation wide Leader board

User 1 User 2 User 3 User M

...

IaaS Provider 1

IaaS Provider2

IaaS ProviderN...

(c) A leader board for a cloud federation

Collaborative LB

Collaborative LB

Collaborative LB Collaborative

LBCollaborative

LBFederation wide

LBCollaborative LB

Collaborative LB

ProviderLB

Leader board Aggregation

Unified user identity association Score Alignment

(d) An aggregation of several kinds of leader boards

Figure 3: Leader board variations

to choose more energy efficient cloud providers participat-ing in the federation. It is important to note that theseleader boards lose their importance if there is no chance forthe users to move between providers as they prefer. Ad-vantages of the federation wide leader boards: (i) allow theindividual providers to concentrate on their infrastructure(they don’t need to set up a provider leader board for theirown users) and (ii) a larger set of users are aggregated onthe federation level thus there are more chances to increasemulti-tenancy on the physical machines. Regrettably, suchan installation could result in over usage of the most effi-cient provider in a federation as it drives users away fromthe less efficient offerings. On the positive side, this casecould highlight the omitted provider the time to updateits infrastructure to a more efficient solution. However,the complete resolution of this possible unbalance remainsfuture work.

4.2.4. Leader board aggregations

It is also possible that third parties set up aggregat-ing leader boards based on various provider or federation

wide leader boards (see Figure 3d). These kind of leaderboards introduce new issues like: (i) user merging (e.g.,there should be an automatic or semi-automatic way toidentify identical users using different accounts on mul-tiple providers), (ii) score alignment (the various leaderboards could use different scoring systems thus they shouldbe transformed and presented on a unified way). Thiscould allow global leader boards covering even unrelatedproviders (i.e., those that are not part of the same fed-eration). Consequently, it would be possible to set up aglobal ranking resulting in the maximum level of competi-tion possible. The operational and management issues ofthis kind of leader boards are out of scope of this paperand will be addressed in future work.

5. Approaching spot pricing like behavior

Although leader boards already achieve significant be-havioral changes, there are several user characteristics thatthey cannot support. Their support is impossible, be-cause leader boards cannot reveal the temporal behavior

10

of the infrastructure (e.g., letting users know that there isa chance of multi-tenancy at a particular moment). There-fore, first, we analyzed the behavioral patterns of Table 1,then we identified those cases where academics could havea chance to achieve higher scores if they would have someinsight on the behavior of others or the inner workings ofthe infrastructure. In the following, we list these casesand also provide discussion on how academics could reachhigher scores in these cases:

Instantiate a smaller VM If an academic has a com-putational task then he/she has to formulate the re-source requirements for it. Very few academics knowtheir resource demands precisely, thus showing themthe possible scoring changes a resource requirementwould result in could really help their decisions. E.g.,if the academic would know that utilizing lesser re-sources would result in bigger scores, then the ex-pected future score could push the academic to a com-promise on resource requests.

Limit tasks to reserved VMs In this case, an aca-demic already has several virtual machines that yieldhigh scores. When a new computational task arises forsuch academic, he/she should be tempted to maintainhis/her high scores. Thus the academic has to decideto either postpone the task or utilize a new VM forit. This decision is however dependent on what kindof resources can be acquired for the task. The userthus decides whether acquiring a resource right nowwould hinder his/her score or would strengthen it. Ifthe new VM would strengthen the score then the taskwill be executed in this new VM, otherwise the taskwill be postponed until the already available VMs ofthe user can process it.

Price & need triggered If the user’s task is not urgent,then one could limit the execution of the task only onresources which would generate higher scores. How-ever, one would need to know when such higher scor-ing resources are available, otherwise its VM requestswould end up on resources that negatively affect theuser’s scores.

Abrupt VM abortion Because of shared physical ma-chines, the score of the user’s VM will be heavily de-pendent on those who share the same machine. E.g.,if a VM is terminated then the remaining VMs onthe same host will start to receive lesser scores. Thusthese users have three basic options: (i) ensure theutilization of the currently not utilized resources (e.g.,by requesting a VM on their own or attracting oth-ers), (ii) checkpoint and terminate their own VM thenuse the previous – price & need triggered – case to de-termine when to resume, or (iii) migrate their VM toanother machine that could result higher scores.

Based on these cases, we identified the need for a mecha-nism that allows users to know the state of the underlying

Optional components for Spot pricing model

Engaging Option

Marketplace

Engaging Option Broker

Engaging Option

Manager

Issue Policy

Validity Checker

Issuer

Foun

datio

n fo

r Spo

t pric

ing

mod

el

Power meter

Physical Machine N

VM xVM xVM x

Power meter

Physical Machine 2

VM xVM xVM x

Power meter

Physical Machine 1

VM xVM xVM x

Virtual Machine Manager

IaaS Frontend

WebUI

CLIVMAPI

Academic User

Exte

nded

IaaS

Clo

ud P

rovi

der

Figure 4: Architectural extensions towards engaging op-tions

infrastructure. The remainder of this section introducesthe concept of engaging options that acts as the founda-tion for such mechanism.

5.1. The definition of engaging options

We define an engaging option as an electronic documentthat represents a non binding, non exclusive, on the otherhand time limited offer to utilize a particular set of re-sources on a physical machine. This document (e.g., anXML file) contains the following items: (i) the descriptorof the resources represented by the offer (e.g., 1 CPU, 500MB of memory or a single micro instance etc.), (ii) theunique identifier of the physical host where the resourcesare available, (iii) expected maximum energy footprint ofthe resource, (iv) the issue date, (v) the maximum lengthof validity (e.g., until two hours from the issue date),(vi) access details to the provider that offers the resources(e.g., a URL to the provider’s IaaS frontend or a textualdescription on how to get access) and (vii) a signature ofthe issuer (i.e., to show that the document is authentic).

Engaging options allow users to know when and in whichinfrastructure there is an underutilized physical machineavailable. Thus users are no longer deemed to requestVMs at arbitrary moments. Instead, they can utilize en-gaging options (if available and valid) to approach theirproviders and receive their VMs on a shared machine. Inother words, the engaging option is a mechanism to attractusers at the right time to the right machine (so the virtualmachine manager of the IaaS system will be able to utilizethe underlying physical machines more efficiently – whiletheir users will end up with higher scores on the leaderboards).

Next, according to Figure 4, we detail (i) how an IaaSsystem would operate with engaging options, and (ii) howacademics are expected to behave after the introductionof the concept, especially in the scope of the behavioralpatterns identified in Table 1.

11

alt[Free resources]

[else]

IaaS FrontendUser Issuer

request VM

Issue PolicyVirtual Machine Manager

Physical Machine

Virtual Machine

new VM

deploy VM

host VM

VM created

EO issuing needed?

Get Load on VM's host

Load

Issue (load)

Engaging Optioncreate options

return created options

return options

return VM & options

Figure 5: Sequence of emitting an engaging option during VM creation

5.2. Issuing engaging options

Within the engaging option manager (see Figure 4), theengaging option issue policy is the main entity that is re-sponsible to decide when, to whom, how many and withwhat length of validity should the system emit engagingoptions. The issue policy is customizable by the providerwho applies it (e.g., it can define a new function that de-termines the length of validity). When the issue policydecides on all aspects of a new engaging option, then itpasses this information to the engaging option issuer. Theissuer then prepares the electronic document of the optionin the user desired output format and hands it over to theuser. This issuing process is exemplified in Figure 5, inwhich we present the IaaS behavior during the creation ofa new virtual machine. Since it is the issue policy thatis responsible for the decisions, we are going to detail itsbehavior in the following paragraphs.

First of all, the policy decides when to issue an option.Whenever a virtual machine arrives to or leaves from aphysical machine the policy will try to emit new engagingoptions for the abundant capacities. There are three stopconditions that can halt the issuing process: (i) the physi-cal machine is fully utilized – the last VM that has arrivedoccupied all previously available resources –, (ii) the phys-ical machine hosts no VMs – the latest VM that has leftwas the last one –, and (iii) the expected length of validityfor the option is too short – i.e., under the useful runtimeof a VM.

Next, the policy identifies those users who should re-ceive options. If the options are issued because of a newVM, then the requestor of the VM will receive the options.Otherwise, the options will be sent to those who still havevirtual machines running on aforementioned physical ma-

chine. In both cases, the system ensures that one par-ticular user would never have more engaging options athis/her possession than the currently available resourceswould necessitate. This last requirement significantly lim-its the number of possible engaging options in circulation,but avoids flooding the users with unnecessary options.

Afterwards, the policy decides how many options shouldbe issued. Basically, the policy will try to issue engagingoptions for all available resources of a physical machine. Toavoid facing a bin packing problem (because of mismatchesin abundant capacities and various sized engaging options)and to ensure that the issued options fit every user’s need,the issued options should be of the smallest possible gran-ularity. Therefore, the issuer is instructed to create an op-tion per processor core or per smallest VM instance typepossible on the physical machine. Figure 5 presents thisoperation as a request to the function Issue(load), wherethe load represents the amount of smallest granularity op-tions needed.

When there are multiple users for a physical machine,the issue policy often emits as many engaging options for aparticular resource as many users are. This approach notonly allows wider visibility of the issued engaging options,but as a consequence it also increases the chances to fillthe not so energy efficiently used resources. The visibilityimproves because it is not guaranteed that any particularuser shares the engaging options he/she receives, but themore users receive them the more likely that the optionsare going to be shared. This is particularly useful whena new VM from a new user arrives to a host for whichthe system already issued some engaging options. If thisnew user was not attracted by a previously issued engag-ing option then we can assume the previous options did

12

not reach a wide enough community. As a result, the newuser’s community can be also targeted by issuing new en-gaging options (in parallel to the already issued ones) forthe remaining capacities of the host.

Finally, the issue policy defines the length of validity ofthe options before they are sent to the users. Similarly tothe virtual machines on the host, issued engaging optionsshould also have a limited length of validity (i.e., it is notdesired that one is attracted to a host when others alreadyterminated their VMs). The length of validity of the en-gaging option is an important factor to the liveliness of thecommunity. If it is too short, then fellow users might notbe able to notice it. If it is too long then the VM place-ment algorithm of the provider might already have takenthe resources that the engaging option represents. Con-ceptually, the engaging options should be valid as long asthere is still one running VM on the host. Unfortunately,during issue time, the future execution time of the origi-nal VMs are unknown, thus the engaging options shouldbe issued according to provider policies (e.g., the lengthof validity could be set as the median run time of its pastVMs).

5.3. Attracting academic users

In this subsection, we assume that a user has just re-ceived one or more engaging options. With the options,the user is now informed about the level of inefficiencieson the physical machine its VM is scheduled on. Althoughthe options are issued so the user would have the incentiveto refer others to the same physical machine, in two casesit is not beneficial for the user to advertise the receivedoptions. These are the following cases: (i) plans for ad-ditional VMs in the near future, (ii) expected short VMlifetime on the host.

In the first case, the user already has plans to executeadditional VMs, and therefore it is recommended to usethe received engaging options for his/her next virtual ma-chine requests. This strategy allows immediate energy ef-ficiency increases for the user because the next VMs areprovisioned on the same physical machine. In the secondcase, the user either has a task that will run on its newVM for a relatively short amount of time or expects to finda physical machine where the VM can be migrated soon.Therefore, if such engaging options would be advertised,their benefits would be negligible or even nonexistent forother users (one would attract users to a resource whichhe/she does not plan to use in parallel to the attractedusers).

In any other cases, the received engaging options wouldbetter serve their original owners if they are shared. Theuser has two basic options: (i) sharing manually and(ii) sharing on a marketplace. Manual sharing involvesdirect interaction between academics (i.e., one sends theoptions to his/her fellows) and thus strengthening the com-munities formed around leader boards. On the other hand,manual engaging option circulation is a tedious task asit needs significant work from the sender (selecting and

IaaS FrontendUser Validity Checker

request VM(EO)

Virtual Machine Manager

Virtual Machine

check Validity(EO)

isResourceAvailable

[true]

valid

Create VMon host

create

[false]

notvalid

check & crateSimilar VM(EO)

isResourceAvailable

Virtual Machinecreate

check & crateSimilar VM(EO)

Not valid EO

alt[equivalent]

[else]

alt

[Available]

[else]

Figure 6: Using an engaging option to create a VM

sending to those who might plan using a VM within a fewminutes/hours) and the receiver also (filtering the alreadyexpired options). To remove the burden of engaging optionhandling from users we introduce the concept of engagingoption marketplaces (see Figure 4).

An engaging option marketplace is a component thatallows users to promote their engaging options to otherusers. Users can add new engaging options while otherscan query, acquire (and remove) them from a browsableindex. In a marketplace, users browse the advertised en-gaging options so they can find resources that meet theirrequirements (regarding specific energy efficiency measuresor resource models). The wider the community a mar-ketplace reaches is the better options it can offer to itsusers. Therefore, it is expected that marketplaces are of-fered alongside leader boards. Thus, those users who al-ready use leader boards can also exploit the offers on mar-ketplaces to increase their score.

Finally, marketplaces can be operated independentlyfrom an the IaaS provider revealing the usefulness of en-gaging options across cloud federations. For example, weenvision federation wide engaging option marketplaces.With these large marketplaces, it becomes possible toguide those users who are hesitant about their choice ofinfrastructure (e.g., one can decide to use a particular in-frastructure for its computations when he/she sees a con-vincing engaging option from that infrastructure).

13

5.4. Utilizing the acquired options

5.4.1. Basic use of engaging options

To make sure of the highest scores possible on the leaderboards, users with immediate computing needs would ac-quire engaging options (representing highly energy efficientresources) either from their fellow academics or from mar-ketplaces. When requesting a virtual machine from theprovider, users would also present the acquired engagingoptions (see Figure 6). If multiple engaging options areused for a single virtual machine request, then users mustensure that the presented options have identical identifierfor the hosting machine (otherwise their VM request can-not be fulfilled). At the provider, this will result in a VMrequest composed from all the resources represented bythe specified engaging options. As engaging options arenon binding and non exclusive, it might happen that therequested resources are already taken by someone. There-fore, before creating an actual VM for the user, the IaaSmust check the validity of the engaging options it was ap-proached with.

There could be three reasons behind an invalid engagingoption: (i) expiration (the option is older than its lengthof validity), (ii) independent VM placement (some user al-ready acquired the resources in question despite not hav-ing the relevant engaging option) , and (iii) concurrentengaging option use (some user also acquired and used anengaging option that represent the same resource). To re-duce the chances of invalid engaging options, users canapproach the providers and check if an engaging optionstill refers to available resources. If, in spite of this check,an engaging option is invalid, the provider tries to replaceit with a similar one that has the same amount of resourcesand the same or better energy characteristics (but mightbe located on other physical machine). If there is no validengaging option after these steps, then the VM request iscancelled.

Therefore, the provider now can safely assume that therewill be enough resources for the virtual machine at a hostwhich already runs some VMs (from those users who theengaging options were originally issued for). Consequently,in its next step, the provider will create the new user’s vir-tual machine on the host referred by the engaging optionspresented in the VM request. Because of the placementof this new virtual machine, the energy consumption ac-counted to the other VMs will be reduced in their remain-ing lifetime (thus engaging options positively impact thescores of those who shared them).

5.4.2. Automated use of engaging options

While engaging options are already a powerful tool forusers who compete for higher scores in the leader boards,automating their usage is still a must. This enables theirinclusion in such scenarios where academics use automatedexperiments like workflows. We have identified a semi au-tomatic (which looks for a specific engaging option andensures its validity but leaves the VM creation to the user)

and a fully automatic (which automates the VM creationprocess) approach that supports these scenarios. Both ofthese scenarios are built on our last component from Fig-ure 4: the engaging option broker.

In the semi automatic approach, the broker is ap-proached by a user who would like to have a virtual ma-chine with a particular resource set and energy properties.The broker monitors the marketplaces for the user andwhenever an suitable engaging option appears, it auto-matically checks its validity at the provider. Next, if thisoption is found valid, then the broker will immediatelyforward it to the user. This requires the user to createthe VM on its own. Despite looking like a disadvantage,this is beneficial for the user if he/she cannot automate thepreparation of his experimental environment in the virtualmachine. With the returned engaging option the user canstart the VM whenever he/she is ready for the prepara-tion task. Since there is a manual operation at the end ofthis process, the user must specify the expected length ofthe task in the VM and the expected time when he/shewill be able to utilize the resources first. With these twodetails, the broker delays the marketplace monitoring pro-cess until the chance of utilizing a successfully identifiedand validated engaging option is high.

In the automatic approach, the broker is approached bya similar user as before. The only difference that this timethe user is capable to automate the preparation of the ex-periment in a newly created virtual machine. Since therequested virtual machine can operate automatically (i.e.,it starts processing user tasks without human interven-tion and terminates if there are no more tasks), the brokerwill start monitoring the marketplaces immediately afterreceiving the user request. When there is a suitable validengaging option, the broker will even instantiate the neces-sary VM for the user. With this procedure, academics willexercise the same behavior as those commercial users whoapply price & need triggered virtual machine instantiationstrategy in case of spot pricing models (see Table 1).

6. Evaluation

In this section, we reveal our analysis on how scientificinfrastructures are impacted while academic user behavioris transformed to a more commercial like behavior. For ouranalysis, the behavior of scientific users is really important.Thus we briefly turn our attention on the infrastructure’sgeneral properties, then we discuss the simulated user be-havior on the system.

In all of our simulations, we assume an unlimited cloudinfrastructure that offers a single VM type (with one core)and a single kind of physical machine with 8 cores. We ap-plied a simple VM placement policy: new VMs are placedon the first physical machine with available cores. If thereare no physical machines with free capacities then a newphysical machine is switched on and the new VM is placedon that physical machine. If a physical machine does nothost a single VM then it is switched off. We also assumed

14

Overall Users/cluster Tasks/clusterWorkload No. of clusters Min Max Median Min Max MedianGrid5000 11593 (42) 1 (2) 98 (=) 2 (13) 1 (1004) 5025 (=) 47 (1504)

LCG 186 (73) 4 (17) 54 (=) 24 (28) 40 (1000) 4378 (=) 815 (1510)AuverGrid 4380 (71) 1 (12) 94 (=) 2 (47) 1 (1003) 15058 (=) 5 (2181)

DAS2 7905 (35) 1 (=) 30 (24) 4 (6) 1 (1009) 3737 (=) 108 (1210)SHARCNET 4692 (204) 1 (=) 92 (=) 2 (30) 1 (1004) 29700 (=) 5 (2190)

Table 2: Overview of the clustering results for the different workloads – numbers in parentheses reveal the details onclusters with over 1000 jobs, later on we only use these clusters for the statistical analysis

that amongst the switched on physical machines only onehas less VMs than cores. When this is not the case weassumed that the IaaS automatically migrates VMs fromless crowded to more crowded physical machines to mini-mize the count of necessary physical machines switched on.With these infrastructural assumptions we ensure that themomentary resource utilization will not experience largevariance because of the internals of the simulated IaaS sys-tem. The elimination of this variance is important becauseotherwise it would be impossible to tell whether resourceutilization reductions are caused by internal IaaS opera-tions or by the behavioral changes we intend to simulatefor the users.

To simulate user behavior, we have turned our attentiontowards workload traces offered by the scientific commu-nity. We have looked for traces that fulfill the followingthree criteria: (i) represent extensive durations becausethe usage patterns for particular users become more clearin the long term, (ii) focus on a large user base so wewill have a chance to see the behavior of a multitude ofusers and (iii) offer a description which clearly associatesactivities and users. After employing these criteria, wehave selected the grid workload archive (GWA – [32]) as agood representative for the behavior of our expected users(unfortunately our last criteria eliminates the possibilityof using some of the well known traces like PlanetLab).Although, this archive is not cloud oriented, we assumedthat the frequency of the user tasks and the user’s overallbehavior regarding the use of the infrastructure is inde-pendent from the underlying technologies. In the followingsubsections we first present how we processed the scientificworkload in GWA for our analysis. Then we present an indepth overview on the expected impacts of our architec-ture on the resource utilization of academic clouds.

6.1. Trace processing methods

Before our analysis, we have processed the traces so wecan simulate both commercial and academic user charac-teristics (in terms on how and when they would instantiatevirtual machines for their tasks). Throughout the simula-tion, we have focused our attention to three of the mainuser characteristics listed in Table 1: (i) delay instantia-tion, (ii) early VM termination and (iii) ensure continuoususe. In our simulations, we did not analyze the perfor-mance compromise characteristic because of our assump-

tions on the simulated infrastructure (i.e., there is onlyone VM type usable), and because of the exact resourceutilization pattern of the user tasks are not specified inthe traces (thus one cannot tell if a smaller VM would besufficient for the user).

As our first trace processing step, we have analyzed thetasks and identified those that have the highest chanceto be delayed and grouped together in a virtual machine(so that the tasks can be grouped like a commercial userwould group them). We assumed that no particular userwould delay its tasks for an indefinite amount of time.Therefore, after some time, even the most scoring con-scious users would run their tasks on resources (or newVMs) that would negatively impact their score. The morecommercial like the user is the more he/she is willing todelay his/her tasks hoping for a more energy efficient (thusbetter scoring) resource opening. To identify those tasksthat together could lead to the minimum amount of VMinstantiations, we have processed the traces with a datamining algorithm called K-Means. Based on submissiontime proximity, this algorithm clusters those tasks togetherthat are the least likely to cause impatience in users (we as-sumed that a user becomes impatient when he/she wouldhave to wait for more than two hours for a more energyefficient resource). Table 2 shows details about the fiveselected workload traces and their clustered tasks. Fromthe formed clusters, we have selected those that could sta-tistically represent an entire trace and we only kept thosefor further analysis (the properties of these clusters arerevealed in parentheses). These clusters all contain over1000 tasks. In the rest of the paper, we only focused ouranalysis on these clusters.

As the next processing step, for each cluster we haverun a user simulator which assigned the user’s tasks in thecluster to virtual machines (ensuring that users can onlyassign tasks to their own VMs). Based on the commercialuser characteristics, we have identified three assignmentapproaches: (i) act as a non-transformed academic user,(ii) employ early VM termination if possible, (iii) delaytasks until they would form a continuous block and createa VM for them. In the next paragraphs, we detail thesethree options.

With the first approach, we simulated the academicusers before their transformation: we assumed users wouldcreate a VM as soon as there is a task to be executed and

15

the user does not have a VM that could execute it immedi-ately. This could lead to several VMs in parallel dependingon the number of tasks that must be executed in a partic-ular instance of time. When a task finishes in a particularVM, the user could choose to terminate the VM. This isonly done when the remaining tasks of the cluster have aparallelism less than the amount of virtual machines theuser currently has. For example, when the trace has threesimultaneous tasks for a while, the user would have threeVMs, but immediately when there is no more chance tohave three simultaneous tasks within the the cluster, theuser would terminate one of his/her VMs.

The second approach partially simulates commerciallike behavior: the users terminate their VMs as soonas feasible. The simulation of this approach only differsmarginally from the previous solution. We set up an ini-tial set of virtual machines utilizing that approach. Thenwe split these VMs by repeating the following procedureuntil there are no new VMs formed: (i) we identify thelongest gap between the tasks of the original VM, (ii) ifthe length of this gap is over a threshold then we removethe original VM from the set of necessary VMs and addtwo new ones (one that includes all tasks before the gap,one for the rest). We define this threshold as the shortesttime period for which two new VMs would render biggeroverall score for the user than the original one (because ofthe impact of Equation 2 on the score of VMs with tooshort lifetimes).

Then, our last approach provides our most sophisticatedapproximation of the commercial like behavior. Here weassume users would postpone their tasks so they form acontinuous block (one task immediately starts after theprevious has finished). To ensure that we can postponetasks indefinitely within the clusters, we assume that thereare no dependencies between user tasks (this informationis not included in the available traces). Thus we first delaythe tasks as much as possible, then we apply the secondapproach to define the shortest lifetime VMs. This combi-nation of task delays and early VM termination will resultin VMs that are in continuous use for their entire lifetime.

6.2. Analysis

To prove our architecture’s effectiveness, we have setup a simulation environment where every selected clusterfrom the GWA traces can be evaluated. A single exper-iment was done in the following phases: first we allowedthe specification of user behavior (e.g., non-transformedacademic, commercial like), then based on this behaviorwe determined the necessary VMs and their properties(request and termination time), next we run the virtualmachine placement algorithm for these VMs in our sim-ulated cloud infrastructure, finally we have estimated theenergy use and the total CPU hours the VMs spent in thesimulated environment.

In our first experiments, we assumed uniform likelinessthat our architecture transforms a selected user. Thus wehave specified a random order of users in which they are

Sheet1

Page 1

0%10%20%30%40%50%60%70%80%90%100%1

1.2

1.4

1.6

1.8

2

2.2

2.4

2.6

Grid5000LCGAuverGridDAS2SHARCNET

% of users who still apply their past bad practices

Re

lativ

e r

eso

urc

es

acq

uir

ed

(c

om

pa

red

to

10

0%

co

mm

erc

ial l

ike

be

ha

vio

r)

Figure 7: The effect of transforming randomly chosen usersto commercial like behavior

Sheet1

Page 1

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1Grid5000LCGAuverDAS2SHARCNET

User heaviness

Nor

mal

ized

ene

rgy

impa

ct

Figure 8: Chance of energy saving based on user heaviness

transformed to commercial like. At first, we have set allusers to behave like commercial ones. We run an initialexperiment and collected the total CPU hours spent inthe system. This formed our baseline. During our firstexperiments, we have collected the total CPU hour figuresfrom the experimental runs and divided by the previouslyacquired baseline. Figure 7 shows the possible extra re-source demands (compared to our baseline) for the fiveanalyzed traces depending on the percentage of the userswho behave as non-transformed academics. Although wehave evaluated the trends for all selected clusters, we pre-sented them as an average per analyzed trace. Based onthe results, we see that if only non-transformed users areusing the system, then the energy and resource consump-tion is at its maximum. AuverGrid is impacted the mostby user behavior. About 2.6 times more resources are usedwhen all users apply the bad practices they are used to.The figure also shows, that users can be served by signif-icantly (23%–33%) smaller infrastructure even if just halfof them are converted to behave more like a commercialuser.

When we looked at the non averaged trends, it clearlyshowed that some users have heavy impact on the resource

16

Sheet1

Page 1

0%10%20%30%40%50%60%70%80%90%100%1

1.2

1.4

1.6

1.8

2

2.2

2.4

2.6Grid5000LCGAuverGridDAS2SHARCNET

% of users who still apply their past bad practices

Re

lativ

e r

eso

urc

es

acq

uir

ed

(c

om

pa

red

to

10

0%

co

mm

erc

ial l

ike

be

ha

vio

r)

Figure 9: Prioritizing transformation of the heaviest users

utilization in the system. Therefore, we have analyzedthe traces from a different point of view: if a given usertransforms its behavior what is its impact on the overallresource demands? We have constructed an experimentwhere only one user is changed to non-transformed, therest already behaves like commercial users. With this ex-periment we evaluated all the users in all the selected clus-ters of the five traces, then we checked the difference inenergy consumption between the case when the user wasnon-transformed and when he/she was already acting as acommercial one. The bigger the difference the heavier theuser is for the system. In Figure 8, we show how the heav-iest users relate in their energy consumption to the otherusers in the traces. On its x axis the individual users areshown (ordered by their heaviness). To allow a less com-pressed figure, we omitted those users who have less than1% of the heaviest user’s energy impact. According to thefigure, only a few users are responsible for the vast ma-jority of the extra energy consumption. Thus providersshould put extra effort in the advertisement of the leaderboards to those users. Also, it might be beneficial to setup a hall of shame – an inverse ranking leader board – thatprominently highlights those users who misbehave.

Following the analysis of the impact of the heavier userson the general saving, we evaluated a scenario where heav-ier users are the first to adopt commercial like behavior.We expect that targeted leader board advertisements andthe hall of shame will increase the likeliness of such a sce-nario. With this new adoption order, we have re-evaluatedour first experiments and presented the new results in Fig-ure 9. The figure not only highlights the impact of heavyusers, but also reveals that converting just half of the usersresult in an even more dramatic decrease in the requiredinfrastructure size as before. The reductions are now be-tween 37% and 61%.

Our evaluation has shown the possible transformationsof academic user towards more commercial like behaviorand their impact on the overall energy and resource us-

age. In real life systems, we expect the introduction of theleader boards will immediately turn some users towardsthe behavior of commercial users. Thus even though ourexperiments start from 100% non-transformed users, it isonly a hypothetical case. The saving curve that can beexpected in realistic scenarios is expected to be close tothe one shown in Figure 7.

7. Conclusion & Future Work

Regrettably, even if an academic cloud provider appliessimilar resource management approaches as commercialproviders, their resource needs and energy footprint hardlydecreases because of the bad practices the academic usersdeveloped in the unconstrained environments of the past.Resources are occupied longer than needed as the lim-iting factor from commercial cloud providers is missing:cost. We have proposed an architecture that helps to raisethe resource and power usage awareness of academic cloudusers by two technologies: (i) leader boards which providedirect feedback on how closely academics behave comparedto commercial users and (ii) engaging options that directusers towards more economical resource usage.

Our evaluation shows potentially high savings both interms of utilized resources and energy if users are willingto change their usage patterns. Our architecture tries toincrease their willingness to change by allowing academicsto compete with fellow scientists. This competition en-courages academics to self-ration their resource usage forbetter scores on the leader boards. As a result of self ra-tioning, academic providers will have a chance to utilizestate of the art resource management approaches to re-duce energy expenses just like commercial providers. Suchresource provisioning algorithms can have high influenceon power draw of infrastructure and in combination withenergy aware users this can lead to considerable powersavings. Also, self-rationing practices often increases thethroughput of the infrastructure, thus academics can usethe infrastructure more productively than before the ap-plication of our architectural extensions.

We plan to continue our work into multiple directions.First, we intend to evaluate different scoring schemes indetail and we will analyze their impact on the overall re-source and energy utilization. We will investigate highlevel leader board organizational issues like proposing newschemes to reduce possible imbalances caused by leaderboards between individual providers in a cloud federation.Next, we will evaluate the possible positive effects of en-gaging option brokers and marketplaces on user behaviorand we will investigate new behavioral patterns made pos-sible by advanced engaging option handling. Afterwards,we plan to research engaging option issue policies and howthey can result in user behavior that resembles reservedand spot pricing schemes more. Finally, we expect thatthis article raises the awareness of academic providers sothe proposed system could be implemented and tested withreal users and its general usability could be demonstrated.

17

[1] M. Zaharia, D. Borthakur, J. Sen Sarma, K. Elmeleegy,S. Shenker, I. Stoica, Delay scheduling: a simple technique forachieving locality and fairness in cluster scheduling, in: Pro-ceedings of the 5th European conference on Computer systems,EuroSys ’10, ACM, New York, NY, USA, 2010, pp. 265–278.doi:10.1145/1755913.1755940.

[2] S. Bhardwaj, L. Jain, S. Jain, Cloud computing: A study ofinfrastructure as a service (iaas), International Journal of engi-neering and information Technology 2 (1) (2010) 60–63.

[3] J. Diaz, G. von Laszewski, F. Wang, A. J. Younge, G. Fox,Futuregrid image repository: A generic catalog and storagesystem for heterogeneous virtual machine images, in: IEEEThird International Conference on Cloud Computing Tech-nology and Science (CloudCom), IEEE, 2011, pp. 560–564.doi:10.1109/CloudCom.2011.85.

[4] D. Nurmi, R. Wolski, C. Grzegorczyk, G. Obertelli, S. Soman,L. Youseff, D. Zagorodnov, The eucalyptus open-source cloud-computing system, in: 9th IEEE/ACM International Sympo-sium on Cluster Computing and the Grid (CCGRID’09)., IEEE,2009, pp. 124–131. doi:10.1109/CCGRID.2009.93.

[5] A. Harutyunyan, P. Buncic, T. Freeman, K. Keahey, Dy-namic virtual alien grid sites on nimbus with cernvm, Journalof Physics: Conference Series 219 (2010) 1–8. doi:10.1088/

1742-6596/219/7/072036.[6] D. Milojicic, I. M. Llorente, R. S. Montero, Opennebula: A

cloud management tool, Internet Computing, IEEE 15 (2)(2011) 11–14. doi:10.1109/MIC.2011.44.

[7] D. Durkee, Why cloud computing will never be free, Queue 8 (4)(2010) 20:20–20:29. doi:10.1145/1755884.1772130.

[8] I. Sfiligoi, D. C. Bradley, B. Holzman, P. Mhashilkar, S. Padhi,F. Wurthwein, The pilot way to grid resources using glidein-WMS, in: WRI World Congress on Computer Science andInformation Engineering, Vol. 2, 2009, pp. 428–432. doi:

10.1109/CSIE.2009.950.[9] P.-O. Ostberg, D. Espling, E. Elmroth, Decentralized scal-

able fairshare scheduling, Future Generation Computer Systems29 (1) (2013) 130–143. doi:10.1016/j.future.2012.06.001.

[10] S. Zhuravlev, S. Blagodurov, A. Fedorova, Addressing sharedresource contention in multicore processors via scheduling, SIG-PLAN Not. 45 (3) (2010) 129–142. doi:10.1145/1735971.

1736036.[11] I. T. Foster, Y. Zhao, I. Raicu, S. Lu, Cloud computing and

grid computing 360-degree compared, in: Grid Computing En-vironments Workshop, 2008. GCE ’08, 2009, pp. 1–10. doi:

10.1109/GCE.2008.4738445.[12] E. Deelman, G. Singh, M. Livny, G. B. Berriman, J. Good, The

cost of doing science on the cloud: the montage example, in:Proceedings of the 2008 ACM/IEEE conference on Supercom-puting, SC ’08, IEEE Press, Piscataway, NJ, USA, 2008, pp.50:1–50:12. doi:10.1145/1413370.1413421.

[13] D. Kondo, B. Javadi, P. Malecot, F. Cappello, D. P. Anderson,Cost-benefit analysis of cloud computing versus desktop grids,in: IEEE International Symposium on Parallel Distributed Pro-cessing (IPDPS), 2009, pp. 1–12. doi:10.1109/IPDPS.2009.

5160911.[14] G. Feng, S. Garg, R. Buyya, W. Li, Revenue maximization

using adaptive resource provisioning in cloud computing envi-ronments, in: Proceedings of the 2012 ACM/IEEE 13th In-ternational Conference on Grid Computing, GRID ’12, IEEEComputer Society, Washington, DC, USA, 2012, pp. 192–200.doi:10.1109/Grid.2012.16.

[15] M. Mazzucco, D. Dyachuk, R. Deters, Maximizing cloudproviders revenues via energy aware allocation policies, in:IEEE 3rd International Conference on Cloud Computing(CLOUD), 2010, pp. 131–138. doi:10.1109/CLOUD.2010.68.

[16] A. B. Yoo, M. A. Jette, M. Grondona, Slurm: Simple linuxutility for resource management, in: D. Feitelson, L. Rudolph,U. Schwiegelshohn (Eds.), Job Scheduling Strategies for Paral-lel Processing, Vol. 2862 of Lecture Notes in Computer Sci-ence, Springer Berlin Heidelberg, 2003, pp. 44–60. doi:10.

1007/10968987_3.

[17] D. Jackson, Q. Snell, M. Clement, Core algorithms of the mauischeduler, in: D. G. Feitelson, L. Rudolph (Eds.), Job Schedul-ing Strategies for Parallel Processing, Vol. 2221 of Lecture Notesin Computer Science, Springer Berlin Heidelberg, 2001, pp. 87–102. doi:10.1007/3-540-45540-X_6.

[18] C. Fischer, Feedback on household electricity consumption: atool for saving energy?, Energy Efficiency 1 (1) (2008) 79–104.doi:10.1007/s12053-008-9009-7.

[19] D. P. Anderson, J. Cobb, E. Korpela, M. Lebofsky,D. Werthimer, Seti@home: an experiment in public-resourcecomputing, Communications of the ACM 45 (11) (2002) 56–61.doi:10.1145/581571.581573.

[20] J. J. Dongarra, H. W. Meuer, E. Strohmaier, et al., TOP500supercomputer sites, Supercomputer 11 (2–3) (1995) 133–194.

[21] S. Darby, The effectiveness of feedback on energy consumption.a review for defra of the literature on metering, billing anddirect displays., Working paper, Oxford Environmental ChangeInstitute (April 2006).URL http://www2.z3controls.com/doc/

ECI-Effectiveness-of-Feedback.pdf

[22] W. Chengjian, L. Xiang, Y. Yang, F. Ni, Y. Mu, System powermodel and virtual machine power metering for cloud computingpricing, in: Third International Conference on Intelligent Sys-tem Design and Engineering Applications (ISDEA), 2013, pp.1379–1382. doi:10.1109/ISDEA.2012.327.

[23] P. Xiao, Z. Hu, D. Liu, G. Yan, X. Qu, Virtual machine powermeasuring technique with bounded error in cloud environments,Journal of Network and Computer Applications 36 (2) (2013)818 – 828. doi:http://dx.doi.org/10.1016/j.jnca.2012.12.

002.[24] Y. Jin, Y. Wen, Q. Chen, Energy efficiency and server virtu-

alization in data centers: An empirical investigation, in: IEEEConference on Computer Communications Workshops (INFO-COM WKSHPS), 2012, pp. 133–138. doi:10.1109/INFCOMW.

2012.6193474.[25] A. Kansal, F. Zhao, J. Liu, N. Kothari, A. A. Bhattacharya,

Virtual machine power metering and provisioning, in: Pro-ceedings of the 1st ACM symposium on Cloud computing,SoCC ’10, ACM, New York, NY, USA, 2010, pp. 39–50. doi:

10.1145/1807128.1807136.[26] D. Huang, D. Yang, H. Zhang, L. Wu, Energy-aware virtual

machine placement in data centers, in: Global CommunicationsConference (GLOBECOM), 2012 IEEE, 2012, pp. 3243–3249.doi:10.1109/GLOCOM.2012.6503614.

[27] S. Takahashi, A. Takefusa, M. Shigeno, H. Nakada, T. Ku-doh, A. Yoshise, Virtual machine packing algorithms for lowerpower consumption, in: IEEE 4th International Conference onCloud Computing Technology and Science (CloudCom), 2012,pp. 161–168. doi:10.1109/CloudCom.2012.6427493.

[28] C.-C. Lin, P. Liu, J.-J. Wu, Energy-efficient virtual machineprovision algorithms for cloud systems, in: Fourth IEEE Inter-national Conference on Utility and Cloud Computing (UCC),2011, pp. 81–88. doi:10.1109/UCC.2011.21.

[29] P. Schmitt, B. Skiera, C. Van den Bulte, Referral programs andcustomer value, Journal of Marketing 75 (1) (2011) 46–59.

[30] G. Ryu, L. Feick, A penny for your thoughts: Referral rewardprograms and referral likelihood, Journal of Marketing 71 (1)(2007) 84–94.URL http://www.jstor.org/stable/30162131

[31] M. A. Hogg, D. Abrams, Group motivation: Social psycholog-ical perspectives., Harvester Wheatsheaf, Hertfordshire, HP27EZ, England, 1993, Ch. Towards a single-process uncertainty-reduction model of social motivation in groups., pp. 173–190.

[32] A. Iosup, H. Li, M. Jan, S. Anoep, C. Dumitrescu, L. Wolters,D. H. J. Epema, The grid workloads archive, Future Genera-tion Computer Systems 24 (7) (2008) 672–686. doi:10.1016/

j.future.2008.02.003.

18