J. Parallel Distrib. Comput. 66 (2006) 647–658www.elsevier.com/locate/jpdc

Running OpenMP applications efficiently on an everything-sharedSDSM�

J.J. Costa∗, T. Cortes, X. Martorell, E. Ayguade, J. LabartaDepartament d’Arquitectura de Computadors, Universitat Politècnica de Catalunya, Barcelona, Spain

Received 16 July 2004; received in revised form 14 February 2005; accepted 6 June 2005Available online 19 September 2005

Abstract

Traditional software distributed shared memory (SDSM) systems modify the semantics of a real hardware shared memory system byrelaxing the coherence semantic and by limiting the memory regions that are actually shared. These semantic modifications are done toimprove performance of the applications using it. In this paper, we will show that a SDSM system that behaves like a real shared memorysystem (without the afore-mentioned relaxations) can also be used to execute OpenMP applications and achieve similar speedups as theones obtained by traditional SDSM systems. This performance can be achieved by encouraging the cooperation between the SDSM andthe OpenMP runtime instead of relaxing the semantics of the shared memory. In addition, techniques like boundaries alignment and pagepresend are demonstrated as very useful to overcome the limitations of the current SDSM systems.© 2005 Elsevier Inc. All rights reserved.

Keywords: OpenMP; Software distributed shared memory; Scalability; Adaptability; Cooperation

1. Introduction

Clusters, or networks of workstations, are becoming oneof the most widely used platforms to run parallel applica-tions. The reasons behind this trend are mainly two: the costof these systems is quite affordable and the technology tobuild them has reached a maturity point to make them stableand easy enough to be exploited.

In these architectures, when the shared-memory program-ming model is used, applications are run on top of a soft-ware distributed shared memory (SDSM) runtime, but un-fortunately it is not a real shared memory one (as in Fig.1(a) that represents the address space of a real shared mem-ory architecture where the code, data and stacks of the

� This work has been supported by the Spanish Ministry of Education(CICYT) under the TIC2001-0995-C02-01 Contract and by the FETprogram of the EU under Contract IST-2001-33071.

∗ Corresponding author. Fax: +34 934017055.E-mail addresses: jcosta@ac.upc.edu (J.J. Costa), toni@ciri.upc.es

(T. Cortes), xavim@ac.upc.edu (X. Martorell), eduard@ac.upc.edu(E. Ayguade), jesus@cepba.upc.es (J. Labarta).

0743-7315/$ - see front matter © 2005 Elsevier Inc. All rights reserved.doi:10.1016/j.jpdc.2005.06.018

threads are shared). In order to improve performance, allimplementations change the semantics of the shared mem-ory to something easier to implement [1,9,14,17,18]. Thetwo main changes are the relaxation of the coherence se-mantic and the limitation of the shared areas. The idea ofthe first modification to the semantic is that a change donein a node will not be visible on the other nodes till some-thing special occurs (synchronization, new thread creations,etc.). The other modification usually done to the “shared-memory” semantics is that not the whole memory is sharedand thus the specific areas to be shared have to be explic-itly defined. An illustration of the resulting address spaceis shown in Fig. 1(b), where three nodes have their privateaddress spaces and a specific memory area is shared. Thesetwo changes mean that in order to run a real shared-memoryapplication, it has to be modified both in the memory reallyshared and in the way it is accessed. In addition, if we wantto achieve performance, many tricks have to be done by theprogrammer, taking into account the side effects of the re-laxations and their implementation and that are not portablefrom one implementation to another. This is not an approachwith future.

648 J.J. Costa et al. / J. Parallel Distrib. Comput. 66 (2006) 647–658

Fig. 1. Different address spaces: (a) real hardware shared memory, (b)usual distributed shared memory and (c) our approach, NanosDSM.

Our proposal (see Fig. 1(c)) is to first implement aneverything-shared distributed memory system (NanosDSM)and then prove that it can be used to run OpenMP appli-cations efficiently. To achieve this efficiency, the problemsthat appear in the SDSM performance will have to be solvedby a tight cooperation between the application runtime andthe SDSM system. To prove our point, we will implementan OpenMP runtime that cooperates with the SDSM systemand achieves speedups comparable with the ones achievedover a relaxed-semantic SDSM, but without paying theprice of relaxation.

2. Running OpenMP on SDSM

In this section, we will first present the main difficultiesfound when trying to run efficiently OpenMP applicationson a distributed memory system such as a cluster. Then, wewill present the solutions traditionally proposed by OpenMPimplementators and SDSM builders.

2.1. Main problems and solutions

The main problem in a SDSM system is that moving orcopying a page from one node to another is a quite time-consuming task. This overhead can be assumed as long aspages are not moved from one node to another very fre-quently. Due to the locality found in most programs, only afew pages are needed by a node at any given time.

One problem appears when a page is needed in exclusivityby two different nodes at the same time. If the two nodesmodify different data located in the same page, we have aping-pong of the page from one node to another when it is notreally shared. Should the data be placed in different pages,the “sharing” would be avoided. This problem is known asfalse sharing.

A popular solution to this problem is to relax the idea thatany modification has to be seen immediately by all nodes.

This will allow several nodes to hold the same page andmodify it as long as they do not modify the same data. Oncethe application reaches a synchronization point, the modi-fied data is forwarded to the rest of the nodes. This solutionsolves the problem but lies a tougher one to the application:programmers have to change their way of thinking as theycannot assume a modification is done till the next synchro-nization point. In addition, this is not even always true asthreads in the same node will see this modification while“threads” in a different node will not.

Even when there is no false sharing, we still have the prob-lem of copying or moving pages from one node to another.A traditional solution proposed to solve this problem is tooverlap the data movement with the execution of the appli-cation. Prefetching is the mechanism used, but this mecha-nism does not cooperate with the application and thus it isvery difficult to prefetch the pages on the right moment, justwhen it will not cause any extra problems such as a pingpong or unnecessary network/CPU consumption. Althoughprefetching may work in some cases, there are many wherethis prediction and finding the appropriate time to do it arenot feasible.

Finally, current SDSM systems do not share the wholememory, but only the regions explicitly declared as shared.This reduces the potential overhead as the number of sharedpages is minimized. The drawback is that, like in the pre-vious solution mentioned, it also places the burden on theprogrammer that has to have, a priori, a clear idea of theregions that are shared and the ones that are not.

2.2. Related work

As we have mentioned, several OpenMP runtime systemshave already been implemented on top of SDSM systems.The most significant ones are the OpenMP translator de-veloped by Hu et al. [8], OpenMP on the SCASH system[7,14], and ParADE [11].

Hu et al. [8] develop an OpenMP translator to runOpenMP applications on top of TreadMarks on a clusterof IBM SP2 nodes. As TreadMarks stacks are private, thetranslator deals with variables declared locally in proce-dures, allocating them in a shared heap in order to allowthem to be accessed from parallel regions.

OpenMP has also been implemented on top of the SCASHsystem [7,14]. This approach uses the Omni compiler [15]to transform OpenMP code to parallel code with calls to aruntime library. The SCASH system is based on a releaseconsistency memory model which allows multiple writers.The consistency is maintained at explicit synchronizationpoints, such as barriers.

ParADE [11] is implemented on a lazy release consistency(HLRC [18]) system with migratory home. The communi-cations subsystem relies on MPI. The translator for ParADEis also based on the Omni Compiler. The synchronization di-rectives are translated to collective communications on MPI.

J.J. Costa et al. / J. Parallel Distrib. Comput. 66 (2006) 647–658 649

The work-sharing constructs are mapped by the compiler toexplicit message passing.

Although they may have some differences among them,the main differences compared to our proposal are: (i)that they all use a relaxed semantic SDSM while our pro-posal is to use a sequential-semantic memory system andencourage the cooperation between the shared memoryand the OpenMP runtime; and (ii) that they generate spe-cific code for the SDSM, which is different from the codegenerated for SMP machines. Our proposal does not re-quire any modification to the OpenMP compiler, as all themessage management is done at the level of the OpenMPruntime.

3. Our environment

3.1. Main philosophical ideas

As we have already mentioned, our environment (Nanos-DSM) offers an everything-shared SDSM. Actually, this isjust one issue that differentiates our work from previouswork done. In this section, we will try to explain the mainissues in our design that are different to what has been doneso far.

• The whole address space is shared: This issue is veryimportant because it reduces the stress placed on the filesystem and the administration. If the code and librariesare shared, we only need to have them in the initial node(where we start the application) and the rest of nodeswill fault the pages and get them. It is also importantto have shared stacks because they will be needed if wewant to have several levels of parallelism or some kindof parallelism within functions when a local variablebecomes shared among all threads.

• Adapt parallel structure instead of implementing fancyconsistency models: NanosDSM offers a sequential se-mantic. This semantic guarantees that any modificationdone in any node is visible to any other node immedi-ately (as would happen in a hardware shared memorysystem). This approach has normally not been followedfor performance reasons. We propose to use it and solvethe performance problem by adapting the structure ofthe parallelism as will be seen later. This changes willbe done by the runtime and will be completely trans-parent to the programmer that will only see OpenMPapplications.

• Sender initiated transactions instead of demand driven:In order to improve performance, issues such as prefetch-ing are normally implemented. These mechanisms arealways started from the node that will need the informa-tion, and this may be done at wrong time for the nodethat has the information. We propose to design mecha-nisms that are initiated by the sender when it knows tobe the right time to send the information.

• Cooperation between layers: Finally, but probably themost important issue, is that all layers have to cooperatein order to make the execution efficient. For instance,the SDSM layer has to inform about some events tothe OpenMP runtime, while the OpenMP runtime has toinform the memory layer about structure and semanticsof the application.

How these philosophical ideas are finally implementedwill be described in the rest of this paper, but by now it isimportant to notice that our system will offer a shared mem-ory equal to a hardware shared memory (as far as seman-tics are concerned) and thus applications that run on a realshared-memory machine will run, with no modification, inour system.

3.2. NanosDSM: An everything-shared SDSM

3.2.1. Managing sequential consistencyIn order to offer a sequential semantic, we have imple-

mented a single-writer multiple-readers coherence protocolin NanosDSM. Any node willing to modify a page has to askpermission to the master of the page, which will take carethat only one node has write permission for a given page.It will also invalidate the copies of the page to be modifiedthat are located in the other nodes.

In order to avoid overloading any node with master re-sponsibilities, we can migrate masters to any node and atany time. The current policy is that the first node to modifya page is the master of that page. If nobody modifies it (aread only page), the node where the application was startedwill behave as the master for that page.

3.2.2. Support to allow cooperation with higher levelsThe most important support consists on offering up-calls.

This mechanism, shown in Fig. 2, allows the application (theOpenMP runtime in our case) to register a memory region,which means that NanosDSM will notify the higher levelwhenever a page fault occurs within this memory region. Themechanism to notify this page faults consists of executingthe function that was passed as a parameter when registeringthe region. As this function is part of the application, itallows the higher level to know what is happening at theNanosDSM level, which is normally transparent. Later inthis paper, we will present mechanisms that use these up-calls to build the cooperation between the OpenMP runtimeand NanosDSM.

NanosDSM also needs to offer the possibility to movepages from one node to another (presend) and to invalidatepages when requested by the application or runtime. A moredetailed description of these mechanisms will also be pre-sented later in this paper. At this moment, we only need toknow that a call is offered for the runtime to specify whichpages have to be moved from which nodes to which othernodes. The NanosDSM also offers a call for a node to vol-untarily invalidate a page it has (preinvalidation).

650 J.J. Costa et al. / J. Parallel Distrib. Comput. 66 (2006) 647–658

Application

2

Shared AreasRegister1

Page @

4

Registered Area?

3NanosDSM in

Page Fault

OpenMP Runtime

Page Fault Upcalls

Fig. 2. Up-calls mechanism.

It is important to keep in mind that these mechanismsare not thought to be used by regular programmers, butby runtime implementators, compiler developers, etc. Thesemechanisms should be transparent to regular applications.

3.2.3. CommunicationAnother important component of NanosDSM is the com-

munications subsystem. Besides providing a mechanism tomove pages from one node to another, it also provides an in-terface to the application (the OpenMP runtime in our case)to implement specific message-based communications withthe goal to avoid the much more costly page faults.

On the one hand, we have implemented what we call re-mote queues. These queues are a communication mechanismused by the runtime to send information from one node toanother. In the shared memory version, this information isplaced in a queue, but as this would be too costly due to pagemovements, we send a message with the information to thereceiver’s queue using sockets. It is important to notice thatthis mechanism is used to improve the efficiency of the run-time, but that user or OpenMP applications will not see it.

On the other hand, if the runtime has to write a fewbytes to a page that knows for sure in which node it re-sides, then we have also implemented a direct write mecha-nisms that allows a node to modify a few bytes on a remotepage without having to fault it. Once again, this mechanismis only available for the runtime that knows how to use itwithout loosing the sequential consistency, but not for userapplications.

Later in the paper, we will describe how the OpenMPruntime uses these communication mechanisms for threadcreation and synchronization.

3.2.4. Other implementation detailsThe current version of NanosDSM is able to run on both

TCP and UDP. In the later case (UDP), it executes a controlmechanism that is able to handle UDP packet loses. In orderto increase the potential usage of NanosDSM, we have alsoported it on top of PadicoTM [4,5], which is a layer designedand implemented at IRISA with the objective of decouplingthe network interface from the higher layers.

The current version is also thread safe, which means thatcan be run in SMP clusters and take advantage of the dif-ferent processors that each SMP has.

Finally, and although in this paper we focus on runningOpenMP applications on top of NanosDSM, we have alsotested other shared-memory programming models suchas Pthreads. We have been able to execute unmodifiedpthread applications on top of our system. This portabilityis achieved because our system is a real shared-memorysystem and not an SDSM with relaxed consistency. In thelater case, we would have needed to modify the pthreadapplications to fit the relaxed model.

3.3. Nanos OpenMP runtime

In our environment, OpenMP applications are parallelizedusing the NanosCompiler [3,6]. This compiler understandsOpenMP directives embedded in traditional Fortran codes,such as the NAS benchmarks 2.3 [10] and generates paral-lel code. In the parallel code, the directives have triggered aseries of transformations: parallel regions and parallel loopbodies have been encapsulated in functions for an easy cre-ation of the parallelism. Extra code has been generated tospawn parallelism and for each thread to decide the amountof work to do from a parallel loop. Additional calls havebeen added to implement barriers, critical sections, etc. Andvariables have been privatized as indicated in the directives.

Nthlib [12,13] is our runtime library supporting this kindof parallel code. Nthlib has been ported to several plat-forms, including Linux/Pentium, Linux/IA64, IRIX/MIPS,AIX/POWER and SPARC/Solaris. We are currently work-ing with the Linux/Pentium version in order to support adistributed memory environment.

Following our philosophy, the first try was to run Nth-lib as a shared library in the NanosDSM environment. Afterthat first milestone was achieved (with terrible performanceresults, as expected), we started the adaptation of the ser-vices in Nthlib to take advantage of the message layer inNanosDSM.

As a result, only four aspects of Nthlib were finallychanged: the way the parallelism is spawned, the imple-mentation of barriers and spin locks, and the reductions.

Nthlib spawns parallelism using an abstraction calledwork descriptor. A thread sets up a work descriptor and itprovides the other threads with it. A work descriptor con-tains a pointer to the function to be executed in paralleland its arguments. Usually, the work descriptor is set up ina shared memory area. In the NanosDSM implementation,the work descriptor is set up in a local memory area andthen it is sent through the message queues described inprevious section to reach the other threads. This solutionallows to distribute work among different nodes avoidingany page fault while spawning parallelism. It is true, thatthis mechanism would not allow a simple or efficient im-plementation of work stealing scheduling policies, but as

J.J. Costa et al. / J. Parallel Distrib. Comput. 66 (2006) 647–658 651

locality is one of the most important issues, we believe thatsuch policies would not be adequate regardless of the workdistribution mechanism implemented.

Nthlib joins the parallelism using barriers. In its simplestform, a barrier contains the number of threads participatingand the number of threads that have reached it. The threadsarriving spin-wait until both number are the same. Afterthat, they continue the execution. In NanosDSM, the samefunctionality is implemented through messages. All threadssend a message to the master thread, and a message is sentback to the threads when the master detects that all thethreads have reached the barrier. This way, there are no pagefaults when the application reaches a barrier.

In the same way as barriers, spin locks have been reim-plemented for this new environment. Spin locks are usedboth internally by Nthlib to protect shared structures and bythe parallel application when using critical regions. They arealso implemented on top of the message system in Nanos-DSM.

Generally, reductions are done directly on the reducedvariable, or to avoid cache problems in a vector that is finallyreduced by the master. In a SDSM environment, this cannotbe implemented directly in any of these ways as the pagewith the variable or the vector would produce too manypage faults. To solve this problem, we implement the vectorversion of reductions, but instead of writing normally, weuse the direct write mechanism described in the NanosDSMsection that allows a node to write on a remote page (viamessage passing) without faulting the page. This mechanismspeeds up reductions a lot avoiding unnecessary page faults.

In addition to that, all the shared data structures in Nthlibare the same that in shared-memory implementations, exceptthat they have been padded to page boundaries in order toavoid false sharing among them.

3.4. Nanos compiler

The last piece in this environment is the compiler. Mostof the time, it works regardless of the fact that the appli-cation will be executed on a SDSM, but in order to makeapplications run efficiently there is only one issue that haveto be taken into account: the alignment of vectors. In or-der to avoid false sharing, vectors (and matrices) should bealigned to page boundaries and should not share pages withother data. This modification in the compiler is very easy toimplement.

4. Our approach: cooperation between runtimes

As we have already mentioned, our approach does notconsist on modifying the behavior of the application northe semantics of the SDSM software, but to encourage thecooperation between the OpenMP runtime and the SDSMsoftware. In this section, we present the three kinds of co-operations we have already implemented and tested, it is

thread 1 thread 2

page p+1

page p-1 page p+1

thread 2thread 1

page p-1 page p

page p

(a)

(b)

Fig. 3. Iterations per thread (a) before and (b) after aligning to theboundaries.

important to notice that these are not the only possibilities,but the ones we have found necessary for our goal.

4.1. Boundaries alignment

The problem: Parallelizing a loop is a widely used tech-nique. The idea is to divide the number of iterations amongthe processors that will execute them in parallel. On a SDSMsystem, if this distribution does not take into account thepage boundaries, we may have several processors writing onthe same page causing false sharing and thus degrading theperformance. This problem is shown in Fig. 3(a), where thedistribution of iterations between two processors producesa sharing with the p page.

The solution: As most parallel loops are executed morethan once, our proposal consists of scheduling the iteration intwo steps. In the first execution of a parallel loop, the runtimestarts with an static scheduling of the iterations (where alliterations are evenly distributed among all processors) andthen learns which iterations access to which pages. Once thisis known, the runtime reschedules the iterations avoiding thesharing of a page among two processors, as shown in Fig.3(b). This mechanism has some overhead the first time theloop is executed, but the benefits are then seen in all furtherexecutions of the loop.

How is the solution implemented: To compute the newschedule of iterations we follow these steps:

(1) Register the memory regions where writes are done (us-ing the up-call mechanism). We only care about writeareas because they are the important ones for page align-ment. Read pages can be replicated in all nodes thatneed them.

(2) When a page fault occurs, the SDSM sends an up-call,and the OpenMP runtime checks if the address is thefirst one in the page. In this case, it marks that thecurrent iteration corresponds to the beginning of a page.Otherwise, it does nothing.

(3) Once each node has its list of iterations that correspondto the beginning of a page, they send them to the master,who will do the redistribution taking into account thelist of iterations and the time spent by each thread. We

652 J.J. Costa et al. / J. Parallel Distrib. Comput. 66 (2006) 647–658

System boundaries

Iterationsat page

NanosDSM

Iteration

Library TrackingLoop Scheduling

Information

Application

OpenMPRuntime

Schedulers

Memory map

feedback

Page Fault Upcalls

Loop Scheduling

Fig. 4. Components that take part in the alignment mechanism.

Prefetchpage P

P(r) P(r)Thread 1

Thread 2

Time

PF

PF

PF

P(r) P(w) P(r)

P(-)P(-)

P(w)

Fig. 5. Prefetch problem, the prefetched page is still being used.

have to note that these times include the page faults andthus may not correspond to the reality. For this reasonwe have to do the task several times till it becomesstable (repeat steps 1–3).

This algorithm generates a new schedule that is thenreused every time the loop is executed. It could also be usedby the parallel loops with the same characteristics.

The modules in the Nanos OpenMP runtime that take partinto the alignment mechanism are presented in Fig. 4.

This mechanism does the best possible load balance takinginto account the page granularity and it adds little overhead.

4.2. Presend

The problem: In order to overlap data movement withcomputation, we need to know which pages will be neededby which nodes and when they will be needed. Prefetching,the traditional solution, can easily detect the pages, but notthe exact time when the data movement will be best donewithout interfering with the application. This is speciallyimportant when using a single writer protocol like us. Thisproblem can be better seen in the Fig. 5, where two threadsappear. One of them prefetchs the p page while the othernode is still writing it. This causes a ping-pong effect movingthe page between the two threads unnecessarily.

The solution: Our solution is to allow a cooperation be-tween the runtime and the SDSM, who will actually do thepresend. The idea is to detect the end of a loop and send thepages that each node has to the nodes that will need themin the next loop. As the work is normally a little bit unbal-

Calculate

PresendPages

Page Access

NanosDSM

Tracking System

ApplicationAfter Each

Presend Pages

Register Shared Areas+ Current LoopLoop

OpenMPRuntime

Memory map

Page Fault Upcalls

Fig. 6. Components that take part in the presend mechanism.

anced (specially if we align boundaries), we can start send-ing pages from one node while others are still computing.The only remaining question is to know if there is enoughtime to send the pages between loops. We will show that inall our experiments, there is enough time to do it.

How is the solution implemented: To compute the list ofpages that have to be copied when presending pages, wefollow these steps:

(1) Learn the sequence of loops in the application to beable to know which loop comes after the current one.

(2) Register the memory regions that are accessed by theparallel loop (note that in this case regions that are readare also important, not like in page alignment wherewrite regions where the only ones to check).

(3) Each thread keeps a list of the page faults it has gener-ated for each loop (using the up-call mechanism) andsends it to the master.

(4) The master makes a new list with the pages that eachnode has that should be sent, once the loop is over, towhich nodes. For performance reasons, if more than onenode have a page that another one will need, all nodesholding the page will have this page in their list of pagesto send. In the execution, only the first one to requestthe copy will actually do it. With this mechanism weguarantee that pages are copied as soon as possible.

(5) Once the thread has this list back, whenever it finishesa loop, it sends the pages specified in the list using thepresend mechanism implemented in the NanosDSM.

The modules that take part into the presend mechanism arepresented in Fig. 6.

4.3. Preinvalidation

The problem: A very similar problem consists on inval-idating the copies of a page once a node wants to modifythem (remember our SDSM implements a sequential seman-tic). This task is also time consuming and it would be desir-able to be able to overlap it with the computation as we dowith presends.

J.J. Costa et al. / J. Parallel Distrib. Comput. 66 (2006) 647–658 653

Table 1Detailed parameter values for the different benchmark classes

Benchmark Param. A B

EP m 28 30CG na 14 000 75 000

non-zero 11 13Ocean n + 2 1024 5000

The solution: Our approach is very similar to the onepresented for presends. When we detect which nodes willneed a page, we also detect if it will need it for writing.If this is the case and a node that holds the page will notneed the page, then we invalidate our copy and inform thepage master that we do not have a copy anymore. Hope-fully, when the node wants to write the page, it will be theonly one holding it as all other nodes will have preinvali-dated it and thus it will be able to write it with no extraoverhead.

How is the solution implemented: The mechanism used isexactly the same as in the presend but taking into accountthe page writes to invalidate the pages a node has that willbe written by other nodes in the next loop.

4.4. Integration of all techniques

As the reader may have guessed, the process of finding theborder to be aligned cannot be done at the same time as thelearning of which pages have to be present an preinvalidated.The effects of one of the techniques affects the results of theother. In order to solve this problem, the learning is done intwo steps. In the first steps, presending and preinvalidationare deactivated and only border alignment is active. Oncethe borders have been found and are stable, the presend andpreinvalidate mechanisms are activated.

This two step mechanism has not produced any problem inthe presented experiments because the borders where foundin one or two iterations.

5. Methodology

5.1. Benchmarks

In order to test our proposal we have executed three stan-dard OpenMP applications. Two of them are NAS bench-marks [10] (EP and CG) and the third one is the Ocean ker-nel from the Splash2 benchmark suite [16,19]. The size ofthe different classes used are specified in Table 1, where thedifferent parameters of each benchmark is detailed by eachproblem class.

5.1.1. The EP benchmark kernelThis kernel generates pairs of Gaussian random deviates

according to a specific scheme. This is a really parallel

!Weighted average computingdiff = 0.0;

C$OMP PARALLEL DO PRIVATE (i,j, tmp)C$OMP& REDUCTION (+:diff)

do j = 2, n+1do i = 2, n+1tmp = A(i,j)A(i,j)=0.2*(A(i,j)+A(i,j-1)

+A(i-1,j)+A(i,j+1)+A(i+1,j))diff = diff + abs(A(i,j) - tmp)

enddoenddo

Fig. 7. Main parallel loop found in Ocean.

benchmark, all the data in the loop is private and it finallydoes a reduction.

5.1.2. Ocean

The Ocean application studies the large-scale ocean move-ments based on eddy and boundary current. It takes a sim-plified model of the ocean based on a discrete set of pointsequally spaced and simplified again as a set of 2D pointplanes. In this situation, it solves a differential equationvia a finite difference method using a Gauss–Seidel up-date, computing a weighted average for each point basedon its 4 neighbors. And it repeats this update until thedifference for all points is less than some tolerance level.The main parallel loop of this benchmark can be seen inFig. 7.

5.1.3. The CG benchmark kernelThe CG benchmark kernel uses a conjugate gradient

method to compute an estimate to the largest eigenvalue ofa symmetric sparse matrix with a random pattern of non-zeros. The problem size of the benchmark class depends onthe number of rows (na) of the sparse matrix and the num-ber of non-zero elements per row (nz). We use the classesA and B as distributed in the NAS benchmarks suite for ourexperiments.

This kernel have the following four consecutive paral-lel loops: (i) matrix–vector product, (ii) dot-product, (iii)AXPY/dot-product combination and (iv) axpy.

5.2. Testbed

All the experiments presented in this paper have beenrun in two different clusters: Kandake and Crossi. Table 2presents their characteristics. For availability reasons, wehave been able to execute them on as much as 7 nodes.

6. Performance results

The results of the different benchmarks will be shownin this section. The speedups are calculated using as thebase case the sequential time of the original OpenMP

654 J.J. Costa et al. / J. Parallel Distrib. Comput. 66 (2006) 647–658

Table 2Platforms used in our tests

Kandake Crossi

Nodes 8 24Available nodes 6 7Processors per node 2 2

(Hyper threaded)Processor speed 266 MHz 2.4 GHzRAM per node 128 MB 2 GBNetwork Myrinet Myrinet

0Number of nodes

0

1

2

3

4

5

6

Spee

dup

EP Class B (Crossi)EP Class A (Crossi)EP Class A (Kandake)

2 4 6

Fig. 8. Speedups obtained by EP.

application. Sequential time means that the original OpenMPcode is compiled using any available Fortran compilerwithout OpenMP support.

All the execution times are shown in the appendix.

6.1. EP

As we have mentioned while describing the benchmark,this is the best possible case for any kind of SDSM. It sharesno pages between the different nodes and thus a SDSM doesnot penalize its executions except for the first copy of thedata.

As this benchmark does not modify shared data, pagealignment does not make sense and thus our mechanism de-tects it and maintains the original static schedule. Regardingthe presend mechanism, there is nothing we can presend,because the parallel loop is executed only once and after itsexecution there is no exchange of data among the nodes.

Fig. 8 presents the speedups obtained by this benchmarkin both machines and using both sizes (classes A and B). InKandake, we only run class A because class B was too big(this happens with all benchmarks).

Observing the graph, we can observe that, as expected,the speedup obtained is quite good. A perfect speedup is notachieved due to the reductions that need to be done at theend of the loop and because the schedule used (STATIC) isnot fully balanced. Some nodes have some more work to dothan others.

Although we have not been able to do a direct comparisonof our results with other SDSM system, we can tell thatsimilar results were obtained by Müller et al. using a relaxedconsistency SDSM [7].

6.2. Ocean

In this benchmark, we have a potential horrible situationfor a SDSM, which is a true sharing among nodes. Manydifferent cells in the array are read by a node and written byanother. This implies that there are no boundaries because nomatter how we split the computation, some elements on oneside will be written by the nodes assigned to the other side.Once again our mechanism detects it and does not align. Ina similar way, the presend mechanism is not useful becausepages are either only accessed by a node, or are read andwritten by several nodes. This last case cannot be taken intoaccount by the presend as we may invalidate a page thatmay be written latter on in the same node. Our granularityis the loop and we cannot use a smaller granularity withinthe loop. In this case, the OpenMP runtime also avoids todo any presend.

Fig. 9 presents the speedup obtained by this benchmark,and once again we can see that they are very good ones.Although there is potential true sharing among nodes, whena node needs a page, the other nodes are not using it. Thisbehavior is quite frequent due to the order in the iterations.It is clear that if the granularity becomes too small and thenodes conflict in the true sharing area, then the performancewill be degraded significantly. Nevertheless, the results pre-sented here show that in all tested cases, the speedup ob-tained is good enough.

6.3. CG

The last benchmark presented in this paper is the CG. Thisbenchmark does not run efficiently on an everything-sharedSDSM if there is no cooperation between the layers. Themost important reason is that the elements of a vector areread by some nodes in a loop and written by different nodesin another loop. This situation is perfect for the presend andalignment mechanisms.

In order to present a more detailed study of the behaviorof this benchmark, we present three different graphs. Thefirst one (Fig. 10) shows the behavior of this benchmarkclass B on Crossi. Then, we present the behavior of thesame benchmark in a smaller class (A) on the same machine(Fig. 11). This will help us to see the effects of the differentproposals when the granularity is smaller and thus will give

J.J. Costa et al. / J. Parallel Distrib. Comput. 66 (2006) 647–658 655

0Number of nodes

0

1

2

3

4

5

Spee

dup

Ocean Class B (Crossi)Ocean Class A (Crossi)Ocean Class A (Kandake)

2 4 6

Fig. 9. Speedups obtained by Ocean.

0Number of nodes

0

1

2

3

4

5

6

7

Spee

dup

NanosDSMNanosDSM+AlignNanosDSM+Align+Presend

2 4 6

Fig. 10. CG class B on Crossi.

us an idea of how well this application will scale. Finally, were-execute CG class A on Kandake and compare its speedupwith the one obtained by TreadMarks (Fig. 12). This experi-ment will show us how well our automatic mechanism doescompared to a version specifically written for TreadMarksand using a relaxed-consistency SDSM.

The first experiment (CG class B on Crossi), shows that agood speedup can be achieved (Fig. 10). It also shows thatas the number of nodes grows, the alignment and presendmechanisms become more important. This makes sense be-cause as we increase the number of nodes, we also increasethe number of boundaries and the number of pages that haveto be copied/moved.

0Number of nodes

0.0

0.5

1.0

1.5

Spee

dup

NanosDSMNanosDSM+AlignNanosDSM+Align+Presend

2 4 6 8

Fig. 11. CG class A on Crossi.

0Number of nodes

0

1

2

Spee

dup

NanosDSMNanosDSM+AlignNanosDSM+Align+PresendTreadMarks

2 4 6

Fig. 12. CG class A on Kandake.

When executing the same benchmark but using a smallerdataset on the same machine (Fig. 11), we clearly see that thealignment and the presend are necessary if some speedup isto be achieved. We can also see that this speedup stops whenmore than 3 nodes are used. The reason behind this behavioris the presence of two variables alpha and beta, which arewritten in sequential and read in parallel, producing a bigcontention. This could be solved if the compiler detects thissituation and informs the other threads with the written valueavoiding any page fault (and we are currently working on it).Even though the load balance has improved the performancea lot, as we divide iterations on a page basis, a given node

656 J.J. Costa et al. / J. Parallel Distrib. Comput. 66 (2006) 647–658

Table 3Measured overhead of shared-everything SDSM

Bench Pages %Time

CG.A 15–16 0.033EP.A 10 0.003Ocean.A 9 0.036

has all the iterations that modify a page or none. This limitsthe possibility of load balancing and thus if very few pagesare used, a good schedule will be impossible. For instance,if the dataset has as many pages as nodes plus one, we willhave all nodes with the iteration of one page and one nodewith the iteration of 2 pages, which means that it will havetwice as many iterations (and thus work) than any othernode.

Finally, we repeated the execution of the benchmarkon Kandake (Fig. 12). The objective was to compare ourspeedup with the one observed when the “same” applicationis run on TreadMarks. We could only test the TreadMarksversion on this machine because we only have a license forthis machine.

The first thing we can see is that it has a similar behavior(speedup wise) than the execution on Crossi. We also seethat this speedup stops growing after 4 nodes and the reasonis also the same as in the previous experiment.

When comparing our behavior with the one achieved byTreadMarks, we can see that we do as well as they do butusing a relaxed-semantic SDSM. In addition, we should re-member that the CG executed in TreadMarks is not theOpenMP version, but a version specially coded for Tread-Marks. Finally, we can also observe that TreadMarks contin-ues to increase its performance when the number of nodesgrows beyond 4. Observe also that, even when using Tread-Marks and relaxed consistency, the speedup is limited to 2.5on 4 processors, confirming the point about the small sizeof the class A of CG.

6.4. Shared-everything overhead

The NanosDSM is a shared-everything DSM, a conse-quence is that the pages containing the code section of theapplication are faulted by the remote nodes instead of beingmoved using an rsh command. From a functional point ofview, this approach is interesting because it does not needany further action rather than launch the OpenMP applica-tion at the master node. All the remote nodes will get thecode without any user interaction.

On the other side, the overhead of this approach must bereally small in order to maintain the performance.

In order to calculate this overhead, the different pagescontaining the code section of the different benchmarks havebeen identified. These code pages are faulted only once by allthreads, and they are faulted at the beginning of the executiontime.

The percentage of time solving these page faults are mea-sured in Table 3 using a pessimistic average time per pagefault of 2 ms, and it shows that this overhead is really smallcompared to the total execution time.

7. Conclusions

We have presented some applications that have achievedvery good speedups. The ones that did not achieve it havebeen compared to the execution on top of other SDSMsystems such as TreadMarks observing a very similarbehavior.

Finally, we have also detected the main limitation in ourapproach. As we have to distribute work on page bases,when the data needed by each nodes reached the size ofjust a few pages, then our alignment mechanism will notbe able to build a good load balance and thus the per-formance will be penalized. On the other hand, we willbe able to run our applications on a system much moresimilar to what we have on a hardware shared memorysystem.

Our future work is to evaluate this proposals using morebenchmark applications, both from the NAS Parallel Bench-marks [10] and the SPEComp 2001 [2]. The experiencetaken from them will then be used to improve the execu-tion environment with new proposals oriented to solve theperformance problems that we could find with them.

Acknowledgements

We thank Ernest Artiaga, Josep Maria Vilaplana and Se-bastia Cortes, for their help in developing part of the soft-ware presented in this paper. We also thank Prof. M. Müllerfor letting us use one of his clusters and for his help.

Appendix. Benchmark’s execution time

Sequential time means that the original Open MP codeis compiled using any available Fortron compiler withoutOpen MP support. The execution times are shown in TablesA.1–A.5.

Table A.1Execution time (seconds) for the Benchmark CG Class A at Kandakecompared with TreadMarks

OnlyDSM Align Presend TreadMarks

Seq 89.0 89.0 89.0 89.02 79.85 56.7 52.15 60.933 60.13 46.39 41.16 43.044 70.46 43.29 37.27 35.535 203.8 42.74 37.36 32.196 255.75 42.17 37.20 31.47

J.J. Costa et al. / J. Parallel Distrib. Comput. 66 (2006) 647–658 657

Table A.2Execution time (seconds) for the benchmark CG Class A at Crossi

OnlyDSM Align Presend

Seq 7.0 7.0 7.02 7.03 5.47 4.343 7.37 5.25 4.214 22.66 5.47 4.355 22.13 5.65 4.546 24.19 6.00 4.767 26.28 6.22 4.978 22.20 6.67 5.38

Table A.3Execution time (seconds) for the benchmark CG Class B at Crossi

OnlyDSM Align Presend

Seq 1421.19 1421.19 1421.192 — — —3 — — —4 378.94 371.32 350.145 343.22 310.18 302.666 — — —7 297.11 280.10 267.79

Table A.4Execution time (seconds) for the benchmark EP

EP.A Crossi EP.B Crossi EP.A Kandake

Seq 101.81 381.05 532.72 58.63 — 307.893 39.28 — 205.174 29.5 117.64 156.815 23.79 106.28 126.056 19.9 — 104.817 16.98 78.08 —

Table A.5Execution time (seconds) for the benchmark Ocean

Ocean.A Crossi Ocean.B Crossi Ocean.A Kandake

Seq 7.36 45.38 54.132 4.34 24.64 28.863 3.21 17.15 21.324 2.58 13.18 16.455 2.21 10.84 13.946 1.98 9.25 12.297 1.8 8.17 —

References

[1] C. Amza, A.L. Cox, S. Dwarkadas, P. Keleher, H. Lu, R. Rajamony,W. Yu, W. Zwaenepoel, TreadMarks: shared memory computing onnetworks of workstations, IEEE Comput. 29 (2) (1996) 18–28.

[2] V. Aslot, M. Domeika, R. Eigenmann, G. Gaertner, W.B. Jones, B.Parady, SPEComp: A New Benchmark Suite for Measuring ParallelComputer Performance, Lecture Notes in Computer Science, vol.2104, 2001.

[3] E. Ayguade, M. Gonzalez, J. Labarta, X. Martorell, N. Navarro, J.Oliver, NanosCompiler: a research platform for OpenMP extensions,in: Proceedings of the First European Workshop on OpenMP, Lund,Sweden, 1999.

[4] A. Denis, C. Prez, T. Priol, PadicoTM: an open integration frameworkfor communication middleware and runtimes, in: IEEE InternationalSymposium on Cluster Computing and the Grid (CCGRID2002),2002.

[5] A. Denis, C. Prez, T. Priol, A. Ribes, Padico: a component-basedsoftware infrastructure for grid computing, in: International Paralleland Distributed Processing Symposium (IPDPS), 2003.

[6] M.Gonzalez, E. Ayguade, X. Martorell, J. Labarta, N. Navarro, J.Oliver, NanosCompiler: supporting flexible multilevel parallelismexploitation in OpenMP, Concurrency: Practice Experience (12)(2000) 1205–1208.

[7] M. Hess, G. Jost, M. Müller, R. Rühle, Experiences using OpenMPbased on compiler directed software DSM on a PC cluster, in:Workshop on OpenMP Applications and Tools (WOMPAT’02), 2002.

[8] Y.C. Hu, H. Lu, A.L. Cox, W. Zwaenepoel, OpenMP for networks ofSMPs, J. Parallel Distributed Comput. 60 (12) (2000) 1512–1530.

[9] W. Hu, W. Shi, Z. Tang, JIAJIA: an SVM system based on a newcache coherence protocol, in: Proceedings of the High PerformanceComputing and Networking (HPCN’99), Lecture Notes in ComputerScience, vol. 1593, Springer, Amsterdam, Netherlands, 1999, pp.463–472.

[10] H. Jin, M. Frumkin, J. Yan, The OpenMP implementation of the NASparallel benchmarks and its performance, Technical Report NAS-99-011, NASA Ames Research Center, October 1999.

[11] Y. Kee, J. Kim, S. Ha, ParADE: an OpenMP programmingenvironment for SMP cluster systems, in: Supercomputing 2003(SC’03), 2003.

[12] X. Martorell, E. Ayguade, N. Navarro, J. Corbalan, M. Gonzalez,J. Labarta, Thread fork/join techniques for multi-level parallelismexploitation in NUMA multiprocessors, in: 13th InternationalConference on Supercomputing (ICS’99), 1999.

[13] X. Martorell, J. Labarta, N. Navarro, E. Ayguade, A libraryimplementation of the nano-threads programming model, in: Euro-Par, vol. II, 1996, pp. 644–649.

[14] M. Sato, H. Harada, Y. Ishikawa, OpenMP compiler for a softwaredistributed shared memory system SCASH, in: WOMPAT2000, 2000.

[15] M. Sato, S. Satoh, K. Kusano, Y. Tanaka, Design of OpenMP compilerfor an SMP cluster, in: EWOMP ’99, 1999, pp. 32–39.

[16] J.P. Singh, W.-D. Weber, A. Gupta, Splash: stanford parallelapplications for shared-memory, ACM SIGARCH Comput.Architecture News 20 (1) (1992) 5–44.

[17] W.E. Speight, J.K. Bennett, Brazos: a third generation DSM system,in: Proceedings of the USENIX Windows NT Workshop, 1997.

[18] L. Whately, R. Pinto, M. Rangarajan, L. Iftode, R. Bianchini, C.L.Amorim, Adaptive techniques for home-based software DSMs, in:13th Symposium on Computer Architecture and High PerformanceComputing, 2001.

[19] S.C. Woo, M. Ohara, E. Torrie, J.P. Singh, A. Gupta, The SPLASH-2 programs: characterization and methodological considerations, in:Proceedings of the 22th International Symposium on ComputerArchitecture, Santa Margherita Ligure, Italy, 1995, pp. 24–36.

Juan Jose Costa is a Ph.D. student atthe Universitat Politecnica de Catalunya(Barcelona, Spain) since 2002. He obtainedhis M.S. degree in Computer Science atthe same university. His main interests aredistributed shared memory and cluster com-puting. Currently, he is doing his researchat the CEPBA-IBM Research Institute.

658 J.J. Costa et al. / J. Parallel Distrib. Comput. 66 (2006) 647–658

Toni Cortes is an Associate Professor atthe Universitat Politecnica de Catalunya(Barcelona, Spain) since 1999. He obtainedhis M.S. and Ph.D. Degrees in ComputerScience at the same university, and is cur-rently the Coordinator of the single-systemimage technical area in the IEEE TaskForce on Cluster Computing (TFCC). Hismain interests are cluster computing andparallel I/O. Dr. Cortes has also been work-ing on several European industrial projectssuch as Paros, Nanos, and POP. As a resultof his research, both in basic research and

technology transfer, Dr. Cortes has published more than 35 papers ininternational journals and conferences, has published 2 book chapters andhas edited a book on parallel I/O (“High-Performance Mass Storage andParallel I/O: Technologies and Applications”, IEEE press, New York). Healso acts a program committee for many international conferences andjournals.

Xavier Martorell received the Engineeringdegree in Computer Science in 1991 andthe Ph.D. degree in Computer Science in1999, both from the Technical University ofCatalunya (UPC), Spain. Since 1992, he hasbeen lecturing on operating systems and par-allel run-time systems. Currently, and since2001, he is an Associate Professor of theComputer Architecture Department at UPC.His research interests cover the areas of op-erating systems, run-time systems and com-pilers for high-performance multiprocessorsystems. He has participated in several long-

term research projects with other universities and industries, mostly in theframework of the European Union ESPRIT and IST programs. Currently,he is a visiting scientist at the IBM T.J. Watson Research Center.

Eduard Ayguade received the Engineeringdegree in Telecommunications in 1986 andthe Ph.D. degree in Computer Science in1989, both from the Technical University ofCatalunya (UPC), Spain. Since 1987, he hasbeen lecturing on computer organization andarchitecture and optimizing compilers. Cur-rently, and since 1997, he is a Full Professorof the Computer Architecture Department atUPC. His research interests cover the ar-eas of parallelizing compilers and runtimesystems for high-performance multiproces-sor systems. He has published more than

125 papers in these topics and participated in several long-term researchprojects with other universities and industries, mostly in the frameworkof the European Union ESPRIT and IST programs. Currently he is oneof the project managers of the CEPBA-IBM Research Institute.

Jesus Labarta is a Full Professor at theComputer Architecture Department of UPC(Technical University of Catalonia) since1990. Since 1981, he has been lecturingon computer architecture, operating systems,computer networks and performance evalua-tion. His research interest has been centeredon parallel computing, covering areas frommultiprocessor architecture, memory hierar-chy, parallelizing compilers, operating sys-tems, parallelization of numerical kernels,metacomputing tools and performance anal-ysis and prediction tools. He has leaded

the technical work of UPC in 15 industrial R+D projects. Since 1995,he is Director of CEPBA where he has been highly motivated by thepromotion of parallel computing into industrial practice, and especiallywithin SMEs. In this line he has acted as responsible of three technologytransfer cluster projects where his team managed 28 subprojects.His major directions of current work relate to performance analysis toolsand OpenMP. The most representative result of his work on tools areParaver and Dimemas. The work on OpenMP and OS scheduling isreflected in the NANOS platform. He actively participates in the OpenMPARB Futures Committee proposing and evaluating the potential of futureextensions to the standard.Since 2000, he is strongly committed in carrying out and promotingproductive research co-operation with IBM as part of the CEPBA-IBM