DAP: A Generic Platform for the Simulation of Distributed Algorithms∗

Ioannis Chatzigiannakis Athanasios Kinalis Athanasios PoulakidasGrigorios Prasinos Christos Zaroliagis

Computer Technology Institute, P.O. Box 1122, 26110 Patras, Greece, andDepartment of Computer Engineering and Informatics, University of Patras, 26500 Patras, Greece.

{ichatz,kinalis,green,zaro}@ceid.upatras.gr, poulak@cti.gr

Abstract

DAP (Distributed Algorithms Platform) is a generic andhomogeneous simulation environment aiming at the imple-mentation, simulation, and testing of distributed algorithmsfor wired and wireless networks. In this work, we presentits architecture, the most important design decisions, anddiscuss its distinct features and functionalities. DAP allowsthe algorithm designer to implement a distributed protocolby creating his own customized environment, and program-ming in a standard programming language in a style verysimilar to that of a real-world application. DAP provides agraphical user interface that allows the designer to moni-tor and control the execution of simulations, visualize algo-rithms, as well as gather statistics and other information fortheir experimental analysis and testing.

1. Introduction

Distributed systems are today ubiquitous in many as-pects of social life, in business, academia and home. Itseems that this trend will continue and we will witness theemergence of distributed applications with increased powerand complexity.

The design of distributed systems presents many chal-lenges both at the theoretical and the implementation level[23]. At the theoretical level the designer has to tacklethe fundamental problems of asynchrony (events take placeat times that cannot be known in advance), limited localknowledge (each computing entity is only aware of infor-mation that it acquires and has no global view of the wholeenvironment), and failures. At the implementation level thedifficulties stem from the fact that it is almost impossibleto achieve scalability, that is, to create a testbed that resem-bles the enviroment where the application is going to be

∗ This work was partially supported by the IST Programme of EU undercontract no. IST-1999-14186 (ALCOM-FT).

deployed (e.g., a network of hundreds of computers). Thiscritical issue of scalability can only be addressed via simu-lations. Simulation environments play a crucial role in thedevelopment of distributed algorithms and their practical as-sessment, provided that they model accurately the applica-tion environment and they are easily usable by the end-user.However, it is not at all obvious how to achieve these goals.

Simulation environments that provide all the necessaryprimitives to allow simulation of any distributed algorithmare not so many. To the best of our knowledge there aretwo such environments: DSP [4], and IOA [12]. On theother hand, there are several environments for the simula-tion of network algorithms (i.e., distributed protocols withlow-level functionality), or environments that focus on thesimulation of a specific area of research in distributed com-puting. Important examples of such environments includens [19], YACSIM [15], and SimUTC [24]. Another crucialcharacteristic is that some simulation environments requestthat a user develops a protocol using a specific descrip-tion language provided by the environment, or a scriptinglanguage (this is the case for ns, IOA, DSP). This shouldbe contrasted to simulation environments (e.g., YACSIM,SimUTC) that allow users to develop programs in a stan-dard programming language (C or C++), which could beadvantageous in the sense that the same program can be di-rectly run on a real distributed environment. We give a briefoverview of the above mentioned environments.

The network simulator, ns, [19] is a discrete event sim-ulator aiming at simulating network protocols for low-levelfunctionality, that is, simulation of TCP-like protocols, aswell as of routing and multicast protocols over wired andwireless (local and satellite) networks. ns requires that theprotocols and the simulation setup is written in the OTclscripting language (Tcl with object-oriented extensions byMIT). A user may develop protocols in a standard program-ming language (C++), but needs to bind them to OTcl in or-der to be simulated by ns. An extended library of network-level services has gradually been built for ns, including ananimation tool (nam) for animating the simulation.

The IOA project [12] provides a formal language for de-scribing processes that are modeled using I/O automata.To model a distributed algorithm or system, a user has toprogram in the IOA language. A set of tools for this lan-guage (including a simulator) that will provide support forthe development, analysis, and simulation of IOA programs,is under current research. The set of tools is developed inJava. Using a tailored language like IOA, or Esterel [6] (asynchronous language for programming reactive systems,e.g., real-time systems and control automata), may be moreconvenient in some applications, but it has a portabilitypenalty (efficient compilers may not be available on someplatforms) and a learning curve.

The Distributed Systems Platform (DSP) [4] is a soft-ware platform designed for the implementation, simulation,and testing of distributed algorithms, and it offers a set oftools which allow the researcher and the algorithm designerto work under a familiar graphical and algorithmic envi-ronment. On the other hand, DSP provides its own set ofsimple, algorithmic languages (DSPL) which can describethe topology and the behavior of distributed systems and itcan support the testing process (on-line simulation manage-ment, selective tracing, and presentation of results) duringthe execution of specific and complex simulation scenarios.DSP has been implemented in C. It uses a client/server ar-chitecture where the client edits and sends the necessary in-formation (the description of the protocol, the topology, thesimulation initializations and actions, like when some fail-ure should happen) to the server. The server compiles an ex-ecutable. In doing that it may also link some protocol fromits database that the client requested. The executable runs atthe server and its simulation is transmitted and viewed at theclient. This scheme is particularly advantageous for teach-ing.

The advantages of extending a general programming lan-guage by adding simulation routines has been noticed be-fore. A specific example is YACSIM [15], a discrete eventsimulator implemented in C that provides a basic set of Csubroutines that the developer can link with his program inorder to produce an executable that performs the simula-tion (i.e., YACSIM does not provide a separate simulator,but the simulator is contained in the executable producedby the user). YACSIM extends C by adding simulation ob-jects, which are data structures and a set of operations onthem. These objects represent activities, queues or statisti-cal records. The activities are either events (for event-drivensimulation), or processes (for process-oriented simulation).The queues are used to delay the occurrence of an eventor the progress of a process until some conditions are met.In order to perform a simulation the user has to manipulateYACSIM’s simulation objects in his code and has to com-pile everything into a single executable. YACSIM is nor-mally used as a base to produce more sophisticated simula-

tors. For example, it was used to produce NETSIM [14], ageneral purpose interconnection network simulator for par-allel architectures.

SimUTC [24] is a simulation toolkit intended for the spe-cific problem of clock synchronization. It is round-basedand built on C++SIM [17]. The latter is a toolkit writtenin C++ that implements facilities provided in SIMULA [5].It uses threads and is a process-oriented continuous-timediscrete-event simulator. It is interesting that the C++SIMdevelopers observe in [17] that C++ compilers producemuch more efficient code than SIMULA, thus resulting infaster simulations.

In this paper we present DAP (Distributed AlgorithmsPlatform), a homogeneous environment, for the implemen-tation, simulation, and testing of any distributed algorithm.DAP focuses on implementing distributed algorithmic ideasdeveloped either for wired or for mobile wireless networksystems. In developing DAP, we set up the following goalsthat actually determine its distinct features:

• To allow the client programmer to create his own en-vironment (abstracting away irrelevant layers, such asthe physical layer) with sufficient flexibility to modeldiverse situations (e.g. wired or wireless mobile net-works).

• To allow the client programmer to program in a naturalstyle, very similar to the style of programming for thefinal real-world application, using a well-known lan-guage.

• To be of substantial help to the algorithm designer byproviding him with the means to monitor the execu-tion of the algorithm and to extract detailed statisticalinformation.

• To be used both as a tool for experimental analysis andfor demonstration or educational purposes.

• To be easy to extend or integrate in other systems.

• To be efficient and easy to use.

Let us now discuss why these issues are important andhow DAP addresses them.

Many systems (e.g., ns) are too generic and incorpo-rate much more detail than it is needed for distributed al-gorithm simulation. In DAP all the less relevant lower lev-els are abstracted in a Topology layer. In such a simple butpowerful model it is easier to capture some important as-pects of a distributed environment such as asynchrony andfailures of computing entities or communication channels.The user is free to customize every aspect of the simula-tion environment according to his own needs by setting theproperties of every simulation object to constant or vari-able values (by selecting a random variable implementingsome probability distribution). Furthermore, the user may

define scenarios, which are series of actions (like node fail-ures, interrupts, etc.) to be executed at specific points (alsorandom variables, if needed) during the simulation, thus al-lowing rigorous testing of particular situations. In additionto the simplicity of the model, DAP provides a graphicaltopology and scenario editor for constructing the requiredenvironment.

Another disadvantage of the majority of the availablesystems is that they require the developer to code his al-gorithm in a language and a style that are specific for thesystem. In this case, the programmer is faced with a steeplearning curve and there is a lot of wasted effort until heis able to utilize the full potential of the system. If, in ad-dition, the language is inefficient (e.g., an interpreted lan-guage) the development/testing cycle becomes too long andthe developer cannot assess easily the quality of his algo-rithm. In DAP the developer can program in a natural style(as in any other program) using C++. C++ is a very efficientlanguage, with a vast array of available tools. It is suitedfor simulations and is a language most developers feel com-fortable with. Furthermore, there is another interesting per-spective: after an algorithm has been evaluated using DAP,the same code can be deployed on a real distributed envi-ronment, without serious modifications and without the useof the simulator, just by linking with libraries we can pro-vide.

The whole point of a simulation is to assess the qual-ity of an algorithm. For this purpose DAP includes a graph-ical user interface (GUI) that can be used to monitor mostaspects of the simulation in real time. This feature is quiteimportant since it allows the developer to pause the simula-tion, make changes to the simulation environment and con-tinue the simulation in order to measure the effect of cer-tain events on the robustness of the algorithm. The GUIincludes modules for designing the topology, constructingscenarios, recording the simulation and viewing statistics.A powerful feature of DAP is that the same simulation canbe observed by more than one user interfaces (but controlledonly by one) on different machines making it an ideal plat-form for demonstration or educational purposes. In addi-tion to the statistics shown by the GUI, the developer is ableto measure any parameters he is interested in through sim-ple method calls. Both the statistics gathered by the user andthe statitics gathered by the system can be presented in var-ious ways (as graphs) or exported (in standard format) foruse by other programs.

DAP has a flexible modular architecture and uses anopen format (encoded in simple XML) for representingand exchanging information (e.g., topology, statistics, com-munication between modules, etc.). These features resultin a variety of interesting possibilities, giving DAP someunique characteristics. The capability of having more thanone users from different computers monitoring the same

simulation is one of them. A more interesting one is that thesimulation itself can be distributed: the simulation of somecomputing entities may be run on different computers (thatare of course connected to the computer running the simu-lation core). Thus the simulator can take better advantage ofavailable computing resources. The modularity of the sys-tem is such that a developer can even code his algorithmin any language of his choice by following our communi-cation protocol (our library, of course, abstracts this com-munication and provides C++ bindings directly). Anothermore practical feature made possible by the use of standardopen formats is that statistical data can be exported to dif-ferent programs for further processing (e.g., statistical pack-ages, spreadsheets, etc.).

An important consideration is the efficiency of the sim-ulation environment. In DAP this is achieved by the use ofC++, and a straightforward programming model (i.e., thereis not any “intermediate” state, or any interpreter involved).

The use of DAP is rather intuitive and does not distractthe developer from his main task, that is, the design, devel-opment and testing of his algorithm.

The rest of this paper is organized as follows. In Sec-tion 2, we present the high-level design of DAP. In Section3 we discuss the design of the Simulator Engine which is theheart of the simulator. The library that provides the interfacebetween an algorithm and the simulator core is described inSection 4. Section 5 discusses the graphical user interfacewhich is the prime medium between the user and the sys-tem. The communication protocols of DAP and the featuresthat enable interoperability with other applications are pre-sented in Section 6, while in Section 7 we explain how an in-terested user can develop an algorithm using DAP. We con-clude in Section 8.

2. The Architecture of DAP at High-level

The main difference between DAP and similar systems isthat every simulated process is actually run by DAP as a dif-ferent “real” (in the operating system sense) process. Thisdecision adds some amount of complexity to the system butit has also four, very important benefits:

1. The developer can code each process without havingto follow any specific programming model (e.g., event-driven).

2. The execution of the simulation is asynchronous, bythe nature of how operating systems manage execu-tion.

3. The processes need not be executing all on the samemachine. On the contrary, the simulation itself may bedistributed.

4. A process may block without affecting the others. Thisobservation is important: under this model it is easy to

implement the blocking receive directive, a commonchallenge for distributed simulators.

On the basis of this design choice, DAP consists mainlyof three components (see Fig. 1).

The simulator engine is the component that actually sim-ulates the distributed system. It handles the set-up of thesimulation, the processing of simulation events, the gather-ing of statistics, the communication with the graphical userinterface, etc. We will examine it in detail in the next sec-tion.

The Graphical User Interface (GUI) is used by the userto set initialization parameters, to control the simulation andgenerally to monitor its execution. The GUI is a separate ex-ecutable and communicates with the engine through normalTCP/IP sockets. This fact gives DAP a lot of flexibility. Theuse of GUI is not strictly necessary as the simulation can beset up and started by using appropriate command-line op-tions on the main simulation engine. Furthermore, the GUIcan disconnect and reconnect at any time, so the user canset-up a lengthy simulation, start it up, disconnect and thenconnect only at regular intervals to monitor its execution.Since the engine and the GUI communicate over TCP/IPit is possible that the simulation runs on a fast computingserver and the GUI on a less powerful computer (e.g., thedeveloper’s laptop). The engine supports also the connec-tion (at any time) of more than one GUI programs. In thiscase one is designated as a controller and is used to controlthe simulation, while the others are passive observers. Therestriction of only one controller is not a technical limita-tion, but just a feature in order to allow centralized controlwhich is typically required; for example, in classes wherestudents observe the simulations controlled by the instruc-tor. Although we currently provide a GUI written in C++(with the use of LEDA [20]), it is possible to write ob-servers in any language by following our simple commu-nications protocol.

The third component is the DAP library (usually abbre-viated to libDAP). The library is linked to every executablethat a developer produces and allows it to communicate withthe simulator engine. The DAP library implements all thedirectives that are needed for the implementation of a dis-tributed algorithm (e.g., send/receive).

All the modules are written in C++ with some help fromLEDA, the library of efficient data types and algorithms[20]. A user of DAP develops the code for his distributedalgorithm in C++ (possibly using data types and algorithmsfrom LEDA), compiles it using a C++ compiler, and linksit with the DAP librariy to produce an executable. The usercan produce more executables exactly in the same way, forthe case where more than one algorithms are needed.

Engine

Event Handling

Statistics

Process LayerComputing LayerTopology Layer

Parser

Communication

Mailbox

Auxiliary modulesController

Observer

Observer

Engine

Process Process

DAP Library

Algorithm

Help

AlgorithmVisualization

GUI

EditorTopology

SystemControl

ScenarioEditor

Figure 1. High-level architecture diagram for DAP.

3. The Simulator Engine

The Simulator Engine (henceforth abbreviated to engine)is the component of DAP which performs the simulationof the distributed system. The characteristics of the simu-lation system – most of them defined through the GUI –are stored here. The engine performs the simulation inter-acting with the GUI and the executables. It updates the vi-sualization data for the GUI and changes the characteristicsof the simulation if the user specifies so through the Con-troller.

3.1. Simulation model

The engine operates on a discrete simulation model, thatis, a model where the system can change its state at onlya countable number of points in time. The simulation isdiscrete-event [16].

The central component of the Engine is an Event queue.Most actions on the system (e.g., the transmission of a mes-sage) generate an appropriate event which is inserted in theevent queue along with a timestamp that specifies the timeof its execution.

The Engine receives communication from some simu-lated process, or from the GUI. In the former case, it maybe a simulated message transmission, coordinating commu-nication with a simulated process, or output from a simu-lated process intended for the GUI. In the latter case, wherethe communication comes from the GUI (in particular, theController), it may be events that should be inserted in the

event queue (like node/channel should go up/down), or in-structions to control the simulation run (like start, pause,resume, step, set delay). Each event is modeled by a classthat includes a handling method that specifies its executionlogic. The simulator engine gets the next event to be simu-lated, updates a Global Clock accordingly and gives controlto the handling method, passing it as an argument a classthat encapsulates the characteristics of the simulated sys-tem. The handling method may change the characteristics ofa system (e.g., a node failure event) or generate more events(e.g., a “send message” event generates “receive message”events with the appropriate delay).

The Global Clock indicates the current simulation time,say t. All events that are specified to happen at time t needto be handled. In particular, the Global Clock is increasedfrom the current value t to the value t′ when all of the fol-lowing conditions are satisfied:

• All events scheduled for time t have been handled.

• t′ is the value of the earliest event in the Event Queuethat has not been handled yet.

• No simulated process can generate an event scheduledfor some time before t′. This is true if each simu-lated process has reached time t′, has completed, oris blocked waiting for some event (like a message re-ceipt).

3.2. Sources of input

The Engine receives input from many sources: the run-ning processes (requesting for example to send a message),the controller, the observers, or from appropriate local fileswhen running without the GUI.

3.2.1. Scenarios The engine has the functionality to sup-port user-defined scenarios, that is, a series of (simulation)actions (like node failures, interrupts, etc.) to be executed atspecified points in time during the simulation. The main useof scenarios is to allow repetitive testing of particular situa-tions. It is technically possible that actions are sent to the en-gine at any moment (if, of course, their intended executiontime is later than the current time). Indeed, since some ac-tions can also be triggered by user actions on the GUI (likenode failures), actions related to scenarios use the same pro-tocol as the ones coming from the GUI. However, the usermay also group a series of actions into a scenario and savethem to a file that can be loaded by the engine. Currentlythe engine supports the scenario actions provided in the fol-lowing list, but more can be easily added.

• Boot/start a node/process at a specified time; if no suchaction is specified for a node/process, it starts by de-fault at time 0.

• Create interrupt on a node/process at a specified time;optionally repeat every some time. These interrupts arehandled by interrupt functions that the user registerswith libDAP.

• Schedule a node or channel to fail perma-nently/temporarily at a specified time. The usercan specify also the probability that the event oc-curs, its duration1, or the repeat interval.

• Schedule a node to recover (with a specified probabil-ity); can be used to recover from permanent failures.

In all the actions, time, interval, duration and probabilityvalues can be specified to be random.

3.2.2. Handling Since input comes from many differentsources, and the input may trigger a variety of situations(from generating an event to stopping the simulation), itshandling becomes a complex task. In order to manage thecomplexity we use the command design pattern [8]. All thepossible actions are represented by command classes whichencapsulate their processing logic. When input arrives, aparser constructs the appropriate command and sends itback to the engine core. The engine then executes the com-mand, passing it as an argument a class that encapsulatesthe characteristics of the system. Under this framework it isa simple matter to add new features and capabilities to oursystem.

3.3. Modeling of the simulated environment

The most important feature of the engine is the way thesimulated environment is modeled. Instead of representingthe full detail of the environment (a situation that wouldplace an unnecessary burden on the developer), its wholecomplexity is abstracted in three layers:

Topology Layer. It describes the location properties of thecomputing nodes. It includes topology nodes that spec-ify positions and topology links that specify connec-tions between these positions.

Computing Layer. It describes the properties of the com-puting nodes. Each computing node resides on a topol-ogy node.

Process Layer. It represents the processes that make up thesimulated algorithm. Each process executes on a com-puting node. The processes may modify characteris-tics of the lower layers (e.g., close a communicationschannel or move the computing node from one topol-ogy node to another).

This model is simple yet powerful and can capture the el-ements of many environments. The exact meaning of each

1 Only valid for temporary failures.

layer depends on the application domain that is being simu-lated. For example, the links of the topology layer may cor-respond to physical connections (e.g., cables) or to virtualchannels (e.g., the transmission range of a mobile phone).Every object on this model has some associated proper-ties (e.g., channel bandwidth). Each property value may beconstant or random (drawing numbers from common ran-dom distributions). DAP provides default configurations forthese properties, but the user is free to provide his own val-ues and even his own properties. This way the model canbe customised to account for any distributed environment.These properties as well as the whole configuration of theenvironment may be static or may change over time. As anexample, nodes and links may fail, or computing nodes maymove to different locations.

DAP focuses mainly on algorithms for wired or mobilewireless networks. On the wired case, topology nodes andchannels specify the actual configuration of the network, i.e.the locations of the computers and the connections betweenthem. The computing nodes describe the properties of thecomputers (e.g., cpu speed). The process layer representsthe actual processes running on the computers.

For the mobile wireless case, DAP utilizes the motiongraph model presented in [10] to abstract the environmentwhere the stations move. The three-dimensional space is ab-stracted by a motion graph (i.e., the detailed geometric char-acteristics of the motion are neglected). This model quan-tizes the motion space into cubes. Each cube has a vol-ume that approximates (from below) the volume of a spherewhich represents the transmission range of a mobile host.The motion graph has a vertex for each cube of the quanti-zation. Two vertices are connected by an edge if their cor-responding cubes are adjacent. In our three-layer model themotion graph corresponds to the Topology layer with eachcube being a node and each edge a link. The computinglayer describes the mobile nodes (also called mobile hosts)of the system (e.g., mobile phones) and the process layerthe different programs that may run inside the mobile nodes(e.g., the program handling incoming calls).

Especially for the mobile wireless domain DAP offersextra functionality about the movement of the computingnodes. The developer may decide to use his own methodsfor handling movement or let the engine move the nodesaccording to a set of commonly used motion patterns. DAPprovides three motion patterns.

Random Waypoint: In this model [1, 13], the mobile hostselects a destination randomly within the area of mo-tion and moves towards that location using the short-est path. The mobile host stays at the new position forsome time and then starts over.

Random Walk: In this model [2, 10, 11], the mobile hostselects randomly one of the outgoing links adjacent to

its current position and moves to the location at the op-posite end of the link. The mobile host stays at the newposition for some time and then starts over.

Arbitrary Motion: In this model [3], the developer gives aseries of positions and delays. The mobile host staysat each of the positions for the specified delay. Afterthe last position the mobile host follows the list back-wards, and then it starts over.

3.4. Auxiliary modules

In addition to the above, there are a few modules for amanifold of small but important tasks.

3.4.1. Communications Communication between the en-gine and the rest of the system (processes and con-troller/observers) is done through the tranceiver mod-ule. This module sets up and maintains a mapping be-tween the process IDs, the controller and the observers onthe one side with their corresponding network (TCP/IP) ad-dresses on the other. This way the engine core doesnot need to have any knowledge of the network de-tails.

3.4.2. Statistics We distinguish between hard-wired anduser-defined statistics. The former refers to the statistics thatare built-in the platform and are provided to any algorithmimplemented in DAP. The latter refers to a mechanism thatallows the user to easily define and display his own (addi-tional) statistics.

Hard-wired statistics refer to the most commonly re-quested statistics from a distributed algorithm. Assume thateach message is assigned a tag denoting its type. Then,hard-wired statistics refer to the number of messages sentper tag, number of messages received per tag, number ofmessages lost per tag, number of node failures and recover-ies, number of channel failures and recoveries, average de-lay of messages per tag, etc.

Depending on the particular algorithm being imple-mented, the user may wish to gather his own specific statis-tics (e.g., memory usage per node of the network). Togather user-defined statistcs, the following libDAP func-tion is provided which can be called by any process at anytime t:

void plot(string tag, double yValue,bool total)

This function registers under the name tag (and perhapsdisplays on a window on the same name, if desired) a two-dimensional plot of the y-value that the user wishes tomeasure as a function of time (x-value in the plot). The pa-rameter total takes boolean values. True denotes that we

want this statistic to act cumulatively, that is, the given y-value will be added to the last y-value and stored as y-valuefor the specific time.

A special module handles the gathering of these statis-tics. This module is available to all the event handling rou-tines, so that every event can record appropriate details.

3.4.3. Mailbox This module handles the storage andtransmission of messages. Messages arriving for a pro-cess are first buffered here and are forwarded to the processone at a time when a process issues a message request di-rective. This strategy allows DAP to be independent of theunderlying transmission mechanism and to have greatercontrol on the implementation of the “blocking receive” di-rective. The processes are unaware of the buffering so thatthe message handling on their side can be kept very sim-ple. The mailbox understands, and handles with prior-ity, messages related to simulation flow of control (e.g., aquit message).

3.4.4. Timekeeper Timekeeper keeps the estimate of theengine about the clocks of each process. This information,and especially the smallest clock value, is used by the en-gine core to coordinate the simulation. A data structure thatprovides efficient extraction of the minimum value alongwith frequent value updates is needed in this module.

3.4.5. Recorder It is possible to record the whole simula-tion run and replay it later by running only the engine. TheRecorder module is used to store all the needed informa-tion. In addition to be replayed by the engine, the recordedinformation can be stored and used for further analysis.

3.4.6. Logger The Logger is a low-level module thatstores detailed information about the simulation run. Themodule is available to all the other components. Even theprocesses may make an entry (through the engine) by is-suing a specific directive. The level of detail can becustomized and the output can be used for debugging or de-tailed analysis.

4. The DAP library – libDAP

The DAP library lies between the code developed by auser and the simulator. It provides to the processes writtenby the user access to certain services (or directives) avail-able to a distributed system. These services are, of course,simulated by the DAP simulator. Our aim was to providea set of services powerful enough to represent most dis-tributed systems of interest, yet easy to learn and use (seeSection 7 for an example of using the library). If needed,we can easily extend libDAP to support other functional-ities that are useful in our setting (taking e.g., ideas fromMPI [21] and PVM [22]).

Current services include finding the ID of a process, thecomputing and the topology node it resides on, the adja-cent channels, its neighbors in adjacent nodes or other pro-cesses in the same node and querying the properties of theseentities. A process may send a tagged message to a partic-ular other process, to all processes at an adjacent node, toall neighbors, or to a particular port2. To receive messages,blocking and non-blocking mechanisms are supported. De-pending on the application domain, there are also directivesfor moving the process or the computing node, or addingnew links.

A process may also change the color of a node or achannel, or the label of the current node, providing sim-ple user-control mechanisms to enhance the visualizationof the simulation run. The input/output of a process is per-formed throught the GUI.

A timeout mechanism is supported, allowing to schedulea wake-up call. A process may register its own functions tohandle such calls or other interrupts.

The DAP library is modular, and the user may specifywhich directives he needs in his application domain. Themodularity of the library combined with the modularity ofthe engine and the simple communication protocol make itstraightforward to extend the system with more functional-ity.

5. The Graphical User Interface

The DAP Graphical User Interface (GUI) provides twopredominant ways of presenting information to the user inorder to assist him in understanding and analyzing complexexecutions, and simultaneously monitor the state of a poten-tially large number of nodes. The information is presentedto the user using both textual techniques like event and mes-sage traces and also with graphical means, by using LEDA’swindows data type that provides an interface for the graph-ical input and output for both the X11 system on Unix plat-forms and for Microsoft Windows systems [20, Ch. 11].

The GUI comes in two flavors. The strong (complete)flavor is called the Controller and includes the followingmodules (see Fig. 1 and 3):

• A Topology and Scenario editor module which can beused to define the topology (connectivity graph) of awired or a fixed wireless distributed system, to definethe connectivity graph as well as the graph describingthe space of motions of mobile hosts in a mobile wire-less distributed system and to define simulation scenar-ios (see Section 3.2.1).

• An Algorithm Visualization module that allows theuser to view the execution of the simulation.

2 Ports are defined as part of the topology of a network and are used tospecify direction, like left-right or north-east-west-south, etc.

• An Event & Log Viewer component that displays themessages generated by the simulator engine (and isdriven by the Algorithm Visualization module).

• A System Control module that allows the user to con-trol the other GUI modules and the simulation (e.g., in-crease the speed, pause it, change the state of a link tofailed, etc).

• A Help module which provides online help.

The weaker flavor of GUI is called the Observer. An Ob-server includes only the Help and the Algorithm Visualiza-tion modules. Many Observers may be connected to the en-gine and view the execution of a simulation at any time. Incontrast, only one Controller is allowed to connect to the en-gine and control the simulation. This scheme allows the useof DAP as demostration platform.

The various components of GUI perform necessary op-erations in a user-friendly manner and greatly enhance theusability of DAP. The Topology and Scenario editor sup-plies the user with the means to customize every aspect ofthe simulation enviroment. Furthermore the topology andscenarios created can be saved in XML format (see Sec-tion 6) and reloaded later thus allowing the reusability oftopologies and scenarios. The System Control module givesto the user fine-grained control of the simulation flow andthe ability to act upon the simulation objects by temporarilysuspending the execution (pause) and performing variousoperations, like changing the status of a mobile host (e.g.,switching it off, or restarting it) or causing links to fail. TheAlgorithm Visualization module displays a detailed viewof the current simulation enviroment and depicts the stateof all simulated objects (see Fig. 2). Using these tools, theuser can easily setup a simulation enviroment and controlthe simulation without knowledge of the underlying details.

The GUI is implemented in C++, using LEDA’s window-ing system for the graphical part of the system. Apart fromthe graphical modules, there exist other modules that imple-ment the required functionality and form a complex but ef-ficient architecture.

The core of the GUI architecture consists of three com-ponents: the Topology manager, the Scenario manager andthe XML manager. The Topology manager holds all in-formation related to the topology, computing and processlayers, and exposes this information to the other modulesthrough simple function calls. Furthermore, the Topologymanager can export and import all topology information toand from the XML representation, with the help of the XMLmanager. The Scenario manager is similar to the Topologymanager but instead of the topology it holds all actions re-lated to the simulation objects and is responsible for theXML representation of scenarios. The XML manager is di-vided into submodules which implement functionality forthe generation, parsing and handling of information repre-

Figure 2. GUI windows while executing the algo-rithm given in Fig. 4. The windows shown are thealgorithm visualization module in the center, theevent viewer on the right, the controller on the top-right and the topology editor in the background.The elected node is the red (shaded) node in theupper area of the algorithm visualization window.Node 1, enclosed in a box, has just received thenotification message through channel (0,1), whichappears dashed, and has changed its color. Theuser has paused the simulation.

sented in the XML format. It is one of the most importantmodules of the GUI since all data exchange between theGUI and the simulation engine is done using an XML pro-tocol (see Section 6).

On top of the core lie the graphical modules, with themost important being the System Control module. The Sys-tem Control module is responsible for the invocation andcoordination of the other graphical modules as well as thecontrol of the simulation engine and the dispatch of simu-lation events to the appriopriate components. The SystemControl module relies heavily on the XML manager for thecommunication with the engine. The Topology and Sce-nario editor module is tightly coupled to the Topology andScenario managers and provides an association betweentthe topology information and the LEDA graph data type al-lowing the powerful graph editing capabilities of LEDA tobe used. It provides additionally a graphical interface to editthe information that can’t be handled by LEDA for the restof the topology and scenario objects. The Algorithm Visu-alization module is invoked by the System Control mod-ule when the simulation begins. It acquires the visual in-formation from the Topology editor module and visualizesthe topology and the execution of the algorithm as messagesfrom the simulation engine are received. It is also responsi-ble for the logging of events. Figure 3 depicts the GUI archi-tecture with arrows representing the direction of data flow.

XMLhandler

XMLgenerator

Scenario Manager

Topology Manager

XMLparser

XML Manager

GUI CoreFilesystem

Engine

Algorithm Visualization

Event&Log

ViewersSystem ControlHelp

Topology andScenario editor

Figure 3. Major modules of the GUI architecture.

6. Communication protocols and interoper-ability

Since DAP is separated into many components there isa need for a mechanism that all components can use to in-teract with each other. The open-ended architecture of DAPmakes it possible to combine DAP-supplied modules withmodules supplied from third-parties. In order to ensure thatthird-party developers can produce new modules with ease,DAP uses simple and open communication protocols.

These protocols are formulated in the XML descriptionlanguage, a specification developed by the World Wide WebConsortium [25]. To use the terms defined for XML the in-termodule communication of DAP is an “application” ofXML. XML is an open standard used in many different ap-plication areas. Its grammar for a specific area can be de-scribed clearly and precisely and is extendable. There are al-ready a lot of tools for parsing and manipulating XML datain many programming languages which means that a third-party module developer is not constrained in his choice oflanguage.

We employ XML to describe the environment of the sim-ulation, that is the Topology, Computing and Process layers.Especially for the topology we use the GraphML emergingstandard [9].

We define a protocol for the communication of the DAPlibrary with the engine as well as a protocol for the commu-nication of the engine with the controllers and observers.By following these protocols it is easy to write a DAP li-brary in a programming language other than C++ since allthe logic resides in the engine. Is is also simple to write an-other flavor of a graphical user interface; an example is aJava applet on a web page monitoring a simulation on theweb server. We plan to provide libraries to ease the devel-opment of external modules.

We use also XML for recording all events that are pro-cessed. In this way the recorder data can be passed to anexternal tool for further processing, from pretty-printing(e.g., through stylesheets) to behaviour analysis (e.g., find-ing event patterns).

Finally, statistics can be written in either appropriateXML or in simple space-delimited text files (in gnuplotstyle).

7. Developing Distributed Algorithms withDAP

In order to run a simulation in DAP four types of ex-ecutable programs are involved: an executable for the En-gine, one for the Controller, one for the Observers (if de-sired), and one or more for the simulated processes. Auser who wishes to simulate a process needs to create anexecutable for it. Note that two simulated processes mayuse the same executable. Even then, they may differentiatefrom each other since they may be run with different argu-ments/input and, also, they have different IDs.

A typical user program would access DAP throughlibDAP. Using libDAP is easy and the best way todemonstrate this is through an example. Consider theLeLann/Chang&Roberts algorithm [18] for leader elec-tion on a ring. Each process participating in the al-gorithm starts by sending its unique code to its nextneighbor. Then, it forwards to the next neighbor ev-ery code it receives that is larger than any code it has seenso far. The process that receives back its own code be-comes the leader, since its code went round the ring andis clearly the larger one. Finally, the leader sends a spe-cial message round the ring notifying the other processesthat the election has ended. Implementing this algo-rithm in DAP is rather easy (see Fig. 4). Observe thatto use DAP the program only needs to include a headerfile and have main() start with the declaration of aDapSystem variable. Through this variable it has ac-cess to all DAP services. Our aim was to make the use ofthese services as easy as possible, in order to keep the fo-cus of the programmer not on using DAP but on develop-ing the algorithm. Furthermore, a reader of the programshould not be distracted from the algorithm by complex li-brary calls nor should the length of the program increase inorder to use DAP.

We can also easily support protocol stacking (i.e., tostack protocols in layers where the lower layers providehigher level functionalities for the higher layers). The usermay implement protocol stacking in DAP by implementingthe lower level protocols as different processes running onthe same computing node with functionalities that are usedby the higher level protocols.

#include "DapSystem.h" /* required by DAP */

const int PREV = 0; // port to previous nodeconst int NEXT = 1; // port to next nodeconst string CODE = "codeTag";

// tag for message with codeconst string END = "endTag";

// tag for ending message with leader code

int main(int argc, char **argv){

DapSystem dap(argc, argv);// required by DAP, has to be 1st line

/* initializations */StringMessage msg;

// use the string-based message implementationint myCode =

atoi(dap.input("Enter unique code >0: ");int maxCode = myCode, inCode, senderID;string tag;int leader = -1; // no leader initiallydap.setColor(WHITE);

/* send your code to the next process */msg << myCode;dap.sendToPort(NEXT, msg, CODE);while (leader < 0){ // run till you learn about the leader/* receive a code from previous process */msg.reset();dap.receive(&senderID, msg, tag, true /* block */);if (tag == CODE){ msg >> inCode;if (inCode > maxCode){ // forward incoming codedap.sendToPort(NEXT, msg, CODE);maxCode = inCode;

}else if (inCode == myCode){ // I am leaderleader = myCode;dap.setColor(RED);/* notify others */msg.reset(); msg << leader;dap.sendToPort(NEXT, msg, END);do{ // wait until the notification comes// around (we need the loop for the// non-FIFO channels)msg.reset();dap.receive(&senderID, msg, tag, true);

}while (tag != END);

}}else{ // notification from elected leader (END)dap.setColor(YELLOW);msg >> leader;dap.sendToPort(NEXT, msg, END);

// pass it on before exiting}

}}

Figure 4. The code for a leader election algorithm.

8. Concluding Remarks

We have presented the architecture and the design of themain components of the Distributed Algorithms Platform,and discussed several distinct features along with their im-plementation and functionality, as well as extensions uponwhich we are currently working. The main strengths of DAPare:

• The user programs in a standard programming lan-guage (C++) and the programs can be easily and ef-ficiently ported to a real distributed environment.

• A user is able to control, monitor, and visualize thesimulation of his algorithm through a Graphical UserInterface.

• Many users can simultaneously monitor and visualize(through a lighter version of the GUI) the simulationof some algorithm from remote locations.

• The execution of the simulation can be distributedamong several hosts (if required).

• Randomness can be introduced in all areas of the sim-ulated distributed system.

• Scenarios can be defined for repetetive simulation ofparticular situations.

The current version of DAP may not be very efficientin simulating algorithms over very large networks (due thehuge number of processes required). To address this issue,the current architecture should be enhanced. For example, ifan algorithm invokes only non-blocking directives, it maybe possible to run all the simulated process in one exe-cutable (since there is no danger that one process couldblock the whole simulation). We are currently investigat-ing several alternatives to enhance the scalability of the sys-tem.

References

[1] J. Broch, D.A. Maltz, D.B. Johnson, Y.C. Hu, J. Jetcheva.A Performance Comparison of Multi-Hop Wireless Ad HocNetwork Routing Protocols. In Pro. 4th ACM/IEEE Interna-tional Conference on Mobile Computing and Networking –MOBICOM’98, pp. 85-97, 1998.

[2] I. Chatzigiannakis, S. Nikoletseas, P. Spirakis. Analysisand Experimental Evaluation of an Innovative and EfficientRouting Approach for Ad-hoc Mobile Networks. In Algo-rithmic Engineering – WAE 2000, Lecture Notes in Com-puter Science, Vol. 1982 (Springer, 2000), pp. 99-110.

[3] I. Chatzigiannakis, S. Nikoletseas, P. Spirakis. An EfficientCommunication Strategy for Ad-hoc Mobile Networks. InProc. 15th Symp. on Distributed Computing – DISC 2001,Lecture Notes in Computer Science, Vol. 2180 (Springer,2001), pp. 159-171.

[4] DSP. http://helios.cti.gr/alcom-it/dsp.

[5] O. Dahl, K. Nygaard. SIMULA — an Algol-based Simula-tion Language. Communications of the ACM, 9(9), pp. 671-678, 1966.

[6] The Esterel language. http://www-sop.inria.fr/meije/esterel/esterel-eng.html.

[7] S. J. Garland and N. Lynch. Using I/O automata for devel-oping distributed systems. In Foundations of Component-Based Systems (G. T.Leavens and M. Sitaraman, eds), Cam-bridge University Press, 2000, pp. 285-312.

[8] E. Gamma, R .Helm, R. Johnson, J. Vlissides Design Pat-terns: Elements of Reusable Object-Oriented Software Ad-dison Wesley, 1995

[9] The GraphML File Format.http://graphml.graphdrawing.org/

[10] K. Hatzis, G. Pentaris, P. Spirakis, V. Tampakas, and R. Tan.Fundamental Control Algorithms in Mobile Networks. InProc. 11th ACM Symp. on Parallel Algorithms and Architec-tures – SPAA’99, pp. 251–260, ACM Press 1999.

[11] G. Holland, N. Vaidya. Analysis of TCP Performance overMobile Ad Hoc Networks. In Proc. 5th ACM/IEEE Interna-tional Conference on Mobile Computing and Networking –MOBICOM’99, pp. 219-230, ACM Press 1999.

[12] The IOA homepage.http://theory.lcs.mit.edu/tds/ioa.html.

[13] P. Johansson, T. Larsson, N. Hedman, B. Mielczarek, andM. Degermark. Scenario-based Performance Analysis ofRouting Protocols for Mobile Ad-hoc Networks In Proc. 5thACM/IEEE International Conference on Mobile Computingand Networking – MOBICOM’99, pp. 195-206, ACM Press1999.

[14] J. R. Jump. NETSIM Reference Manual, Version 1.0. RiceUniversity, 1993.

[15] J. R. Jump. YACSIM Reference Manual, Version 2.1. RiceUniversity, 1993.

[16] A. M. Law and W. D. Kelton. Simulation Modeling andAnalysis. 3rd ed., McGraw-Hill, 2000.

[17] M. C. Little and D. McCue. Construction and Use of a Sim-ulation Package in C++. In C User’s Journal, 12(3), 1994.Also at http://cxxsim.ncl.ac.uk/.

[18] N. Lynch. Distributed Algorithms. Morgan Kaufmann, 1996.[19] S. McCanne and S. Floyd. ns Network Simulator.

http://www.isi.edu/nsnam/ns/.[20] K. Mehlhorn and S. Naher. LEDA: A Platform for Combi-

natorial and Geometric Computing. Cambridge UniversityPress, 1999.

[21] Message Passing Interface Forum. MPI: A message passinginterface. In Proc. Supercomputing’93, pp. 878-883, IEEEComputer Society, 1993.

[22] V. Sunderam, PVM: A framework for parallel dis-tributed computing. Concurrency: Practice and Experience,2(4):315-339, 1990.

[23] P. Spirakis and C. Zaroliagis, “Distributed Algorithm Engi-neering”, Chapter 10 in Experimental Algorithmics - FromAlgorithm Design to Robust and Efficient Software Eds. R.Fleischer, B. Moret, and E. Meineche-Schmidt, Springer-Verlag, 2002, pp. 197-228.

[24] B. Weiss, G. Gridling, U. Schmid, and K. Schossmaier.The SimUTC Fault-Tolerant Distributed Systems SimulationToolkit. In Proc. 7th Symp. on Modeling, Analysis and Simu-lation of Computer and Telecommunication Systems – MAS-COTS’99, 1999.

[25] The XML homepage at the W3 consortium.http://http://www.w3.org/XML/.