UNCO

RREC

TED

PRO

OF

FUTURE: 1595 + Model pp. 1–8 (col. fig: NIL)

ARTICLE IN PRESS

Future Generation Computer Systems xx (xxxx) xxx–xxxwww.elsevier.com/locate/fgcs

A framework for distributed content-based web services notification inGrid systemsI

Andres Quiroz∗, Manish Parashar

Department of Electrical and Computer Engineering, Rutgers University, 94 Brett Road, Piscataway, NJ 08854, United States

Received 1 March 2007; received in revised form 23 May 2007; accepted 10 July 2007

Abstract

We describe a content-based distributed notification service for Publish/Subscribe communication that implements the WS-Notification (WSN)standards. The notification service is built on a messaging framework that supports an associative rendezvous messaging mechanism on astructured overlay of peer nodes. The entire system acts as a notification broker, so that notification publishers and subscribers outside the networkcan achieve loosely coupled communication through a decentralized, scalable service, by interacting with any of the broker peers. Self-optimizingmechanisms are built into the framework to reduce notification traffic within as well as from the peer network. We describe the framework and itsperformance and evaluate the effectiveness of the self-optimizing mechanisms.c© 2007 Published by Elsevier B.V.

Keywords: Distributed systems; Web-based services; Network communication

1. Introduction1

Web services have emerged as one of the key enabling2

technologies for Grid systems, providing platform-independent3

interactions between distributed applications and resources.4

The WS-Notification (WSN) specification [1] is a set of5

web service standards that define protocols for realizing6

the publish/subscribe communication pattern. Most existing7

implementations of this emerging standard (see Section 6) are8

essentially bindings to programming languages that include9

extensible API’s for application developers, and are thus10

purposefully generic. As such, they are not meant to address11

some important issues not regulated by the standard, such as12

mechanisms for the matching of publishers to subscribers, and13

the efficient and scalable management of subscriptions and14

routing of notifications. Of these issues, the former pertains15

to the degree of coupling between publishers and subscribers,16

I The research presented in this paper is supported in part by NationalScience Foundation via grants numbers ACI 9984357, EIA 0103674, EIA0120934, ANI 0335244, CNS 0305495, CNS 0426354 and IIS 0430826, andby the Department of Energy via grant number DE-FG02-06ER54857.

∗ Corresponding author.E-mail addresses: aquirozh@caip.rutgers.edu (A. Quiroz),

parashar@caip.rutgers.edu (M. Parashar).

while the latter deals with the distribution and interaction 17

between the providers of the notification service. 18

With respect to these issues, the nature of pervasive Grid 19

environments creates specific challenges. Grids are large-scale 20

systems where potentially many publishers and subscribers 21

(sensors, computation services, agents, etc.) need to exchange 22

information. As a result, it is critical that the notification service 23

be distributed, decentralized, and scalable. 24

The degree of coupling between Grid elements can be 25

characterized in the following way. Tight coupling implies that 26

interacting elements have specific information about each other 27

and are able to communicate directly and efficiently to achieve 28

a given task or workflow. In general, however, the tasks and 29

workflows that run on Grids are highly dynamic and may be 30

made across application and organizational domains, requiring 31

decoupled interactions without global synchronization of 32

addresses or naming conventions. However, some tasks or 33

workflows might still be more tightly coupled than others, 34

given the physical or logical proximity (control) of computing 35

elements. 36

This paper describes the design and implementation of 37

a notification broker service for subscription management 38

and notification dissemination targeting highly dynamic 39

pervasive Grid environments that adopt the WSN standards. 40

0167-739X/$ - see front matter c© 2007 Published by Elsevier B.V.doi:10.1016/j.future.2007.07.001

Please cite this article in press as: A. Quiroz, M. Parashar, A framework for distributed content-based web services notification in Grid systems, Future GenerationComputer Systems (2007), doi:10.1016/j.future.2007.07.001

UNCO

RREC

TED

PRO

OF

FUTURE: 1595

ARTICLE IN PRESS2 A. Quiroz, M. Parashar / Future Generation Computer Systems xx (xxxx) xxx–xxx

A notification broker is an entity defined by the WS-1

BrokeredNotification (WSBrN) specification for mediating2

decoupled interactions between publishers and subscribers, but3

it is left up to particular implementations to determine how this4

mediation will take place. Our service design is based on a5

distributed and decentralized architecture that supports loosely6

coupled communication by semantic synchronization between7

communicating entities (i.e. a content-based [2] communication8

mechanism). Additionally, the notification service builds on9

a self-managing overlay that can handle coupling within and10

between separately communicating node clusters for enacting11

different kinds of tasks or workflows. To further support the12

efficiency and scalability of our approach, we design self-13

optimization mechanisms for reducing the number of messages14

transmitted by the system. These optimizations are meant15

to alleviate the overhead of notification flows. We show16

experimental data that demonstrates the effectiveness of the17

self-optimizing behavior.18

The rest of the paper is organized as follows. Section 219

presents a brief overview of the WSN family of specifications.20

Section 3 then presents our system design and functionality, and21

Section 4 provides the details of our distributed framework.22

Section 5 then describes and evaluates the system’s self-23

optimizing mechanisms. We outline related work in Section 624

and finally present our conclusions and directions for further25

work in Section 7.26

2. Overview of WSN27

WSN was developed within the context of the WS-Resource28

Framework (WSRF), which describes how to implement OGSA29

capabilities using Web services [3]. The WSN specification30

consists of three interrelated standards: WS-BaseNotification31

(WSBN) [4], WS-BrokeredNotification (WSBrN) [5], and32

WS-Topics (WST) [6]. WSBN specifies the basic elements33

of the notification pattern: the NotificationConsumer (NC)34

that accepts notification messages, the NotificationProducer35

(NP) that formats and generates these messages, and the36

Subscription, a consumer-side initiated relation between a37

producer and a consumer. The only fixed field in a subscription38

is the consumer reference, which by itself implies the39

consumer’s interest in all of the notifications generated by40

the producer to which the subscription was made. Optionally,41

a subscription can contain a filter, specified as a FilterType42

element, that forces a producer to send only those notifications43

that pass (match) the filter. WSBN does not regulate the syntax44

or use of the FilterType element, but suggests three basic types:45

topic filters, message content filters, and producer properties46

filters. WSBN also regulates subscription management, which47

the consumer can perform given the reference it receives in48

response to a subscription. This reference is meant to contain49

enough information to enable it to contact and interact directly50

with the subscription as a resource, as defined by WSRF.51

In addition to the push-style pattern of notification, where52

producers send notifications directly to consumers, WSBN53

defines a pull-style pattern, where messages are stored at a pre-54

defined location (a pull-point) until they are retrieved by the55

consumer.56

WSBrN defines the NotificationBroker (NB) entity and its 57

expected functionality. A notification broker is an intermediary 58

Web service that decouples NC’s from NP’s [5]. A broker is 59

capable of subscribing to notifications on behalf of consumers 60

and is capable of disseminating notifications on behalf of 61

producers. Consumers and producers thus interact dynamically 62

through the NB without the need for explicit knowledge of each 63

other’s identities or locations. Management of this knowledge 64

is delegated to the broker. An NB essentially implements 65

the NC, NP, and other interfaces defined in WSBN. As a 66

specific functionality, a notification broker can accept producer 67

registrations, which is meant for realizing the demand-based 68

publishing pattern. Using this pattern, publishers avoid the 69

(possibly expensive) task of creating notifications when no 70

subscriptions (and, thus, no consumers) exist for them. To this 71

end, an NP must register with the NB, providing a set of topics. 72

When subscriptions are made that correspond to or include 73

topics in a particular producer’s registration, the NB subscribes 74

to the producer for those topics. Only then does the producer 75

start sending notifications. 76

Finally, WST tries to standardize the way in which topics 77

are defined, related, and expressed. It defines the notion of a 78

topic space, where all of the topics for an application domain 79

should be defined and organized, possibly in a hierarchical way. 80

A topic expression is the representation of a particular topic or 81

range of topics. The syntax of a topic expression is identified 82

by the topic expression’s dialect. WST defines three dialects: 83

Simple, Concrete, and Full. A simple topic expression is just 84

the qualified topic name. A concrete topic expression is used 85

for hierarchical topic spaces, and is given in a path notation, 86

such as in myNamespace:news/tv/cnn. Here myNamespace 87

identifies the topic space, and each of the subsequent identifiers 88

belong to successively deeper levels in the hierarchy. A full 89

topic expression is the same as a concrete expression, except 90

that it uses special operators and wildcard sequences for 91

spanning multiple topics within the topic hierarchy. 92

3. Distributed content-based notification broker 93

Our system, referred to henceforth as the notification 94

service, is a distributed and decentralized notification broker. 95

Each of the nodes within the system is a peer that 96

implements the NotificationBroker interface, and an external 97

client (producer or consumer) can interact with any of them. 98

Thus, the whole system essentially acts as a single NB, as 99

illustrated in Fig. 1. This is important because the interface is 100

not a bottleneck, and the system has no single point of failure. 101

Service providers participate by making nodes available to the 102

notification system, and can in turn make use of the system 103

through these nodes. A peer-to-peer design avoids the need for 104

centralized control and gives the service providers the flexibility 105

to join or leave the system at will. 106

Through the NB interface, clients and brokers realize the 107

message exchanges defined in the WSN specifications, using 108

XML messages that represent subscriptions, notifications, etc. 109

Topic expressions are used by the notification service as 110

identifiers for these messages, and provide the means for 111

Please cite this article in press as: A. Quiroz, M. Parashar, A framework for distributed content-based web services notification in Grid systems, Future GenerationComputer Systems (2007), doi:10.1016/j.future.2007.07.001

UNCO

RREC

TED

PRO

OF

FUTURE: 1595

ARTICLE IN PRESSA. Quiroz, M. Parashar / Future Generation Computer Systems xx (xxxx) xxx–xxx 3

Fig. 1. Layout of the broker network. Matching subscriptions and notificationswill be routed to the same rendezvous node, which will perform the matchingand relay the notification.

matching between them. Unlike a purely topic-based system,1

such as WSN with topic filtering, topics in the notification2

service are meant to be content-based. However, unlike3

message content filtering as described in WSN, which requires4

parsing the entire payload content, the idea is to remain as close5

as possible to the notation and semantics defined in WS-Topics6

for topic expressions. This is meant both to support applications7

that implement this standard and to simplify content-based8

indexing.9

Topics in the notification service are expressed using the10

same notation as in the Concrete dialect of WST (see Section 2);11

the difference is that, while normally the topic is an atomic12

unit that should correspond to an actual path in the topic space13

definition, here each of the identifiers is taken as a value from14

a separate dimension in a multidimensional information space,15

where only the type of each dimension is defined. Thus, the16

range of topics is limited only by the possible combinations of17

the values along each dimension.18

To observe the difference, consider a weather monitoring ap-19

plication that subscribes to sensor data. In this example, the20

application may define the information space with three di-21

mensions: geographic area, barometric pressure, and temper-22

ature. An application may define a content-based interest, such23

as weatherService:*,NJ/>25/<50, for which a matching24

content-based topic could be weatherService:Piscataway,25

NJ/29/48.9. Defining a hierarchical topic space to enable this26

type of subscription would not be practical, since individual27

topic identifiers would be needed for each geographic location,28

and, worse still, for each numeric value.29

The system uses a rendezvous-based messaging model [7],30

in which matching messages “meet” at some node within the31

network, referred to as a rendezvous node. The matching and32

routing of messages to service nodes is done by parsing topic33

expressions and mapping their constituent values to the node34

identifier space, as explained in the next section. The mapping35

used ensures that matching topics will be routed to at least one36

common rendezvous node. The messaging model also applies37

the concept of reactive behaviors, by which the behaviors at38

rendezvous nodes are determined by actions embedded in the 39

message request (in this case, subscribe, notify, etc.). The 40

application of this model to the main operations of WSN is 41

described below. 42

Subscribe and notify: Both subscription and notification 43

messages handled by the notification service must contain a 44

topic expression within the FilterType element as explained 45

in Section 2. Subscriptions are first received and routed to 46

one or more rendezvous nodes (a subscription topic can 47

contain wildcards or ranges, so that the subscription may 48

span multiple topics, which may correspond to one or several 49

nodes), which store each subscription under a unique identifier. 50

Since notifications with matching topic expressions will be 51

routed to an intersecting set of rendezvous nodes, matches 52

can be determined at each node, which relays notifications 53

to subscribers based on the contact data provided in each 54

subscription message. Fig. 1 illustrates this rendezvous process. 55

Subscription management: As mentioned earlier, subscrip- 56

tions are stored with a unique identifier at the rendezvous nodes. 57

This identifier is returned to the subscriber when the subscrip- 58

tion is completed, and thus can be used by it to realize the op- 59

erations defined in the SubsriptionManager and PausableSub- 60

scriptionManager interfaces. Note that, since service nodes can 61

enter and leave the system at any time, in which case stored 62

subscriptions are offloaded to other nodes, the full topic expres- 63

sion must still be used for routing to the rendezvous node(s) 64

currently holding the subscription. This differs from the use of 65

endpoint references in WSRF that are not suitable for this dy- 66

namic context. 67

Pull-style notification: Pull-style notification in the notifica- 68

tion service is done in a very similar way to the way regular sub- 69

scriptions are handled. The only difference is that, when a pull- 70

point creation request is made to a broker, a message repository 71

is also created at each rendezvous node where the subscription 72

is stored. Notifications are stored in these repositories rather 73

that being relayed directly to the consumers. Finally, when a 74

consumer invokes the GetMessages command on a broker, it 75

queries the network with the subscription reference to obtain 76

the notifications stored at the repositories, constructs a single 77

response with all of these notifications, and sends them back to 78

the client. 79

Demand-based publishing: Publisher registration occurs 80

in the notification service in exactly the same way as 81

a subscription. The registration topic is used to route a 82

registration message to a node or nodes in the network. In 83

order to accommodate demand-based publishing, however, the 84

procedure for a subscription detailed above must now include a 85

query for publisher registrations that match (or, rather, overlap 86

with) the subscription’s topic expression. If such registrations 87

exist, then the NB that received the original subscription 88

subscribes in turn to the producer(s) for the topic(s) given in 89

the registration(s). Fig. 2 illustrates this mechanism. 90

4. Framework components 91

The infrastructure that supports the notification service is 92

made up of three layers, shown in the left side of Fig. 3. These 93

Please cite this article in press as: A. Quiroz, M. Parashar, A framework for distributed content-based web services notification in Grid systems, Future GenerationComputer Systems (2007), doi:10.1016/j.future.2007.07.001

UNCO

RREC

TED

PRO

OF

FUTURE: 1595

ARTICLE IN PRESS4 A. Quiroz, M. Parashar / Future Generation Computer Systems xx (xxxx) xxx–xxx

Fig. 2. Publisher registration and subsequent subscription. Notice that asubscription and notification are not necessarily the same, but as long as theyoverlap as some node, the registration will be retrieved.

Fig. 3. Architecture and behavior of the distributed system. An XML messageis received by the WS interface, and the action and topic expression areextracted. The topic expression is mapped from its multidimensional spaceto a node ID, which it then uses to route the message on the overlay to therendezvous node, which must execute the corresponding action.

layers support each of the requirements of the notification1

service as follows. First, the WS-Interface is responsible for2

parsing and formatting messages received and sent by the3

broker nodes, as per the WSN specification. Correspondingly,4

this layer also constructs and interprets the messages exchanged5

within the peer network. These messages are handled by the6

Rendezvous layer, so that matching topics are delivered to7

intersecting sets of nodes, as described in Section 3. This is8

ensured by the overlay layer and the content-based routing9

mechanism. The following sections give further details of the10

functionality and implementation of each of these layers.11

4.1. WS-interface12

The Web Service interface for the notification broker13

was implemented in Java using the JWSDP 2.0 API and14

development tools. First, the XML schema for base notification15

and brokered notification, provided in [1], were transformed16

into Java objects using the JAXB binding tools, modifying some17

optional elements to conform to both the Java platform and our 18

own implementation architecture. 19

After the Java objects were created from the schema, 20

the WSDL documents were used to create the Java service 21

interfaces and implementing classes, which were then deployed 22

as a service endpoint for an NB. This endpoint can then be run 23

on an Apache or similar web service container to receive and 24

respond to client requests like the one shown in Fig. 3(1). 25

4.2. Rendezvous layer 26

The implementation of the rendezvous messaging model and 27

reactive behaviors in terms of which the WSN operations were 28

defined is based on the Meteor content-based communication 29

middleware [8]. Meteor defines the exchange of messages of the 30

form (header, action, data) (Fig. 3(2)), where header contains 31

the keywords extracted from topic expressions and used for 32

routing to the appropriate rendezvous node(s) (Section 4.3); 33

action specifies the operation (subscribe, notify, etc.) to be 34

carried out at the receiving node; and data holds the rest of 35

fields and payload associated with each WSN message. 36

4.3. Content-based routing 37

To support content-based routing necessary to ensure that 38

messages are delivered to peers based on topic expressions, 39

we use the Squid routing engine [9]. Squid uses Hilbert Space 40

Filling Curves (SFC’s) [10] to realize the mapping between 41

the multidimensional topic space and the one-dimensional 42

identifier space used by peers in the overlay. Fig. 3 shows how 43

a topic in the multidimensional space is mapped to a node 44

identifier. Notice that, with this mapping, continuous ranges 45

in the identifier space correspond to continuous regions in 46

the multidimensional space. This enables Squid to efficiently 47

handle queries with partial keywords, wildcards, and ranges, as 48

these queries map to a reduced number of peer nodes. Notice 49

also that matching topics, corresponding to intersecting regions 50

in the multidimensional space, will map to intersecting ranges 51

in the id space, which ensures that matching topics will “meet” Q1 52

at rendezvous nodes. SFC’s are discrete mappings, so that 53

continuous data types used for any of the dimensions must be 54

discretized by rounding or truncating values to a fixed precision. 55

Squid’s routing mechanism ensures that queries will be 56

resolved with bounded costs in terms of number of messages 57

and number of nodes involved. Both Squid and Meteor (which 58

builds on Squid) have been evaluated on a university-wide Grid, 59

as well as on PlanetLab in terms of these performance bounds as 60

well as end-to-end latency and load distribution and balancing 61

(see [9,8,11]). 62

4.4. Network overlay 63

Our overlay design is based on the Chord overlay [12]. 64

Chord has a ring-based topology, where every node is 65

randomly assigned an identifier in the range [0, 2m). As nodes 66

dynamically arrive and depart, they self-organize according 67

to their identifier to form the ring. Nodes join by contacting 68

Please cite this article in press as: A. Quiroz, M. Parashar, A framework for distributed content-based web services notification in Grid systems, Future GenerationComputer Systems (2007), doi:10.1016/j.future.2007.07.001

UNCO

RREC

TED

PRO

OF

FUTURE: 1595

ARTICLE IN PRESSA. Quiroz, M. Parashar / Future Generation Computer Systems xx (xxxx) xxx–xxx 5

any other node already in the overlay, from which they obtain1

references to their successor and predecessor in the ring, as well2

as a number of extra references (chords or fingers) to allow for3

faster traversal of the ring (O(log n) hops). References are kept4

in a data structure of size m called a finger table. Successor5

and predecessor references are enough for successful routing in6

Chord.7

We have implemented a two-level overlay design, in which8

nodes are organized into independent Chord rings (clusters)9

based on physical or logical proximity or locality, so that10

most interactions take place within these clusters. For each11

cluster, nodes’ finger tables are no different than in a single-12

level Chord implementation. However, each node additionally13

keeps a reference to its successor in each of the other clusters,14

which can be done by querying any node in a remote cluster15

as in a regular join operation. The result of this design is a16

fully connected higher level overlay at relatively small cost17

(assuming the number of individual clusters is much smaller18

than the number of nodes), where any node can be used as19

a link between clusters. Thus, there are no specialized nodes20

or converging paths as potential bottlenecks or single points21

of failure. Because the successor–predecessor relationship is22

maintained for nodes between clusters, routing between clusters23

is expected to take a better than average number of hops,24

where the average case corresponds to starting a search from25

an arbitrary node in the remote cluster [13]. This is because26

a query will already be closer to the destination node when it27

enters the remote cluster than if an arbitrary node were used.28

Inter-cluster routing in the two-level overlay requires an29

extra parameter that indicates the cluster within which the30

search is to be carried out. Two wildcard values are accepted for31

this parameter: ANY, which is used to route to a node on any32

cluster, starting with the local cluster, and ALL, which is used to33

route to nodes on all clusters. Both values first initiate routing34

normally to a node on the local cluster. For ANY, a query is35

propagated further only if it cannot be resolved on that node.36

For ALL, the query is sent in parallel to every other cluster.37

Note that in both cases, because the access point to remote38

clusters is the successor of the responsible rendezvous node in39

the local cluster, routing in remote clusters is expected to be40

resolved in a better than average number of hops, as explained41

above.42

5. Self-optimization mechanisms43

The number of messages sent within the system can be44

reduced at the notifications level, which is important because45

any reduction in the number of messages leads to a reduction46

in the overhead involved in packaging and delivering each47

individual message, and to an improvement in scalability. In the48

case of Web Services, this overhead is incurred mainly by XML49

and SOAP headers. In addition, the messaging within the JXTA50

framework that supports our current overlay implementation51

also adds considerable overhead. To see how much bandwidth52

is actually consumed by overhead in one implementation, a53

sniffer program was used to capture the packet flows between54

the nodes in the network for notifications. For messages55

between network nodes, the combined overhead of XML and 56

JXTA for each message is just over 3.5 KB, which amounts to 57

about 28 Kbps in a message flow of one message per second. 58

The following mechanisms are used by the system to reduce the 59

number of individual messages in the network. 60

5.1. Grouping of notifications by buffering 61

This optimization is meant to reduce the flows of small 62

and frequent notifications. A simple way to deal with these 63

notification flows is to buffer and group several notifications 64

within a single notification message, a mechanism which is 65

allowed by the WSN XML schema. This way, the headers that 66

would have been transmitted with every individual message are 67

reduced to a single header on a grouped message. Determining 68

when and how many messages to buffer, however, depends on 69

several factors, and thus it is worthwhile to equip the system 70

with logic that allows it to autonomously determine the most 71

appropriate level of message aggregation based on high-level 72

constraints. 73

Without application-specific considerations, messages can 74

be grouped based on two criteria. The first is on messages 75

that correspond to the same topic, and the second is on 76

messages that match the same subscription. These criteria 77

are not necessarily the same, since, depending on how broad 78

a subscription is made (with wildcards or ranges), several 79

different topics may match a single subscription. The system 80

can benefit from applying both criteria, since grouping based 81

on topic equality can be done when messages enter the 82

system at an interface node, which does not necessarily know 83

about subscriptions for that topic, and then subscription-based 84

grouping can be determined at the rendezvous nodes. The 85

mechanism, however, is the same in both cases, so we will 86

describe grouping based on topic equality. 87

The mechanism for grouping and packaging of notifications 88

is as follows. Each interface node keeps a separate buffer of 89

messages for every topic it receives (garbage collection can be 90

employed to eliminate buffers for which no messages arrive for 91

a period of time). Each buffer is configurable by setting the 92

length of the period during which messages are accumulated. 93

This buffering level is determined by managers associated with 94

each buffer, the design of which is described below. 95

If the buffering period is determined only with respect 96

to bandwidth utilization (the number of messages), then the 97

solution is trivial because a higher buffering level (more 98

messages grouped together) always increases the saving 99

achieved. If a limit is set on the buffering period, according 100

to the maximum latency allowed for each individual message, 101

then the solution would always be set to this limit. However, 102

a more balanced solution should consider the tradeoff between 103

bandwidth utilization and message latency. An optimal point 104

can be found between a buffering period of zero (minimum 105

latency, maximal bandwidth consumption) and one equal to 106

the maximum allowed latency (highest buffering level, minimal 107

bandwidth consumption). This is the range used in Eq. (1) 108

below, although the reciprocal of the incoming rate is used as 109

a lower bound instead of zero (any period set smaller than the 110

Please cite this article in press as: A. Quiroz, M. Parashar, A framework for distributed content-based web services notification in Grid systems, Future GenerationComputer Systems (2007), doi:10.1016/j.future.2007.07.001

UNCO

RREC

TED

PRO

OF

FUTURE: 1595

ARTICLE IN PRESS6 A. Quiroz, M. Parashar / Future Generation Computer Systems xx (xxxx) xxx–xxx

incoming message period would result in no buffering). Instead1

of manually assigning a weight to each extreme, a dynamic2

solution is determined based on the relative size of the payload3

with respect to the total message size (Eq. (2) below). The4

rationale behind this is that the relative saving in bandwidth is5

greater for small messages because the overhead constitutes a6

larger fraction of the total data sent, whereas for large messages7

the overhead becomes relatively insignificant. In the former8

case, there is greater payoff for sacrificing latency, and thus9

buffering should have a larger weight. For the latter case, the10

reverse is true. Finally, the period is calculated by obtaining a11

value within the range determined by the weight, using Eq. (3).12

If the incoming rate is very low, with a period higher than the13

maximum latency, then Eq. (3) is not used, and rather the period14

is set directly to zero.15

range = maxLatency − avgIncomingRate−1 (1)16

weight =avgPayload

overhead + avgPayload(2)17

period = maxLatency − weight × range. (3)18

As a proof-of-concept, experiments were conducted for19

single message flows of different incoming rates and payload20

sizes. The maximum latency allowed for messages at each node21

is 1 s. The buffering period, as well as the actual groups of22

messages transmitted by the system, were observed to obtain23

the overhead bandwidth consumption. Fig. 4 plots the results.24

Notice that savings in bandwidth utilization are substantial,25

even though buffering periods are distributed within the range26

of allowable latencies. The lowest buffering period set in this27

case is 373 s for message rate of 20 messages per second and28

10 000 B per message.29

For irregular notification flows, possibly originating from30

several producers publishing notifications on the same topic at31

different time intervals, several complications are possible, such32

as short bursts of notifications at high rates, high variability in33

the incoming rate, and concurrency. A number of mechanisms34

were used to reduce the sensitivity of the system to these35

conditions. To emphasize the self-managing aspect of the36

system, the use of fixed low level parameters was avoided. For37

example, instead of using a fixed threshold for the minimum38

change in the buffering period, the threshold is calculated39

dynamically based on whether or not the change in buffering40

would cause at least one message more or less to be buffered41

at the current estimated incoming message rate. This new42

parameter (the change in the number of messages for which43

a change in period is allowed) is at a higher level and is more44

meaningful than the period alone.45

To test the behavior of the system under these conditions, an46

interface node was set to receive messages with the same topic47

from 32 different producers, each one of which sent messages48

of random payload size between 10 and 500 B at random49

intervals of up to 5 s. The combined effect of these notifications50

produces a high message rate, with high variability. Fig. 551

shows the changes in the buffering period during the time of52

the test.53

Fig. 4. Results for buffering with different incoming rates and payload sizes.Lines correspond to the different payload sizes, bracketed by the minimumand maximum values for each measure. Top: Overhead bandwidth, grows withpayload size. Bottom: Buffering period set, decreases with payload size.

Fig. 5. Change in the buffering period with highly variable incoming periods.

5.2. Demand-based notification relay 54

Ideally, notifications should not be sent if no subscribers 55

exist for them. Demand-based publishing, explained in 56

Section 3, is WSN’s provision for dealing with this issue. 57

However, demand-based publishing depends on producers 58

registering their topics with the notification broker, which 59

particular publishers may not choose to do or may not be able 60

to do if they do not implement the NP interface. To further 61

optimize messaging, the system implements a mechanism 62

which is similar to that of demand-based publishing but that is 63

based on the topics of individual notifications. The idea is that 64

interface nodes should determine when not to relay notifications 65

to rendezvous nodes based on the existing subscriptions. 66

Please cite this article in press as: A. Quiroz, M. Parashar, A framework for distributed content-based web services notification in Grid systems, Future GenerationComputer Systems (2007), doi:10.1016/j.future.2007.07.001

UNCO

RREC

TED

PRO

OF

FUTURE: 1595

ARTICLE IN PRESSA. Quiroz, M. Parashar / Future Generation Computer Systems xx (xxxx) xxx–xxx 7

Unlike publisher registrations that define the topics that1

will be produced beforehand, an interface node has no way2

of knowing which topics it will have to handle. Registering3

for every topic received would also be inefficient. Thus, the4

mechanism devised is implemented as follows. Each interface5

node keeps subscription caches associated with particular6

topics. If there is no cache associated to a particular topic7

when a notification for it is received, the interface node queries8

the network for subscriptions for that topic. If any are found,9

they are placed in the subscription cache, which is marked as10

empty otherwise. Subsequent notifications with the same topic11

will only be relayed if the corresponding subscription cache is12

not empty. To avoid making a query for every topic received,13

locality is exploited by checking a topic against all cached14

subscriptions. New queries are only made if no subscriptions15

exist in these caches (note that for these topics, the notification16

is relayed in any case).17

Meanwhile, at the rendezvous nodes that responded to18

the query, a temporary registration is kept of the interface19

nodes and their corresponding queries. This ensures that if20

a subscription did not exist at the time of the query, a21

matching subscription made thereafter can be made known22

to the interested nodes, so that a notification for which a23

subscription exists is never dropped. The same happens for the24

cancellation of subscriptions. Because they are potentially more25

numerous than publisher registrations, rather than keeping these26

registrations indefinitely, they are deleted once they are used.27

Thus, interface nodes must requery the network once a cache28

for a particular topic becomes empty.29

The overhead of this mechanism for each topic are the query30

and its corresponding response (two messages), as well as31

one message per update of a subscription or its cancellation.32

New queries are only triggered after cancellations. Thus, the33

overhead is small and can easily be made up when large34

notification flows are not relayed while no subscriptions exist,35

unless rates of subscriptions and cancellations are in the same36

order as the rate of notifications.37

6. Related work38

To date, there are several implementations of WSN,39

including Apache’s Pubscribe [14], for Java, WSRF.NET40

from the University of Virginias Grid Computing Group41

[15], for Microsoft’s development platform, pyGridWare [16],42

a Python-based implementation, and GT4 from the Globus43

Toolkit [17], with bindings for both Java and C. Apache’s44

project is derived from GT4-Java. The primary focus of these45

implementations is WSRF, and, as a result, they provide46

different levels of functionality for WSN. For example, the47

pyGridWare and GT4 implementations of WSN are meant48

primarily for providing notifications about the state of resource49

properties. Pubscribe extends these capabilities and fully50

supports both WS-BaseNotification and WS-Topics, but does51

not implement WS-BrokeredNotification. WSRF.NET, which52

was developed using ASP.NET and the IIS infrastructure,53

supports all of the specifications. A thorough comparison of54

these implementations can be found in [18].55

The above implementations are meant to be development 56

tools, providing technology-specific bindings of the standards 57

and extensible API’s. Like the standards themselves, they do 58

not address the issues that arise when actually composing 59

systems that make use of the notification protocols and 60

standards, such as service discovery, and efficient and scalable 61

routing of requests and messages. These issues have been 62

addressed in the context of messaging infrastructures such as 63

the Enterprise System Bus (ESB) architecture [19,20], which 64

mediates the interactions of different web services, including 65

WSN service implementations, by service virtualization. An 66

ESB hides implementation and location details of the services 67

that register to it and is capable of spanning wide area networks 68

and involving multiple infrastructure servers. However, as its 69

name suggests, an ESB is an enterprise-level solution and there 70

is no reference implementation for it. NaradaBrokering [21] 71

is a distributed middleware framework that supports peer-to- 72

peer systems and message-oriented interactions. It implements 73

a variety of messaging technologies and is currently working 74

on an implementation for WSN. The objectives and approach 75

of the NaradaBrokering framework are essentially the same 76

as those that underlie the present work; it manages a network 77

of brokers through which end systems can interact, providing 78

scalability, location independence, and efficient routing. The 79

difference is that Narada brokers are organized in a hierarchical 80

structure which must be maintained through tighter coupling 81

and control mechanisms that do not allow uncontrolled 82

connections and disconnections. 83

Content-based publish/subscribe over DHT’s is a topic for 84

which there is much current work. DHT functionality is usually Q2 85

built using some sort of structured overlay network, the most 86

popular of which are Chord [12], used here, Pastry [22], and 87

CAN [23], because they provide scalability, search guarantees 88

and bounds on messaging within the network, as well as 89

some degree of self-management and fault tolerance with 90

respect to the addition/removal of nodes. With this foundation, 91

designing content-based publish/subscribe systems requires an 92

efficient mapping between content descriptors and nodes in the 93

overlay network, as well as efficient techniques for routing and 94

matching based on these content descriptors, which can contain 95

wildcards and ranges for complex queries. The work in [24– 96

27] addresses these issues to some extent. Meteor and Squid 97

differ from these approaches mainly in the locality-preserving 98

mapping used. The Meghdoot system [27] also uses a locality- 99

preserving mapping, but the multidimensional address space 100

used by its overlay CAN is costlier to maintain than Chord’s one 101

dimensional overlay. To our knowledge, none of these systems 102

are as yet used to implement the WSN standards. 103

Of these systems, [24] also proposes buffering of messages, 104

but it does so statically by setting the buffering level 105

as a multiple of the incoming period. Another messaging 106

optimization designed for web services message exchanges is 107

described in [28]. Here, minimum overhead is achieved by 108

setting up headerless message streams from messages with 109

common headers. These headers are instead stored separately 110

in a context-store, implemented according to the WS-Context 111

standard, and used to disambiguate the messages in the streams. 112

Please cite this article in press as: A. Quiroz, M. Parashar, A framework for distributed content-based web services notification in Grid systems, Future GenerationComputer Systems (2007), doi:10.1016/j.future.2007.07.001

UNCO

RREC

TED

PRO

OF

FUTURE: 1595

ARTICLE IN PRESS8 A. Quiroz, M. Parashar / Future Generation Computer Systems xx (xxxx) xxx–xxx

While this solution is optimum is terms of overhead, the1

management of the context-store adds to system complexity,2

and, more importantly, streams have to be known and set up3

a priori, which is contrary to the dynamicity and burstiness of4

notification traffic.5

7. Conclusion6

We have described the implementation of a distributed7

content-based notification broker service for WS-Notification8

in the context of large-scale, dynamic Grid environments.9

The issues of scalability and dynamism are addressed by our10

system design, and by our implementation, which is based11

on a scalable and self-managing underlying infrastructure. We12

also described and evaluated self-optimization mechanisms to13

reduce the number of notification messages transmitted within14

the network.15

The two-level overlay organization of the messaging16

framework can support different degrees of coupling within17

and between physically or logically organized clusterings of18

Grid elements that execute tasks or workflows. In this sense,19

remote communication is highly decoupled and event-driven,20

and local communication, while more tightly coupled, must still21

be created dynamically and require content-based interactions22

to deal with this dynamism.23

This system is yet to be tested in a WAN setting with24

real application data. As described in [18], compatibility with25

other implementations must be ascertained due to differences26

in the tools and development platforms used. We are currently27

developing a more lightweight overlay implementation that28

does not depend on JXTA. Other optimization mechanisms,29

as well as improvements to those presented here, can also be30

explored.31

References32

[1] OASIS WSN Technical Committee. http://www.oasis-33

open.org/committees/tc home.php?wg abbrev=wsn.34

[2] P.T. Eugster, P.A. Felber, R. Guerraoui, A.-M. Kermarrec, The many faces35

of publish/subscribe, ACM Computing Surveys 35 (2) (2003) 114–131.36

[3] Globus Alliance OGSA webpage. http://www.globus.org/ogsa/.37

[4] S. Graham, D. Hull, B. Murray, Web Services Base Notification 1.3 (WS-38

BaseNotification), Oasis Public Review Draft 01, July 2005.39

[5] D. Chappell, L. Liu, Web Services Brokered Notification 1.3 (WS-40

BrokeredNotification), Oasis Public Review Draft 01, July 2005.41

[6] W. Vambenepe, Web Services Topics 1.3 (WS-Topics), Oasis Public42

Review Draft 01, December 2005.43

[7] I. Stoica, et al., Internet indirection infrastructure, in: Proceedings of ACM44

SIGCOMM, Pittsburgh, PA, 2002, pp. 73–86.45

[8] N. Jiang, C. Schmidt, V. Matossian, M. Parashar, Enabling applications46

in sensor-based pervasive environments, in: Basenets 2004, San Jose, CA,47

2004.48

[9] C. Schmidt, M. Parashar, Flexible information discovery in decentralized49

distributed systems, in: 12th IEEE International Symposium on High50

Performance Distributed Computing, HPDC-12’03, 2003.51

[10] H. Sagan, Space-Filling Curves, Springer-Verlag, 1994.52

[11] N. Jiang, C. Schmidt, M. Parashar, A decentralized content-based53

aggregation service for pervasive environments, in: Proceedings of the54

International Conference of Pervasive Services, ICPS, June 2006.55

[12] I. Stoica, et al., Chord: A scalable peer-to-peer lookup service for internet56

applications, in: Proceedings of ACM SIGCOMM, San Diego, CA, 2001,57

pp. 149–160.58

[13] A. Quiroz, Two-level structured overlay design for cluster management in 59

peer-to-peer networks, Technical Report TR-275, CAIP Center, Rutgers 60

University, 2006. 61

[14] Apache Pubscribe project home. http://ws.apache.org/pubscribe/. 62

[15] WSRF.NET project homepage. http://www.cs.virginia.edu/gsw2c/wsrf. 63

net.html. 64

[16] pyGridWare project homepage. http://dsd.lbl.gov/gtg/projects/ 65

pyGridWare/. 66

[17] GT4 tutorial. http://gdp.globus.org/gt4-tutorial/multiplehtml/index.html. 67

[18] M. Humphrey, et al., State and events for web services: A comparison 68

of five WS-Resource framework and WS-notification implementations, 69

in: 14th IEEE International Symposium on High Performance Distributed 70

Computing, HPDC-14, Research Triangle Park, NC, 2005, pp. 24–27. 71

[19] P. Niblett, S. Graham, Events and service-oriented architecture: The 72

OASIS web services notification specifications, IBM Systems Journal 44 73

(4) (2005) 869–886. 74

[20] M.T. Schmidt, B. Hutchison, P. Lambros, R. Phippen, The enterprise 75

service bus: Making service-oriented architecture real, IBM Systems 76

Journal 44 (4) (2005) 781–797. 77

[21] S. Pallickara, G. Fox, NaradaBrokering: A middleware framework and 78

architecture for enabling durable peer-to-peer grids, in: Proceedings 79

of ACM/IFIP/USENIX International Middleware Conference, 2003, pp. 80

41–61. 81

[22] A. Rowstron, P. Druschel, Pastry: Scalable, decentralized object location 82

and routing for large-scale peer-to-peer systems, in: Proceedings of 83

IFIP/ACM International Conference on Distributed Systems Platforms 84

(Middleware), Heidelberg, Germany, 2001, pp. 329–350. 85

[23] S. Ratnasamy, et al., A scalable content-addressable network, in: 86

Proceedings of ACM SIGCOMM, San Diego, CA, 2001, pp. 161–172. 87

[24] R. Baldoni, C. Marchetti, A. Virgillito, R. Vitenberg, Content-based 88

publish-subscribe over structured overlay networks, in: Proceedings of 89

the 25th International Conference on Distributed Computing Systems, 90

ICDCS ’05, Columbus, OH, June 2005. 91

[25] D. Tam, R. Azimi, H.-A. Jacobsen, Building Content-based Pub- 92

lish/Subscribe Systems with Distributed Hash Tables, in: Lecture Notes 93

in Computer Science, vol. 2944, 2004, pp. 138–152. 94

[26] I. Aekaterinidis, P. Triantafillou, Internet scale string attribute pub- 95

lish/subscribe data networks, in: Proceedings of the ACM 14th Con- 96

ference on Information and Knowledge Management, CIKM, Bremen, 97

Germany, October 2005. 98

[27] A. Gupta, O.D. Sahin, D. Agrawal, A.E. Abbadi, Meghdoot: Content- 99

Based Publish/Subscribe over P2P Networks, in: Lecture Notes in 100

Computer Science, vol. 3231, 2004, pp. 254–273. 101

[28] S. Oh, G. Fox, Optimizing web service messaging performance in mobile 102

computing, Future Generation Computer Systems 23 (4) (2007) 623–632. 103

104

Andres Quiroz is a Ph.D. student of Computer 105

Engineering at Rutgers University. He obtained an 106

M.S., also in Computer Engineering, from Rutgers 107

in 2007, and a B.S. in Systems Engineering at Eafit 108

University, Colombia, in 2004. His research interests 109

are in the area of distributed systems, currently in 110

middleware for peer-to-peer, grid, and autonomic 111

computing. 112

113

Manish Parashar is Professor of Electrical and 114

Computer Engineering at Rutgers University, where 115

he also is co-director of the Center for Advanced 116

Information Processing (CAIP) and director of the 117

Applied Software Systems Laboratory (TASSL). 118

He received a B.E. degree in Electronics and 119

Telecommunications from Bombay University, India, 120

and M.S. and Ph.D. degrees in Computer Engineering 121

from Syracuse University. He has received the Rutgers 122

Board of Trustees Award for Excellence in Research 123

(2004–2005), NSF CAREER Award (1999) and the Enrico Fermi Scholarship 124

from Argonne National Laboratory (1996). His research interests include 125

autonomic computing, parallel and distributed computing, scientific computing, 126

and software engineering. 127

Please cite this article in press as: A. Quiroz, M. Parashar, A framework for distributed content-based web services notification in Grid systems, Future GenerationComputer Systems (2007), doi:10.1016/j.future.2007.07.001