Pathways: augmenting interoperability across scholarly repositories

arX

iv:c

s/06

1003

1v1

[cs.

DL]

5 O

ct 2

006

Accepted forInternational Journal on Digital LibrariesSpecial Issue on Digital Libraries and eScience

Pathways: Augmenting interoperability across scholarly repositoriesSimeon Warner1, Jeroen Bekaert2, Carl Lagoze1, Xiaoming Liu 3, Sandy Payette1, Herbert Van de Sompel3.

1 Computing and Information Science, Cornell University, Ithaca, NY 14853, USA2 Ghent University, Faculty of Engineering, Jozef Plateaustraat 22, 9000 Gent, Belgium3 Research Library, Los Alamos National Laboratory, Los Alamos, NM 87545, USA

Abstract. In the emerging eScience environment, reposito-ries of papers, datasets, software, etc., should be the foun-dation of a global and natively-digital scholarly communica-tions system. The current infrastructure falls far short ofthisgoal. Cross-repository interoperability must be augmented tosupport the many workflows and value-chains involved inscholarly communication. This will not be achieved throughthe promotion of single repository architecture or contentrep-resentation, but instead requires an interoperability frame-work to connect the many heterogeneous systems that willexist.

We present a simple data model and service architecturethat augments repository interoperability to enable scholarlyvalue-chains to be implemented. We describe an experimentthat demonstrates how the proposed infrastructure can be de-ployed to implement the workflow involved in the creation ofan overlay journal over several different repository systems(Fedora, aDORe, DSpace and arXiv).

1 Introduction

The manner in which scholarly research is conducted is chang-ing rapidly. This is most evident in Science and Engineer-ing [42], but similar revolutionary trends are becoming ap-parent across disciplines [43]. Improvements in computingand network technologies, digital data capture techniques,and powerful data mining techniques enable research prac-tices that are highly collaborative, network-based, and data-intensive. Moreover, the notion of a unit of scholarly com-munication is changing fundamentally. Whereas in the paperworld, the concept of a journal publication dominated the def-inition of a unit of communication, in the emerging eScienceenvironment, units of communication are increasingly com-plex digital objects. The digital objects can aggregate datas-treams with both a variety of media types and a variety of

intellectual content types, including papers, datasets, simula-tions, software, dynamic knowledge representations, machinereadable chemical structures, etc.. Repositories that host suchcomplex digital objects are appearing on the network at arapid pace.

In the light of these profound changes, we envision theemergence of a natively digital scholarly communication in-frastructure that has this wide variety of repositories as itsfoundation. This infrastructure would leverage the value ofthe digital objects in the underlying repositories by makingthem accessible for use and re-use in many contexts. In thisinfrastructure, repositories are not regarded as static nodesin a scholarly communication system that are merely taskedwith archiving digital objects that were deposited there byscholars. Rather, repositories are perceived as the buildingblocks of a global scholarly communication federation in whi-ch each individual digital object can be the starting point ofvalue chains with global reach.

Implementation of this infrastructure brings up a varietyof intriguing prospects and associated questions across thewhole sociological-economical-legal-technical spectrum. Inthe Pathways project, a joint project between Cornell Univer-sity and the Los Alamos National Laboratory, we are explor-ing the technical problem domain. We focus on identifyingand specifying the fundamental components required to fa-cilitate the emergence of a natively digital, repository-basedscholarly communication system. Our research tries to findthe appropriate level of cross-repository interoperability thatwill provide a sufficiently functional technical basis for therealization of the vision, and will stand a realistic chanceofbeing implemented in existing and future repository systems.

This work is important because the current level of cross-repository interoperability is inadequate to support advancedforms of communication. Different communities have fol-lowed their own perspectives on repository design, imple-mentation and management as well as on digital object rep-resentation and identification. Current interoperabilityis pro-vided mainly by support of the OAI-PMH [34] and its manda-

http://arXiv.org/abs/cs/0610031v1

2 Warneret al.: Pathways: Augmenting interoperability across scholarlyrepositories

tory Dublin Core metadata format [20]. Realizing the visionwill require significantly augmented cross-repository interop-erability.

The remainder of this paper is organized as follows. Sec-tion 2 presents several motivating scenarios, and then sec-tion 3 describes related interoperability work. The follow-ing sections then introduce ideas for a cross-repository inter-operability framework that have resulted from the Pathwaysproject. The proposed high-level requirements for participat-ing repositories can be summarized as follows:

– Support for a shareddata modelfor digital objects(sec-tion 4.1).

– Support for asurrogateformat that serializes thedigitalobjectin accordance with thedata model(section 4.2).

– Support for three core repository interfaces:obtain, har-vestandput, to allow dissemination and ingest ofsurro-gates(section 5).

The proposed framework also requires a shared serviceregistry that lists the network location of the core interfacesfor participating repositories. Section 6 describes the serviceregistry and possible format and semantic registries that wouldfurther empower the environment. In section 7 we describeexperiments to implement an overlay journal scenario (an ex-ample we will use repeatedly throughout this paper) usingthis framework over existing repositories. Section 8 presentsplans for future work, and section 9 draws some conclusions.A less technical exposition of these ideas is given in [16].

2 Motivating context

In order to gain insights into the characteristics of the desiredinteroperability framework, it is helpful to investigate scenar-ios that drive this need for augmented interoperability. Wesee two classes of cross-repository value-chains: rich cross-repository services and cross-repository scholarly communi-cation workflows.

In the first class of cross-repository value chains, repos-itories are regarded as sources of materials that can be usedin services with a reach beyond the boundaries of a singlerepository. Materials should be exposed by repositories inamanner that allows for the seamless emergence of rich andmeaningful services. Discovery services are an obvious ex-ample of this class, and, although support of the OAI-PMHhas resulted in a suite of cross-repository discovery capabil-ities, their functionality remains limited. For example, imag-ine creating a special-purpose search engine that collectsonlymachine-readable chemicals structures, expressed using theXML Chemical Markup Language (CML), contained in dig-ital objects hosted by repositories worldwide. The currentin-teroperability environment provides neither the ability to ex-pose digital objects at a repository interface in a manner thatunambiguously reveals the digital object’s constituent datas-treams, nor the language to express their intellectual contenttype (e.g. chemical structure). As a result, the creation ofthe cross-repository chemical search engine would currently

be truly complex, and would involve numerous repository-specific trial and error procedures.

Consider the case where monitoring agencies make se-mantically tagged data on Arctic sea ice available in inter-operable repositories. An automated alerting service mightthen be able to discover and use both raw and processed data(with raw data provenance accurately indicated) to provideearly warning of events such as the abrupt shrinkage in Arc-tic sea ice in 2005. The output might be a report, a new dig-ital object, containing both static ‘snapshot’ results andim-porting dynamically computed elements. Accurate version-ing of datasets would allow readers to be made aware of lateramended inputs and perhaps even to recompute the resultsincluded in the report based on machine-actionable descrip-tions of the transform and visualization service. A newspaperarticle on the findings might reference the source reports al-lowing readers to delve into and understand the sources andthe basis of the claims as far as their understanding permits.

In the second class of cross-repository value chains, repos-itories are regarded as the basic building blocks of a digitalcommunication system, and scholarly communication itselfis seen as a global cross-repository workflow [18]. Digitalobjects contained in repositories are the subjects of the work-flows, and are used and re-used in many contexts.

Citation is probably the most obvious example of this.In today’s scholarly communication system, citation is im-plemented by inserting textual information describing a citedpaper at the end of the citing paper, either by just typing it,bycopy/pasting it from a Web page, or by importing metadatafrom a personal bibliographic citation tool. Thus, citationsthat are included in a digital manuscript are purely textualandare not natively machine readable or machine actionable. Asa result, variouspost-factumapproaches have been devisedto connect citing paper to cited paper by means of hyper-links in the Web environment [13]. These approaches includefuzzy metadata-based citation matching [23], the DOI-basedCrossRef linking environment [11], and the OpenURL frame-work for context-sensitive linking [14]. The variable qualityof citation metadata, among other factors, means that noneof these approaches is foolproof. Furthermore, it is challeng-ing to imagine how these approaches would extend beyondconventional scholarly papers, into the realm of complex dig-ital objects that contain datasets, simulations, visualizationsand so forth. It is therefore intriguing to think about citationas the re-use of the cited digital object in the context of theciting digital object.

To understand this expanded view of citation, imagine be-ing able to drag a machine readable representation of a digitalobject hosted by some repository, and to drop it into the citingobject that, once finalized, is submitted into another reposi-tory. Now imagine being able to do the same for the citing ob-ject ad infinitum. Assuming that the machine readable repre-sentations that are being dragged and dropped contain the ap-propriate properties, the result would be a natively machine-traversable citation graph that would span across repositoriesworldwide. With appropriate user tools this would not only

Warneret al.: Pathways: Augmenting interoperability across scholarlyrepositories 3

be vastly more functional than current forms of citation, butalso simpler to use and to manage.

Collectively, these scenarios lead to a number of high-level observations:

Long-term perspective — Scholarly communication is along-lasting endeavor, and, as a consequence, a long-termperspective should inspire the thinking about a future dig-ital scholarly communication infrastructure. Clearly, thisyields requirements related not just to the longevity ofrepositories and their collections, but also to the inter-operability framework. The framework should be definedwith sufficient abstraction to allow implementation usingdifferent technologies as time goes by, and should not betied to a specific type of identifier, but rather support allcurrent and future identification systems.

Content-transfer is often unnecessary— Most of the valuechains illustrated in the above scenarios do not require thetransfer of all digital object content. Instead just a subsetappropriate to the particular value chain. For example, thecitation scenario requires only the transfer of the biblio-graphic metadata of the cited paper, whereas the searchengine scenario only requires the transfer of the chemicalformula. Full content-transfer as required for repositorymirroring is just one of many use cases that should beenabled by a desired solution.

Fine grained identification — Identifiers of journal articles,such as DOIs, are typically repository independent in thesense that copies of a paper with a given identifier storedby multiple repositories share the same public identifier.This level of identification granularity is sufficient for ci-tation purposes. However, it becomes inadequate whentrying to record the chain of evidence for cross-repositoryvalue chains because these have a specific digital objectfrom a specific repository as their subject. This means thata finer level of identification granularity is required thanprovided by the existing bibliographic infrastructure.

3 Related work

Pathways is focused on defining a commondata modelandservice interfaces. These are designed to enable re-use andre-combination of digital objects and their components, tofa-cilitate workflows over distributed repositories, and to enablecomputation and transformation of digital objects with dy-namic service linkages. A key aspect of this work is that itexplicitly handles the notion of provenance or lineage whencontent is re-used.

A significant amount of work exists in the design andspecification of data models for digital objects, and in thecreation of XML representation formats to promote the inter-operable transmission and exchange of digital objects. XMLrepresentation formats include the Metadata Encoding andTransmission Standard (METS) [37], the MPEG-21 DigitalItem Declaration Language [8], the IMS Content PackagingXML Binding [26], and the XML Formatted Data Unit (XF-DU) [45]. Many of these formats have been used to enable the

transfer of digital assets among systems. A notable exampleis the use of MPEG-21 DIDL in the transfer of the AmericanPhysical Society collections to Los Alamos National Labora-tory [6].

There is no doubt that multiple data models for complexobjects exist and will continue to be favored by different com-munities. The challenge is to develop a simple and flexibleoverlay data model that does not depend upon asset transfer,and can accommodate the essence of these different contentmodels, yet can provide a simple low-barrier entry point forinteroperability among repositories.

The Content Object Repository Discovery and Registra-tion/Resolution Architecture (CORDRA) [33] is similar toPathways in its goal to provide an open, standards-based mo-del for repository interoperability. However, CORDRA is pri-marily focused on enabling interoperability between learningobject repositories via federated registries of metadata cat-alogs. Unlike Pathways, CORDRA is specified upon a land-scape of authored metadata. The Pathways data model is builtupon a generic, graph-based abstraction that does not pre-scribe specific metadata other than small set of key attributesfor objects. While CORDRA offers support for retrieval ofcontent, Pathways addresses both retrieval and write for com-plex objects among heterogeneous repositories. Finally, whilea distributed name resolution system (e.g., the handle system)is a necessary architectural pillar in CORDRA, the Pathwaysidentifier scheme does not depend on a shared digital objectidentifier resolution service shared by distributed reposito-ries.

There are a number of other projects in the higher edu-cation community devoted to the goal of repository interop-erability within service-based architectures. Similarlymoti-vated work is being done with the EduSource CommunityLayer (ECL) [21], the DLF Asset Action Experiments [19],the Open Knowledge Initiative Open Service Interface Defi-nitions (OSIDs) [40]. These projects each specify a middle-ware service layer to enable applications to be built over het-erogeneous data sources. Pathways is distinguished in thatitis focused on defining a minimal set of read/write servicesnecessary to enable access and re-use of complex objects ina distributed, heterogeneous repository environment. Thein-tent of Pathways is to specify relatively lightweight servicesthat are easy to deploy over existing architectures. At thesame time, Pathways is motivated to provide a model thatcan record and exploit provenance relationships as contentisre-used across different services.

The challenge of service-based repository interoperabil-ity is being taken up by many other communities, often withdifferent definitions of the basic concept of a “repository”.In terms of service interfaces and APIs, repository interop-erability is being addressed both from an access perspectiveand an authoring perspective (i.e., write, put). Many effortsare positioned around a limited view of “content”, typicallyindividual content byte streams (an image, a web page, a PDFdocument), or hierarchies of content byte streams with simpledescriptors.


For example, there are many new services that positionthe web as both a readable and writable space, albeit in alimited manner. Atom [3] provides an API for an applicationlevel protocol for publishing and editing web resources. Italso provides an XML data format that can be used in boththe syndication and authoring of content. The Amazon S3 [2]web service provides an interface to support reading and writ-ing, ultimately providing an internet data storage servicethatis scalable, reliable, and fast. SRW Search/Retrieve and Up-date [41] defines a web-service interface for retrieving andupdating metadata records. Web-based Distributed Author-ing and Versioning (WebDAV) [44] enables the web serversto be exposed as writable, in addition to readable, by provid-ing an interface for uploading content using a file and direc-tory paradigm. Each of these services share with Pathwaysthe notion of simple web-based interfaces for creating andaccessing content over the web. However, a key distinctionof Pathways is its focus on complex digital objects as unitsof content as compared to single-content byte streams (e.g.,a file). Another distinction of the Pathways work is that it isprimarily intended to be an interoperability model for man-aged repositories, as distinguished from more nebulous stor-age services on the open web.

Pathways employs a graph or tree-based data model tooverlay heterogeneous data sources, which is also the basisof several other efforts. Recently, JSR 170 [30] has garneredmuch attention. This is a specification of a Java-based APIfor interacting with heterogeneous “content repositories” andrepository-like applications in a uniform manner. The basicmetaphor for interaction is that of a hierarchy of nodes withproperties, where node properties can be either simple datatypes or binary streams. JSR 170 is positioned similarly tohow JDBC is for relational databases. It is most useful fordeveloping Java-based applications with a standard interfacefor connecting with content storage components (i.e., “con-tent repositories”). Since it is not web services oriented,it isnot clear the impact it could have in providing interoperabilityamong distributed institutional repositories, and in non-Javaenvironments.

The Pathways framework is intended to be consistent withexisting and emerging web architecture principles and shouldbe easily implemented using existing web protocols and stan-dards. In considering the W3C recommendation for the Ar-chitecture of the World Wide Web [28], the Pathways frame-work has been influenced by the need for URI-based iden-tifiers for resources, the notion that resources can have oneor more “representations”, and that these representationscanbe sent or received via simple protocols. Pathways is influ-enced by work in the semantic web community, particularlythe Resource Description Framework (RDF) [32] as a mo-del for expressing resources in a graph-oriented manner asresource nodes with property and relationship arcs.

4 Digital objects, the Pathways Core data model, andsurrogates

The goal of our work on data models and interfaces is thecreation of an interoperability layer, as indicated in figure 1.We expect that this layer will overlay data models and serviceinterfaces that are distinct to individual repository implemen-tations. These repository-specific models and interfaces mayprovide functionality outside and above the models and in-terfaces described here, which are intended to represent theintersection (rather than union) of individual repositoryfea-tures.

We use the following definitions:

Digital object — In the manner of the seminal Kahn andWilensky paper [31] we use the notion of a digital objectto describe compositions of digital information. This ispurposely abstract, and is not tied to any implementationor data model. The principal aspects of a digital objectare digital data and key-metadata. Digital data can be anycombination and quantity of individual datastreams, orphysical streams of bits, and can consist of nested digitalobjects. Key-metadata, at a minimum, includes an identi-fier that is a key for service requests on the digital objectat a service point.

Data model — We describe a data model, the Pathways Core,that provides a formalization for overlaying digital ob-jects on a network of heterogeneous repositories and ser-vices. We use UML to describe this data model, but itcould be described in other formalizations such as XMLschema or OWL.

Surrogate — We use the term surrogate to indicate concreteserializations of digital objects according to our data mo-del. The purpose of this serialization is to allow exchangeof information about digital objects from one service toanother and thus propagate them through value chains.We use RDF/XML for constructing our surrogates, be-cause it is useful for representing arbitrary sub-graphs.

The primary goal of the data model, and consequently thesurrogates that represent it on the wire, is not asset or content-transfer. Rather we have designed the data model primarilyas a framework that describes the abstract structure of infor-mation objects, and the properties of that abstract structuresuch as lineage, identity, and semantics. The linkage in themodel from the abstract structure to the physical content isby-referencerather thanby-valuecontainment.

There are a number of good reasons to not mandate as-set transfer in the interoperability fabric. Full asset transferis necessary for only a subset of possible applications. Onenotable one is preservation mirroring, and thus preservationframeworks such as the Reference Model for an Open ArchivalInformation System (OAIS) [39] include the notion of infor-mation packages that imply full transfer of information units.Many other applications such as the overlay journal exampledescribed later can be accommodated without the overheadof shipping all the bits between repositories and services.Bysupporting by-reference content, the Pathways Core model


ob

tain

har

vest

pu

t

Shared Serializationof ModelModel

Shared Data

Interoperability Layer

Shared Serviceson Model

Individual Models and Interfaces

DS

pac

e

arX

iv

Fed

ora

aDO

Re

ePri

nts

m

Fig. 1. Interoperability layer over heterogeneous repositories.

enables services to selectively decide when and if to derefer-ence and pull content into the service environment. This pro-motes the notion of “service-tuned” asset transfer, where eachservice can be configured to respond to by-reference contentin a manner appropriate to the context.

In a number of cases, full asset transfer is forbidden orundesirable. For example, a rights holder may be willing toallow inclusion of their asset in another context by means ofreference through a surrogate, but may be unwilling to trans-fer the datastream itself. Or, the rights holder might allowtheasset transfer if the assets are placed in some digital rightsmanagement (DRM) wrapper.

Finally, static transfer of an asset may be undesirable inthe case of dynamic information objects, such as data setsderived from sensor networks. We foresee a number of appli-cations in the scholarly domain where such dynamic objectsare desirable, such as astronomy publications that includethelatest sky survey data.

4.1 Pathways Core data model

The Pathways Core data model is based on the notion of agraph of abstractentitieswith concretedatastreamsas leaves.In this model, a digital object is a sub-graph rooted at an en-tity. The data model is designed to meet the following re-quirements:

1. It permits recursion for arbitrary levels ofentitycontain-ment.

2. It provides an explicit link to the concrete representation,or componentdatastreams, of the digital object.

3. It includes a notion of object identity that is independentof specific identifier schemes.

4. It expresses lineage among objects, providing evidence ofderivation and workflow among objects.

5. It accommodates the linkage of semantic tags to informa-tion entities that extend the functionality of format tags tothe domain of complex, multi-part objects.

6. It allows the maintainer of the object to assert persistenceof the availability of a surrogate.

A UML structure diagram of the Pathways Core is shownin figure 2. The correspondenceof features of the model to therequirements list above is indicated by the numbered proper-ties. Each feature of the model is explained in more detail inthe following sections. Our goal has been to find the minimalset of features necessary, the core properties. Certain uses orapplications may require refinement of these relationshipsorthe addition of new relationships, and we believe that suchextensions can be added without breaking the core function-ality.

4.1.1 Entity recursion

At the root of the Pathways Core is the notion of anen-tity. As shown in figure 2, this is the attachment point of aset of properties that associate the entity with its requiredand optional features. One property ishasEntity, which ex-presses recursive containment of entities. This maps to theKahn/Wilensky [31] notion that digital objects can containnested digital objects. An example of the utility of this re-cursive relationship is modeling of an overlay journal. In thiscase, a top level entity could represent the journal itself,withsemantic, persistence, and identity attributes that correspondto the journal. A journal “contains” issues, which themselvesmay be entities, with associated properties. This recursionnaturally continues, with issues “containing” articles.

As indicated in figure 3, an entity is an abstract concept,distinct from concrete datastreams described in the next sec-tion. This abstract/concrete distinction is fundamental to the


4 1

563

2

Fig. 2. UML diagram of the Pathways Core data model with parts that fulfill particular requirements numbered.

model — removing assertions of identity, persistence, lin-eage, and semantics from individual physical manifestationsof intellectual objects. This separation of abstract and con-crete properties (or attributes) is similar to that in the FRBRmodel [38].

4.1.2 Concrete representation

As indicated in figure 2, an entity can have severalhasDatas-tream properties. The motivation for this is well-establishedin compound document formats such as METS, MPEG-21DIDL, and Fedora FOXML, which allow a single object tohave multiple datastreams with different media types (e.g.,the availability of a scholarly paper in PDF, Word and TeX).

A datastream has both aformat (e.g. a format registeredin GDFR [1] or PRONOM [12]) and alocation, a URL torequest a dissemination of the datastream. The datastream as-sociation is intentionallyby-referencerather thanby-value, toavoid mandating asset transfer for the reasons given earlier.

A typical digital object will contain one or more datas-treams. The digital object represented in figure 3 comprisesa top-level entity E1, with sub-entities E2 and E3. The entityE2 has two datastreams, D1 and D2, which might be alternateexpressions of the entity E2. As semantic assertions appearonly at the level of the entity, both D1 and D2 are assumed tohave any semantics expressed for E2. The entity E3 has justa single datastreams D3 and may thus be used to express se-mantics that apply just to D3, as separate from the semanticof E1. For example, E1 might be a “journal issue” with twoarticles E2 and E3, and the article E2 happens to be availablein both PDF and Word formats.

Abstract, recursive(entities)

(datastreams)Concrete

E

E E

D D D

2 3

1

1 2 3

Fig. 3. Entity recursion and concrete representation.

4.1.3 Identity

We recognize the reality that one identifier technology willnever dominate and have thus incorporated two notions ofidentity. First, thehasIdentifier property allows expression ofURIs associated with a digital object, a DOI for example.Second, thehasProviderInfo property introduces a relativelysimple repository-centric identifier paradigm which permitsprecise identification of digital objects in the particularrepos-itory, facilitating re-use and accurate provenance records. Thisparadigm is not intended to replace existing identifier mech-anisms or to interfere with future technologies in this area.Rather it is intended as a future-proof long-term scheme thatcan co-exist with other identifier mechanisms.

ThehasProviderInfo property has three components:


provider — The identity of the repository (i.e.; the servicepoint providing access and ancillary services on the digi-tal object). We assume that the participants in this infras-tructure — institutional repositories and the like — havea commitment to of their repository identity. Indirectionvia a repository identifier presumes some technology forregistering repositories and resolution to the location oftheir service interfaces. Registries are discussed later,insection 6.

preferredIdentifier — The identity of the entity within therepository. This serves as the key for making service re-quests upon the digital object at the service point definedby the repository (provider) identity. As explained later inthis paper, the basic repository service is a request for asurrogate of the digital object. We expect, however, that ahost of other services will evolve. We emphasize that thesyntax, semantics, and resolution of the identity of the ob-ject is local to the individual repository, rather than beingglobal as in more ambitious identifier schemes.

versionKey — This is a means of parameterizing a servicerequest on an object according to version semantics. Theintention here is to provide an opaque hook into individ-ual repository versioning implementations, rather than as-suming or imposing some universal cross-repository ver-sion schema.

Two copies of the same object in two different reposito-ries may have the same identifier expressed via thehasIden-tifier property. However, they will have differentproviderInfobecause they are available from different repositories.

4.1.4 Lineage

Isaac Newton wrote “If I have seen further it is by standing onthe shoulders of Giants” [27]. In the face of massive changesin scholarship since Newton’s time, one constant is the evolu-tion of scholarship, whereby new results are built on the inno-vations of earlier scholars. We believe therefore that the inter-operability infrastructure must support the notion of lineage,natively linking entities to other entities from which theyarederived.

As shown in figure 2, entities in the model can link toother entities through thehasLineage relationship. This link-age leverages thehasProviderInfo identity of the entity (orentities) from which the new entity derives, thus allowing anentity to express its derivation from another entity and specif-ically state both the repository origin of the source objectandits version semantics. Furthermore, since the model is recur-sive, entities can contain entities and the derivation of con-tained parts of objects can be similarly expressed.

This lineage capability is illustrated in figure 4. The en-tity labeled E1 is derived from that labeled E2. For example,E1 may be translation of E2 into a new language. E2, as illus-trated, contains sub-entities with respective derivations fromE3 and E5. For example, E2 may be an issue of an overlayjournal with articles that are edited versions of the preprintsE3 and E5, where E5 is itself a sub-entity of the preprint seriesE4. These cases illustrate re-use at different granularities.

E2

E4

E3

1E

hasLineage

E5

hasLineage

hasLineage

Fig. 4. Relating entities by lineage.

The result of these lineage links among entities at the in-teroperability layer is a web ofevidential citation. This graphindicates both the workflow origins of an information object— the partial ordering of information objects from which itderives — and also the curatorial heritage of the object —the repositories and services responsible for its legacy. Thisnew, uniquely networked and digital form of citation pro-vides a finer level of identification than convention biblio-graphic citation. In the case that a repository has an objectderived from an object in another repository, there is a localchoice as to whether the same object identifier is used or anew one generated. This choice would be presumably be in-fluenced by repository policy, community agreements and bythe kind of value chain implemented. In either case, two ob-servations can be made. First, theproviderInfo includes theprovider which make the complete identification unique anddistinguished the objects. Second, thehasLineage property ofthe derived entity provides and unambiguous link back to theoriginal entity.

Both cases are illustrated in figure 5, which shows theentity labeled E1 taking part in value chains that result innew entities, E2 and E3, in different repositories. In all casesthe entities are uniquely identified by theproviderInfo, eventhough E2 has the samepreferredIdentifier as E1. Also, bothE2 and E3 indicate their lineage from E1 with theproviderInfoextracted from E1.


E3

1E

E2

value chain

value chain

providerInfo={repo1,id1}

repository 2

repository 3


hasLineage

hasLineage

repository 1

{repo1,id1}

{repo1,id1}


Fig. 5. Identification and lineage of derived entities in differentrepositories.The shorthand{provider,preferredIdentifier} is used forproviderInfo, andversionKey is omitted.

We imagine that thehasLineage relationship is a super-class of the many types of inter-entity derivation relationshipsthat could be expressed. Thus, future evolution of the infras-tructure might refine this relationship.

4.1.5 Semantics

We envision applications that need to know about the “se-mantic” composition of digital objects in addition to know-ing the media-format types of the individual datastreams. Acomplex digital object might represent a “dissertation” ora“journal article”, each of which might have datastreams thatare images, data sets, spreadsheets, or text in various formats.

One particularly interesting application is service match-ing. The utility of automated match of preservation servicesto information objects has been demonstrated by the PANICwork [24]. While PANIC demonstrates the utility of automa-tion for individual datastreams based on media type, we wouldlike to enable similar services over complex objects and basedon intellectual content types.

The Pathways Core therefore associates thehasSeman-tic property with each entity. The target of this property isa URI specifying the semantic typing of the entity. Admit-tedly, no universal semantic registry exists at this time. How-ever, the property could be exploited by individual commu-nities that develop local schemes, and later extended to morewidespread use.

4.1.6 Persistence

The history of persistence of information artifacts, especiallydigital objects, is riddled with examples of the gaps betweenintention, expectation, and reality. Despite our best intentionsto provide storage of and access to digital information “for-

ever” (or even a few months!), the realities of hardware fail-ures, format rot, and mismanagement frequently interfere.Thismust be considered in the design of any information interop-erability framework.

Therefore, we have taken a purposely modest approachto persistence that is oriented towards surrogates and ser-vices over surrogates, rather than towards digital objects. ThehasProviderPersistence property associated with an entity isa slot in which the repository can declare, by means of aURI, the longevity of its commitment towards providing ser-vices over the respective entity. The repository making thiscommitment is identified as theprovider in the entity’shas-ProviderInfo property. Since the core service in the interoper-ability fabric is the dissemination of a surrogate for the entity,hasProviderPersistence indicates the level of commitment ofthe respective repository to provide access to a surrogate forthe entity. While there is clearly scope for subtle refinementof persistence declaration, at this point we propose a set ofjust two persistence declarations:

– The entity is transient and the repository makes no com-mitment to providing services for it over time.

– The entity is persistent and the repository intends to re-spond to service requests for it over time.

4.2 Surrogates and serialization

An individual instance of the Pathways Core data model, arepresentation of an individual digital object, is packaged andtransmitted as a surrogate: a serialization that conforms to thedata model. We note possible terminological confusion herebut have not found a word with less baggage. By surrogatewe mean a serialization that substitutes for the digital objectand must therefore reveal all essential characteristics, and isthus distinguished from some arbitrary representation. Theobtain and harvest interfaces (described in sections 5.1 andsection 5.2) provide the means for clients to request a surro-gate. Similarly, a put request (described in section 5.3), whichrequests deposit of a digital object in a repository, contains asurrogate as a payload.

We have found that RDF [32] is a useful tool for mod-eling the graph-like structure of information in the PathwaysCore. We have done this by associating URIs with the prop-erties in the Pathways Core and similarly associating URIswith a number of controlled vocabularies such as persistence,formats, and semantics that are the values of Pathways Coreproperties. RDF modeling naturally led to the adoption of theRDF/XML syntax [4] as the serialization syntax for PathwaysCore surrogates. A fragment of an example of this syntax isshown in figure 6.

5 Repository interfaces: obtain, harvest and put

We have described the Pathways Core data model and a sur-rogate that serializes the model. For these to enable repositoryinteroperability, a set of essential services are required. Three


<?xml version=”1.0” encoding=”UTF-8”?>

<rdf:RDF xmlns:core=”info:pathways/core#” xmlns:rdf=”http://www.w3.org/1999/02/22-rdf-syntax-ns#”><core:entity rdf:about=”info:pathways/entity/info%3Asid%2Flibrary.lanl.gov%3Apathways/info%3Adoi%2F10.1016%2Fj.dyepig.2004.12.010”>

<core:hasSemantic rdf:resource=”info:pathways/semantic/journal-article”/><core:hasIdentifier>info:doi/10.1016/j.dyepig.2004.12.010</core:hasIdentifier><core:hasProviderPersistence rdf:resource=”info:pathways/persistence/persistent”/><core:hasProviderInfo>

<core:providerInfo>

<core:preferredIdentifier>info:doi/10.1016/j.dyepig.2004.12.010</core:preferredIdentifier><core:provider>info:sid/library.lanl.gov:pathways</core:provider>

</core:providerInfo>

</core:hasProviderInfo>

<core:hasEntity>

<core:entityrdf:about=”info:pathways/entity/info...(shortened)...lanl-repo%2Fssm%2Fdoi-10.1016%2Fj.dyepig.2004.12.010”><core:hasSemantic rdf:resource=”info:pathways/semantic/bibliographic-citation”/><core:hasIdentifier>info:lanl-repo/ssm/doi-10.1016/j.dyepig.2004.12.010</core:hasIdentifier><core:hasProviderPersistence rdf:resource=”info:pathways/persistence/persistent”/><core:hasProviderInfo>

<core:providerInfo>

<core:preferredIdentifier>info:lanl-repo/ssm/doi-10.1016/j.dyepig.2004.12.010</core:preferredIdentifier><core:provider>info:sid/library.lanl.gov:pathways</core:provider>

</core:providerInfo>

</core:hasProviderInfo>

<core:hasDatastream>

<core:datastream>

<core:hasFormat rdf:resource=”info:pathways/fmt/pronom/1000”/><core:hasLocation>http://purl.lanl.gov/demo/adore-arcfile/00e682eb-a87eb27b0c79</core:hasLocation>

</core:datastream>

</core:hasDatastream>

...

Fig. 6.Excerpt from a sample surrogate that serializes the Pathways Core in RDF/XML.

repository interfaces with the following functions fulfillthisneed and are described below:

– An obtain interfacewhich, in its most basic implemen-tation, allows the request of a surrogate for an identifieddigital object from a repository.

– A harvest interfacethat exposes surrogates for incremen-tal collection or harvesting.

– A put interfacethat supports submission of one or moresurrogates into the repository, thereby facilitating the ad-dition of digital objects to the collection of the repository.

5.1 Obtain interface

Pathways defines an obtain interface that supports the requestof services pertaining to an identified digital object withina repository. The simplest implementation of the obtain in-terface allows requesting a surrogate for an identified digitalobject from a repository. Such an interface can be regarded asan identifier-to-surrogate resolution mechanism that resolvesthe preferredIdentifier of a digital object into a surrogate ofthat digital object.

The information needed to construct an obtain request isrecorded in theproviderInfo property of the surrogate itself.The providerInfo is a triple consisting of the identifier of therepository that exposes the surrogate, thepreferredIdentifierof the digital object and an optionalversionKey. By using theidentifier of the repository, the location of the obtain inter-face of the identified repository can be found by a look-up ina service registry (see section 6). Once known, one can usethe preferredIdentifier of the digital object (and the optional

versionKey) to obtain a surrogate using the repository’s ob-tain interface.

Higher levels of the obtain functionality have been ex-plored theoretically by Bekaert [5]. Straightforward exten-sion of the obtain concept allows the request of any supportedservice pertaining to an identified digital object. This includesthe request of services pertaining to datastreams of the digitalobject. Possible examples are requests to obtain a surrogateof an identified article, requests to obtain a PDF datastreamof that same article, requests to obtain an audio version ofthat article by applying a text-to-speech service upon the PDFdatastream, and so forth. Such services can be considered asuperclass of the basic obtain functionality described above,and do not have to be supported by all repositories. Rather,such services would typically be supported by autonomousservice applications that overlay one or more repositoriesanduse surrogates that are obtained through interaction with thecore obtain interface of the underlying repositories.

One technology that lends itself for implementing theseobtain interfaces is the OpenURL Framework for Context-Sensitive Services [46]. The OpenURL standard originatesfrom the scholarly information community where it was pro-posed a solution to the provision of context-sensitive refer-ence links for scholarly works such as journal articles andbooks [14]. The initial standard was generalized to createthe current NISO OpenURL Framework which describes anetworked service environment, in which packages of con-text information (ContextObjects) are used to request context-sensitive services pertaining to a referenced resource. EachContextObject contains various types of information that are


needed to provide context-sensitive services. Such informa-tion may include the identifier of the referenced resource, theReferent, the type of service that needs to be applied upon theReferent (theServiceType), the network context in which theresource is referenced, and the context in which the servicerequest takes place.

In this way, the core obtain interface can be implementedas an OpenURL Framework Application. TheContextObjectused in the obtain request conveys the following information:

A Referent — The digital object for which an obtain requestis formulated. TheReferent is described by means of itspreferredIdentifier.

A ServiceType — The service that generates a surrogate ofthe identified digital object.

Beyond meeting our basic requirements, the OpenURLFramework has the following attractive properties:

– it makes a clear distinction between the abstract defini-tion of concepts and their concrete representation and theprotocol by which such representations are transported. AContextObject may be represented in many different for-mats and transported using many different transport pro-tocols, as technologies evolve. Yet, the concepts underly-ing the OpenURL Framework persist over time.

– it does not make any presumptions about the identifiernamespace used for the identification of digital objects (orconstituents thereof), and hence, provides for an obtaininterface that can be implemented across a broad varietyof repository systems.

– it allows information about the context in which the ob-tain request took place to be conveyed. This informationmay allow delivery of context-sensitive service requests.Of particular interest is information about the agent re-questing the obtain service (theRequester). This infor-mation could convey identity, and this would allow re-sponding differently to the same service request depend-ing on whether the requesting agent is a human or ma-chine. Similarly, different humans could receive differentdisseminations based on recorded preferences or accessrights. The OpenURL Framework is purposely genericand extensible, and would also support to convey the char-acteristics of a user’s terminal, the user’s network con-text, and/or the user’s location via theRequester entity.Though, this type of context-related tuning may not beimportant when requesting surrogates of digital objects,it may prove to be essential when requesting rich servicespertaining to datastreams.

5.2 Harvest interface

A harvest interface allows collecting or harvesting of surro-gates of digital objects. In addition to the facility to harvestall the surrogates exposed by a repository, we believe it isnecessary to provide a facility allowing some forms of selec-tive harvesting. The simplest, and perhaps most useful, formof selective harvesting is to allow downstream applications

to harvest surrogates only for those digital objects that werecreated or modified after a given date. This echoes the OpenArchives Initiative Protocol for Metadata Harvesting (OAI-PMH) [34] with the same motivation: downstream applica-tions may need an up-to-date copy of all the surrogates froma repository in order to provide some service, and incremen-tally harvesting surrogates of newly added or modified digitalobjects is an efficient way to do this.

A harvest interface could be implemented using varioustechnologies such as the OAI-PMH, RSS or Atom, or witha subset of more complex technologies such as SRU/SRW.The OAI-PMH is a well established harvesting technologywithin the digital library community and allows aggregationof metadata from compliant repositories using a datestamp-based harvesting strategy. Although the OAI-PMH was firstconceived for metadata harvesting, it can be used to transferany metadata or data format, including complex-object for-mats, expressed in XML according to an XML Schema [17].The OAI-PMH is thus capable of providing the harvest func-tionality, and the ability to leverage existing OAI-PMH im-plementations is a significant benefit.

To support the harvest interface, the underlying OAI-PMHinterface must follow these conventions:

– Each OAI-PMH item identifier must match thepreferredI-dentifier of the Pathways Core digital object. This avoidsthe need for clients to record relationships between OAI-PMH identifiers and digital object identifiers which canbecome complex in various aggregation scenarios.

– The OAI-PMH datestamps must be the datetime of cre-ation or modification of the digital objects as discussedin [17].

– It must provide a metadata format for surrogates as de-scribed in section 4.2.

It is worth noting one possible issue. The OAI-PMH spec-ification is bound to the HTTP protocol and the XML syntaxfor transporting and serializing the harvested records. Whilethis approach proves to be satisfactory in the current techno-logical environment, it may prove to be inadequate as tech-nologies evolve. If this work were to be tightly bound withthe OAI-PMH then an abstract model would need to be cre-ated. However, if OAI-PMH is used simply as one possibletechnology to implement harvest functionality then it couldlater be replaced.

5.3 Put interface

Pathways defines a put interface to promote interoperable trans-mission of surrogates to one or more target digital reposito-ries. As with the obtain interface, digital objects are expressedas surrogates. At the interface level, put operations are sim-ple and unassuming. They can be understood as arequest fordepositof a digital object. This distinguishes the put inter-face from similar operations found in other push-oriented ser-vices whose purpose is to facilitate upload of binary contentstreams, or transfer of assets using community-specific con-tent packages such as METS, IMS-CP, or MPEG-21 DIDL.


The put interface does not presuppose that target repos-itories conform to any particular underlying storage scheme(e.g., hierarchal file system, a web server with directories, re-lational database, etc.). Additionally, the put interfaceis neu-tral about the underlying data model of target repositories.The only requirement is that digital objects be representedassurrogates expressed in the Pathways Core, which is specif-ically designed to transcend the particulars of heterogeneousdata models. The graph-based nature of this model providesthe flexibility to support the submission of both simple andcomplex digital objects.

The put interface, in combination with a surrogate, is in-tended as a means for transmitting just enough informationto enable a receiving repository to make decisions on how toprocess a surrogate — without anticipating or assuming anunderlying repository’s requirements for ingest. Datastreamcontent is expressed by-reference in the surrogate, via thelo-cation property. With this constraint, a surrogate represents a“shallow copy” of a complex object since there is no trans-mission of raw content within the surrogate. As discussedearlier (section 4), this constraint is motivated by the needfor simplicity, and the desire to keep authentication and au-thorization concerns out of the functional definition of theput interface. Authentication, authorization and policy are ex-pected to be handled at service implementation layers.

Unlike the obtain and harvest interfaces, no protocol ortechnology stands out as an obvious implementation optionfor the put interface.

6 Registries

The proposed interoperability framework requires at leastonesupporting infrastructure component: a service registry to as-sociate providers with services. Additional format and se-mantic registries would significantly enrich the environment.No particular technical implementation is implied by the useof the term registry, but rather the general ability to record,share and retrieve terms of a controlled vocabulary alongsidetheir associated properties.

6.1 Service registry

A service registry is fundamental to the framework, as it fa-cilitates locating the core service interfaces of participatingrepositories. This registry has the identifier of a repository(provider from providerInfo in the Pathways Core) as its pri-mary key, and it minimally stores the actual network loca-tion of the obtain, harvest and put services, where supported.Thus, given a surrogate withproviderInfo (provider, preferre-dIdentifier, versionKey), it is possible for an application to useprovider as a look-up key in the service registry to retrievethelocation of the core service interfaces for the repository iden-tified by provider. Once this information is available, actualservice requests can be issued against those interfaces. For

example, in order to retrieve an up-to-date surrogate, the ap-plication can issue an obtain request using thepreferredIden-tifier and optionalversionKey of theproviderInfo as shown infigure 7. The use of a registry permits repository interfacestochange their network location, allows different services (in-cluding those not yet imagined) to be associated with a repos-itory, and makes the combination of repositories trivial.

It should be noted that, in contrast to other repositoryfederation approaches such as CORDRA [33], ADL-R [29],aDORe [15], and the Chinese Digital Museum Project [10],the proposed framework does not require a registry of all dig-ital objects in all contributing repositories thereby allowinglocation of a digital object given its identifier. In the proposedframework, a surrogate carries its self-identifying provider-Info, which, through the intermediation of the service reg-istry, allows location of the service interfaces of the originat-ing repository. This approach alleviates two major drawbacksinherent in the use of digital object registries. First, given anidentifier, how does one know that it is an identifier of a dig-ital object from repositories contributing to the federation,and hence that a look-up in the federation’s object registryismeaningful? It seems that this question can only be answeredif all repositories in the federation share a common, recog-nizable identifier scheme. This is a significant requirement,especially in light of the considerations regarding the long-term horizon of desired solutions. Second, the scale of objectregistries is several orders of magnitude larger than that of theproposed service registry because the latter only containsanentry per repository, not per digital object. The repercussionsfor operating the registry infrastructure are obvious.

6.2 Format and semantic registries

While the service registry is essential for the operation oftheproposed framework, two other registries, while less funda-mental, would significantly enrich the functionality of thean-ticipated environment.

First, it is now widely recognized that repositories, espe-cially in preservation environments, must support more finelygrained identification of digital media formats than is pro-vided by MIME types. A format registry that has the iden-tifier of a digital format as its primary key and that recordsvarious properties of the format have been proposed by boththe PRONOM [12] and GDFR [1] efforts. Format identifierswould be used for the format property available at the datas-tream level of surrogates. Such a fine level of format identifi-cation would, for example, enable rich format-based servicematching as explored in the PANIC [24] and aDORe [7] ef-forts.

Second, automated object use and re-use would be en-hanced by identification of the intellectual content type ofmaterials. A semantic registry that has the identifier of a schol-arly content type as its primary key and that records vari-ous properties of the content type would support this. To fa-cilitate syndicating, aggregating, post-processing and multi-purposing magazine, news, catalog, book, and mainstreamjournal content, the PRISM effort [25] has created such a


pu

t

har

vest

ob

tain

Service

Ser

vice

Reg

istr

y

surrogate

1) Service extractsproviderInfo from

obtain interface

3) Service obtains up−to−date surrogate

provider obtain harvest put

put1

put2harvest2

harvest1obtain1

obtain2

obtain1

put1 harvest1

repo1

2) Lookup for repo1

from repo1

repo1

repo2

Fig. 7.Use of the service registry.

vocabulary, but for materials typically used in a scholarlycontext it is lacking, making the semantics registry proba-bly more critical to pursue than the format registry for whichthe MIME types can serve as a pragmatic stand-in. Semanticidentifiers would be used for the semantic property availableat the entity level of surrogates. Returning to the chemicalsearch engine scenario, appropriate semantic identification ofan entity would allow an agent to recognize it as a machinereadable chemical formula, and thus choose to ingest the as-sociated datastreams, the format of which can also be pre-cisely described.

7 Experiments

To test the ideas presented above, we created obtain, har-vest and put services to disseminate and ingest surrogatesfrom and to several different repository architectures: Fedora,aDORe, DSpace and arXiv. We then used these interfaces tosupport the assembly of a number of articles from differentrepositories into a new issue of a hypothetical overlay jour-nal. Instead of relying just on the user interfaces of the partic-ipating repositories, we also created a resource-centric searchservice using the harvesting infrastructure provided by theharvest interfaces. To further enhance the demonstration,wecombined these techniques with Live Clipboard [35] technol-ogy to allow surrogates to be moved among repositories viathe usualdrag-and-dropmetaphor. We first describe the twoparts of this experiment, and then discuss implementation is-sues and experiences.

7.1 Harvesting journal articles to produce aresource-centric search service

A number of projects have attempted to use OAI-PMH har-vested metadata for the creation of resource-centric or full-text discovery services. The principal problem is that resourcescannot be unambiguously located from the simple DublinCore metadata exposed by most OAI compliant repositories.This issue was discussed in detail in [17], where the use ofcomplex object formats was proposed as a solution. Whileskeletal compared with formats such as METS and MPEG-21 DIDL, the surrogates proposed here also meet the require-ments for resource harvesting. We implemented a search ser-vice based on the Nutch [36] crawler and search service. In-stead of simply doing a web crawl, surrogates were harvestedfrom the participating repositories using the harvest inter-face. Each surrogate was then introspected upon to selectonly those with the semantics “journal-article” (we agreedon a small ontology for these experiments). All the appropri-ate surrogates were examined to extract format and locationinformation to dereference the datastreams. The datastreamswere then fetched and indexed while retaining their associa-tion with the surrogate. This process is illustrated in figure 8.

In addition to the usual links back to the source reposi-tory and content excerpt, the search results display was aug-mented with a Live Clipboard icon allowing the surrogate tobe copied into the copy/paste buffer on the user’s computer,and thus easily passed to other applications as described be-low.

7.2 Creation of a new issue of an overlay journal

The scenario we have referred to most frequently is the com-position of a new issue of an overlay journal from articles in


har

vest

datastream

datastream

har

vest

ob

tain

har

vest

inte

rfac

eu

ser

aDORe

DSpace

Nutch

Aggregation &Search Service

Fig. 8.Use of Nutch to create a resource-centric search service over repositories supporting the Pathways harvest interface.

different repositories. When combined with the search ser-vice just described, this scenario demonstrates the use of allthree repository interfaces in a realistic scholarly value-chain.This scenario revolves around the editor of the overlay jour-nal, “Ed”. The key data flows as Ed interacts with one sourcerepository are shown in figure 9, and the complete sequenceof actions required to create the new overlay journal issue aredescribed below.

1. Select:Ed applies whatever selection and review policiesthe overlay journal uses to decide which articles should beincluded in the new issue. Ed selects three articles; oneeach from arXiv, from an aDORe based repository, andfrom a DSpace repository.

2. Obtain: Consider first the selection of an article fromarXiv. Ed navigates to the normal splash page for this arti-cle in whichever way is convenient, perhaps from Google,or from arXiv’s own interface. The splash page not onlydisplays the usual metadata, links to associate resourcesand links to the full-text, but also a Live Clipboard iconas shown in figure 10. By clicking on this icon, the LiveClipboard JavaScript uses arXiv’s obtain interface to get asurrogate for the article which is stored in the copy/pastebuffer of Ed’s computer.

3. Compose:Ed then goes to the editorial web-interface forthe overlay journal and pastes the surrogate via the LiveClipboard JavaScript on that page. Behind the scenes, thesurrogate is put into the Fedora repository hosting thejournal. Here it is a matter of local policy whether theingest mechanism simply stores the surrogate with ref-erences to included entities and datastreams, or whetherthese are dereferenced and also ingested. For this demon-

stration we chose to ingest only the structural information— the entities — which simulates a “pure” overlay whichsimply links to articles in trusted repositories (perhapswith cyptographic signatures to guarantee that the origi-nal has not been altered). It would also be possible for theingest system to implement a deep-copy and duplicate allthe datastreams of a digital object. Note that in a real sys-tem Ed would have to authenticate with the repository forthe overlay journal in order to be granted the privileges toput new content into the journal, presumably any attemptto put content by a non-authenticated and or unprivilegeduser would be denied.

4. Complete composition:A similar process is repeated forarticles from DSpace and from aDORe. Here Ed uses thesearch service described in 7.1. The search results showa Live Clipboard icon with each result. By clicking onthis icon, the Live Clipboard JavaScript uses the searchservice’s obtain interface to get a cached surrogate for thearticle which is stored in the copy/paste buffer of Ed’scomputer. The overlay journal issue then has three articlesqueued.

5. Submit: When Ed is happy that all articles for the newissue are ready, the issue can be created as an entity in itsown right by clicking the “Submit Issue” button. Whenthis is complete, a surrogate for the new issue is availablefrom the obtain and harvest interfaces of the overlay jour-nal repository and may be used by all the same servicesthat interoperate with the underlying repositories.

6. Visualize: To illustrate how other services can work withinthis framework, and to allow easy visualization of surro-gates, we created an additional OpenURL-based serviceto visualize the surrogate graph using WebDot [22]. Ex-


ob

tain

use

rin

terf

ace

inte

rfac

eu

ser

pu

t

repositoryOverlay journal

Source repository

View page Submit page

Live Clipboard pasteLive Clipboard copy

Fig. 9.Addition of an article from a participating repository to anoverlay journal using Live Clipboard technology to copy a surrogate.

ample output for an arXiv article containing an additionaldata datastream is shown in figure 11. The OpenURL re-quest simply includes the providerInfo of the surrogatewhich is enough to enable the surrogate to be obtained,rendered as an image of a graph with links to sub-entities,datastreams and registry entries for format and semanticURIs.

Though only a demonstration, the process of compiling anew issue for an overlay journal described above uses manyinteroperability features provided by the Pathways frameworkwhich are simply not available in existing systems. By imple-menting this over several of the most popular repository tech-nologies we have demonstrated that this technology couldreadily be deployed.

7.3 Implementation of the obtain and harvest services

The obtain interface is the simplest service and was the firstthat we implemented for each repository. By choosing to baseit on OpenURL we were able to leverage existing OpenURLimplementations for some of the repositories, simply addinganother service identifier (svc id) for the obtain service. Inimplementing and obtain interface, one must work with thenative data model of the underlying repository. The key deci-sions arise in translation of digital objects from their under-lying representations to the Pathways Core model. Since thePathways Core model is flexible, this can be done in differentways, depending on how much visibility vs. encapsulation ofcomponent parts is desired. Parts that are made available forre-use should be modeled as entities with associatedprovider-Info.

All of the repositories used in this experiment alreadysupported the OAI-PMH so the implementation of the harvestinterface was simply a matter of adding another “metadata”format for the Pathways Core surrogate to the existing OAI-PMH interfaces. It is a requirement of the OAI-PMH thatall metadata formats be expressed in XML according to anXML Schema. Thus, we created surrogates using RDF/XML

Live Clipboard"scissors" iconplus graph and RDF links

Fig. 10. Screenshot of the arXiv wrapper page augmented with Live Clip-board links.

according to the W3C RDF/XML Schema [9]. Additionally,we agreed that, for convenience, all repositories would useacommonmetadataPrefix=pwc.rdf for this metadata format.


Fig. 11.Screenshot of the graph visualization showing an article from arXivthat contains a PDF file and a dataset.

7.4 Implementation of the put service

While the put interface is agnostic to the particulars of un-derlying repository technology and models, any concrete im-plementation of a put service must be attentive to the specificcapabilities and limitations of the underlying repositoryar-chitecture. There are many questions that arise pertainingtohow a put service interprets a surrogate and the assumptionsthe service makes in interacting with a particular underlyingrepository.

To support the overlay journal experiment, a put servicewas developed to interact with a Fedora repository. As no ex-isting protocol already provided the required functionality, anew REST-based service for Fedora was created for our ex-periments. The flexibility of Fedora made it well suited foraccepting and ingesting both simple surrogates (single en-tity) and complex surrogates (a graph of entities). However,this flexibility provoked the realization that there are a num-ber issues to be considered in implementing an effective putservice, which we detail in the following sections.

7.4.1 Identifiers and lineage

The put interface, itself, imposes no requirement on a receiv-ing repository in terms of how it should deal with identifiers.However, whether the repository assigns new identifiers forsurrogates ingested or not, it is important that there is a wayto later determine that an entity is a new instance of an exist-ing entity. The means that theproviderInfo of the original sur-rogate should be retained in thehasLineage property of thenew entity as described in section 4.1.4. Thus theprovider-Info provides the basis for “a chain of lineage” across multi-ple distributed repositories where each repository representsthe same entity or entities in different contexts.

7.4.2 Ingesting hierarchies or networks of objects

There are many cases when a put service will receive a surro-gate that models a hierarchy or graph of related entities. This

presents a challenge in terms of determining an appropriateingest policy for how surrogates will be processed, and whatkinds of digital objects will ultimately be created in a receiv-ing repository.

When surrogates contain a hierarchy of entities, some as-sumptions must be made as to the nature of the relationshipsof the entities in the hierarchy. Do parent-child relationshipsof a hierarchy imply a part-whole composition? Is the pres-ence or absence of each part essential to the integrity of thewhole? Alternatively, is the hierarchy to be interpreted asalooser containment relationship, where the integrity of thewhole is not compromised if its parts are disassociated?

In our experiment, the put service assumed that any en-tity within a surrogate that containedproviderInfo should bemanaged as its own digital object within the target Fedorarepository. Thus, in the case of the journal overlay example,all journal, issue, and article entities were to be represented asseparate Fedora digital objects with appropriate relationshipsasserted among them. Furthermore, any sub-entities of articleentities withproviderInfo were also represented as digital ob-jects in their own right. In the experiments, there were articleentities that were comprised of both a document and datasets,and each of these was represented as a separate digital object.

The end result of creating a journal issue via the put in-terface, was the creation of a graph of related digital objectsin the target Fedora repository. From a management perspec-tive, this modular and atomic arrangement can enable flexiblemanagement of objects. For example, it is easier to discoverand do something with all dataset objects than it would be tofind all types of objects that may encapsulate datasets. Froman access standpoint, each component is registered as a dig-ital object with its own public identifier and is available forre-use. This approach facilitated the ability to obtain theen-tire journal, or any sub-part down the hierarchy via the obtainservice is a simple and generic manner. It was not necessaryto create special services to discover and extract entitiesthatwere encapsulated within other objects. On the other hand,additional processes would be required to implement facil-ities that depend on handling all the constituent parts of agiven digital object.

8 Future work

Much of the intellectual effort in this work has been to parethe Pathways Core data model down to its essential compo-nents. Having successfully carried out some initial experi-ments we intend to explore other scenarios, including thosedescribed in the introduction, to see where additional rich-ness is required. We envision extensions or refinements to re-lationships in the model. One example is the notion of entitycontainment to include more specific semantics: distinctionof part/whole vs. alternative or auxiliary; indication of equiv-alence; and the notion of ordering or not. In the overlay jour-nal example, how and where should the order of articles beexpressed in a surrogate for the journal issue?


The hooks from URI expression of semantic informationand format identifiers open up tantalizing ties to current ser-vice matching work, the semantic web, and to ontology-basedreasoning. It seems likely that PRONOM and GDFR formatregistries will coexist and be used by different segments ofthe scholarly community. How can we use ontologies in sys-tems that will “understand” the equivalences and differencesbetween these specifications? How can notions of generaliza-tion be applied to semantic information created in differentcontexts, and how can we service match on compositionalsemantics? For example, if an object contains a set of JPEG-2000 entities, should it be treated as a scanned book or aphoto album?

If we imagine a landscape of widespread re-use of digi-tal objects, there will undoubtedly be many copies and ver-sions in different repositories and this provokes a number ofquestions. When designing a put interface, how can one un-derstand if a surrogate duplicates an existing digital object?When a duplicate is discovered as part of a put request, shouldthe current object be replaced? Or should multiple versionsofthe digital object be managed? These are repository-specificdecisions, but whatever is decided may have significant im-plications in collaborative scholarly workflows. How shouldsurrogates be validated?

The experiments described here have been performed overrepositories that have primarily document content. The Path-ways framework was conceived with a rich environment ofdocuments, data and other media-types in mind. Future workwill involve collaboration with other repositories that includesignificant data-repositories and other repository architectures.

9 Conclusions

Our experiments successfully demonstrated the ability to movesurrogates of digital objects among repositories, and to re-usethem in new contexts. The proposed interoperability frame-work allowed us to show how the basic workflow necessaryto create a new issue of an overlay journal could be supportedacross heterogeneous repositories. The simplicity and gen-erality of the Pathways Core data model allowed its use tocreate surrogates for digital objects held in Fedora, aDORe,DSpace and arXiv repositories, each with significantly differ-ent internal data models and architectures. Furthermore, byleveraging existing implementations of OpenURL resolversand OAI-PMH interfaces for the repositories, it was remark-ably easy to provide the dissemination (obtain) and harvestinterfaces necessary for each repository to participate. The in-gest (put) interface was implemented only for a Fedora repos-itory and involved considerably more design decisions, manyof which require further investigation to determine best prac-tices.

These results are serving as the basis for further experi-ments with even more heterogeneous repository architectures,to include data repositories in particular. These experimentswill implement other important value chains that are neces-

sary to move toward the goal of the creation of a global schol-arly communication system.

Acknowledgements

We thank Rob Tansley for implementing the Pathways Coredata model and associated dissemination services in DSpace;Chris Wilper for work on Fedora, including the put interface;and Lyudmila Balakireva for work on the WebDot service,and Zhiwu Xie for work on the the Live Clipboard infrastruc-ture. This work was supported by NSF award number IIS-0430906 (Pathways).

References

1. Stephen L. Abrams and David Seaman. Towards aglobal digital format registry. In 69th IFLA (Inter-national Federation of Library Associations and Insti-tutions) General Conference and Council, Berlin, Ger-many, August 2003. http://www.ifla.org/IV/ifla69/papers/128e-Abrams Seaman.pdf [Accessed 29 June 2006].

2. Amazon S3 - Simple Storage Service.http://aws.amazon.com/s3 [Accessed 28 June 2006].

3. The Atom Publishing Protocol. http://www.ietf.org/internet-drafts/draft-ietf-atompub-protocol-09.txt [Accessed 30 June2006].

4. RDF/XML Syntax Specification (Revised), W3C Recommen-dation, February 2004. http://www.w3.org/TR/2004/REC-rdf-syntax-grammar-20040210/ [Accessed 28 June 2006].

5. Jeroen Bekaert.Standards-Based Interfaces for Harvesting andObtaining Assets from Digital Repositories. PhD thesis, GhentUniversity, Belgium, 2006.http://hdl.handle.net/1854/4833.

6. Jeroen Bekaert and Herbert Van de Sompel. AStandards-based Solution for the Accurate Transfer ofDigital Assets. D-Lib Magazine, 11(6), June 2005.http://www.dlib.org/dlib/june05/bekaert/06bekaert.html.

7. Jeroen Bekaert, Patrick Hochstenbach, Lyudmila Balakireva,and Herbert Van de Sompel. Using MPEG-21 DIP andNISO OpenURL for the Dynamic Dissemination of Com-plex Digital Objects in the Los Alamos National Labora-tory Digital Library. D-Lib Magazine, 10(2), February 2004.http://www.dlib.org/dlib/february04/bekaert/02bekaert.html.

8. Jeroen Bekaert, Patrick Hochstenbach, and Herbert Vande Sompel. Using MPEG-21 DIDL to Represent Com-plex Digital Objects in the Los Alamos National Lab-oratory Digital Library. D-Lib Magazine, 9(11), 2005.http://www.dlib.org/dlib/november03/bekaert/11bekaert.html.

9. RDF Vocabulary Description Language 1.0: RDFSchema, W3C Recommendation, February 2004.http://www.w3.org/TR/rdf-schema/ [Accessed 29 June 2006].

10. The China Digital Museum Project: A DSpace federation.http://wiki.dspace.org/ChinaDigitalMuseumProject [Accessed 28June 2006].

11. CrossRef.org: DOIs for Research Content.http://www.crossref.org/ [Accessed 28 June 2006].

12. Jeffrey Darlington. PRONOM – A Practical Online Com-pendium of File Formats.RLG DigiNews, 7(5), October 2003.http://www.rlg.org/legacy/preserv/diginews/v7 n5 feature2.html.


13. Herbert Van de Sompel. Roadblocks, June 2003.http://www.sis.pitt.edu/∼dlwkshop/paper sompel.html [Ac-cessed 27 June 2006].

14. Herbert Van de Sompel and Oren Beit-Arie. OpenLinking in the Scholarly Information Environment Us-ing the OpenURL Framework. D-Lib Magazine, 7(3),March 2001. http://www.dlib.org/dlib/march01/vandesompel/-03vandesompel.html.

15. Herbert Van de Sompel, Jeoen Bekaert, Xiaoming Liu, LudaBalakireva, and Thorsten Schwander. aDORe: A Modular,Standards-Based Digital Object Repository.The ComputerJournal, 48(5):514–535, 2005.doi:10.1093/comjnl/bxh114.

16. Herbert Van de Sompel, Carl Lagoze, Jeroen Bekaert, Xi-aoming Liu, Sandy Payette, and Simeon Warner. An In-teroperable Fabric for Scholarly Value Chains.D-Lib Mag-azine, 12(10), 2006. doi:10.1045/october2006-vandesompel.http://dx.doi.org/10.1045/october2006-vandesompel.

17. Herbert Van de Sompel, Michael L. Nelson, CarlLagoze, and Simeon Warner. Resource Harvestingwithin the OAI-PMH Framework. D-Lib Magazine,10(12), 2004. doi:10.1045/december2004-vandesompel.http://www.dlib.org/dlib/december04/vandesompel/-12vandesompel.html.

18. Herbert Van de Sompel, Sandy Payette, John Erickson,Carl Lagoze, and Simeon Warner. Rethinking scholarlycommunication: Building the system that scholars deserve.D-Lib Magazine, 10(9), 2004. doi:10.1045/september2004-vandesompel. http://www.dlib.org/dlib/september04/-vandesompel/09vandesompel.html.

19. DLF Aquifer - Asset action experimentation.http://rama.grainger.uiuc.edu/assetactions/index.asp [Accessed28 June 2006].

20. Information and documentation - The Dublin Core metadata el-ement set, ISO (ISO 15836) and ANSI/NISO (Z39.85-2001),2001. http://www.niso.org/standards/resources/Z39-85.pdf [Ac-cessed 29 June 2006].

21. EduSource Communication Layer (ECL).http://ecl.iat.sfu.ca/#LORNET [Accessed 28 June 2006].

22. John Ellson. WebDot Home Page.http://www.site.uottawa.ca:4333/webdot/ [Accessed 28 June2006].

23. Steve Hitchcock, Donna Bergmark, Tom Brody, Christo-pher Gutteridge, Les Carr, Wendy Hall, Carl Lagoze,and Stevan Harnad. Open citation linking: The wayforward. D-Lib Magazine, 8(10), October 2002.http://www.dlib.org/dlib/october02/hitchcock/10hitchcock.html.

24. Jane Hunter and Sharmin Choudhury. A Semi-Automated Dig-ital Preservation System based on Semantic Web Services. InProceedings of the 4th ACM/IEEE-CS Joint Conference on Dig-ital libraries (JCDL2004), pages 269–278, Tucson, Arizona,USA, June 2004.http://doi.acm.org/10.1145/996350.996415.

25. IDEAlliance and PRISM Working Group. PRISM:Publishing Requirements for Industry Stan-dard Metadata, Version 1.2h, September 2003.http://www.prismstandard.org/PAM 1.0/PRISM 1.2h.pdf [Ac-cessed 29 June 2006].

26. IMSCP: The IMS Content Packaging Specification.http://www.imsproject.org/content/packaging/ [Accessed 28June 2006].

27. Isaac Newton to Robert Hooke in a letter dated 5th Febru-ary 1676. Written as “If have seen further it is by standingon ye sholders[sic] of Giants.”. Quote reproduced in Never at

Rest: A Biography of Isaac Newton, Richard S. Westfall, Cam-bridge, 1980, p274. Amusingly, many forms of this phrase are(mis)attributed to Newton. The phrase draws on the much oldermetaphor of “dwarfs standing on the shoulders of giants” andhence being able to see further by virtue of their vantage pointrather than stature.

28. Ian Jacobs and Norman Walsh (editors). Architectureof the World Wide Web, Volume One, December 2004.http://www.w3.org/TR/2004/REC-webarch-20041215/ [Accessed29 June 2006].

29. Henry Jerez, Giridhar Manepalli, Christophe Blanchi, andLaurence W. Lannom. ADL-R: The First Instance of aCORDRA Registry. D-Lib Magazine, 12(2), February 2006.http://www.dlib.org/dlib/february06/jerez/02jerez.html.

30. JSR 170: Content Repository for JavaTM technology API.http://www.jcp.org/en/jsr/detail?id=170 [Accessed 28 June2006].

31. Robert Kahn and Robert Wilensky. A frameworkfor distributed digital object services. Interna-tional Journal on Digital Libraries, 6(2), April 2006.doi:10.1007/s00799-005-0128-x. Original 1995 report at:http://www.cnri.reston.va.us/cstr/arch/k-w.html.

32. Resource Description Framework (RDF): Concepts andAbstract Syntax, W3C Recommendation, February 2004.http://www.w3.org/TR/2004/REC-rdf-concepts-20040210/ [Ac-cessed 28 June 2006].

33. Wilbert Kraan and Jon Mason. Issues in FederatingRepositories: A Report on the First International COR-DRA Workshop. D-Lib Magazine, 11(3), March 2005.http://www.dlib.org/dlib/march05/kraan/03kraan.html.

34. Carl Lagoze, Herbert Van de Sompel, Michael Nelson,and Simeon Warner. The Open Archives Initiative Pro-tocol for Metadata Harvesting, version 2.0, June 2002.http://www.openarchives.org/OAI/2.0/openarchivesprotocol.htm[Accessed 29 June 2006].

35. Live Clipboard Technical Introduction.http://spaces.msn.com/editorial/rayozzie/demo/liveclip/-liveclipsample/techPreview.html [Accessed 2006-06-25].

36. Nutch: Open-source web search software built on lucenceandjava. http://lucene.apache.org/nutch/ [Accessed 29 June 2006].

37. The Library of Congress. METS: Metadata Encoding andTransmission Standard.http://www.loc.gov/standards/mets/.

38. IFLA Study Group on the Functional Requirements for Bib-liographic Records. Functional Requirements for Biblio-graphic Records, UBCIM Publications, New Series, 1998.http://www.ifla.org/VII/s13/frbr/frbr.pdf.

39. Reference Model for an Open Archival Informa-tion System, CCSDS Secretariat, NASA, July 2002.http://ssdoo.gsfc.nasa.gov/nost/wwwclassic/documents/-pdf/CCSDS-650.0-B-1.pdf.

40. Open Service Interface Definitions (OSIDs).http://www.okiproject.org/ [Accessed 28 June 2006].

41. SRW/U Record Update service.http://srw.cheshire3.org/docs/update/ [Accessed 28 June2006].

42. D. Atkins et al. National Science Foundation Blue-RibbonAdvisory Panel on Cyberinfrastructure, Revolutionizing Sci-ence and Engineering through Cyber-infrastructure, 2003.http://www.communitytechnology.org/nsf ci report/.

43. Donald J. Waters. Cyberinfrastructure and the Humanities.http://www.cni.org/tfms/2003b.fall/handouts/Fall2003Handouts/H-Watersplenary.doc [Accessed 27 June 2006].


44. WebDAV: Web-based Distributed Authoring and Versioning.http://www.graphviz.org/webdot [Accessed 28 June 2006].

45. XFDU: XML Formatted Data Unit.http://sindbad.gsfc.nasa.gov/xfdu/ [Accessed 29 June 2006].

46. ANSI/NISO Z39.88 - 2004 The OpenURLFramework for Context-Sensitive Services,2004. http://www.niso.org/standards/resources/-Z39 88 final ANSIpending.pdf [Accessed 28 June 2006].

Pathways: augmenting interoperability across scholarly repositories

Documents

Transcript of Pathways: augmenting interoperability across scholarly repositories