Awareness in Collaborative Information Seeking

Chirag ShahSchool of Communication & Information (SC&I) Rutgers, The State University of New Jersey 4 Huntington St,New Brunswick, NJ 08901. E-mail: chirags@rutgers.edu

Gary MarchioniniSchool of Information and Library Science, University of North Carolina at Chapel Hill, 100 Manning Dr.,Chapel Hill, North Carolina 27599. E-mail: chirag@unc.edu; march@ils.unc.edu

Support for explicit collaboration in information-seekingactivities is increasingly recognized as a desideratum forsearch systems. Several tools have emerged recently thathelp groups of people with the same information-seekinggoals to work together. Many issues for these collab-orative information-seeking (CIS) environments remainunderstudied.The authors identified awareness as one ofthese issues in CIS, and they presented a user study thatinvolved 42 pairs of participants, who worked in collabo-ration over 2 sessions with 3 instances of the authors’CIS system for exploratory search. They showed thatwhile having awareness of personal actions and historyis important for exploratory search tasks spanning multi-ple sessions, support for group awareness is even moresignificant for effective collaboration. In addition, theyshowed that support for such group awareness can beprovided without compromising usability or introducingadditional load on the users.

Introduction

Information seeking is typically considered a solo activ-ity, but there is evidence to suggest otherwise. For instance,Twidale and Nichols (1996) pointed out a problem: “The useof library resources is often stereotyped as a solitary activity,with hardly any mention in the substantial library science andinformation retrieval literature of the social aspects of infor-mation systems.” They argued that introducing support forcollaboration into information retrieval systems would helpusers to learn and use the systems more effectively. Simi-larly, based on their extensive study of patent office workers,Hansen and Järvelin (2005) also concluded that the assump-tion that information retrieval performance is purely indi-vidual should be reconsidered. Morris (2007) proposed thatfour features of exploratory search experience—coverage,confidence, exposure, and productivity—could be enhanced

Received January 6, 2010; revised March 2, 2010; accepted May 3, 2010

© 2010 ASIS&T • Published online 21 June 2010 in Wiley Online Library(wileyonlinelibrary.com). DOI: 10.1002/asi.21379

by providing explicit support for collaborative search andsubsequent sensemaking processes.

These demonstrations of collaborative information seek-ing (CIS) as a phenomenon inspired a variety of systemsthat aim to support collaborative information seeking. Sys-tems such as Ariadne (Twidale, Nichols, & Paice, 1997)and SearchTogether (Morris and Horvitz, 2007) provide userinterface-level support for collaboration. Other approaches(e.g., Pickens, Golovchinsky, Shah, Qvarfordt, & Back, 2008)have focused on providing system-mediated collaborativesupport. Despite these encouraging efforts, there is a sig-nificant need to address a number of CIS-related issues byunderstanding and addressing the users’ needs in a collab-orative environment. Grudin (1994, p. 93) noted: “Manyexpensive failures in developing and marketing software thatis designed to support groups are not due to technical prob-lems. They result from not understanding the unique demandsthis class of software imposes on developers and users.”One such demand or requirement for CIS systems is thesupport for awareness (Schmidt, 2002, p. 285). During ourparticipatory design sessions,1 we also discovered the impor-tance of providing appropriate awareness functionality ona CIS interface. This led us to investigate the impact, cost,and implementation related issues of awareness support inuser-centric CIS systems.

To facilitate this investigation, we have developed a CISsystem, Coagmento,2 which provides support for communi-cation and awareness of personal and group histories whileworking on CIS tasks. Using Coagmento, we conducted auser study that involved 42 pairs of participants, who workedon exploratory search tasks. We assigned each pair randomlyto three different versions of Coagmento, where the indepen-dent variable was the level of awareness support provided.

1The participatory design sessions were conducted in Dr. Marchionini’sHCI seminar class at UNC Chapel Hill during Fall 2008 and Fall 2009semesters.

2In Latin, Coagmento means working or joining together.

JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY, 61(10):1970–1986, 2010

The participants were brought to the lab for two differentsessions, thus allowing them to get used to the system andfor studying multisession collaboration. In the present article,we describe this user study, showing how providing differ-ent levels of awareness support to collaborators workingon exploratory search tasks can impact their productivity,physical and cognitive efforts, satisfaction, and engagement.

The rest of the article is organized as follows. We firstpresent a brief overview of related works and place ourresearch in context in the next section. We then describeour methodology, explaining different aspects of our userstudy. The analysis and discussion are followed by designimplications and concluding remarks.

Background

Three components specific to group-work or collabora-tion that are highly predominant in the CIS or Computer-Supported Cooperative Work (CSCW) literature are control,communication, and awareness (Rodden, 1991).

Several systems supporting collaboration have identifiedthe above-mentioned issues (control, communication, andawareness) as critical to their design and associated userresponse. For instance, Farooq, Ganoe, Carroll, and Giles.(2009) presented a collaborative design for CiteCeer, a searchengine and digital library of research literature in the com-puter and information science disciplines. Based on a surveyand follow-up interviews with CiteSeer users, the authorspresented four novel implications for designing the Cite-Seer collaboratory: (a) visualize query-based social networksto identify scholarly communities of interest, (b) providean online collaborative tool support for upstream stages ofscientific collaboration, (c) support activity awareness forstaying cognizant of online scientific activities, and (d) usenotification systems to convey scientific activity awareness.

Awareness is one of the most important issues that is iden-tified and addressed in the CSCW literature.A clear definitionand a methodology for providing awareness, though, arelacking. One of the often-asked questions about awarenessin CSCW is “awareness of what?” Schmidt (2002, p. 288)argued that we should talk about awareness not as a sepa-rate entity, but as someone’s being aware of some particularoccurrence. In other words, the term “awareness” is mean-ingful only if it refers to a person’s awareness of something.Heath, Svensson, Hindmarsh, Luff, and Lehn (2002) sug-gested that awareness is not simply a “state of mind” or a“cognitive ability,” but rather a feature of practical action thatis systematically accomplished within developing course ofeveryday activities.

Several related terms and definitions are used in the CSCWliterature to refer to awareness in collaborative projects.For instance, Dourish and Bellotti (1992, p. 107) definedawareness as “an understanding of the activities of others,which provides a context for your own activity.” Dourishand Bly (1992, p. 541) suggested the following defini-tion for awareness: “Awareness involves knowing who is‘around,’ what activities are occurring, who is talking with

whom; it provides a view of one another in the daily workenvironments. Awareness may lead to informal interactions,spontaneous connections, and the development of sharedcultures—all important aspects of maintaining working rela-tionships which are denied to groups distributed acrossmultiple sites.”

A set of theories and models for understanding and pro-viding awareness emerged in the early works reported inthe CSCW literature. Gaver (1991) argued that an intensesharing of awareness characterizes focused collaboration inwhich people work closely together on a shared goal. Hefurther claimed that less awareness is needed for division oflabor, and that more casual awareness can head to serendip-itous communication, which can turn into collaboration. Heproposed a general awareness model that incorporates andsupports all of such activities.

Bly, Harrison, and Irwin (1993, p. 29) also identified theimportance of such general awareness by saying, “Whengroups are geographically distributed, it is particularly impor-tant not to neglect the need for informal interactions, sponta-neous conversations, and even general awareness of peopleand events at other sites.”

Some of the early works, which reported using ethno-graphic field studies in CSCW (e.g., Harper, Hughes, &Shapiro, 1989b; Harper, Hughes, & Shapiro, 1989a; Heath &Luff, 1991 not cited), identified the need to seamlesslyalign and integrate the activities of the participants of acollaborative project. Although they did not refer to it asawareness, soon, the term awareness was adopted to addresssuch practices that support connecting collaborators withoutthe activities of asking, suggesting, requesting, ordering, orreminding.

Several works argued that providing audio-video com-munication channel could suffice for awareness (Mantei,Baecker, Sellen, Buxton, & Milligan, 1991; Gaver et al.,1992). However, the use of communication as a substitute forawareness turned out to be very limited in its applicability(Gaver, 1992). Another line of research focused on pro-viding awareness using computational environments basedon “event propagation mechanisms” for collecting, dissemi-nating, and integrating information concerning collaborativeactivities.

To summarize, there are several ways of defining andimplementing awareness.Various research projects have usedtheir own taxonomy and interpretation of awareness for cre-ating frameworks and systems. For instance, Gutwin andGreenberg (2002) classified awareness in two types: situa-tional, and workspace, and they suggested that situationalawareness underlies the idea of workspace awareness ingroupware systems. Their definition of workspace aware-ness included how people interact with the workspace, ratherthan just awareness of the workspace itself. Simone and Ban-dini (2002) identified two kinds of awareness: (a) by-productawareness that is generated in the course of the activities peo-ple must do to accomplish their collaborative tasks; (b) andadd-on awareness that is the outcome of an additional activ-ity, which is a cost for the collaborators to what they must do

JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY—October 2010 1971DOI: 10.1002/asi

and is discretional in that it depends on collaborators’ evalu-ation of the contingent situation. Chalmers (2002), likewise,divided the awareness in two kinds: awareness of people andof information artifacts. He suggested implementing activity-centered awareness tool, in that it focuses on presenting theongoing appearance and activity of people.

For the purpose of the work reported here, a more com-prehensive and well-accepted taxonomy of awareness, whichaddresses four kinds of awareness (Liechti & Sumi, 2002) aslisted below, will be used.

• Group awareness. This kind of awareness includes provid-ing information to each group member about the status andactivities of the other collaborators at a given time.

• Workspace awareness. This emphasizes the fact that aware-ness generally emerges when people share a space. In otherwords, this kind of awareness refers to a common space thatthe group members share and where they can bring and discusstheir findings and create a common product.

• Contextual awareness. This type of awareness relates to theapplication domain, rather than the users. Here, we want toidentify what content is useful for the group and what thegoals are for the current project.

• Peripheral awareness. This refers to the human ability toprocess information at the periphery of the attention, witha very low overhead. In other words, peripheral awarenessrelates to the information that should be kept separate (ontheir periphery) from what a participant is currently viewingor doing.

We started our investigation on the issue of awareness byposing the following questions.

• How should different kinds of awareness be implemented(mappings from the kind of awareness to components)?

• How should these awareness components be presented to theuser (design)?

• What are the costs and benefits associated with each of theseawareness components (evaluation)?

As far as the design goes, several design decisions arealready made for Coagmento based on previous works (e.g.,Shah, Marchionini, & Kelly, 2009; Shah, 2010a), cognitivewalkthroughs, pilot runs, and general understanding of creat-ing a familiar, easy to use, and accessible interface. Therefore,the primary objective of the present work is not to study theoptimal design or compare different designs for a CIS system,but to evaluate the impact of different awareness componentsfor their effectiveness, efficiency, usefulness, and usability insupporting collaborative activities and inducing higher levelsof satisfaction and engagement in collaboration.

Different CIS systems have different ways of providingawareness to the collaborators, depending on the domainand the kind of application. Take, for example Ariadne(Twidale, Nichols, & Paice, 1995), developed to support thecollaborative learning of database browsing skills. To facil-itate complex browsing processes in collaboration, Ariadnepresents a visualization of the search process. This visualiza-tion comprises thumbnails of screens, looking like playingcards, which represented command-output pairs. Any suchcard can be expanded to reveal its details. The support for

awareness, in this case, is driven by the specific domain(library) and application (catalogue search).

SearchTogether (Morris & Horvitz, 2007), on the otherhand, was based on information seeking (application) on theWeb (domain). SearchTogether instantiates awareness in sev-eral ways, one of which is per-user query histories, which isdone by showing each group member’s screen name, his orher photo, and queries in the “Query Awareness” region. Theaccess to the query histories is immediate and interactive, asclicking on a query brings back the results of that query fromwhen it was executed. The authors identified query awarenessas a very important feature in collaborative searching, whichallows group members to not only share their query termsbut also learn better query formulation techniques from oneanother. Another component of SearchTogether that facili-tates awareness is the display of page-specific metadata. Thisregion includes several pieces of information about the dis-played page, including group members who viewed the givenpage, and their comments and ratings. The authors claim thatsuch visitation information can help one to choose to eitheravoid a page already visited by someone in the group, toreduce the duplication of efforts or, perhaps, visit such pages,as they provide a sign of promising leads as indicated bythe presence of comments and ratings. Thus, SearchTogetherprovides awareness support via person, query, or Web page.

The design of our CIS system Coagmento (describedbelow) was greatly influenced by SearchTogether, but unlikeSearchTogether, Coagmento provides tools and support fornot only searching and sharing but also organizing and syn-thesizing information. In addition to query and results aware-ness, Coagmento incorporates contextual and workspaceawareness. This allowed us to explore the impact of variouskinds of awareness in a CIS environment.

To our knowledge, there has not been any recent studyof online collaborative information seeking that looks atawareness in the context of different interface designs. Ourreported work here, thus, is highly valuable in adding to ourunderstanding about awareness in CIS.

Coagmento—A Tool for CollaborativeInformation Seeking

We have developed Coagmento (Shah, 2010b), a Firefoxplug-in that helps multiple people, working in collaboration,to communicate and to search, share, and organize informa-tion. Coagmento allows its users to be colocated or remote,working synchronously or asynchronously. The design ofCoagmento is based on several personal interviews, mock-ups, cognitive walkthroughs, and pilot runs (Shah et al.,2009).

A screenshot of Coagmento is given in Figure 1. As wecan see, it includes a toolbar and a sidebar. The toolbar hasseveral buttons that help one collect information and be awareof the progress in a given collaboration. The toolbar has threemajor parts as follows:

• Buttons for collecting information and making annotations.These buttons help one save or remove a Web page, make

1972 JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY—October 2010DOI: 10.1002/asi

FIG. 1. A screenshot of Coagmento with enhanced views of its toolbar and sidebar.

annotations on a Web page, and highlight and collect textsnippets.

• Page-specific statistics. The middle portion of the toolbarshows various statistics, such as the number of views, annota-tions, and snippets, for the displayed page.A user can click ona given statistic and obtain more information. For instance,clicking on the number of snippets information will bringup a window that shows all the snippets collected by thecollaborators from the displayed page.

• Task-specific statistics. The last portion of the toolbar dis-plays task name and various statistics, including number ofpages visited and saved, about the current task. Clicking onthat portion brings up the workspace, where one can view allthe collected objects (pages and snippets) brought in by thecollaborators for that task.

The sidebar features a chat window, under which there arethree tabs with the history of search engine queries, savedpages, and snippets. With each of these objects, the user whocreated or collected that object is shown.Anyone in the groupcan access an object by clicking on it. For instance, one canclick on a query issued by anyone in the group to rerun thatquery and bring up the results in the main browser window.

In the present design of Coagmento, different kinds ofawareness discussed earlier are mapped to system compo-nents as follows.

• Group awareness is facilitated by providing informationabout one’s partner’s current status (online/offline) and activ-ities (e.g., document or rank list being viewed).

• Workspace awareness is given by providing a common space,where the group members can see and compile their collectiveresults (described later with Figure 5).

• Contextual awareness is instantiated by display of informa-tion about the task and the goals.

• Peripheral awareness is given by presenting a history of var-ious actions, such as documents viewed and queries used aswell as products such as saved documents and snippets, in away that does not interrupt a user’s current activity.

The study reported here varied conditions that includedor excluded these awareness components in certain configu-rations. More specifically, the study manipulated peripheralawareness in three different ways by providing: (a) noperipheral awareness, (b) personal peripheral awareness, and(c) group peripheral awareness. The value of focusing onperipheral awareness is inspired by some of the works inthe CSCW literature. For instance, Bly et al. (1993), withtheir experiments in media space, found that although itwas seemingly invisible, peripheral awareness was the mostuseful one. Gaver (1992) also argued for the provision ofperipheral awareness that provides unobtrusive information

JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY—October 2010 1973DOI: 10.1002/asi

FIG. 2. Interface for condition 1, baseline.

of the people, artifacts, and environment, while working ona collaborative project.

Method

To address the issue of awareness in CIS, we conducteda user study, in which we invited 42 pairs of participants towork on two different tasks over two separate sessions. Thedetails of this study are given below.

Participants

We recruited 84 participants in 42 pairs from the Uni-versity of North Carolina at Chapel Hill. These participantswere asked to come to the lab for two different sessions,which were 1 to 2 weeks apart. The participants were ableto choose the day and time convenient to them. Because theparticipants had to sign up in pairs, both the participants ina given pair already knew each other. In addition to this,we required that the participants in a given pair should havedone some collaborative work with each other before, thus,making sure they not only know each other, but also are com-fortable working with each other on a collaborative project.The approval of a pair’s participation in this study was basedon these requirements.

Of the 84 participants, 27 were male and 57 were female,and their ages ranged from 17 to 50 years with a median of21 years. Several of the pairs were coworkers or spouses. Amajority of the participants were undergraduate or graduate

students, while a few were university employees. Participantswere compensated $25 each for their participation in twosessions and competed for two Apple iPods presented to theteam with the best performance (based on a scoring rubricthat took into account the amount and kind of informationcollected, and the level of coordination the team had).

Conditions

The participants were assigned to one of the three condi-tions randomly. These conditions were defined based on theprovision of different levels of awareness-related support tothe participants, which are as follows:

1. Baseline: support of contextual awareness (current taskname and topic description) and workspace awareness(where the team can compile their final report; Figure 2).

2. ‘Personal peripheral awareness: support of personal his-tory (documents and snippets saved, queries used, etc.) inaddition to the support provided in the baseline (Figure 3).

3. Group peripheral awareness: support of group history(documents and snippets saved, queries used, etc.) inaddition to the support provided in the baseline (Figure 4).

Thus, the main independent variable here was the kindof peripheral awareness provided (1 = none, 2 = personal,3 = group). All the conditions had communication, contex-tual awareness, and workspace awareness. No condition hadthe traditional group awareness (where a participant can seeexactly what another collaborator is doing at a given time).

1974 JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY—October 2010DOI: 10.1002/asi

FIG. 3. Interface for condition 2, personal peripheral awareness.

FIG. 4. Interface for condition 3, group peripheral awareness.

JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY—October 2010 1975DOI: 10.1002/asi

FIG. 5. A screenshot of Coagmento workspace.

This was because of the following two main reasons. For one,because the participants were required to work on the sametask at the same time, they knew the other person was not onlyonline, but also working on the same task. This realizationeliminated the need to know the immediate status of one’s col-laborator. Furthermore, introducing this additional conditionwould weaken the statistical power of the study. Therefore,we decided to keep the traditional group awareness featureomitted from this study interface.

Note that peripheral awareness does not indicate the natureof the information, but how it is presented (at the periphery).For our purpose, we divided such awareness into personaland group peripheral awareness, corresponding to the infor-mation relating to personal and group history, respectively.Both conditions 2 and 3 had the same components andpresentations for instantiating peripheral awareness, but theinformation displayed in each of them was different: condi-tion 2 interface included a user’s personal information only,whereas condition 3 interface provided the team’s cumulativeinformation.

To support workspace awareness, each participant wasgiven access to a common workspace (Figure 5). Thisworkspace is updated in real time and shows the collectedobjects (Web pages and snippets).

Our random assignment of each pair of participantsyielded 14 pairs for each of the three conditions.

Sessions

We brought each individual pair of participants to the labfor two sessions that were 1 to 2 weeks apart. Each session

lasted about one and a half hours. The flow for each sessionis depicted in Figure 6.

During the first session, the participants were shown avideo tutorial that demonstrated the use of Coagmento and theprocess of collecting relevant information (snippets of text).After the tutorial, the participants were placed in differentrooms so that they could not talk to each other directly or seewhat the other person was doing (Figure 7). Both participantsused typical mid-end PC workstations, running WindowsXP, with Ethernet connectivity and 19-inch monitors. Aresearcher was stationed so that he could see both partici-pants. Once the participants logged in, they filled out a demo-graphic questionnaire and began working on the first task.

As discussed below, the tasks were simulated work tasks(Borlund, 2003). About 20 minutes into their work, theresearcher sent out a message via the sidebar chat askingthem to stop the task and fill in an online questionnaire, as iftheir supervisor had requested an update. Once both the par-ticipants finished their individual questionnaires, they wereasked to start working on the second task. The participantswere once again interrupted about 20 minutes later, and askedto complete the same questionnaire.

After the post-task questionnaire for the second task, theparticipants were asked to fill in an end-of-session question-naire. After the participants submitted their individual end ofsession questionnaires, they were brought together and askeda few questions.

For the second session, the participants were given arefresher of the system and shown how to compile their finalreport by grouping their collected snippets into different cate-gories for each of the two tasks.The categories were presented

1976 JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY—October 2010DOI: 10.1002/asi

FIG. 6. Flow of the two sessions.

FIG. 7. Subjects in a study session.

JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY—October 2010 1977DOI: 10.1002/asi

in the task statement and corresponded to different aspects ofthe work task (see task statements below). The participantswere then asked to take their places in the room other thanthe one they had the last time, to take care of any bias theparticipants may have for the machine or the room they used.

After 20 minutes of additional work on task one, they com-pleted the post-task questionnaire and were asked to organizetheir collected snippets by placing each relevant snippet intoone of the categories for a given task. When they had finishedorganizing their snippets, they worked through the secondtask, including collecting their information, post-task ques-tionnaire, and organizing the snippets. Upon completing boththe tasks, the participants completed an exit questionnaire,after which they were brought together and interviewed.

Tasks

The participants were asked to collect relevant informa-tion for two exploratory tasks, which were designed to berealistic work tasks, that might be of relevance and interestto the participant pool (Borlund & Ingwersen, 1999). Ratherthan asking participants to create their own organizations forthe pertinent snippets, the task statements identified specificissues that should be addressed, and these issues were usedas organizing bins for the collected snippets.

The task descriptions as given to the subjects are providedin the following sections.

Task 1: Economic Recession

A leading newspaper has hired your team to create a compre-hensive report on the causes and consequences of the currenteconomic recession in the U.S. As a part of your contract, youare required to collect all the relevant information from anyavailable online sources that you can find.

To prepare this report, search and visit anyWeb site that youwant and look for specific aspects as given in the guidelinebelow. As you find useful information, highlight and saverelevant snippets. Later, you can use these snippets to compileyour report. You may also want to save the relevant Web sitesas bookmarks, but remember, your main objective here is tocollect as many relevant snippets as possible.

Your report on this topic should address the followingissues: reasons behind this recession, effects on some majorareas, such as healthcare, home ownership, and financial sec-tor (stock market), unemployment statistics over a period oftime, proposal, execution, and effects of the economy stimu-lation plan, and people’s opinions and reactions on economy’sdownfall.

Task 2: Social Networking

The College Network News Channel wants to do a docu-mentary on the effects of social networking services andsoftware. Your team is responsible for collecting various rel-evant information (including statistics) from the Web. As apart of your assignment, you are required to collect all therelevant information from any available online sources thatyou can find.

To prepare this report, search and visit anyWeb site that youwant and look for specific aspects as given in the guidelinebelow. As you find useful information, highlight and save

relevant snippets. Later, you can use these snippets to compileyour report. You may also want to save the relevant Web sitesas bookmarks, but remember, your main objective here is tocollect as many relevant snippets as possible.

Your report on this topic should address the followingissues: emergence and spread of social networking sites, suchas MySpace, Facebook, Twitter, and del.icio.us, statisticsabout popularity of such sites (How many users? How muchtime they spend? How much content?), impacts on studentsand professionals, commerce around these sites (How do theymake money? How do users use them to make money?), andexamples of usage of such services in various domains, suchas healthcare and politics.

Analysis

The core question of the study described here is to look atthe impact of awareness in collaborative information-seekingprojects. The three conditions we used were defined based onthe amount and kind of awareness provided to the collabora-tors. To measure how aware the participants were at differentpoints, we employed several instruments. In this section, wepresent analysis based on studying the question of awarenessusing these instruments. In addition to measuring awarenessdirectly, we also used several other outcome measures, suchas productivity, user satisfaction, and engagement, for a morecomprehensive understanding of the impacts of awarenesssupport in CIS.

For the purpose of the discussion and the analysis pre-sented here, we have collapsed both the tasks and the sessionstogether. Our initial analysis of these two independent vari-ables found no statistically significant differences betweenthe tasks or sessions for the perceived and reported awarenessmeasures, and so they are combined in what follows. Whenanalyzing a questionnaire, if creating an index that combinesall of the questions on that questionnaire was appropriate,then that index’s statistical reliability was checked, and if reli-ability statistic was α > 0.7, then such an index was createdby averaging the responses for all the questions. One-wayanalysis of variance (ANOVA) on this index is done, and iffound significant with p < 0.05, then Sheffe’s post hoc testwas executed. If creating an index was not appropriate or itsstatistical reliability was not sufficient, or if an item for anal-ysis was log data, then straight one-way ANOVA was done,followed by Sheffe’s post hoc test, if appropriate. Note thateach condition had 14 groups, with 28 individuals. Therefore,there was a total of 42 groups, with 84 individuals.

Productivity

To begin our analysis, we first looked at the productivityof each team. This was measured in two ways: looking ateach participant’s activities (Table 1) and finding the Webpages overlapped between the collaborators in a given pair(Table 2).

We found that participants in condition 3 used significantlymore unique queries than those in condition 1 (p < 0.05).Given that there was no difference between these two con-ditions for the total number of queries used, we can say that

1978 JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY—October 2010DOI: 10.1002/asi

TABLE 1. Average use/collection of various objects by each user for eachcondition.

Condition 1 Condition 2 Condition 3

Mean SD Mean SD Mean SD

Total queries 44.50 2037 49.61 18.46 58.07 29.92(F = 2.391, p < 0.098)

Unique queries∗ 24.11 10.37 29.11 9.94 31.93 14.44(F = 3.177, p < 0.149)

Total webpages viewed 124.71 31.22 142.86 45.10 144.64 47.15(F = 1.952, p < 0.149)

Unique webpages viewed 79.43 16.95 92.89 27.00 92.61 27.68(F = 2.788, p < 0.0487)

Webpages saved 13.54 5.75 14.86 8.04 15.54 8.95(F = 0.489, p < 0.615)

Snippets collected 19.95 14.73 18.79 13.07 19.80 9.38(F = 0.141, p < 0.0868)

Note. SD = standard deviation.∗Difference across conditions was statistically significant (p < 0.05).For one-way analysis of variance, between groups degree of freedom

(df) = 2 and within groups df = 81.

TABLE 2. Summary statistics of Web pages overlapped between thecollaborators in a given pair.

Condition 1 Condition 2 Condition 3

Mean SD Mean SD Mean SD

Webpages overlapped 4.00 3.96 4.29 2.27 4.07 2.16within team(F = 0.036, p < 0.964)

Note. SD = standard deviation.For one-way analysis of variance, between groups degree of freedom

(df) = 2 and within groups df = 39.

the participants in condition 3 had much less overlap in theirquery usage and could try more unique searches for the taskat hand.

Looking at the overlap of Web page visits between the twoparticipants in a team (Table 2), we see that those in condition2 had slightly more overlap than the other two conditions.However, these differences were not found to be statisticallysignificant.

These observations inform us that although it appears thatthe teams in each condition are doing the same amount ofwork, condition 3 teams are clearly able to explore morevenues of information overall. This can be attributed to thekind of awareness provided. For example, a participant incondition 1 has no support for knowing the search queries heor his teammate used in the past. Therefore, he is likely torepeat a search even when it is not needed. A participant incondition 2, on the other hand, can avoid repeating his ownsearches, unless necessary because of the personal periph-eral awareness support. A participant in condition 3 will haveknowledge about his as well as his teammate’s searches.This explains why greater differences were found betweenconditions 1 and 3 for unique query usage, and not amongother pairs of conditions.

Another important aspect of the assigned tasks was thesnippets. As shown in Table 1, no significant difference wasfound in the number of snippets collected by the teams indifferent conditions. Much of the snippets collected can beattributed to personal motivation. The team that collected themost number of snippets was in condition 1. Given that timewas limited, the tasks exploratory, and a prize to be won forthose collecting many snippets, this team quickly divided thetasks and went through as many Web pages as possible tocollect numerous snippets. This team gathered 223 snippetscombined for both the tasks during the two sessions. Thiswas significantly higher than the average, which was about78. The minimum number of snippets collected was 38, by ateam in condition 1.

To summarize, it is hard to claim difference in productivityamong different conditions. On the other hand, through queryusage, it was found that teams in condition 3 were being moreeffective in their work, that is, they managed to explore morevolume of information with the same amount of time andwork.

Perceived Awareness

To obtain feedback on how aware the participants felt, thefollowing statements, inspired by Govern and Marsch (2001),were presented individually to the participants at the end ofeach task for both sessions. The participants were asked toselect a value between 1 (strongly disagree) to 7 (stronglyagree) for these statements.

Q1. Right now, I am keenly aware of everything in myenvironment.

Q2. Right now, I am conscious of what is going onaround me.

Q3. Right now, I am conscious of all objects around me.Q4. Right now, I am concerned about what my teammate

thinks of me.Q5. Right now, I am aware of what my teammate just did.Q6. Right now, I am conscious that my teammate is aware of

my actions.Q7. Right now, I am aware of how well we performed

together in the team.

Because creating an index is appropriate for this ques-tionnaire, and it was found to be statistically reliable withα = 0.781, an index was created by averaging the responseson all seven questions for every individual. Table 3 reportsvarious statistics for this index across all the conditions. Nosignificant difference was found among the three conditions.This indicates that as far as being conscious of the situationaround them was concerned, participants in each conditionwere similarly aware.

In addition to these post-task questions, the participantswere also asked to rate two specific questions, given below,

JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY—October 2010 1979DOI: 10.1002/asi

TABLE 3. Index for perceived situational awareness questions.

Condition 1 Condition 2 Condition 3

Mean SD Mean SD Mean SD

Index for perceived 4.69 0.859 4.56 1.062 4.79 0.924situational awareness(F = 1.653, p < 0.193)

Note. SD = standard deviation.For one-way analysis of variance on each question, between groups

degree of freedom (df) = 2 and within groups df = 333.

TABLE 4. Summary statistics for responses on perceived awarenessquestions at the end of each session.

Condition 1 Condition 2 Condition 3

Mean SD Mean SD Mean SD

Q1. personal awareness∗ 5.73 0.842 6.12 0.955 5.59 1.141(F = 4.426, p < 0.013)

Q2. group awareness∗ 4.55 1.205 3.95 1.531 5.50 1.250(F = 19.221, p < 0.001)

Note. SD = standard deviation.∗Difference across conditions was statistically significant (p < 0.05).For one-way analysis of variance on the index, degree of freedom (df) = 2

and within groups df = 165.

relating to personal and group awareness at the end of eachsession.

Q1. I was aware of my own history (documents viewed andsaved, snippets saved, queries used, etc.).

Q2. I was aware of my team’s history (documents viewed andsaved, snippets saved, queries used, etc.).

The average responses on the scale, 1 (strongly disagree)to 7 (strongly agree), are shown in Table 4 for each condition.Condition 2 participants reported higher personal awareness(Q1), even higher than those in condition 3 (p < 0.018). Thisis not surprising given that condition 2 had exclusive supportfor personal history. But when it comes to being aware of theteam, condition 3 participants reported significantly highervalues than those in condition 1 (p < 0.001) or 2 (p < 0.001).This is, once again, an indication that the participants in con-dition 3 were much more aware of the status of the team andthe task throughout their collaborative endeavor.

The questions listed above asked the participants to reporttheir level of awareness about certain aspects directly. Thismay not be enough to tell the whole story about awareness.We, therefore, asked the participants more specific questionsabout the status of their ongoing task, as described next.

Task Status Awareness

As listed in the method section, the participants were inter-rupted after about 20 minutes for a given task and asked to

TABLE 5. Index for absolute difference between two collaborators’reported numbers for group’s status.

Condition 1 Condition 2 Condition 3

Mean SD Mean SD Mean SD

Index for group reporting 2.38 3.247 3.99 4.896 0.598 0.756difference∗(F = 13.827, p < 0.001)

Note. SD = standard deviation.∗Difference across conditions was statistically significant (p < 0.05).

For one-way analysis of variance on the index, degree of freedom (df) = 2and within groups df = 165.

report the task status. This status was obtained by asking thefollowing questions.

Q1. How many Web pages do you think you viewed?Q2. How many Web pages do you think your team viewed?Q3. How many Web pages do you think you saved?Q4. How many Web pages do you think your team saved?Q5. How many snippets do you think you saved?Q6. How many snippets do you think your team saved?Q7. How many queries do you think you used?Q8. How many queries do you think your team used?

We will first see how much the collaborators of a giventeam were in agreement of reporting the status of the taskby analyzing questions 2, 4, 6, and 8 in the above-mentionedquestionnaire. It was found that creating an index was appro-priate and statistically reliable (α = 0.759). Table 5 reportsthe statistics with this index for average differences (abso-lute values) in the responses to these questions for a giventeam. As we can see, the teams in condition 3 have the leastdifference for these questions, and these differences were sig-nificant compared with condition 1 (p < 0.024) and condition2 (p < 0.001). In other words, these results indicate that theparticipants in condition 3 were highly coordinated when itcame to being aware of the group’s history and progress. It isalso interesting to note is that even the teams in condition 1had smaller differences than those in condition 2 (p < 0.046).

Because the participants were allowed to chat with eachother while filling in these values, we cannot simply lookat their differences and comment about the strength of theircollaboration. We also need to consider the cost associatedwith achieving the coordination. This was done by analyzingthe time and the number of messages that they exchangedwhile completing this questionnaire.

The results are reported in Table 6. It turns out that therewas no statistically reliable difference between conditions1 and 2 for the amount of time used; however, participantsin condition 1 exchanged significantly more messages thanthose in condition 2 (p < 0.018). It was also found thatcondition 3 had significantly less overhead in terms of com-munication compared with conditions 1 (p < 0.001) and 2(p < 0.001), while filling in this status report. They also

1980 JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY—October 2010DOI: 10.1002/asi

TABLE 6. Average amount of time spent and messages exchanged by each participant while reporting project status.

Condition 1 Condition 2 Condition 3

Mean SD Mean SD Mean SD

Avg. time (sec)∗ 198.18 114.149 180.12 130.626 123.04 71.000(F = 1.714, p < 0.001)

Avg. no. of messages∗ 13.94 11.963 10.12 10.576 2.37 6.103(F = 39.981, p < 0.001)

Note. SD = standard deviation.∗Difference across conditions was statistically significant (p < 0.05).For one-way analysis of variance on the index, degree of freedom (df) = 2 and within groups df = 333.

TABLE 7. Average difference between reported numbers and real numbersfor questions on personal progress.

Condition 1 Condition 2 Condition 3

Mean SD Mean SD Mean SD

Webpages viewed∗ 16.68 10.855 0.35 1.486 5.65 4.960(F = 161.194, p < 0.001)

Webpages saved∗ 2.44 2.660 1.12 1.693 1.61 1.812(F = 11.331, p < 0.001)

Snippets saved∗ 2.86 4.721 0.62 2.023 2.16 3.349(F = 11.832, p < 0.001)

Queries used∗ 2.29 2.399 0.46 1.314 1.83 2.380(F = 23.192, p < 0.001)

Note. SD = standard deviation.∗Difference across conditions was statistically significant (p < 0.05).For one-way analysis of variance on each response, degree of freedom

(df) = 2 and within groups df = 333.

spent less time compared with condition 1(p < 0.001) and2 (p < 0.001). This shows that the participants in condition 3were well aware of the status of their tasks and could reportit without spending much time or communication.

Perceived Awareness Versus Reality

Next, let us look at how close participants’ reporting abouttask awareness was to the reality. For this purpose, eight ques-tions (given in the previous subsection) on which a participantwas asked to report various numbers regarding personal orteam history on the task were considered. The actual num-bers were then calculated using the log data and comparedwith the reported numbers. The results of this comparison arepresented in Tables 7 and 8.

The following observations can be made from theseresults:

• The participants in condition 2 did significantly better thancondition 1 while reporting personal numbers on Web pagesviewed (p < 0.001), Web pages saved (p < 0.001), snip-pets collected (p < 0.001), and queries used (p < 0.018). Infact, condition 2 also did better than condition 3 on Webpages viewed (p < 0.001), snippets collected (p < 0.005), andqueries used (p < 0.001). This makes sense, as they were theonly ones with exclusive support for personal awareness.

• However, the participants in conditions 1 and 2 did nothave group awareness support. Therefore, the participants

TABLE 8. Average difference between reported numbers and real numbersfor questions on group progress.

Condition 1 Condition 2 Condition 3

Mean SD Mean SD Mean SD

Webpages viewed∗ 31.75 18.319 6.25 9.728 1.56 4.152(F = 198.123, p < 0.001)

Webpages saved∗ 4.27 3.435 3.66 3.335 0.72 1.209(F = 49.665, p < 0.001)

Snippets saved∗ 5.95 7.789 2.72 4.752 0.49 1.599(F = 29.554, p < 0.001)

Queries used∗ 4.32 4.215 2.09 3.639 0.91 1.585(F = 30.042, p < 0.001)

Note. SD = standard deviation.∗Difference across conditions was statistically significant (p < 0.05).For one-way analysis of variance on the index, degree of freedom (df) = 2

and within groups df = 333.

in condition 3 could report the numbers for group progressthat were significantly closer to the reality compared withthose reported by condition 1 or 2. Condition 3 outperformedcondition 2 on Web pages viewed (p < 0.017) and saved(p < 0.001), snippets collected (p < 0.008), and queries used(p < 0.030). They also did better than those in condition 1 onall of these fronts (p < 0.001).

• The participants in condition 3 did significantly better, even onpersonal awareness-related questions on Web pages viewed(p < 0.001) and saved (p < 0.013), compared with those incondition 1. The participants in condition 3 did not haveexplicit support for personal awareness, and they were in a bet-ter position to make educated guesses (and sometimes actualcounts) using the group awareness information, than those incondition 1.

It is clear from these observations that the condition 2interface provides the most suitable support for personalawareness, whereas condition 3 has the best support for groupawareness. An ideal interface may include group informationwith the ability to switch to personal information as needed.In other words, such an interface would provide more con-trol for the awareness to the user. During the interviews at theend of each session, a majority of participants in condition 3expressed their desire to have this ability of switching to per-sonal awareness mode, while retaining the group awarenesssupport.

JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY—October 2010 1981DOI: 10.1002/asi

TABLE 9. Average number of times workspace accessed by each partici-pant.

Condition 1 Condition 2 Condition 3

Mean SD Mean SD Mean SD

Avg. no. of times workspace 11.07 4.883 7.93 4.027 5.43 3.511accessed∗(F = 12.818, p < 0.001)

Note. SD = standard deviation.∗Difference across conditions was statistically significant (p < 0.05).For one-way analysis of variance on the index, degree of freedom (df) = 2

and within groups df = 81.

Workspace Awareness

In each of the conditions, the participants were given thesame kind of workspace (Figure 5). This workspace includedthe pages and the snippets saved by the team (both the partic-ipants). Thus, even the participants in condition 1, who hadno personal or group awareness, could access this workspaceand see where the team stood at a given point. Given that theworkspace was a common component among all the condi-tions, we were curious to see if having peripheral (personalor group) awareness would change how often one accessesthis workspace.

We found that on average (Table 9), the participants in con-dition 1 accessed the workspace more than those in conditions2 (p < 0.023) and 3 (p < 0.001). There was no statisticallysignificant difference for workspace accesses between con-ditions 2 and 3. This indicates that the participants withoutany kind of awareness support had to access the commonspace more often to stay updated about personal and teamprogress, thus, further providing support for the usefulnessof the peripheral awareness in collaborative projects.

Physical Effort

To measure the amount of physical work one did duringthis study, the logs were mined, and the key actions that everyteam performed were enumerated. These key actions are asfollows: viewing or saving a Web page, executing a query,

TABLE 10. Average number of key actions, chat actions, and combined actions taken by each team.

Condition 1 Condition 2 Condition 3

Mean SD Mean SD Mean SD

Key actions∗ 460.79 78.861 529.50 95.191 571.14 152.191(F = 3.393, p < 0.044)

Chat actions∗ 219.50 81.408 166.36 79.369 103.71 75.209(F = 7.592, p < 0.002)

Combined actions 680.29 115.133 695.86 92.494 674.86 172.400(F = 0.097, p < 0.908)

Note. SD = standard deviation.∗Difference across conditions was statistically significant (p < 0.05).For one-way analysis of variance on the index, degree of freedom (df) = 2 and within groups df = 39.

collecting a snippet, making or viewing an annotation, click-ing on any item in the sidebar, and accessing the workspace.A summary of the number of such actions for each conditionis reported in the second column of Table 10. As we can see,the participants in condition 3 performed more actions thanthose in condition 1 (p < 0.046).

One important action that is omitted from these key actionsis chat. Let us first look at chat action itself. The third rowin Table 10 shows the average number of chat messages sentbetween the teammates for each condition. As we see, nowthe relationships between different conditions are reversed.The participants in condition 1 exchanged significantly moremessages than those in condition 3 (p < 0.002). It was alreadyshown (Table 6) that the participants in conditions 1 and 2exchanged significantly more chat messages while reportingthe team’s progress. The observations reported in Table 10complements it by indicating a significantly higher volumeof communication for these two conditions throughout thesessions. The implications of exchanging a higher numberof messages are difficult to state; but, later in this section,we will see if the greater amount of communication resultedin a higher level of satisfaction or engagement among theparticipants.

Now, if we include chat into the other key actions (rowfour), then we find that different conditions are not very farfrom each other in terms of the number of actions performed.

Synthesizing the observations from these three columns,we can say that while the teams in different conditions tooksimilar number of key actions, those in condition 3 exploredmore information (and probably reflected more on that infor-mation as well) and spent less effort in communicating witheach other. Of course, not all communication can be consid-ered as cost; but, as we saw earlier, the teams in conditions 1and 2 sent significantly more messages (and spent more time)than those in condition 3, who tried to coordinate with andupdate each other.

Cognitive Effort

Although it seems that providing more (and appropriate)support for awareness allows the users to be more aware oftheir collaborators and progress, we also need to consider the

1982 JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY—October 2010DOI: 10.1002/asi

TABLE 11. Index for cognitive load questionnaire filled by individuals.

Condition 1 Condition 2 Condition 3

Mean SD Mean SD Mean SD

Index for cognitive 3.21 0.825 3.19 0.802 3.41 1.005load questions(F = 2.178, p < 0.115)

Note. SD = standard deviation.For one-way analysis of variance on the index, degree of freedom (df) = 2

and within groups df = 333.

effect of such support on other factors of usability. One suchfactor is perceived effort, primarily mental effort or cognitiveload.

We used six questions based on NASA’s task load indexfor measuring the perceived cognitive load.3 The questionsare given below. These questions were answered at the endof each task in each session.

Q1. How mentally demanding was this task?Q2. How physically demanding was this task?Q3. How hurried or rushed was the pace of the task?Q4. How successful were you in accomplishing what you

were asked to do?Q5. How hard did you have to work to accomplish your level

of performance?Q6. How insecure, discouraged, irritated, stressed, and

annoyed were you?

The participants were asked to rate each of the six ques-tions, from 1 (very low) to 7 (very high).An index was createdfor these questions (α = 0.730) and is reported in Table 11.

No statistically significant differences were found amongthe three conditions. This informs us that even those with amore complex interface, such as condition 3, felt no morementally loaded than those with a simple interface, such ascondition 1. This is good news for system designers, as itindicates that we could provide additional support for CISwithout adding any burden to the user.

Given that the participants in each condition did a similaramount of work, the question is: Were they happy doing it?To learn this, we employed two more questionnaires at theend of each session: one for engagement and another for easeof use and satisfaction.

Engagement

Because this study was done in an interactive environ-ment, one important factor to analyze was how engaged theparticipants felt while working on their tasks. To obtain thisfeedback, the participants were presented with the followingquestionnaire at the end of each session. This questionnaire

3The actual instrument was taken from http://www.cc.gatech.edu/classes/AY2005/cs7470_fall/papers/manual.pdf

TABLE 12. Index for engagement questions responded by individuals.

Condition 1 Condition 2 Condition 3

Mean SD Mean SD Mean SD

Index for engagement∗ 5.58 0.719 5.35 0.886 5.81 0.825(F = 4.344, p < 0.014)

Note. SD = standard deviation.∗Difference across conditions was statistically significant (p < 0.05).For one-way analysis of variance on the index, degree of freedom (df) = 2

and within groups df = 165.

was adopted from the full instrument presented in (Ghani,Supnick, & Rooney, 2007).

Using the system wasQ1. Uninteresting . . . InterestingQ2. Not enjoyable . . . EnjoyableQ3. Dull . . . ExcitingQ4. Not fun . . . Fun

How did you feel while collaborating with this system?Q5. Not absorbed intensely . . . Absorbed intenselyQ6. Attention was not focused . . . Attention was focusedQ7. Did not concentrate fully . . . Concentrated fullyQ8. Not deeply engrossed . . . Deeply engrossed

An index was created for these eight questions (α = 0.889)and reported in Table 12. It was found that the participants incondition 3 gave significantly higher ratings compared withthose in condition 2 (p < 0.015). This indicates that althoughthe condition 3 interface was more complex, the participantsfelt more engaged while working in collaboration with thesystem.

These results indicate that engagement in a user-centricCIS environment depends on not only the method of pro-viding awareness (here, peripheral awareness), but also thenature of that awareness (personal vs. group). Condition 3mirrored condition 2 in terms of the design for providingawareness (at the periphery), but the kind of awareness pro-vided in the same screen space was different (group, insteadof personal). This allowed the participants in condition 3 tocarry on their tasks without being distracted and, at the sametime, to be aware of the status of their teammates and theoverall task.

Ease of Use and Satisfaction

Finally, to obtain feedback on ease of use and satisfaction,we presented the participants with the following statementsat the end of each session and asked them to rate themon a scale of 1 (strongly disagree) to 7 (strongly agree).The questionnaire was derived from the original ComputerSystem Usability Questionnaire (Lewis, 1995),4 removing

4Also available from http://oldwww.acm.org/perlman/question.cgi?form=CSUQ

JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY—October 2010 1983DOI: 10.1002/asi

TABLE 13. Index for ease of use and satisfaction questions responded byindividuals.

Condition 1 Condition 2 Condition 3

Mean SD Mean SD Mean SD

Index for ease of use 5.99 0.661 5.04 1.789 5.31 2.120and satisfaction∗(F = 5.065, p < 0.007)

Note. SD = standard deviation.∗Difference across conditions was statistically significant (p < 0.05).For one-way analysis of variance on the index, degree of freedom (df) = 2

and within groups df = 165.

those questions that were not relevant for this study.Responses to these questions shed light on their perceivedease of use and satisfaction.

Q1. Overall, I am satisfied with how easy it is to use thissystem.

Q2. I can effectively complete my work using this system.Q3. I am able to efficiently complete my work using this

system.Q4. I feel comfortable using this system.Q5. It was easy to learn to use this system.Q6. I believe I became productive quickly using this system.Q7. It is easy to find the information I need.Q8. The information provided for the system is easy to

understand.Q9. The organization of information on the system screens

(toolbar, sidebar) is clear.Q10. The interface of this system is pleasant.Q11. I like using the interface of this system.Q12. This system has all the functions and capabilities

I expect it to have.Q13. Overall, I am satisfied with this system.

An index was created for the 13 questions (α = 0.986),which is reported in Table 13. We found it interesting to seethat compared with condition 2, participants in condition 1reported higher values (p < 0.010). One of the implicationsof these results is that although the participants in both theconditions liked the interface provided to them, they haddifferences in the expected functionalities and satisfactionfrom the system. Because they had personal awareness sup-port, they realized how valuable it could be to have similarawareness about their teammates.

No difference was found between conditions 1 and 3, and2 and 3. This, once again, indicates that a complex interface,such as condition 3, can do equally well as a simple interface,such as condition 1, when it comes to providing ease of useand satisfaction. Even though condition 3 had more infor-mation on the interface, the organization of that informationwas similar to condition 2 (on the periphery). This allowedthe participants in condition 3 to carry on their tasks withoutbeing distracted and, at the same time, to be aware of the sta-tus of their teammates and the overall task. In other words,

the condition 3 interface could provide a better support forawareness without sacrificing ease of use and satisfaction.

Design Implications

There are larger implications for designing a CIS systembased on the kind of awareness desired. To understand this,we have placed a number of systems and tools that supportCIS in Figure 8. This is based on a classical way of organizingcollaborative or groupware systems (Rodden, 1991).

The placement of a CIS system on this figure has implica-tions for its implementation, functionalities, and evaluation.For instance, Adobe Connect facilitates online meetings,where the participants can share and discuss information.Such an environment will fall under Synchronous-Remotecollaboration in Figure 8. Thus, this environment needsto have a way to connect remote participants in real timeand a shared space for exchanging and processing infor-mation. Our proposed system, Coagmento, is designed toprovide support for collaborators who work in synchronousor asynchronous mode and are primarily remotely located.This study simulated the situation of remotely located syn-chronous collaboration. We asked the participants during theinterviews what they would like to see more of in this sys-tem if they were using it for their real-life projects, and theyoffered many interesting suggestions. Because they may notbe working synchronously with their teammates most of thetime, they desired to have a shared notepad, such as ShrEdit(Olson, Storrøsten, & Carter, 1993). During the study, the par-ticipants were aware of their teammate being online and readyto respond. However, when there is no prerequisite of workingin synchronous mode in collaboration, the participants askedfor a way to know when their teammates were online. Theyalso wanted more active notifications of chat messages andother critical actions, such as collecting relevant information.As we can see, although some of these components that pro-vide awareness were not required for our lab study (colocated,synchronous), they were desired in other situations.

We started our study with a hypothesis that more aware-ness would help in making the collaborative experience moreproductive, engaged, and satisfactory for users. However, wefound that although the condition with the most awarenessperformed best overall, the condition that fell in between thehighest and the no awareness systems did the worst. This indi-cates the importance of providing an interface that matchesthe complexity and nature of the task. Because collabora-tively seeking information in an interactive and multisessionsetting is a relatively more complex activity than an indi-vidual seeking information in a single session, condition 3provided appropriate interface for the situation. Condition 1incurred significantly more cost for coordinating events andkept the participants less engaged and aware, but the par-ticipants reported high level of satisfaction and ease of use.Condition 2, on the other hand, had complex interface and didnot provide enough added benefits as condition 3 did. Thus,it achieved low ratings for usability, while not offering anysignificant benefits over the baseline.

1984 JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY—October 2010DOI: 10.1002/asi

FIG. 8. CIS systems organized according to time and space aspects.

Overall, the results of this study suggest that designersprovide support for group awareness. Although this studyfocused on synchronous collaboration, it is likely that groupawareness support will be even more important when peoplesearch together asynchronously. The results suggest that theadded complexity of group awareness can be implementedin such a way to avoid adding cognitive load to the users.

Conclusion

Web-based search services have become ubiquitous indaily life and are beginning to move beyond the simplequery-based lookup and result list paradigm. Collaborativeinformation seeking has emerged as one important kind ofextension to this paradigm. In this article, we examinedawareness in CIS projects, where the collaborators wereworking remotely in synchronous mode. We described a CISsystem, Coagmento, which was inspired by previous work onuser-centric, collaborative interfaces. Using three versionsof this system, each of which provided different kinds andlevels of awareness, we conducted a user study to addressvarious questions related to role and impact of awarenessin CIS. This study generated a large amount of data across84 participants, who conducted two exploratory search tasksspanning two sessions, and focused attention on how aware-ness functionalities affect collaborative search. The datastrongly demonstrate that basic group awareness componentsoffer substantial advantages to collaborative search withoutadding new burdens to users. In particular, we found thatalthough having personal awareness support could be usefulin multisession, exploratory search tasks, it is not enough forcollaborative projects. In our study, the participants withoutany kind of awareness support reported comparable loadsand satisfactions to those with full team awareness support;however, these searchers executed far fewer queries, exam-ined fewer pages, and used much more chat to complete theirtasks. They were able to work without the group awarenesssupport by using their memories and instant communicationcapabilities. It is likely that people could find having team

awareness even more valuable and effective in many practicalsituations where they are not working synchronously.

Another way to summarize the results of the reported studyis that we need to provide adequate and appropriate inter-face depending on the task. The version of Coagmento usedin condition 3 had complex interface with proper awarenesssupport, but because the CIS tasks and setup were com-plex enough, the participants could benefit from the addedfunctionality provided by this interface. The same interfacemay not be suitable for a single-session, information-seekingtask by an individual. The condition 2 interface had thesimilarly complex interface, but not enough awareness infor-mation. This resulted in lower levels of engagement andawareness among the participants. These results can be sum-marized as follows: (a) personal awareness is required butnot sufficient; (b) group awareness support helps the partici-pants be more efficient, engaged, and aware without raisingcognitive load; and (c) one needs to provide adequate andappropriate support for awareness given the task and thesituation.

The work reported here demonstrated the need for goodawareness support functions using Coagmento and shouldbe helpful to designers of other CIS systems. The data col-lected here bear additional analysis, especially with respectto the dynamics within pairs of collaborators using the dif-ferent awareness support services. The present work can beextended in several ways. One of them is by a field studythat allows the participants to work without the limitationsthe controlled lab study had. The participants in a field studycould work with the system as they please, creating their ownprojects of interest, and initiating collaborators with their col-leagues and friends as they see fit. Running a field studyover a long period of time (at least a few weeks) will alsoallow one to study long-term adoption effects, appropriationfactor, and specialization with various features of the system.Specifically, regarding awareness support, group awarenesscan be highly desired in many real life situations, allowingthe collaborators to be aware of each others’ current status(online/offline, present working subtask, etc.).

JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY—October 2010 1985DOI: 10.1002/asi

Acknowledgments

This work was supported by the National Science Foun-dation under grant no. IIS 0812363. We thank all of ourparticipants for their valuable time and inputs.

References

Bly, S.A., Harrison, S.R., & Irwin, S. (1993). Media spaces: Bringing peopletogether in a video, audio, and computing environment. Communicationsof the ACM, 36(1), 28–46.

Borlund, P. (2003). The IIR evaluation model: A framework forevaluation of interactive information retrieval systems. InformationResearch, 8(3).

Borlund, P., & Ingwersen, P. (1999). The application of work tasks inconnection with the evaluation of interactive information retrieval sys-tems: Empirical results. In M. Dunlop, S. Draper, K. van Rijsbergen,I. Ruthven, (Eds.), Mira ‘99 (pp. 29–46). British Computer SocietyElectronic Workshops in Computing.

Chalmers, M. (2002). Awareness, representation and interpretation. Com-puter Supported Cooperative Work, 11(3–4), 389–409.

Dourish, P., & Bellotti, V. (1992). Awareness and coordination in sharedworkspaces. In Proceedings of the Conference on Computer-SupportedCooperative Work (CSCW) (pp. 107–114). New York: ACM Press.

Dourish, P., & Bly, S.A. (1992). Portholes: Supporting awareness in a dis-tributed work group. In Proceedings of the ACM SIGCHI Conferenceon Human Factors in Computing Systems Conference on Computer-Supported Cooperative Work (CSCW) (pp. 541–547). New York: ACMPress.

Farooq, U., Ganoe, C.H., Carroll, J.M., & Giles, C.L. (2009). Design-ing for e-science: Requirements gathering for collaboration in CiteSeer.International Journal of Human Computer Studies, 67, 297–312.

Gaver, W.W. (1991). Sound support for collaboration. In Bannon, L.J.,Robinson, M., & K. Schmidt (Eds.), Proceedings of European Confer-ence on Computer Supported Cooperative Work (CSCW) (pp. 293–308).Norwell, MA: Kluwer Academic Publishers.

Gaver, W.W. (1992). The affordances of media spaces for collaboration. InJ. Turner, & R. Kraut (Eds.), Proceedings of the Conference on Computer-Supported Cooperative Work (CSCW) (pp. 17–24). New York: ACMPress.

Gaver, W.W., Moran, T.P., MacLean, A., Lövstrand, L., Dourish, P.,Carter, K., et al. (1992). Realizing a video environment: EuroPARC’sRAVE system. In Bauersfeld, P., Bennett, J., & Lynch, G. (Eds.). Pro-ceedings of ACM SIGCHI Conference on Human Factors in ComputingSystems Conference on Human Factors in Computing Systems Confer-ence on Human Factors in Computing Systems (pp. 27–35). New York:ACM Press.

Ghani, J.A., Supnick, R., & Rooney, P. (1991). The experience of flow incomputer-mediated and in face-to-face groups. Proceedings of Interna-tional Conference on Information Systems, Minneapolis (pp. 229–237).University of Minnesota.

Grudin, J. (1994). Groupware and social dynamics: Eight challenges fordevelopers. Communications of the ACM, 37(1), 92–105.

Gutwin, C., & Greenberg, S. (2002). A descriptive framework of workspaceawareness for real-time groupware. Computer Supported CooperativeWork, 11(3–4), 411–446.

Hansen, P., & Järvelin, K. (2005). Collaborative information retrieval in aninformation-intensive domain. Information Processing & Management,41, 1101–1119.

Harper, R.H., Hughes, J.A., & Shapiro, D.Z. (1989a). The functional-ity of flight strips in ATC work. The report for the Civil AviationAuthority. Technical report, Lancaster Sociotechnics Group, Departmentof Sociology, Lancaster University.

Harper, R.H., Hughes, J.A., & Shapiro, D.Z. (1989b). Working in harmony:An examination of computer technology in air traffic control. In Proceed-ings of European Conference on Computer Supported Cooperative Work(CSCW) (pp. 72–86). London: Gatwick.

Heath, C.C., & Luff, P. (1991). Collaborative activity and technologi-cal design: Task coordination in London underground control rooms.In Bannon, L.J., Robinson, M., & Schmidt, K., (Eds.), Proceedingsof European Conference on Computer Supported Cooperative Work(ECSCW) (pp. 65–80). Dordrecht, The Netherlandst: Kluwer AcademicPublishers.

Heath, C., Svensson, M.S., Hindmarsh, J., Luff, P., & vom Lehn, D. (2002).Configuring awareness. Computer Supported Cooperative Work, 11(3–4),317–347.

Lewis, J.R. (1995). IBM computer usability satisfaction questionnaires: Psy-chometric evaluation and instructions for use. International Journal ofHuman-Computer Interaction, 7(1), 57–58.

Liechti, O., & Sumi, Y. (2002). Editorial: Awareness and the WWW.International Journal of Human Computer Studies, 56(1), 1–5.

Mantei, M.M., Baecker, R.M., Sellen, A.J., Buxton, W.A.S., & Milligan,T. (1991). Experiences in the use of a media space. In S. Robertson, G.Olson, & J. Olson, (Eds.), Proceedings of ACM SIGCHI Conference onHuman Factors in Computing Systems (pp. 203–208). New York: ACMPress.

Morris, M.R. (2007). Interfaces for collaborative exploratory Web search:Motivations and directions for multi-user design. Proceedings of ACMSIGCHI 2007 Workshop on Exploratory Search and HCI: Designingand Evaluating Interfaces to Support Exploratory Search Interaction(pp. 9–12). New York: ACM Press.

Morris, M.R., & Horvitz, E. (2007). SearchTogether:An interface for collab-orative Web search. In Proceedings of the 20th annual ACM Symposiumon User Interface Software and Technology (UIST ’07) (pp. 3–10).New York: ACM Press.

Olson, J.S., Olson, G.M., Storrøsten, M., & Carter, M. (1993). Groupworkclose up: A comparison of the group design process with and without asimple group editor. ACM Transactions on Information Systems, 11(4),321–348.

Pickens, J., Golovchinsky, G., Shah, C., Qvarfordt, P., & Back, M. (2008).Algorithmic mediation for collaborative exploratory search. In Proceed-ings of the 31st Annual ACM SIGIR Conference on Research andDevelopment in Information Retrieval (pp. 315–322). New York: ACMPress.

Rodden, T. (1991). A survey of CSCW systems. Interacting with Computers,3(3), 319–353.

Schmidt, K. (2002). The problem with ‘awareness’: Introductory remarkson ‘awareness in CSCW.’ Computer Supported Cooperative Work, 11(3),285–298.

Shah, C. (2010a, February). Working in collaboration—What, why, andhow? Paper presented at the Conference on Computer-Supported Coop-erative Work (CSCW), 2010 Workshop on Collaborative InformationSeeking, Savannah, GA. Retrieved June 14, 2010, from http://workshops.fxpal.com/cscw2010cis/submissions/tmp11.pdf

Shah, C. (2010b, February). Coagmento—A collaborative information seek-ing, synthesis and sense-making framework. Integrated demo at theConference on Computer-Supported Cooperative Work (CSCW), 2010Workshop on Collaborative Information Seeking, Savannah, GA.

Shah, C., Marchionini, G., & Kelly, D. (2009). Learning design principlesfor a collaborative information seeking system. In Extended Proceedingsof ACM SIGCHI Conference (pp. 3419–3424). New York: ACM Press.

Simone, C., & Bandini, S. (2002). Integrating awareness in cooperative appli-cations through the reaction-diffusion metaphor. Computer SupportedCooperative Work, 11(3–4), 495–530.

Twidale, M.B., & Nichols, D.M. (1996). Collaborative browsing andvisualisation of the search process. Proceedings of Aslib, 48,177–182.Retrieved June 10, 2010, from http://www.comp.lancs.ac.uk/computing/research/cseg/projects/ariadne/docs/elvira96.html

Twidale, M.B., Nichols, D.M., & Paice, C.D. (1995). Supporting collabora-tive learning during information searching. In Proceedings of ComputerSupported Collaborative Learning (pp. 367–374). Hillsdale, NJ: LawrenceErlbaum Associates, Inc.

Twidale, M.B., Nichols, D.M., & Paice, C.D. (1997). Browsing is acollaborative process. Information Processing & Management, 33(6),761–783.

1986 JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY—October 2010DOI: 10.1002/asi