Designing and evaluating online communities: research speaks to emerging practice

2 Int. J. Web Based Communities, Vol. 1, No. 1, 2004

Copyright © 2004 Inderscience Enterprises Ltd.

Designing and evaluating online communities: research speaks to emerging practice

Jenny Preece University of Maryland Baltimore County (UMBC), 1000 Hilltop Circle, Baltimore, MD 21250, USA E-mail: [email protected]

Chadia Abras Goucher College, 1021 Dulaney Valley Road, Baltimore, Maryland 21204, USA E-mail: [email protected]

Diane Maloney-Krichmar Bowie State University, 14000 Jericho Park Road, Bowie, Maryland, USA E-mail: [email protected]

Abstract: What makes online communities successful? There are many indicators of success. Some are easily observed and measured, such as the number of people registered in the community, or the number who participate, or the number of messages posted, replied to or read over a certain period. How community members judge their community is also important. What do they like or dislike and how committed are they to the community? In this paper, we describe participatory community-centred development – an approach that relies heavily on iterative evaluation. We then point out that standard evaluation techniques are inadequate for evaluating online communities because they do not address sociability. Finally, we propose two approaches from research that can be tailed for evaluating online communities. Both attempt to draw directly on what community users want. The first is rooted in ethnography and the second is a form of heuristic evaluation. The paper ends with an agenda for developing these approaches to make them practical yet robust.

Keywords: evaluation; online communities; heuristics; ethnography; sociability; usability.

Reference to this paper should be made as follows: Preece, J., Abras, C. and Maloney-Krichmar D. (2004) ‘Designing and evaluating online communities: research speaks to emerging practice’, Int. J. Web Based Communities, Vol. 1, No. 1, pp.2–18.

Biographical notes: Jenny Preece, PhD, is a Professor of Information Systems at the University of Maryland, Baltimore, USA. Dr. Preece is the author of seven books and numerous papers and book chapters. Her books include Online Communities: Designing Usability, Supporting Sociability (2000) and a co-authored text with Yvonne Rogers and Helen Sharp entitled Interaction Design: Beyond Human-Computer Interaction (2002). John Wiley and Sons publishes the two books.

Designing and evaluating online communities 3

Chadia Abras, PhD, is an Assistant Professor of French at Goucher College, Baltimore USA. Dr. Abras is developing heuristics for evaluating a range of different online communities including education communities for students learning modern languages.

Diane Maloney-Krichmar, PhD, is an Assistant Dean at Bowie-State University in Maryland, USA. Dr. Maloney-Krichmar is continuing her work on patient support communities.

1 Introduction

Online communities have a ‘purpose’, are supported by ‘technology’, and are guided by ‘policies’ [1]. Purposes can vary. Online communities that support professional discussion and work are referred to as communities of practice (COPs) [2]. Online communities that support interest groups such as dog-owners, gardening, football, bridge, and book discussions are often called communities of interest (COIs). More descriptive, generic names are frequently used too, including patient support communities, education communities, and political communities.

We use the term ‘online community’ broadly to refer to all communities that have some kind of online presence, but we acknowledge that it is important to recognise the dimensions along which communities differ because these can have a profound influence upon their nature and success. Some of these dimensions include:

• Whether the community exists only virtually, or has a physical presence, or exists primarily through physical connections, e.g., a networked neighbourhood or a mobile telephone-texting community.

• The primary purpose for the community existing, e.g., health support, education, business, and neighbourhood activities.

• The type of software environments that support the community, e.g., listserver, bulletin boards, chat, instant messaging, internet telephone, streaming video, photographs, sound, voice, webcam, blogs, and wikis (an open source collaborative server technology that enables users to access, browse, and edit HyperText pages in a real-time context).

• The size and for how long the community has existed. Small communities of 25 people are different from those comprising1,000 or more people and communities that have existed for several years such as Bob’s WWWACL Board [3] are different from those that spring up over issues like demonstrating against World Trade Organisation meets.

Since developing successful online communities require attention to sociability and usability, we start by reviewing approaches for developing online communities. In particular, we focus on participatory community-centred development for two reasons. The central premises of this approach are: (i) to involve the community, and (ii) to evaluate frequently throughout the community development process. In section three, we review the advantages and disadvantages of using current techniques to evaluate online communities. Sections four and five briefly discuss two research projects that suggest

4 J. Preece, C. Abras and D. Maloney-Krichmar

new evaluation approaches. The paper ends with conclusions about the need for evaluation techniques that address sociability as well as usability, and an agenda outlining subsequent steps for converting these approaches into robust, practical techniques for developers and moderators.

2 Background: developing online communities

Developing online communities is different from designing software. Once software is shipped it is stable until the next upgrade. Users can tailor the software, but they cannot fundamentally change their interaction experience. Participating in an online community is first and foremost a social experience that changes according to who is present, the number of people involved, and the type of discussion that occurs. For example, as we have said above, the ambiance of a new, small community is quite different from a large community that has existed for many years. Even if the software does not change, social interactions will change. For example, communities with a tradition of aggressive responses or put-downs, if they manage to exist at all, will be more like arenas in which there is a relatively small number of participants with many onlookers [4]. Other communities may evolve over time to develop strong norms of acceptable behaviour [3,5]. Both are examples of social adaptations, that is, sociability. In many communities these social changes are at least as influential as software changes. It is therefore essential that evaluation techniques should focus on sociability as well as usability. For this reason, online community developers are looking to theory and practice from sociology, social psychology, anthropology and other disciplines to guide their design practices, as well as traditional user-centred design approaches [6] that have become de facto standards in human-computer interactions.

Most designers and developers agree that it is essential to understand the audience for whom you are designing. Who is this audience? What do the designers and developers need to do? This research can be done through questionnaires and focus groups [7,8]. In participatory community-centred development methodology (see Figure 1), the community members are consulted and involved in the design right from the start [1,9,10]. It is modeled after Norman’s [6] user-centred design model that focuses on the user, rather than the technology.

Each step of the design is followed by an evaluation period and redesign of the community according to the findings. Step one in the design involves understanding the needs of the community and the developer must ask specifically what kind of discussion space the community wants, what the participants’ internet habits are, and how they actually read and send messages. To be able to assess the users’ needs, the developer must understand who the members of the community are and what the main purpose of the community is intended to be. These issues are the foundation for developing the community.

Step two addresses selecting the technology for communication, such as discussion boards or chat areas, and tailoring the software to fit the needs of the intended community. Developers with strong software skills may opt to create their own software environments, whereas developers with little or no programming expertise can select from the platforms and services available on the internet; for example, Yahoo groups or MSN communities. The drawback with the latter approach is that there is little scope for


the community to modify its software environment. A happy compromise for many developers is to adapt one of the many open source software products now available free of charge on the internet. At the same time, the developer is planning for sociability by working with core community members to determine policies and early social structures. These need to be flexible so that they can adapt and change as the community grows and itself takes ownership.

Figure 1 Participatory community-centred development model

Step three requires testing the prototype with users, including all the intended software and sociability structures. This includes the representation of any social policies and social interaction support. For example, if there is a registration policy with a form that members must complete, it must be tested to make sure that it is understandable and usable by the intended membership. If participants are to be represented by an avatar or some other kind of graphical representation, what form will this take and is it acceptable to the membership? In this step, iterative evaluation is done through many tests involving several small tasks and followed by changes and retesting as found appropriate. Step four is more detailed and relies on in-depth usability testing of webdesign, consistency of language, fonts, and navigation. At this stage, more detailed sociability testing is also done that focuses particularly on examining policies, refining the statement of the community’s purpose and representing the community’s identity.

Step five, nurturing the community, is the most important step and is an ongoing process as long as the online community exists. The community has to be well advertised, the members welcomed, supported, and nurtured to ensure that they will come back. Moderators and leaders assume and refine their roles. Evaluation and change in the design should reflect the dynamic nature of the community. This process is very important throughout a community’s life. The software provides the place and structure in which a group of people can evolve to create a community. Community development


is iterative by nature and this development should reflect an evolving community [1, p.209–211].

Each developer and designer has a different view of what one should focus on to ensure success in an online community. Kim [7] and Preece [1] see the purpose as the most important issue in community design. Powazek [8] sees the content as a major issue in the design of the community. Content and purpose are closely related because for the content to be successful it has to serve a purpose. Kollock [11] sees several elements as being crucial for fostering community such as methods for assuring members’ identity persistence, recording and displaying the history of the community, and a coherent sense of space.

A participatory community-centred development approach ensures that the members are heard and their ideas and desires are incorporated in the design. However, Kim [7] warns that people are inaccurate in self-reporting. She suggests that the designer should listen to conversational feedback reported through surveys and interviews and rely on behavioral feedback reported through logging and observations. Also, choosing communication software that meets the needs of the community is important. In the end, all designers agree that designing for trust is what keeps the members coming back and fosters a sense of belonging and intimacy. Developing a social identity is closely related to trust if the members know whom they are dealing with. They can develop a sense of security and are more apt to trust people with whom they can interact [7].

Participatory community-community centered design is based on an iterative design-evaluate-redesign paradigm. In Figure 1, ‘evaluation’ is aptly placed in the centre of the diagram and the main stages of design and evaluation are shown. In practice, however, many developers will carry out more evaluations than the diagram suggests. Some of these are informal ‘quick and dirty’ evaluations in which ideas are discussed with or shown to a small group of users. Other types of evaluation are more formal. While tried and tested evaluation techniques can be used, they usually have to be adapted to suit online communities. There are two key reasons why standard techniques are of limited value. First, it is often not possible to bring evaluators and users together to test software in a laboratory or to participate in a face-to-face focus group because they are distributed in both space and time. There may also be language and cultural differences. Second, while most evaluation techniques address only usability, good sociability is central to the success of online communities and must be evaluated too.

3 Using established techniques for evaluating online communities

The traditional approach for evaluating users’ satisfaction and the efficiency with software is usability testing, which focuses on how well the user interface supports human-computer interactions (HCI) [12]. Usability testing depends upon evaluating users on clearly specified tasks in a controlled laboratory environment, which is not practical for evaluating most online communities.

While some attention has focused on developing new evaluation methods and adapting existing ones, there is still a paucity of techniques. For example, efforts to understand who participates and what they do through visualising results from mining data logs using tools such as Netscan [13–15] are impressive. However, these techniques, compelling though they are, speak only of what happens; they tell little about why. In


HCI, evaluation can be formative in order to check if the product being developed meets users’ needs, or summative in order to assess the success of a finished product and to check if specified standards are being upheld [12]. A brief review of four evaluation paradigms would help to indicate their potential for evaluating online communities:

• Quick and dirty evaluation can be done at any stage of the design and usually involves informal meetings with users to get feedback on the product [16]. The techniques used provide a form of quick, low cost evaluation [12,17]. This approach can be used at any stage in the development of online communities to address usability and sociability, provided users can be located. It is particularly useful in the early stages of development. While computer mediated communication techniques are useful for contacting remote users, running effective focus groups online is more difficult than conducting face-to-face interviews.

• Usability testing measures typical users’ performance in carefully prepared tasks. The tester is in control and the users are videotaped and monitored while performing these tasks. The test is performed with intended users, doing set tasks, in a controlled environment. The outcome is measured by the number of errors users make and time spent on the task [12,18]. Usually these tests are accompanied by surveys, interviews, focus groups discussions, and logging, in which users’ satisfaction is also evaluated [12,18]. Usability testing is useful for evaluating the software that supports online communities, but it was not developed to evaluate sociability.

• Field studies are done in the natural setting of the user. The tools for evaluation are mostly qualitative techniques such as interviews, participant observations, and ethnography. The data is recorded through audio or video, or notes are taken by an observer and later analysed in a variety of ways [12,18]. It is recommended that logging techniques be employed to capture error commands and productivity measures [18]. One type of field study that is widely accepted in studying online communities is ethnography. Many researchers such as Baym [19] , Markham [20], Cherney [21] and Hakken [22] studied online communities through an ethnographic approach. However, in order for ethnography to be used successfully online, it has to be adapted for virtual environments. For example, several researchers use combinations of face-to-face and online interviews because they do not have access to all participants offline. Ethnography and qualitative research methods are particularly well suited for understanding social interaction and sociability. To make these approaches more acceptable to online developers, they need to be ‘packaged’ so that they are practical.

• Predictive evaluation is guided by heuristicsso that usability problems can be predicted [17,23,24]. Existing usability guidelines are not always accurate for testing any HCI productand so new heuristic guidelines are needed to test a range of interactive products, including online communities. Also, as with the other approaches, they need to be geared specifically for evaluating sociability [1,25].

A combination of these approaches may be used to evaluate online communities, depending on the characteristics of the particular communities. Which approach and techniques to choose depends on the questions to be answered, how much time is allowed, access to participants, equipment and evaluators, and the evaluation budget. However, even the techniques described do not adequately address the need to evaluate


sociability and usability. This is because even most computer-supported cooperative work (CSCW) systems have a strong task and work focus and pay comparatively little attention to sociability. In contrast, sociability overshadows usability in many online communities. Provided that the software is usable and users can perform basic tasks such as registering, reading and sending messages, and performing basic archival searches, they are satisfied with software that to experts may appear to have limited functionality. For example, in one online community studied, interviews with participants revealed that they were happy with an older product and saw no need for upgrading to state-of-the-art software. The major concern for these participants was 24/7 availability of the server and consequent access to their community [3,5].

Surprisingly, only some research has addressed the need to identify and develop techniques for evaluating usability and sociability in online communities. In the next two sections we describe two research studies that suggest ways of evaluating usability and sociability online. The first is a multi-level analysis of a single online community in which an ethnographic approach was adopted [3,5]. The second study involved developing and testing emergent heuristics for evaluating sociability and usability that would guide online community design and evaluation [26,27].

4 Study 1: a multi-level evaluation of an online patient support community

This study examined an online support community for knee injury patients that is supported by a bulletin board and a website. This online community is particularly interesting because of its longevity (it has been in existence for almost eight years), the high level of activity within the community (it averages several hundred messages per week), and strong evidence of social support and trust within the community. Members of the community ask and answer medical questions and provide support and encouragement for each other. There is little sign of aggression or inappropriate comments on the bulletin board and, in rare instances when these occur, community members take swift action to point out that the community does not accept this type of behaviour. This norm is sufficiently strong and so formal netiquette policies are not needed.

4.1 Usability and sociability reviews

One of the research questions addressed in this study was concerned with how usability and sociability affects social and technological aspects of online community dynamics. The purpose was to identify those factors that support effective and meaningful online interaction [3,5]. In order to evaluate the community’s usability and sociability, the researcher (Maloney-Krichmar) and expert reviewers used the heuristic guidelines for evaluating online communities presented in Online Communities: Designing Usability, Supporting Sociability [1] and in a subsequent paper [25]. These guidelines represent an early attempt to identify key issues of importance to online community users. The analysis addressed the impact of sociability and usability design and management decisions on the Kneeboard. Social policies are a form of sociability that are particularly important for providing a framework for social interaction within the community and fall under three broad categories: (1) purpose, policies, and procedures; (2) administration;


and (3) encouraging the development of social presence, trust, and empathy. Online community developers and designers believe that providing clear statements about social facilitates are a key for successful social interactions. The usability heuristics were drawn from Nielsen [24] and focused on navigation, access, dialog and information design [1,25].

The evaluators found that the online health community site in its design did not address issues related to sociability very well. However, they noted that the lack of clear statements related to purpose, policies, procedures, administration, moderation, or registration on the Kneeboard do not appear to affect the community in a negative way. Levels of social interaction within the community were strong as indicated by the high level of message posting and viewing. At the time of the study, there were an average of 494 messages posted to the online health community per week; 3.6 postings per active member per week; each message received an average of 43 readings; and, average thread length was 5.8 messages. There were no rules, regulations, or guidelines concerning acceptable levels of free speech or online etiquette and the community functioned with a very low level of moderating. Yet, the community had extremely low levels of aggressive or hostile behaviour [3]. Table 1 presents an overview of the status of key sociability features on the Kneeboard.

Table 1 Key sociability features

Key Sociability Features Present in the online community Statement of purpose No Statement identifying site administrator Present – but weak Statement of moderation policy No Guidelines for netiquette (online etiquette) No Registration policy No

The usability analysis revealed that the community’s technical environment is relatively simple and easy to use, especially when viewing or posting messages which is the major activity on the site. There is good navigational support on the site and moving within the pages of the site is easy and fast. Due to the simplicity of the website design, standard use of colours, and limited use of graphics, download times for the site are within the suggested guidelines for using a variety of computers and browsers. Nonetheless, there are usability problems as users go deeper into some pages or follow links offsite. However, these problems did not interfere with the main focus of the community, which is, the exchange of messages [3].

The major technical issue for the community was when the server broke down and members could not access the bulletin board. Once access was reestablished, a flurry of messages ensued pleading that server breakdown be avoided in the future. In an interview, one community member summed up this concern:

“…when you need to talk to someone and the server went down, you sort of felt more isolated than was actually the case. The first time I experienced losing the server was a three-day event. I dreaded it and finally contacted the server host to learn what the problem was. Turned out I wasn’t the only one having trouble breathing. Many others had done the same (Peter, 7/15/02).”


This comment reflects the opinion expressed by a majority of the twenty (out of 139 active members during November 11–17, 2001) community members interviewed that the most important function of the Kneeboard is to provide a reliable means of communication for its members. Perhaps there is better or different software of which the community is not aware, but their attitude to suggestions of change can be summed up in thesaying, If it isn’t broke, don’t fix it. This may cause a dilemma for human-computer interaction specialists, some of whom may argue that if only users knew what they were missing, they would want the newer technology. However, any change should be made with prudence. The premise of sociability and usability guidelines is that certain minimum standards should be met to insure the success of an online community. However, the online patient support community did not meet many of the criteria,yet, members were extremely satisfied with the community.

The lesson to take away from these findings is that sociability features may not be obvious. In well functioning communities, norms tend to be developed that deal with issues more subtly than the explicit policies that help to set standards for behaviour in new online communities. Another message from this research is that the criteria for success should be derived from users whose opinions may differ from experts. In this case, contrary to popular belief, state of the art technology was not necessary for a successful community – as judged by the community itself.

4.2 Ethnographic observations and interview

The study also took a broad fieldwork approach guided by the ethnographic research techniques of observation, interviewing, and archival research [28–31]. In addition to the sociability and usability heuristic evaluation, messages posted for a one-week period in November 2001 (over 400 messages) were coded using group membership roles analysis [32], interaction process analysis [33] and social network analysis [34]. Additionally, 20 community members were interviewed online. The ethnographic research approach focuses on the ways all aspects of a culture are related [35]. This approach was adapted for use in a textual environment on the internet following the examples of Baym [19,36], Hine [37], and Silver [38]. The study utilised a multi level, in-depth analysis.

Each phase of the research examines different aspects of the social environment and the interactions within the community. By combining a long-term immersion in the community as an observer with systematic data collection, analysis, cross checking and triangulation of observations and data, a thick and rich description of the culture, norms, and dynamic processes of the Kneeboard emerged. The advantages of using this method for evaluating an online community are:

• it provides a theoretical framework for the study that helps to contextualise the data in each phase of the research [31]

• the ethnographic approach focuses on understanding the culture of the online community from an insider’s point of view

• the holistic nature of the ethnographic approach, in which findings are contextualised, allows the researcher to develop a deep understanding of the culture of the online community and to fully understand how issues related to sociability evolve and are expressed.


Ethnographic research is becoming an increasingly popular method for studying the internet because of the unique way in which it can be used ‘to develop an enriched sense of the meaning of the technology and the cultures which enable it and are enabled by it’, [37, p.8].But, the major disadvantage of an ethnographic approach is that it requires a long-term commitment on the part of the researcher. It was only through careful observation and investigation that we were able to develop a deep understanding of the social interaction and mediating effects of technology on the members of the community.

This study suggests some of the limitations that developers would encounter if they employ ethnography to evaluate online communities. The main constraint is that to understand a community in depth, to observe its norms and to become aware of subtle interactions takes time which most developers do not have. However, for community moderators and managers, a quick and practical application of this research could help them assess the functioning of an existing group that may be having problems. For example, a week’s worth of messages can be examined using group membership role and interaction process analysis to provide a profile of the community. If the purpose of the community is to provide information and support and it exhibits a majority socio-emotional interaction, members are not likely to be satisfied with the community. The community moderator ormanager could invite a healthcare expert to lead a discussion on some aspects of the illness or injury in order to promote a more balanced communityinteraction. If a high number of messages are aggressive or hostile, it would be wise to post guidelines for conduct and/or institute a filter to cope with interactions that block effective group functioning until the group has developed sufficiently to self-monitor this type of behaviour. The application of the research tools used in this study to provide a snapshot of group interaction, could be tailored to provide an effective method for evaluating social interaction within an online community.

5 Study 2: emergent community-centred heuristics for evaluating online communities

The results of the previous study and our observations over the years suggested that we should attempt to understand what community members themselves want from their communities and to use this information as a basis for evaluating the success of the online community. Such an approach is also synergistic with our participatory community-centred development philosophy. Therefore, we set out to derive a set of community-centred heuristics for evaluating the success of online health communities [26,27]. These heuristics would also provide a valuable guide for online community development.

This approach was stimulated by the knowledge that compared to other types of evaluation, heuristic evaluations (HE) are cheaper because one does not need a room or special equipment, nor do evaluators need to be co-located with real users. These benefits suggest that HE may offer promise for evaluating online communities. Most of the advantages and disadvantages that we foresee have already been reported for heuristic-based usability inspections with other types of software. Positive support and negative support for heuristic evaluations (HE) seem to be intertwined. Several of the studies we reviewed presented the negative side of HE, but at the same time, they suggested solutions for its shortcomings [39–41]. Others, such as Sears [41] argued that HE has low validity because a significant number of false positives are identified, which


can be attributed to evaluators who apply each heuristic without considering its impact on potential users. Therefore, in this study, we drew the list of usability and sociability heuristics directly from the users’ perspectives, in addition to established sources [23,24], in order to reduce the occurrence of heuristics that the users consider irrelevant for this context. Our approach also addresses a criticism that in HE evaluators are surrogate users and that data obtained through HE is therefore suspect since it does not involve users directly in the testing [40]. As a solution, it has been proposed that user interviews are added to HE in to give the usability specialist insight about user needs [39].

The original list of usability heuristics first introduced by Molich and Nielsen [23] was later refined by Nielsen [24] to include ten usability heuristics. In this study, we adapted Nielsen’s heuristics to fit online communities and we adopted the sociability heuristics that were described by Preece [1,25] as a set to be tested and validated. Both sets of heuristics were then turned into a questionnaire (G) that was iteratively tested with other online communities in order to refine the heuristics and tailor them for online health support communities. Each heuristic was turned into one or many questions as appropriate. The questions solicit answers on a seven-point Likert scale that ranges between not important to extremely important, and an added point for not applicable (NA).

The study draws its data from the users’ perspectives and at each testing, iteration users are interviewed after they have completed the questionnaire containing open-ended questions ito solicit opinions that are not addressed in the questionnaire. The results of the interviews and the open-ended questions are then coded and a list of new questions is identified for addition to a new questionnaire (questionnaire H) for the next iterative testing stage. Questionnaire H is the final questionnaire for health online communities. In all, three health online communities were tested. Testing ended when the interviews did not yield new questions. The research design is presented in Figure 2 below and described in detail elsewhere [26,27].

The results from the final questionnaire were clustered and two sets of heuristics were identified, one for usability and one for sociability among online health support communities. It is recommended that developers use the heuristics as a guide to build the communities. The final questionnaire is meant to further explore and evaluate whether the targeted audience has more specific needs than the community currently supports through its usability and sociability design.

The usability heuristics that were adapted from Nielsen’s [17,24] were quite stable and only one new heuristic with two components concerned with reliability was identified (see Table 2). In retrospect, these results were to be expected since usability is concerned with individual human-computer interaction across the human-computer interface.

In strong contrast, the sociability heuristics varied considerably from the original set. The iterative testing yielded 48 items and eight heuristics as seen in Table 3 below.

When these heuristics were used with education communities we found that the usability heuristics changed very little. However, as expected, the sociability heuristics were more community specific and they varied between the two types of communities [26,27]. These and the earlier results suggest that the basic heuristics offer an emergent set of heuristics that developers and moderators can use to develop questionnaires and interviews to evaluate specific communities.


Figure 2 Iterative testing research design

Table 2 Usability heuristics for health online communities

Usability Heuristics Consistency and Accessibility Consistency Consistent Navigation Familiar Language Necessary navigation buttons provided on each page

Accessibility Able to find the site using any search engine Archived discussions easily accessible Site and board are useful

Navigation Easy navigation Layout of the site and board intuitive Original search page easily accessible Members able to obtain information easily


Table 2 Usability heuristics for health online communities (Continued)

Usability Heuristics Organisation Familiar icons Layout of discussion board organised by topic or string depending on need Directions on how to use the site and board are provided

User Control Feeling in charge of the system Fast pace of interaction Interaction with site and board is similar

Reliability Site and board are accessible at all times Search sequence easy to remember

Table 3 Sociability heuristics for online health communities

Sociability Heuristics Members and Community Discussions

New topics introduced regularly Dynamics of interactions like f2f interactions Natural and active discussions Deep or light discussions depending on topic

Community’s Commitment

Community keeps users interested Community able to attract new members Information on the website updated regularly Community should reach out to lurkers Being able to ask questions in a secure environment Members in control of discussions, not moderators

Members’ Commitment

Members visit the community frequently Members committed to the community Members participate in discussions when necessary Reading messages without posting should be allowed

Connections

Strong ties between members not important Weak ties between members is important Need to feel connected to others


Table 3 Sociability heuristics for online health communities (Continued)

Sociability Heuristics

Policies and purpose Policies Community has strict rules of behaviour Feedback from other members is important Clear policies Prominently displayed policies Policies being enforced Website contains valuable information and links

Purpose Purpose clearly stated Purpose relevant to members’ lives

Social Representation Purpose changes and evolves with the community Putting one’s photograph or avatar on the board not important Community provides a sense of belonging Community adapts and changes to fit the needs of the group

Empathy and Support Password needed to access the community Need to feel support Need to feel empathy Need a caring atmosphere Members should have user profiles Members should have a consistent identity

Trust and Privacy Discussions are positive Trust that the medical information provided is not shared with others Trust the information provided Privacy of my medical information

Feedback Need to share experience with others Feedback from moderators not necessary Moderators are active in the discussions only to guard against flaming

Interesting Topics Topics of discussions are interesting and helpful

Information Information provided by moderators and administrators is accurate


Drawing the heuristics from the perspectives of users is a major strength of this approach, since the developer, when relying on these heuristics, will be designing according to the wishes of the community users. But this advantage may also be a weakness since it has been shown that sometimes users are not very reliable in self-reporting [7]. Therefore, combining user-derived heuristics with expert-derived heuristics may offer the best solution.

6 Conclusions

Several conclusions can be drawn from this discussion. The first is that participatory community-centre development approach, like other user-centred and participatory methods, relies heavily on evaluation for success. The second is that online communities are quite different from most types of software in that an online community is more than just software. The community evolves according to how people interact with each other using software to support their interactions. Sociability and usability are therefore important when designing and evaluating online communities. Third, little attention has been directed to developing effective, practical methods to evaluate collaborative systems. Fourth, one way of rectifying this problem is to look at how established evaluation techniques can be modified. A brief analysis reveals that most methods can be adapted to a limited extent to evaluate usability. However, there are no standards for evaluating sociability. Therefore, one way forward is to examine techniques used in research that aim to understand what makes online communities successful and attempt to make the most promising ones practically applicable by community developers and moderators. Fifth, the two studies that we conducted suggest that identifying heuristics with which to evaluate online communities is one such fruitful approach. The rich in-depth understanding gained from the ethnographic research techniques used in the first study also suggests that we need to develop ethnographically-oriented observation practices for online community moderators.

The next step in our research agenda is to work on both approaches to develop robust, yet practical, approaches for online community developers, moderators, evaluators, and community users. Another approach that we advocate is developing tools to make it easy to examine logging data (data mining) like those developed by Smith and his colleagues [13–15]. A suite of techniques based on our emergent heuristics and the ethnographically-oriented approach used in our first study, together with data mining and visualisation tools, would empower online community developers to evaluate both usability and sociability more effectively. A mixed technique approach has been explored with some success for work-oriented computer supported cooperative work systems [42]. Without better methods for evaluating the sociability and usability of online communities, the success of these systems will continue to be judged superficially. Community-derived heuristics and ethnographically oriented techniques are needed to explain why the numbers obtained from mining data logs are as they are.


References and Notes 1 Preece, J. (2000) Online Communities: Designing Usability, Supporting Sociability,

Chichester, England, John Wiley & Sons. 2 Wenger, E. (1999) Communities of Practice: Learning, Meaning, and Identity, Cambridge,

UK, Cambridge University Press. 3 Maloney-Krichmar, D. (2003) ‘The meaning of online health community in the lives of its

members: roles, relationships and group dynamics’, Unpublished Doctoral Dissertation, UMBC, Baltimore.

4 Preece, J. (2004) ‘Etiquette online: from nice to necessary’, Communication of the ACM, Vol. 47, No. 4, pp.55–61.

5 Maloney-Krichmar, D. and Preece, J. (2004) A Multilevel Analysis of Sociability, Usability and Community Dynamics in an Online Health Support Community (in press).

6 Norman, D.A. (1986) ‘Cognitive engineering’, User-Centered Systems Design, Hillsdale, NJ, Lawrence Erlbaum Associates.

7 Kim, A.J. (2000) Community Building on the Web: Secret Strategies for Successful Online Communities Berkeley, CA, Peachpit Press.

8 Powazek, D. (2002) Design for Community: The Art of Connecting Real People in Virtual Places, Indianapolis, New Riders.

9 Muller, M.J. (1992) ‘Retrospective on a year of participatory design using the PICTIVE technique’, Paper presented at the Proc. CHI’92 Human Factors in Computing Systems.

10 Schuler, D. and Namioka, A. (1993) Participatory Design: Principles and Practices, Mahwah, NJ, Lawrence Erlbaum.

11 Kollock, P. (1998) ‘Design principles in online communities’, PC Update, Vol. 15, pp.58–60.

12 Preece, J., Rogers, Y. and Sharp, H. (2002) Interaction Design: Beyond Human-Computer Interaction, New York, NY, John Wiley & Sons.

13 Fiore, A., Teirnan, S.L. and Smith, M. (2001) ‘Observed behavior and perceived value of authors in usenet newsgroups: bridging the gap’, (Available November 2003: http://research.microsoft.com/~masmith/).

14 Fiore, A. and Smith, M. (2001) ‘Tree map visualizations of newsgroups’, (Available November 2003: http://research.microsoft.com/~masmith/)

15 Smith, M. and Fiore, A. (2001) ‘Visualization components for persistent conversations’, Proceedings of ACM SIG CHI Conference.

16 Michael, T., Randall, D. and Bentley, R. (1994) ‘Situated evaluation for cooperative systems’, Proceedings of the 1994 ACM Conference on Computer Supported Cooperative Work.

17 Nielsen, J. (1993) Usability Engineering, Boston, AP Professional. 18 Shneiderman, B. (1998) Designing the User Interface: Strategies for Effective Human-

Computer Interaction, 3rd Edition, Reading, MA, Addison-Wesley. 19 Baym, N. (1997) ‘Interpreting soap operas and creating community: inside an electronic fan

culture’, in Kiesler, S. (Ed.): Culture of the Internet, Mahwah, NJ, Lawrence Erlbaum Associates, pp.103–119.

20 Markham, A.N. (1998) Life Online: Researching Real Experience in Virtual Space, Walnut Creek, CA, Altamira.

21 Cherny, L. (1999) Conversation and Community: Chat in a Virtual World, Stanford, CA, CSLI Publications.

22 Hakken, D. (1999) Cyborgs @ Cyberspace? An Ethnographer Looks to the Future, New York, Routledge.

23 Molich, R. and Nielsen, J. (1990) ‘Improving a human-computer dialogue’, Communications of the ACM, Vol. 33, No. 3, pp.338–348.


24 Nielsen, J. (1994) ‘Heuristic evaluation’, Usability Inspection Methods, New York, John Wiley & Sons.

25 Preece, J. (2001) ‘Sociability and usability in online communities: determining and measuring success’, Behaviour & Information Technology, Vol. 3, No. 5, pp.347–356.

26 Abras, C. (2003) ‘Determining success in online academic and health communities: developing usability and sociability heuristics’, Unpublished Doctoral Dissertation, UMBC, Baltimore.

27 Abras, C., Ozok, A. and Preece, J. (2004) ‘Emerging heuristics for evaluating online health and education communities’, (in press).

28 Fetterman, D.M. (1998) Ethnography: Step by step, Second Edition, Vol. 17, Thousand Oaks, CA, Sage Publications.

29 LeCompte, M.D. and Schensul, J. (1999) Analyzing and Interpreting Ethnographic Data, Vol. 5, Walnut Creek, CA, AltaMira Press.

30 Miles, M.B. and Huberman, A.M. (1994) An Expanded Sourcebook: Qualitative data analysis (2nd Edition), London, Sage Publications.

31 Wolcott, H. (1999) Ethnography: A Way of Seeing, Walnut Creek, CA, AltaMira Press. 32 Benne, K.D. and Sheats, P. (1948) ‘Functional roles of group members’, Journal of Social

Issues, Vol. 4, No. 2, pp.41–19. 33 Bales, R.F. (1958) ‘Task roles and social roles in problem-solving groups’, in MaCcoby,

E.E., Newcomb, T.M. and Hartley, E.L. (Eds.): Readings in Social Psychology, New York, Holt, Rinehart & Winston, pp.437–458.

34 Wellman, B. and Frank, K. (2001) ‘Getting support from personal communities’, in Lin, N., Burt, R. and Cook, K. (Eds.): Social Capital: Theory and Research, Chicago, IL, Aldine De Gruytere.

35 Nardi, B.A. (1997) ‘The use of ethnographic methods design and evaluation’, in Helander, M.G.T., Landauer, K. and Prabhu, P. (Eds.): Handbook of Human-Computer Interaction (2nd Edition), Amsterdam, Netherlands, Elsevier Science B.V., pp.361–366.

36 Baym, N. (2000) Tune in, log on: Soaps, Fandom, and Online Community, Thousand Oaks, CA, Sage Publications.

37 Hine, C. (2000) Virtual Ethnography, Thousand Oaks, CA, Sage Publications. 38 Silver, D. (1999) ‘Localizing the global village: lessons from the blacksburg electronic

village’, in Brown, R.B. and Fishwick, M.W. (Eds.): The Global Village: Dead or Alive?, Bowling Green, OH, Popular Press, pp.79–92.

39 Jeffries, R., Miller, J., Wharton, C. and Uyeda, K. (1991) ‘User interface evaluation in the real world: a comparison of four techniques’, CHI 91, Conference Proceedings, ACM Press, pp.119–124.

40 Kanter, L. and Rosenbaum, S. (1997) ‘Usability studies of www sites: heuristic evaluation vs. laboratory testing’, Proceedings of the 15th Annual International Conference on Computer Documentation, ACM Press, pp.153–160.

41 Sears, A. (1997) ‘Heuristic walkthroughs: finding the problems without the noise’, International Journal of Human-Computer Interaction, Vol. 9, No. 3, pp.213–234.

42 Steves, M.P., Morse, E., Gutwin, C. and Greenberg, S. (2001) ‘A comparison of usage evaluation and inspection methods for assessing groupware usability’, Proceedings of the 2001 International ACM SIGGROUP Conference on Supporting Group Work, Session, 3, pp.125–134.

Designing and evaluating online communities: research speaks to emerging practice

Documents

Transcript of Designing and evaluating online communities: research speaks to emerging practice