Clarity for Stakeholders:Empirical Evaluation of ScenarioML, Use Cases, and Sequence Diagrams

Thomas A. Alspaugh Susan Elliott Sim Kristina Winbladh Mamadou H. DialloLeila Naslavsky Hadar Ziv Debra J. Richardson

Department of InformaticsBren School of Information and Computer Sciences

University of California, Irvine{alspaugh,ses,awinblad,mamadoud,lnaslavs,ziv,djr}@ics.uci.edu

Abstract

We studied the clarity of three requirements forms, op-erationalized as ease of problem detection, freedom fromobstructions to understanding, and understandability by avariety of stakeholders. A set of use cases for an industrialsystem was translated into ScenarioML scenarios and intosequence diagrams; problems identified during each trans-lation were noted; and all three forms were presented to arange of system stakeholders, who were interviewed beforeand after performing tasks using the forms. The data wasanalyzed, and convergent results were triangulated acrossdata sources and methods. The data indicated that Scen-arioML scenarios best support requirements clarity, thensequence diagrams but only for stakeholders experiencedwith them, and finally use cases as the least clear form.

That I require a clearness ...To leave no rubs nor botches in the work ...

Shakespeare, The Tragedy of Macbeth, III.i.

1. Introduction

High quality requirements are critical to software projectsuccess. Research over the past quarter century has shownthat requirements errors rapidly become more expensive toaddress, and their consequences if not addressed can in-clude a crippled product or cancelled development. Incom-plete requirements, lack of sufficient stakeholder involve-ment, and systems that fail to meet stakeholder needs areall too common [4, 5, 8, 13]. These issues have led usto consider clarity for stakeholders as a fundamental pre-requisite of requirements quality. In this paper, we reportan empirical study evaluating the clarity of three require-ments specification formats (use cases, sequence diagrams,

and the first author’s ScenarioML). We operationalize clar-ity through the following research questions:

1. Which format most readily permits the detection ofproblems in requirements?

2. Which format’s structure least conceals errors?

3. Which format do stakeholders find most clear?

The study has two phases, analogous to first testing thetransmission characteristics of three different telephone ca-bles, before testing the ability of specific humans to hearconversation across the cables. In the first phase (Figure 1),we examine how well the three formats represent the samerequirements specification. We obtain three comparable re-quirements specifications by translating an existing set ofindustrial use cases into ScenarioML scenarios and into se-quence diagrams, and then independently re-translate se-lected scenarios and sequence diagrams back to use casesfor comparison. The use cases were for the prototype ofthe Mirth healthcare middleware system [11] developed forour industrial partner WebReach, Inc. [15]. We analyticallyevaluate the three formats by comparing the number andkind of problems that arise or are identified during eachtranslation. In the second phase (Figure 2), we ask threestakeholders from WebReach to complete a test of theircomprehension of material presented in each of the require-ments specification formats, with pre- and post-interviewsto elicit reactions, opinions, and insights. We use a Latinsquare distribution of stakeholders, tasks, and specifications(Figure 2) in order to obtain effective cross-participant andcross-format comparisons. We evaluate each stakeholder’stask performance and then code (i.e. chunk and categorize)the interview transcripts, which are qualitatively analyzedto identify themes [7, 9].

We chose this mixed-methods approach [7] for severalreasons. Our goals were to explore the concept of clarity for

UCs

ScosSDs

WebReach

UCs

re-translated back to re-translated back to

translated intotranslated into

Figure 1. Translation to obtain three formats

Pre-test Interview ScosUCs SDs

Stakeholder 1Task 1

Task 2

Task 3

Stakeholder 2 Stakeholder 3Test

Scos UCsSDs

ScosUCs SDs

ScosUCsSDs

Post-test Interview ScosUCs SDs

Figure 2. Stakeholder interviews and test

stakeholders in the context of different requirements specifi-cation formats, to validate the clarity of ScenarioML specif-ically, and to gain insights into the views, beliefs, prefer-ences, activities, and goals of various kinds of stakeholdersin their use of requirements specifications. The existence ofpublic use cases for the Mirth system, and our connectionwith stakeholders at WebReach, Inc. provided an opportu-nity for in-depth study using industrial use cases not devel-oped by us and interactions with a range of industrial stake-holders. The context of the study permitted collection ofboth quantitative and qualitative data concurrently, enablingus to obtain focused data for addressing specific questionsand also more-open-ended data to provide insight and un-derstanding and guide future research. Finally, the approachsupported collection of data from multiple sources and tri-angulation across these data sources (Figure 3), thus pro-viding cross-correlation and corroboration in the event thatresults from different sources converged, as they did in thiscase [7]. While this research design does not produce sta-tistically generalizable results, it does provide insights intothe properties of requirements specification formats, whichwas our main goal.

The data from both phases of the study indicated thatthe ScenarioML format was the most clear, followed by se-quence diagrams (but only for those familiar with them),followed by use cases. In general, the researchers whoperformed the translations were able to detect more prob-

Pre-testinterviews

Testresponses

Task results data

Translationtranscripts

Translationproblem data

Post-test interviews

Figure 3. Triangulation across data sources

lems, such as inconsistency, when translating use cases intoScenarioML than when translating them into sequence di-agrams. While it was easier to detect high-level problemswith ScenarioML, it was easier to detect problems with low-level details with sequence diagrams. This both confirmedand delineated the kinds of clarity provided by ScenarioML.Stakeholders also preferred ScenarioML scenarios over usecases, and usually over sequence diagrams. They foundScenarioML to be clearer and expressed a preference forthis format, except in certain specific situations, such ascommunication among different architectural components.

The remainder of the paper is organized as follows. Sec-tion 2 surveys related work. Section 3 presents the study’sthree requirements specification formats. Section 4 dis-cusses the method used. Section 5 presents study results,which are discussed in Section 6. Section 7 lists lessonslearned, and Section 8 concludes and outlines future work.

2. Related Work

Requirements clarity and related concepts have beennoted by a number of researchers. Among the most fo-cal are the following. Heninger proposed operational ob-jectives closely related to our research questions [10]. BenAchour et al. note the advantages of narrative text use casesfor stakeholders, and discuss guidelines for improving theirquality which would also address clarity [3]. Sommervillelists lack of clarity as one of the top problems for naturallanguage requirements [12].

There has been little investigation of clarity as a qualityattribute of requirements. We know of only one contributionthat focuses specifically on clarity, Wasson’s CLEAR (Cog-nitive Linguistic Elicitation and Representation) method forwriting requirements and a case study in which CLEAR wasapplied to the requirements for a medical device [14]. Incontrast to CLEAR which focuses specifically on the word-ing of requirements, we examine how requirements specifi-cation formats support or obstruct clarity.

Williams et al. identify several weaknesses of use casesleading to confusion in their practical use, including weaksemantics and mismatches between text-based use cases

and the UML metamodel of use cases [17]. Whittle pro-poses a more formal modeling paradigm for use cases,called use case charts, strongly influenced by UML seman-tics [16]. Both Whittle and Williams et al. seek to increaseuse-case-based automation, for example for test case gener-ation, validation, or simulation. We distinguish clarity forstakeholders from formality, noting that formality provides(among other benefits) focus, compactness, and precision,but may not be appropriate for most stakeholders, who of-ten are not trained in formal approaches.

3. Requirements Specification Formats

3.1. Use Cases

A use case is a description of a sequence of interactionsbetween the system and one or more external actors that re-sults in an outcome of value to at least one actor. A usecase “describes the system’s behaviors and interactions un-der various conditions as a response to a request on behalf ofone of the stakeholders—the primary actor—showing howthe primary actor’s goal gets delivered or fails” [6].

Use cases are a structured text form, typically written us-ing a use case template whose form may vary widely fromproject to project. The use case author fills in text fieldssuch as use case name, goal, brief description, pre- and post-conditions, and normal, alternative, and exceptional flows.Thus a use case typically contains several scenarios, pro-viding a primary path to reach the stated goal and possiblyalternative paths to the goal or paths describing ways thegoal can fail to be met. The Mirth use cases correspond toCockburn’s “casual” use case structure, customized as herecommends to the project and its situation [6].

3.2. Sequence Diagrams

A sequence diagram is a UML diagram that documentsa sequence of object interactions that take place for a paththrough use case. They are typically used during analysisand design activities but, much like use cases, have beenused at many different levels of detail and abstraction, evenas replacement for and in lieu of use cases. They are wellsuited for illustrating the interplay between several architec-tural or design components, or between several actors and asystem, and show the interactions in terms of message pass-ing. The entities involved in the interactions are representedas vertical timelines, and messages or operations can be fol-lowed in sequence on each timeline from top to bottom.

3.3. ScenarioML

ScenarioML is a language for scenarios, designed to beread and written by people while also accommodating auto-

mated processing. It is an XML language that is presentedto stakeholders through an automatically-produced HTMLrepresentation, or by an Eclipse editor plugin currently indevelopment. The goals of ScenarioML are to make sce-narios that are clearer, more useful, and more effective. Thescenarios in this study are represented using a combinationof recursively-defined events, scenario parameters, refer-ences, and ontologies.

The basis of recursively-defined events are simple events,each of which describes in words one thing happening in theworld. A compound event groups several subevents to de-scribe several things happening in the world in a temporalpattern, either an event chain (the only compound eventsappearing in the study scenarios), directed acyclic graph,or general graph. An event schema compactly expressesa set of related combinations of events, such as an itera-tion representing any of several chains of repetitions of itssubevent, or an alternation representing any one of a groupof alternative subevents. Finally, an episode uses an entirescenario as one event of another scenario, with argumentsfor the reused scenario’s parameters to specialize it for thecontext in question. These recursively-defined events sup-port clarity for stakeholders by providing expressive meansof stating how events unfold over time, and by support-ing refactorings among equivalent or specialization-relatedevent patterns by which a stakeholder can verify what is ex-pressed. Figure 4 shows an excerpt from one of the studyscenarios, presented in HTML and illustrating several ofthese kinds of events.

ScenarioML supports explicit references to entities andterms defined elsewhere. In the XML, references are ex-pressed with markup elements in the text of a simple event.In the HTML presentation, a reference appears as a hyper-link to the referent. References support clarity by allowingscenario authors to link each referring word or phrase to itsreferent or definition, so that a reader can learn or confirmwhat was intended. The referents may be entities or types inthe ontology of the world described by the scenarios, or pa-rameters of the scenario containing the event with the refer-ence. ScenarioML also supports references to entities newlycreated or identified during the course of a scenario’s events.This is particularly useful in a scenario involving two ormore of the same kind of entity. An example in point fromthe current study is the scenario “Add/Edit/Delete Users”,in which the actor (who is a user) can add a new user (nothimself), edit a user (including himself), or delete a user(but not himself). There are a number of references to a‘user’ in this scenario, but not all to the same user. Scen-arioML references and new-instance markup allow this tobe done unambiguously.

ScenarioML has been the basis of work with autonomousanimated social agents, including a novel visualizationof software scenarios as social interactions between au-

Figure 4. ScenarioML scenario in HTML pre-sentation oriented to stakeholders (excerpt)

tonomous animated agents generated automatically fromthe ScenarioML [1]. ScenarioML has supported an ap-proach for goal-driven specification-based testing, in whichthe lower-level goals for a system are organized into plansfor achieving higher-level goals against which the system istested [19], and is the basis for our requirements-based test-ing approach, in which a system is driven by and checkedagainst its requirements scenarios [18].

4. Method

Our goal is to evaluate the clarity for stakeholders ofthree requirements forms. We apply the Goal Question Met-ric approach [2] to produce a measurement model opera-tionalizing our goal into research questions, and associatingeach question with data that can be evaluated (Figure 5).We begin by summarizing definitions of clarity from au-thoritative dictionaries as “easy to understand, fully intelli-gible, free from obscurity, unmistakable,” and derive threeresearch questions.

1. Which format most readily permits the detection ofproblems in requirements?

2. Which format’s structure least conceals errors?

3. Which format do stakeholders find most clear?

Goal Evaluate the clarity of requirements specification forms

QuestionsPermits

detection of errors?

Least conceals errors?

Judged clear by

stakeholders?

Metrics Number of problems

Types of problems

Stakeholder preference

Stakeholder performance

Figure 5. Goal Question Metric model

We decomposed the high-level goal into these three re-search questions, so that we could triangulate the phe-nomenon using data about user behavior, format proper-ties, and stated user preferences. The first research questionseeks to determine which format allows users to find moreerrors in the requirements. The second question probes theunderlying, or structural, properties of the format. The finalquestion addresses subjective experiences.

To answer these questions, we conducted a two-part em-pirical study, with the first focusing on the capabilities ofthe formats and the second focusing on stakeholder perfor-mance using the formats. It was necessary to evaluate thesetwo aspects separately, because we needed to factor out thecapabilities of the formats in terms of their carrying capac-ity, in order to isolate human behavior with the formats.

In this study, we created and studied the requirements fora single system, Mirth, expressed in three different formats,use cases, sequence diagrams, and ScenarioML. In the firstpart, we examined the clarity of the formats. The metricsthat we collected were the number and types of problemsencountered when creating the documents. In the secondpart, we examined the clarity of the formats when beingused by stakeholders. The metrics that we collected weretheir subjective statements on their experiences with thedocuments and their objective performance on tasks test-ing their understanding of the documents presented in eachformat.

4.1. Subject System

We studied use cases for WebReach’s Mirth system [11].Mirth is an open source cross-platform interface engine fortransferring healthcare information between systems thatuse different data formats. There were ten use cases rang-ing in size from 2 to 20 steps (average 7.3), and with zeroto six alternate flows per use case (average 1.9). The usecases had been developed by a team of senior undergradu-ate software development students working with WebReachstakeholders, and then used as requirements for prototypedevelopment.

4.2. Requirements Representation

In the first phase of the study, we investigated the clarityof the three requirements specification formats.

We chose ScenarioML because we wanted to conduct aformative evaluation of it, and it is similar in structure touse cases, but with additional features intended to supportclarity. In this sense, ScenarioML scenarios are more com-plex than use cases, but result in documents that are morestraightforward, because event chains are linear.

We chose use cases because they are similar in someways to ScenarioML scenarios and are widely used in in-dustrial practice, and because of the availability and appro-priateness of the Mirth use cases as a basis for the study.

We chose sequence diagrams because they are alsowidely used, and have distinctive similarities and differ-ences with respect to use cases and scenarios that couldresult in interesting insights. All three formats describe se-quences of events, but sequence diagrams do so graphicallywhile use cases and scenarios are primarily textual.

4.2.1 Procedure

We translated the Mirth use cases into the other two for-mats, ScenarioML and sequence diagrams. Each translationwas done by a different graduate student; both students werecomparably proficient with use cases, and comparably pro-ficient with their target formats (ScenarioML or sequencediagrams). Another researcher logged the translators’ think-aloud verbal protocols of the translation process, includingany problems or challenges that were encountered. A thirdgraduate student analyzed the logs and the resulting require-ments specifications for problems identified in the use casesduring the translations. He characterized the types of prob-lems by inducing categories from the data, as is done in aGrounded Theory approach [9].

Yet another researcher familiar with all three formats re-translated a sample of the scenarios and sequence diagramsback into use cases, with another researcher logging the re-translator’s introspective narration of the process. Problemsand insights from this process were correlated where possi-ble with those obtained in the first translations. We did thisto cross validate the translations that were performed andto evaluate the process of translating requirements into usecases. This step allowed us to verify that the initial trans-lations were done correctly, and that any information losscould be attributed to the specification formats.

4.3 Stakeholder Evaluation

In the second part of the study, we presented the threeformats for evaluation by stakeholders. They were askedabout their impressions of the formats in an interview and

then tested on their ability to perform tasks with them. Weachieved a high level of realism, though not necessarily gen-eralizability, by using a subject system from industry andselecting participants, also from industry, who had prior ex-perience with the subject system. Due to our stringent se-lection criteria for subjects, we were able to obtain a sam-ple size of only four (one in the pilot study, and three inthe main study). We compensated for this small number byusing a Latin square design to ensure that each subject saweach problem and each format. This approach ensured sci-entifically valid qualitative results (although not statisticallygeneralizable), that were highly realistic and provided novelinsights into the underlying research problem.

4.3.1 Procedure

Each interview was conducted individually and lasted forabout one hour. We began by asking participants abouttheir impressions and experiences with the formats. Wethen gave each participant a short test to assess the clarity ofthe three formats for that participant. The test consisted ofthree tasks each involving a different format and a differentscenario to probe their understanding. The tasks were cho-sen to highlight potential problems in clarity by revealingwhether the participants fully grasped complex interactions,understood potentially confusing material, and detected er-rors.

Problem 1: Add/Edit/Clone/Delete a Channel Thisuse case was complex, with challenging inter-relations be-tween the main success scenario and alternate scenarios,and several needed pieces of information missing. The cor-responding scenario and sequence diagram represented thecomplexity in the structures provided by those formats. Aparticipant who completely understood what was presentedby the assigned format would be able to paraphrase the stepsinvolved and identify missing information.

Problem 2: Deleting a User This use case contained analternate flow that did not fit into the main flow, and the usecase title listed more functions than the steps of the use casedescribed. Participants were asked to describe the sequenceof steps necessary to delete a user. A participant who com-pletely understood would be able to point out the ways inwhich the format did not tell them what was needed.

Problem 3: Change Channel Status Although the usecase describes enabling a channel, and purports to describedisabling one, it does not provide all the steps needed fordisabling. A participant who completely understood wouldnotice this and be able to point out where steps are missing.

With three formats, three tasks, and three participants,we used a Latin square design. Each participant saw eachformat and each task but in different combinations, sothat we could compare between—and within—each partici-pant’s performance. Figure 2 depicts the distribution to each

participant of the tasks and formats.The interviews were transcribed verbatim, without im-

posing sentence structure or clarification. We then codedand categorized utterances in the transcripts [7, 9]. We used16 categories derived from the data such as preferences, fa-miliarity, and hostility with regard to each format. Duringanalysis, we identified statements that pertained to the clar-ity of each of the formats. We were careful to consider bothpositive and negative statements when summarizing the in-terviews and selecting examples.

4.3.2 Participants

Four stakeholders at WebReach of varied background andfocus were participants in this study. The first in the listwas a pilot subject and the remainder served in the mainstudy.

Stakeholder 1 An enterprise software architect with over20 years of experience, who has designed and imple-mented solutions for clients in a wide variety of indus-tries and technologies. This participant’s results wereused as a pilot evaluation for the study’s second phase.

Stakeholder 2 A manager with 20 years of software engi-neering and management experience, whose primaryfocus is business development and key-client accountmanagement.

Stakeholder 3 A software engineer who helped lead the ar-chitecture and development of the Mirth Project, and iscurrently a PhD student.

Stakeholder 4 A Mirth developer and an author of theoriginal use cases, whose current focus is business de-velopment and project management.

4.4. Validity

In this subsection, we discuss the internal and externalvalidity of the study, as well as threats to validity.

Internal Validity. Internal validity is the soundness ofthe relationships within a study. In the requirements trans-lation portion of the study, we linked the clarity of a for-mat with the clarity of a document produced using that for-mat. These in turn, were measured indirectly in terms ofthe problems that the format could reveal. A requirementsspecification format demands inclusion of certain kinds ofinformation, and presents certain kinds of information moreprominently, and different formats have different demandsand different presentations. Therefore, translating require-ments from one format to another will reveal shortcomingsof the source document and the source format.

External Validity. External validity is the degree towhich the results from the study can be generalized. In this

study, we used only one subject system, three students, andthree stakeholders. This is a small sample size on all counts.However, we are not trying to create statistically significantresults that can be generalized to a population of subjectsystems, students, or stakeholders. Rather, we are seekingto build theories and deepen understanding of requirementsspecification formats. In the requirements creation part ofthe study, we seek to reason analytically about propertiesand qualities of the formats. These properties will holdacross subject systems and documents. In terms of stake-holders, the data are not as strong, so we use these only ascorroborating evidence. However, we feel this evidence isvalid, because it comes from actual stakeholders.

We are using qualitative and quantitative data from mul-tiple sources regarding both behavior and opinion. Thiswide variety of data allows us to triangulate a phenomenon,the importance of requirements clarity. Because the datawere produced in different ways and show different view-points on the issues, it is unlikely that they will convergedue to chance. If the four metrics converge, the weight oftheir support will be magnified.

Threats to Validity. The students who participated inthis study are authors on this paper and involved in someway in the design of ScenarioML. This may result in abias in favor of ScenarioML. We attempted to minimize thisthreat by having the students work independently and eval-uated their work independently.

5. Results

In this section, we present our findings regarding our fourmetrics. These metrics will be used in Section 6 to answerthe research questions.

5.1. Number and Types of Problems Found

Following the translation exercise, the documents andlogs were analyzed independently by another student. Acatalog of the problems found was created. These problemswere then grouped according to similarities. Based on thegrouping, a taxonomy was created so that each problem en-countered fit into only one category. The result was a com-parison table (Table 1) between the problems found duringthe two translation activities. This table was verified by thetranslators to ensure that the taxonomy was accurate.

Category 1 in Table 1 addresses inconsistencies betweenthe use case diagram and the textual use cases. Some usecases seemed to be represented in both, but under differentnames such as “CreateModifyModule” and “CreateModi-fyViewModule”. Others were visible in the diagram andsimply missing in the textual representation. Another prob-lem that surfaced during the translation was that use casesdo not clearly express iterations, this type of problem is

1. UC diagram - textual UC mismatch

Problem categories Found in SDs

Found in scenarios

0 42. Iteration was not specified3. Redundancy in the UCs4. UC numbering scheme does not show where alternative flows start/stop5. All possible alternatives not specified

0 20 1

3 4

0 26. Mistakes in the UC7. Incomplete UCs8. Two names referring to the same term9. Problems with details of events

1 43 35 5

36

Table 1. Problems found during translation

shown in category 2. Category 3 shows structural problems,where clarity could have been gained by combining simi-lar use cases. For example, three of the use cases performexactly the same steps with different actors. Another prob-lem was the confusing numbering of events that resulted inambiguous alternatives and exception scenarios. Category4 shows such problems, which were found in three of theten use cases. Category 5 shows problems of missing al-ternatives. The option of deleting a user, for example, waslisted as an option but there were no events that describedthis process. Category 6 shows mistakes, such as mixing upthe order of events in the use cases. Category 7 addressesthe problem of incompleteness, such as referencing non ex-isting use cases, while category 8 deals with inconsistenciessuch using different names for the same actors. The last rowcombines problems in individual events, such as where datacan be stored.

The data shows that ScenarioML revealed more prob-lems in the original use cases than the sequence diagrams.However, sequence diagrams were better at showing prob-lems with the details of events. In contrast, ScenarioMLscenarios more effectively show problems at a high level ofdetail and interstitial problems, i.e., problems between usecases or between different parts of use cases.

5.2. Stakeholder Preference

Each participant was interviewed individually and askedabout their preferences regarding the three formats.

In general, the participants preferred ScenarioML overthe other two formats. While they pointed out a few thingsthat were unclear and not intuitive in the scenarios, suchas the star that represents a number of iterations, they didnot perceive these as off-putting or think that other stake-holders would. However, they also thought that use casesmight be superior for initial requirements gathering and forusers to write themselves. Sequence diagrams were dis-liked by the two participants who were not already familiar

with the notation. Sequence diagrams were seen as usefulor advantageous in more restricted contexts and audiences.The participants liked ScenarioML although they had thesame or less experience with this notation. The participantsliked use cases but found ScenarioML scenarios to be animprovement in a number of ways. Below is a summary ofthe results per format.

Use Cases. The participants perceived use cases as“fairly easy to understand”, in particular the steps describedin the use cases, the summaries of what the use cases do,and the main and alternative scenarios that express alterna-tive flows. Things that the participants found difficult tounderstand included the numbering scheme for alternativescenarios, and the terms used in events. In terms of pref-erence, the participants liked that the use cases were easyto read and that the notation was familiar. Some things theparticipants disliked were the lack of detail, the numberingscheme, the inconsistency in terminology, and the lack ofhyperlinks to supporting documents.

Sequence Diagrams. One participant found the nota-tion “pretty easy”, and understood the calls between com-ponents and the separation of layers. Two of the partici-pants perceived the notation as confusing, in particular howto follow branches and options. The subject that understoodthe sequence diagrams liked them because of their graphicalnature. Some things that the other two participants did notlike were the lack of readability, the visual overload, and thelack of hyperlinks to supporting documents.

ScenarioML Scenarios. The participants found the no-tation easy to understand. The participants understood theconcepts of parameters, ontology definitions, event chains,alternations, references, and instances. The participantsliked that the notation was simple and textual with indenta-tions, the declarative statements with parameters, that therewere better and more consistent definitions of terms, thatit was more rigid, and had hyperlinks to supporting docu-ments and terminology.

The overall observation is that the participants thinkScenarioML specifies scenarios with greater clarity, exceptin showing interactions with various layers of a system witha layered architecture.

5.3. Stakeholder Performance

Overall, the subjects exhibited low performance in find-ing problems in the use cases and sequence diagrams de-spite their familiarity with the subject system. We hadexpected better performance because we selected subjectswho had worked with Mirth. Another unexpected resultwas the conflict between attitudes and performance on usecases. All three participants had positive attitudes towardsuse cases and thought they were clear. However, they haddifficulty on the comprehension tasks involving use cases.

This comprehension difficulty implies that they were dif-ficult to use because they were unclear, contradicting theinitial reaction from the stakeholders.

In the remainder of this subsection, the performance onthe three formats will be discussed in detail.

Use Cases. Overall the participants performed poorlywhen trying to identify problems in the use cases (see 4.3.1for the task descriptions). In Task 1, the participant with theuse case failed to identify any problems, although the alter-nate extension did not fit into the main use case and was notappropriate for the “Clone” option. However, the partici-pant did note that “the numbering scheme for the steps canbe confusing.”

In Task 2, the participant with the use case listed a se-quence of steps that did not make sense, since the optionwas missing in the use case. The participant concluded thatthe use case was “kind of ugly”. The participant had not no-ticed this problem before, and did not think that the notationdid anything to point out this problem.

In Task 3, the participant with the use case thought it wasdifficult to determine the starting point of the two scenarios(main and alternate), because they were different. The par-ticipant did not notice this problem when looking throughthe use cases during the interview.

Sequence Diagrams. In Task 1, the participant with thesequence diagram understood the notation well, and waspositive toward it although no problems were identified.

In Task 2, the participant was asked to list the steps thatdelete a Mirth user, although the sequence diagram onlycontained steps for the disallowed case of a user deletinghimself. The user gave a plausible sequence of steps in-ferred from the diagram, but did not realize his error. De-spite this, the participant displayed a positive attitude to-wards sequence diagrams, stating “This specification makesit relatively clear to spot problems with the task.”

In Task 3, the participant could not list the steps to dis-able a channel, saying “The form is a bit incomprehensible.”

ScenarioML Scenarios. In Task 1, the participant withthe ScenarioML scenario understood the scenario well.

In Task 2, the participant with the ScenarioML scenariounderstood the scenario well, and pointed out that the nestedEventChain highlights the fact that something is occur-ring in an otherwise flat sequence of events and draws atten-tion to it, and that “This causes me to further inspect thesenested events more closely as they often seem to be prob-lematic areas.”

In Task 3, the participant with the ScenarioML scenariowas able to determine the steps to disable a channel (whichthe scenario presented correctly), but his attitude conflictedwith this successful performance; he stated “I can not reallyidentify potential problems (as easily as the sequence dia-gram)”, although in fact he had previously been unable toidentify problems in a sequence diagram.

6. Discussion

This section is organized according to our three researchquestions. However, data from each metric may be usedto answer more than one question. We also add qualitativedata to help corroborate our conclusions, and add richnessand immediacy to the discussions.

6.1. Which most readily permits the detec-tion of problems in requirements?

In our study, the translation process is used to comparethe ease of problem detection in each of our target formats.This comparison is based on the reasoning that informationrequired by a target format, but missing from the sourceformat, is indicative of a shortcoming in the source format.

We decide the strengths and weaknesses of a format bycomparing numbers and types of problems identified in thetranslation to that format. A drawback of this approach isit does not permit a side-by-side comparison of use casesto either ScenarioML scenarios or sequence diagrams, asuse cases were the source format in both translations. Thismismatch is compensated for in part by the re-translationfrom these formats back to use cases.

However, data from the interviews and comprehensiontest shows that stakeholders detected few of the problemspresent in either the use cases or sequence diagrams theyexamined, which is consistent with the inference that usecases are not markedly better than sequence diagrams foruncovering problems.

The list of problems uncovered (see Table 1) support thefollowing inferences: that sequence diagrams more read-ily detect problems in the details of events; and that Scen-arioML scenarios more readily permit detection of larger-scale problems related to clarity of requirements. Overall,3 of the problems found in the translations were uncoveredonly using sequence diagrams, 15 problems were uncov-ered by both formats, and 13 additional problems were un-covered solely by ScenarioML scenarios. The data fromthe interviews provided evidence converging with that fromcomparing the lists of problems uncovered during trans-lation, indicating stakeholders thought sequence diagramsmore effective for design tasks in which layers and architec-tural components are important, while ScenarioML scenar-ios would be more effective for requirements, as a contractbetween stakeholder and developer, and for high-level com-munication. This triangulation across two quite differentsources of data strengthen the inference drawn from each.

Overall, we infer that ScenarioML scenarios more read-ily permit the detection of errors than sequence diagrams toa substantial degree, but that sequence diagrams appear tomore readily permit detection of detailed problems withinindividual events. Use cases fall in between the two and

seem to be more similar to sequence diagrams.

6.2. Which least conceals errors?

The degree to which the three formats obstruct under-standing is addressed by the notes from the translation pro-cess, the comprehension tests, and the interviews. Thestudent who translated use cases to ScenarioML scenarioscommented that she frequently had to consult other doc-uments for missing information. The comprehension testshowed that all three stakeholders found the use case formobstructed understanding, with comments such as “it’s nottoo helpful for me in understanding these types of issuesat all”, “doesn’t do anything to point it [the problem] out”,and “more difficult to spot what is a problematic step”. Thestakeholder who was familiar with sequence diagrams didnot find any of the problems they contained (although he feltthey were easy to understand). The stakeholders who exam-ined ScenarioML scenarios indicated the form did not ob-struct their understanding, and noted ways in which they feltthe form may have helped. The interview data revealed es-sentially the same pattern: all stakeholders identified waysin which use cases obstructed their understanding; stake-holders’ responses for sequence diagrams were divided bywhether they understood the form or not; and all stakehold-ers commented on ways in which ScenarioML enhancedtheir understanding,while identifying two minor ways inwhich it obstructed their understanding. (The stakehold-ers did not understand that an iterated event numbered with∗ indicated it may occur many times; and they wished theycould optionally collapse compound events for clarity whenlooking at the overall picture.)

All three sources of data converged on the inferencesthat use cases most obstructed understanding, sequence di-agrams did not obstruct understanding if one already un-derstood the format, and ScenarioML tended to aid under-standing rather than obstructing it.

6.3. Which do stakeholders find most clear?

The tests and interviews were the sources of data for thisquestion. Overall, the stakeholders indicated they foundScenarioML scenarios clear. In contrast, the stakeholdersmade conflicting statements about use cases, and only stake-holders who had expertise in sequence diagrams liked them.

While all three stakeholders indicated that they felt theyunderstood the original use cases, this is contradicted bytheir performance on the tasks, and noncommittal andsometimes contradictory comments such as “it’s hard totell” and “That’s not that clear”, but also “pretty straight-forward” and “they’re easy to read”. Other statements alsoindicated their ambivalence towards use cases. For exam-ple, they said “this form of specification is confusing to me”

and “perhaps this is a problem with the user creating the usecases versus the format of the use case itself”.

Comments about sequence diagrams were also mixed.Stakeholders who had expertise in sequence diagrams foundthem to be clear; those who did not disliked them stronglyand one stakeholder commented that if he had to create se-quence diagrams, “I would be regretting the day I had tocome in and do these”.

In contrast, all three stakeholders found ScenarioMLscenarios easy to understand, with comments such as “Veryeasy”, “unambiguous”, “this is a very good form for a de-veloper to get something in”, “a richer, more defined way ofdoing use cases”, “wow, absolutely, I think that’s a definiteimprovement” (over use cases), “That’s good, I like that”,and “It’s kind of like a UseCase++”.

The data from the tests and interviews converge, support-ing the inference that the use case form is least well under-stood by stakeholders despite their comfort with the format,the sequence diagram form only understood by those famil-iar with it but apparently well understood in these cases,and that ScenarioML is easily understood by stakeholdersdespite their inexperience with it. The interview data onScenarioML is strikingly positive on this point.

7. Lessons Learned

Requirements specification formats vary in clarity for stake-holders.

This statement is so obvious that we did not think tomake it until confronted by data that stakeholders do notconsider all formats equally clear. This issue has re-ceived little consideration because non-developer stake-holders have not been the main target user group for require-ments specification formats. In other words, it was assumedthat stakeholders would find all formats equally unclear orconfusing. This is not the case and we have cataloged waysin which formats can be unclear. For example, the number-ing scheme for event sequences needs to be easy to followand understand. Another problem can be found in graphi-cal notations that obscure readability by visual overload andconfusing notations that lack clear definitions.

Clarity for stakeholders is a significant quality.

Clarity of requirements specification formats for stake-holders is important because it affects how and how muchthey can participate in the requirements engineering pro-cess. In addition to elicitation, cooperative stakeholders arealso needed for other activities including validation, verifi-cation and negotiation. Stakeholders and developers need tobe able to find the information they need easily, understandthat information, and know when they don’t understand theinformation. Therefore, when selecting a format, require-

ments engineers should select one that affords the level ofstakeholder participation that they desire.

Clarity for stakeholders can be operationalized.

Although communication is subjective, we have madeclarity objective by finding ways to empirically measurestakeholder performance using different formats by testingtheir comprehension. By setting out goals, questions andmetrics, we established an operational definition for clar-ity. These concepts can in turn be used as design objectiveswhen improving requirements specification formats.

8. Conclusion

In this paper, we reported on an empirical study thatrevealed the importance of clarity in requirements. Wecompared three requirements specification formats, Scen-arioML scenarios, use cases, and sequence diagrams. Ourempirical study consisted of creating documents in each ofthe formats, collecting data on their creation, and evaluatingthe usability of these documents from the point of view ofstakeholders. We compared the formats themselves, our ex-perience working with the formats, and stakeholder prefer-ence and understanding of information in the formats. Thedata from the translation and user study indicated that Scen-arioML was the format with the greatest clarity, followed byuse cases, followed by sequence charts. Despite stakehold-ers’ greater familiarity with use cases, they both preferredScenarioML and found the format clearer. Our analytic re-sults, stakeholders’ subject opinions, and stakeholders’ ob-jective performance were a strong endorsement of the Scen-arioML format.

While the research design that we used in this study pro-vided us with novel insights into requirements specificationformats, the results were qualitative in nature and not statis-tically generalizable. We plan to address this shortcomingin our next empirical study, which will be an experiment in-volving a much larger sample size. The working hypothesisis that material presented in a clearer format will be remem-bered better. We will be using a between-subjects combi-natorial design where each subject is given a single format(ScenarioML, use cases, or sequence diagrams) and testedon their recognition and recall of details from the require-ments. We expect to be able to perform tests of statisticalsignificance on the quantitative performance measures fromthis planned study.

9. Acknowledgements

The authors wish to thank WebReach, Inc. for their gen-erous cooperation in the study, and Suzanne K. Schaefer forhelping us develop the study protocol. We also thank the

anonymous reviewers for their careful and insightful com-ments.

References

[1] T. A. Alspaugh, B. Tomlinson, and E. Baumer. Using socialagents to visualize software scenarios. In ACM Symposiumon Software Visualization (SoftVis’06), 2006.

[2] V. R. Basili, G. Caldiera, and H. D. Rombach. The goalquestion metric approach. In Encyclopedia of Software En-gineering, pages 528–532. John Wiley and Sons, 1994.

[3] C. Ben Achour, C. Rolland, C. Souveyet, and N. A. Maiden.Guiding use case authoring: Results of an empirical study.In Fourth IEEE International Symposium on RequirementsEngineering (RE’99), pages 36–43, 1999.

[4] B. W. Boehm. Software Engineering Economics. 1981.[5] F. P. Brooks, Jr. No silver bullet: Essence and accidents of

software engineering. IEEE Computer, 20(4):10–19, 1987.[6] A. Cockburn. Writing Effective Use Cases. 2000.[7] J. W. Creswell. Research Design: Qualitative, Quantitative,

and Mixed Methods Approaches. Sage Publications, 2003.[8] A. M. Davis and A. S. Zweig. The missing piece of software

development. J. Syst. Softw., 53(3):205–206, 2000.[9] B. G. Glaser and A. L. Strauss. The Discovery of Grounded

Theory: Strategies for Qualitative Research. Aldine, 1967.[10] K. L. Heninger. Specifying software requirements for com-

plex systems: New techniques and their application. IEEETransactions on Software Engineering, 6(1):1–13, 1980.

[11] Mirth. http://www.mirthproject.org/.[12] I. Sommerville. Software Engineering. 2006.[13] Standish Group. The CHAOS report, 1994.[14] K. S. Wasson. A case study in systematic improvement of

langauge for requirements. In 14th Int. Conference on Re-quirements Engineering, 2006.

[15] WebReach. http://www.webreachinc.com.[16] J. Whittle. Specifying precise use cases. Technical Report

ISE-TR-06-02, George Mason University, Feb. 2006.[17] C. Williams, M. Kaplan, T. Klinger, and A. Paradkar. To-

ward engineered, useful use cases. Object Technology,4(6):45–57, 2005.

[18] K. Winbladh, T. A. Alspaugh, D. J. Richardson, andR. Waltzman. Meeting the requirements and living up toexpectations. Technical Report UCI-ISR-07-1, Institute forSoftware Research, University of California, Irvine, 2007.

[19] K. Winbladh, T. A. Alspaugh, H. Ziv, and D. J. Richardson.An automated approach for goal-driven, specification-basedtesting. In International Conference on Automated SoftwareEngineering (ASE’06), pages 289–292, 2006.