www.elsevier.com/locate/infsof

Information and Software Technology 49 (2007) 366–380

Experimental evaluation of an object-oriented functionpoint measurement procedure

Silvia Abrahao a,*, Geert Poels b

a Department of Computer Science and Computation, Valencia University of Technology, Camino de Vera, s/n, 46022, Valencia, Spainb Faculty of Economics and Business Administration, Ghent University – UGent, Hoveniersberg 24, 9000 Ghent, Belgium

Received 26 May 2005; received in revised form 2 June 2006; accepted 9 June 2006Available online 26 July 2006

Abstract

This paper presents an empirical study that evaluates OO-Method Function Points (OOmFP), a functional size measurement proce-dure for object-oriented systems that are specified using the OO-Method approach. A laboratory experiment with students was conduct-ed to compare OOmFP with the IFPUG – Function Point Analysis (FPA) procedure on a range of variables, including efficiency,reproducibility, accuracy, perceived ease of use, perceived usefulness and intention to use. The results show that OOmFP is moretime-consuming than FPA but the measurement results are more reproducible and accurate. The results also indicate that OOmFP isperceived to be more useful and more likely to be adopted in practice than FPA in the context of OO-Method systems development.We also report lessons learned and suggest improvements to the experimental procedure employed and replications of this study usingsamples of industry practitioners.� 2006 Elsevier B.V. All rights reserved.

Keywords: Functional size measurement; Function point analysis; OO-method function points; Object-orientation; Empirical software engineering

1. Introduction

Functional Size Measurement (FSM) methods measurethe size of software systems by quantifying their functionaluser requirements. These are the user practices and proce-dures that the software system must perform to fulfill theuser needs [1]. The most commonly used FSM method isFunction Point Analysis (FPA), introduced in the late sev-enties in IBM by Albrecht [2], but now universally support-ed by the International Function Point Users Group(IFPUG) [3].

FPA assumes a model of functional user requirementsthat is expressed in terms of logically related groups of data(e.g. product data) and transaction functions that use oract upon data groups (e.g. query product data, add newproduct, change product data, assign to product line,

0950-5849/$ - see front matter � 2006 Elsevier B.V. All rights reserved.

doi:10.1016/j.infsof.2006.06.001

* Corresponding author. Tel.: +34 96 3877000; fax: +34 96 3877359.E-mail addresses: sabrahao@dsic.upv.es (S. Abrahao), Geert.Poels@

UGent.be (G. Poels).

etc.). Such a model is relatively easy to construct whenfunctional user requirements are specified using structuredanalysis and functional decomposition techniques (e.g.Data Flow modeling, Entity-Relationship modeling). It isless straightforward to apply when object-oriented (OO)techniques are employed. The elementary unit of functionaluser requirements in OO analysis is the object type, whichdescribes the properties of a class of similar objects (e.g.products). As an object type encapsulates both data andbehavior, the dichotomy in the functional user require-ments assumed by FPA (i.e. data groups versus transactionfunctions), is not readily observable in an OO model of therequirements. This makes it difficult to uniquely interpretand consistently apply the FPA measurement rules, whichare based on the interactions between transactions anddata groups.

Two approaches to FSM in OO systems developmentcan be distinguished. In the first approach, new FSM meth-ods have been developed for the specific purpose of mea-suring OO software systems. These methods capture and

S. Abrahao, G. Poels / Information and Software Technology 49 (2007) 366–380 367

quantify functional size by means of constructs and modelsthat are expressed in terms of the OO paradigm. Some ofthe methods in the first approach define measures that devi-ate significantly from function points, for instance by sizingsystems in terms of frequencies of OO construct occurrenc-es such as object classes and messages (e.g. [4–10]). Othermethods are analogous to FPA in that they share commonaspects with the FPA structure, for instance a weightingschema to express the relative functional size contributionsof different types of OO constructs (e.g. [11–14]).

The second approach consists of methods that reuse theFPA constructs and model, but map them onto the lan-guage used by the OO analysis method for which theyare developed (e.g. [15–17]). These FSM methods do notdefine new measures, but offer a measurement procedure1

that can be used in combination with FPA. Their mainobjective is to prescribe the supposedly correct interpreta-tion of the FPA measurement rules for a target applicationcontext (i.e., an OO analysis method), and hence facilitateand improve the performance of FSM in this context. Themain advantage of these methods over the first approach isthat they result in a function point value that can be com-pared to other function point values, obtained using differ-ent measurement procedures (such as the ‘regular’procedure proposed by IFPUG [3]) and across applicationcontexts. The advantage is significant as over the yearsthousands of projects have been measured with FPA anda substantial amount of these historical data are publiclyavailable for analysis and benchmarking purposes (e.g. inthe ISBSG repository [19]).

In [20] we have presented an FPA-to-OO mapping thatwe developed as a function point measurement procedurefor software systems specified using OO-Method [21].OO-Method is an automated software productionmethod that takes a model-driven approach to systemsdevelopment based on conceptual modeling, model trans-formation, and code generation. Our proposal, calledOO-Method Function Points (OOmFP), consists of a setof measurement rules and a procedure prescribing how toapply these rules on an OO-Method conceptual representa-tion of the system specifications.

According to March and Smith’s framework forresearch in IT [22], OOmFP is a design science artifactresulting from a build research activity. As a method it isa set of steps used to perform a task (i.e. FSM), based ona set of underlying constructs (i.e. the FPA concepts) anda model of the solution space (i.e. FPA concepts mappedonto OO-Method conceptual modeling primitives). Marchand Smith argue that information technologies (as designscience research outputs in IT) ‘‘once built, must be evalu-ated scientifically’’ (p. 258). Specifically for methods, eval-uation considers ‘‘operationality (the ability to perform the

1 A set of operations, described specifically, used in the performance of

particular measurements according to a given method of measurement’’, asdescribed in the International Vocabulary of Basic and General Terms inMetrology [18].

intended task or the ability of humans to effectively use themethod if it is not algorithmic), efficiency, generality, andease of use’’ (p. 261). Hence, after the build activity, theevaluate research activity must answer the basic questionhow well the new information technology works.

This paper presents an evaluation of OOmFP as adesign science artifact. The framework for the study isthe Method Evaluation Model (MEM) of Moody [23],which is a model for evaluating information system (IS)design methods. Although this model was developed forevaluating IS design methods, its constructs can bedescribed in terms of other criteria to evaluate other typesof methods. Hence, we used the MEM to verify if and howmuch OOmFP improves the performance of function pointmeasurement of systems specified using OO-Method. Alaboratory experiment was conducted in which we com-pared the performance of students employing OOmFPagainst that of students employing the ‘regular’ IFPUGprocedure for function point measurement. Following theMEM, the OOmFP and IFPUG function point measure-ment procedures were compared on objective as well asperception-based variables of task performance, includingefficiency (i.e. effort required to apply FSM), reproducibil-ity of the FSM results, accuracy of the results, perceivedease of use, and perceived usefulness. In addition, theintention to use a method was measured in an attempt toestimate the likely adoption in practice of OOmFP.

The paper is organized as follows. In Section 2, we brief-ly introduce the reader to OO-Method, OOmFP, and theMEM. In Section 3, we present the design of the laboratoryexperiment used to evaluate OOmFP. The data collected inthe experiment are analyzed in Section 4. In Section 5, wedescribe the lessons learned from this first empirical evalu-ation study, assess the validity and generalisability of thestudy results, and suggest improvements to the experimen-tal procedure that can be implemented in future replicationstudies. Finally, in Section 6 we present conclusions andoutline future work.

2. Background

This section starts with a brief introduction to OO-Method and OOmFP. Next, we discuss the problem ofhow to evaluate FSM methods. The section ends withour solution approach in the form of an evaluation modelbased on the MEM [23].

2.1. OO-Method and OOmFP

OO-Method is a model-driven requirements and soft-ware engineering approach developed at the Valencia Uni-versity of Technology [21]. In OO-Method, a cleardistinction is made between two models: the ConceptualModel (centered on the problem space) and the ExecutionModel (centered on the solution space). The OO-Methoddevelopment process is automated by implementing theprovided set of mappings between conceptual primitives

2 FPA categorizes data groups into internal logical files (ILF) andexternal interface files (EIF). Transaction functions are external inputfunctions (EI), external output functions (EO) or external inquiryfunctions (EQ) [3].

3 These elements are data element types (DET) and record element types(RET) for data groups, and DETs and file types referenced (FTR) fortransaction functions [3].

4 For more information on the different subsets of OOmFP measure-ment rules, and their relationship to the IFPUG rules for function pointmeasurement, we refer to [20].

368 S. Abrahao, G. Poels / Information and Software Technology 49 (2007) 366–380

(problem space) and their software representations (solu-tion space).

The Conceptual Model is used to capture the relevantproperties of an object-oriented system in a graphicalway. It is based on the specification of four complementarymodel views using an UML-compliant notation:

• Object Model: captures the structure of the system interms of classes identified in the problem domain andtheir relationships.

• Dynamic Model: captures the behavior of the system interms of valid object lives and interaction betweenobjects.

• Functional Model: captures the semantics associated tothe state changes of the objects.

• Presentation Model: captures the user interaction pat-terns for the system interface.

A system specification is described using these four mod-el views, each one representing a particular perspective, butall together constituting the whole conceptual schema ofthe required system. This conceptual schema is a formalrepresentation of the concepts that characterize the systemas required by its users in a manner that is independent ofthe system design and implementation.

As a conceptual representation of the user requirements,the four model views of an OO-Method conceptual schemacontain all the elements that may possibly contribute to thefunctional size of the system. Hence, the conceptual schemais a sufficient input for FSM and thus FPA can be appliedto an OO-Method conceptual schema to measure the sys-tem’s functional size.

Experience reports on FSM of OO systems have recog-nized that FPA can be used, even if the functional userrequirements are captured in a different way (i.e. usingOO constructs and models) than usually assumed [24,25].However, this does not mean that it is straightforward toapply FPA in such contexts. Therefore, OO-Method Func-tion Points [20] was developed to facilitate the functionpoint measurement of an OO-Method conceptual schema.OOmFP provides a mapping of the OO-Method Conceptu-al Model concepts onto the FPA concepts. This mapping isformalized in a set of measurement rules. A procedure hasbeen devised to apply these measurement rules (Fig. 1).

The process model shown in Fig. 1 is based on the gen-eralized representation for FSM methods of Fetcke et al.[26]. According to this representation, FSM requires twosteps of abstraction: first, the elements that contribute tothe functional size of the system are identified (identifica-tion step); and second, these elements are mapped intonumbers (measurement step).

In the OOmFP identification step, we use a first subsetof measurement rules, called ‘mapping’ rules, to obtainthe FPA view on the functional size of the modeled system.This step consists in identifying the elements in theOO-Method conceptual schema that contribute to thefunctional size of the system and to disregard those that

do not contribute to functional size. The result of this stepis a collection of uniquely identified and categorized datagroups and transaction functions that can be quantifiedin the next step.2

Hence, in the OOmFP measurement step, a functionpoint value is obtained through the application of twoother subsets of measurement rules. First, measurementrules are used to identify the elements that help determin-ing the complexity of the identified data groups and trans-action functions3. These elements along with the datagroups and transaction functions to which they apply, forma system representation according to the OOmFP Measure-

ment Abstract Model of user requirements. Next, a last sub-set of measurement rules, which are standard IFPUGcounting rules [3], are used to rate the complexity of thedata groups and transaction functions, to transform com-plexity ratings into function point values, and to aggregatethe assigned values into an overall functional size value forthe system4.

We remark that the FPA adjustment phase is not includedin OOmFP. The view on functional size measurement artic-ulated in the ISO/IEC 14143 series of standards on FSM [1]prohibits the use of an adaptation factor that does not assessfunctional user requirements (but assesses, for instance,quality or technical requirements as is the case with FPA’sadjustment factor). Therefore, OOmFP expresses functionalsize in unadjusted function point values.

2.2. Evaluating FSM methods

Despite the wide usage of established FSM methodssuch as FPA [3] and Mark II FPA [27], and the prolifera-tion of newer ones such as COSMIC-FFP [28], there cur-rently exists no methodology for their systematicevaluation. The best approximation to an FSM methodevaluation framework is the 14143 series of standards forFSM issued by the ISO/IEC.

Part 1 [1] of these standards defines the concepts of FSMand establishes a basis against which existing and new FSMmethods can be compared. Part 2 [29] provides a processfor checking whether a candidate FSM method conformsto the concepts described in part 1. Part 3 [30] providesguidance for verifying the stated performance of an FSMmethod with respect to its repeatability, reproducibility,accuracy, convertibility, discrimination threshold, andapplicability to functional domains. The standard specifies,for instance, requirements for verification methods to

Fig. 1. The OOmFP measurement procedure [20].

S. Abrahao, G. Poels / Information and Software Technology 49 (2007) 366–380 369

ensure that verification results are objective, impartial, con-sistent and repeatable.

Part 4 [31] defines a reference model to be used whenverifying an FSM method. Although this part providessome examples of reference objects against which anFSM method can be applied and compared, the formula-tion and execution of evaluation tests as well as the inter-pretation of their results is outside the scope of thisdocument. Finally, Part 5 [32] addresses the definition offunctional domains for which FSM can be used. A func-tional domain is a class of software characterized by specif-ic software requirements (e.g. enterprise systems software,embedded software, etc).

The editors for part 3 of the standards, Jacquet andAbran, have also presented in [33] (not part of the stan-dards) a process model for evaluating FSM methods. Thismodel describes all processes and activities involved in thedefinition and validation of FSM methods. However, themodel of Jacquet and Abran is very general and needs tobe specialized and detailed depending on the specific pur-pose for which the FSM method was developed. ForOOmFP this specific purpose is to facilitate the functionpoint measurement of OO-Method conceptual schemas,thereby improving upon the IFPUG measurement proce-dure for FPA. The evaluation must therefore focus onthe performance of OOmFP in achieving its objective,meaning provide a ‘better’ measurement procedure thanIFPUG does within the OO-Method application context.

Some of the performance properties in part 3 of thestandards offer relevant evaluation criteria. The OOmFPand IFPUG measurement procedures can be directlycompared with respect to repeatability, reproducibility,and accuracy. The other properties (i.e. convertibility,discrimination threshold, and applicability to functional

domains) apply to the functional size measure used by anFSM method, hence cannot be used to differentiatebetween OOmFP and IFPUG (as both use Function Points(FPs) as a measure).

The retained properties relate to the effectiveness ofFSM. According to March and Smith [22], efficiency, gen-erality, and ease of use must also be considered when eval-uating methods. Although part 3 of the ISO/IEC standardsfor FSM and the process model of Jacquet and Abranprovide a starting point for evaluating OOmFP, what ismissing is a structured view of the quality of a FSM mea-surement procedure. A simple list of isolated quality crite-ria hides relationships between quality properties. A qualitymodel for FSM measurement procedures should explicitlydescribe how quality properties of different dimensions(e.g. effectiveness versus efficiency) are related and whattheir impact is on overall quality. Furthermore, a systemat-ic evaluation based on such a model should not only seekobjective evidence of the performance of a method inachieving its objectives. It should also test the user’sresponse to a new method and allow predicting its accep-tance in practice. The importance of measuring percep-tion-based performance variables (e.g. ease of use) forassessing software developers’ acceptance of new methodshas been stressed in recent software engineering literature(see e.g. [34]).

2.3. An evaluation model based on the MEM

Moody’s Method Evaluation Model (MEM) [23] pro-vides a suitable basis for a multidimensional quality modelof FSM measurement procedures (Fig. 2). The MEM wasoriginally proposed as an evaluation model for IS designmethods. The main contribution of the MEM, compared

PerceivedEase of Use

PerceivedUsefulness

Intentionto

Use

ActualEfficiency

ActualEffective-

ness

Performance

ActualUsage

External (PerformanceBased) Variables Internal (Perception Based)

Variables

ExternalBehaviour

Perceptions

Intentions Behaviour

ACTUALEFFICACY

PERCEIVEDEFFICACY

ADOPTION IN PRACTICE

Method Adoption Model

Fig. 2. The Method Evaluation Model [23].

370 S. Abrahao, G. Poels / Information and Software Technology 49 (2007) 366–380

to alternative models, is that it incorporates two differentaspects of method ‘‘success’’: actual efficacy and actual

usage. As argued in the previous subsection, both aspectsmust be considered when evaluating FSM methods orprocedures.

In the MEM, efficacy is defined as a separate construct,different from efficiency and effectiveness. The efficacy con-struct is derived from Rescher’s notion of pragmatic suc-cess [35], which is defined as the efficiency andeffectiveness to which a method achieves its objectives.Thus, the evaluation of the efficacy of a method requiresmeasurement of both effort required (efficiency) and thequality of the results (effectiveness).

The core of the MEM, called the Method AdoptionModel (MAM), is based on the Technology AcceptanceModel (TAM) [36], a well-known and thoroughly validatedmodel for evaluating information technologies5. The con-structs of the MEM are:

• Actual Efficacy, which consists of two parts:� Actual Efficiency: the effort required to apply a

method. This represents an input variable to theMAM.

� Actual Effectiveness: the degree to which a methodachieves its objectives. This also represents an inputvariable to the MAM.

• Perceived Efficacy, which consists of two perception-based MAM variables:� Perceived Ease of Use: the degree to which a person

believes that using a particular method would be freeof effort.

� Perceived Usefulness: the degree to which a personbelieves that a particular method will achieve itsintended objectives.

5 The TAM contains three variables or dimensions for evaluatinginformation technologies: ease of use, usefulness and intention to use.

• Intention to Use: an intention-based MAM variablefor predicting adoption in practice, defined as the e-xtent to which a person intends to use a particularmethod.

• Actual Usage: a behavior-based variable, defined as theextent to which a method is used in practice. This re-presents an output variable from the MAM.

To evaluate FSM measurement procedures such asOOmFP, the constructs of the MEM must be operational-ized for use with this kind of method. Efficiency can bedefined as the time it takes to apply the measurement pro-cedure to a system of a given size. Effectiveness can bedefined in terms of the retained performance propertiesfrom part 3 [30] of the ISO/IEC standards for FSM, i.e.repeatability, reproducibility, and accuracy. The percep-tion/intention-based variables of the MAM can be directlyused. Only the behavior-based variable Actual Usage is notused in the evaluation of OOmFP, as it can of course notbe measured for newly proposed methods (only for estab-lished ones).

3. Method

In terms of the GQM template for goal definition[37], our empirical study aimed at analyzing functionpoint measurement for the purpose of evaluating theOOmFP measurement procedure (hereafter referred toas ‘OOmFP’) against the IFPUG measurement proce-dure (hereafter referred to as ‘FPA’) with respect to

their actual and perceived efficacy and likely adoptionfrom the point of view of the researcher. The context

of the study is an advanced software engineeringcourse at the Valencia University of Technology, wherethe OO-Method approach to systems modeling istaught.

Our study was conducted as a laboratory experiment.The design of this experiment was based on the frameworkfor experimentation in software engineering research sug-gested by Wohlin et al. [38]. The broad research questionsaddressed are:

• RQ1: Is the actual efficacy of OOmFP higher than theactual efficacy of FPA?

• RQ 2: Is the intention to use and the perceived efficacyof OOmFP higher than the intention to use and the per-ceived efficacy of FPA?

3.1. Variables and operational hypotheses

The independent variable is the measurement proce-dure used: OOmFP or FPA. Hence, the experimentemployed two treatments: the function point measure-ment of a system with OOmFP and the function pointmeasurement of the same system with FPA. The experi-mental data collected allows comparing the effects ofboth treatments.

6 http://www.ifpug.org/certification/.7 Whether the function points value produced by an CFPS is really the

‘true value’ of the system’s functional size is a question concerning thevalidity of function points as a functional size measure, and given ourfocus on measurement procedure evaluation outside the scope of thepaper. The certification is an assurance that the value produced by anCFPS is ‘true’ in the sense that function point measurement was appliedcorrectly. Because variability (or error rates) by CFPS up to 10% isaccepted, we contacted a number of CFPS from the USA, Australia andBrazil to provide us with a ‘true value’. From the total of four responsesreceived, the mean function points value was used as the ‘true value’.

S. Abrahao, G. Poels / Information and Software Technology 49 (2007) 366–380 371

There are two types of dependent variables for compar-ing the treatments: performance-based and perception/intention-based variables. Performance-based variablesassess how well the experiment participants perform theexperimental task. They are used to evaluate the actual effi-cacy of the measurement procedures. Perception/intention-based variables assess the participants’ perceptions of theirperformance and their subsequent intention to use OOmFPor FPA. These variables are used to evaluate the perceivedefficacy of the measurement procedures, as well as theirlikely adoption in practice.

Three performance-based variables were selected. Thefirst variable is measurement time which operationalizesthe actual efficiency construct of the MEM. It is definedas the time taken by a subject to complete the measurement

task. For a fair comparison of the measurement time, itwas important to ensure the equivalence of the measure-ment task for both treatments. Although OOmFP isapplied on an OO-Method conceptual schema (seeFig. 1), we decided not to include the time spent on theconceptual modeling of the user requirements. Our moti-vation for this decision is that OOmFP was specificallydeveloped to be used in an OO-Method application con-text, and thus an OO-Method conceptual schema will bedeveloped anyway (independent of the need for FSM).On the other hand, we also realized that forcing partici-pants to apply FPA to the OO-Method conceptual sche-ma would disadvantage this measurement procedure,given the aforementioned mapping problem betweenFPA and OO concepts.

Besides, in practice, when documented user require-ments and the conceptual schema are available, users ofFPA can chose which format to use as an input forFSM. Therefore, in the experiment, FPA was directlyapplied to the user requirements specifications and not tothe OO-Method conceptual schema.

The second performance-based variable is reproducibil-

ity, defined as the agreement between the function pointmeasurement results of different participants applying the

same measurement procedure on the same system. This var-iable, taken from part 3 of the ISO standards for FSM[30], operationalizes the actual effectiveness construct ofthe MEM. We postulate that the more similar the func-tion point values obtained by different raters, the moreeffective the measurement procedure is in terms ofconsistency.

However, even when the obtained values lie close toeach other, they can be far from the ‘true value’ for func-tional size. Consequently, we selected accuracy from part3 of the FSM standards [30] as a complementary effective-ness variable. Accuracy is defined as the agreement between

the function point measurement results of different partici-

pants obtained by applying a same measurement procedure

on a same system and the true function point value for thatsystem. To obtain this ‘true value’, the requirements speci-fication documents were measured using the IFPUG func-tion point measurement procedure by four Certified

Function Point Specialists (CFPS),6 who were blind tothe research questions tested in the experiment.7

We wish to remark that repeatability, which is analternative way to express consistency of measurement,is not considered in this experiment. This is becauserepeatability is measured through successive applicationsof the same measurement procedure under the same con-ditions of measurement (i.e. the same subject measuringthe same system at different times). If measurement isperformed by humans, we may assume that they remem-ber (at least to some extent) having measured the samesystem. The learning effects that may result from suchrecall are difficult to control in an experiment of thiskind.

As perception/intention-based variables, we used thePerceived Ease of Use (PEOU), Perceived Usefulness

(PU) and Intention to Use (ITU) variables of the MAM.The hypotheses tested in the experiment are summarized

below. The first three hypotheses relate to the first researchquestion while the last three hypotheses relate to the secondresearch question. Because the goal of the experiment wasto evaluate OOmFP, which was proposed to improve FPAas a function point measurement procedure when usingOO-Method, the direction of the effect has been stated inthe hypotheses.

HTime: OOmFP takes less time to apply than FPA.HRep: OOmFP results in a more reproducible functionpoint measurement than FPA.HAcc: OOmFP results in a more accurate function pointmeasurement than FPA.HPEOU: OOmFP is perceived to be easier to use thanFPA.HPU: OOmFP is perceived to be more useful than FPA.HITU: OOmFP is more likely to be used than FPA.

3.2. Participants

The participants in the experiment were twenty-two stu-dents in the final year of Engineering in Computer Science,majoring in Information Systems. The students agedbetween 22 and 24 years and they had similar backgrounds(they previously attended the same courses about softwaredevelopment and management). The subjects were chosenfor convenience, i.e., they were students enrolled in the

372 S. Abrahao, G. Poels / Information and Software Technology 49 (2007) 366–380

Software Development Environments course during theperiod of February until June of 2003.

The Software Development Environments course wasselected because it was an advanced unit (where studentslearn advanced techniques about software development).The necessary preparation for the experiment and theexperimental task itself fitted well into the scope of thecourse, as one of the course objectives is to learn studentsabout FSM in the OO-Method development context. Toreduce the risk of experimenter bias (i.e., students wantingto please us by indicating a preference for OOmFP), stu-dents were informed beforehand that their answers wouldnot affect their grade for the course and that the experimen-tal tasks were to be considered as exercises on FSM (also tomotivate the students to perform well and take the tasksseriously).

3.3. Design and instrumentation

In order to balance differences in ability between theparticipants as well as to get more data, this experimentused a within-subject design. The students were randomlyassigned into two groups, one group using FPA first andOOmFP second, and another group using OOmFP firstand FPA second. This counterbalancing procedure allowedus to cancel out possible order or learning effects due tosimilarities in the treatments, in particular the requirementto measure the same system in both treatments.

The instrumentation of the experiment consisted of auser requirements specification and the correspondingOO-Method conceptual schema describing a project man-agement system for a fictitious company. The requirementsdocuments used in the experiment followed the IEEE 830standard [39] and also included an Entity-Relationship dia-gram and screen prototypes.8 Prior to the experiment, therequirements specification and conceptual schema wereverified in a small pilot study and the obtained feedbackwas used to improve the understandability of the experi-mental materials. We also stress that it was carefully veri-fied that the conceptual schema obtained with OO-Method conformed to the user requirements specification.Therefore, the results obtained with both procedures canbe compared as they receive the same user requirementsas input, albeit in a different format.

During the experiment, participants could also refer topreviously used FSM training materials in the form ofinstructional slides describing FPA and OOmFP, a com-

8 The IFPUG manual describing the function point measurementprocedure [30] suggests that measurement may be based on one or moreof the following components: user requirements, data models (Entity-Relationship Diagrams), process models (Data Flow Diagrams) anddesigns of windows, screens, and reports. In some examples, the text-onlyrequirements specification document stands alone as the basis for themeasurement, but most of the examples presented in the manual include agraphical data or process model, and designs of windows, screens, andreports. We therefore also included an ER diagram and screen prototypesin the user requirements specifications used in the experiment.

plete example for both procedures, and a measurementguideline summarizing the measurement rules of bothprocedures.

Finally, there was a post-task survey with 14 items (seeAppendix A) used to measure the constructs of theMAM. These items were based on the measurement instru-ment proposed by Moody [23], which was itself based onthe original TAM instrument for PEOU, PU and ITU[36]. The items of the survey instrument were formulatedusing a 5-point Likert scale, using the opposing statementsformat. The order of the items was randomized and halfthe item statements negated to avoid monotonous respons-es. PEOU is measured using five items on the survey (items1, 3, 4, 6, and 9). PU is measured using six items on the sur-vey (items 2, 5, 8, 10, 11, and 13). Finally, ITU is measuredusing three items on the survey (items 7, 12, 14).

3.4. Operation and data collection procedures

All students participating in the experiment receivedtraining sessions on FPA and OOmFP. For each proce-dure, two sessions of 2 h were needed. In the first session,the measurement rules of the procedure were explainedand demonstrated using toy examples. In the second ses-sion, the application of the measurement rules on a com-plete case study was demonstrated. We are aware that atotal training of 4 h per measurement procedure is not suf-ficient to turn the students into experienced function pointcounters. The group of students available for the experi-ment therefore approaches a sample of novice users ofFSM methods, rather than a sample of function point anal-ysis experts. On the other hand, the main objective of thetraining was to enable a fair comparison between bothfunction point measurement procedures. Hence, from thepoint of view of the study’s internal validity, more impor-tant than the intensity and duration of the training was thatall participants were equally trained in OOmFP and FPA,so that this study variable was effectively controlled.

During the first run of the experiment, each participantperformed the measurement task using either OOmFP orFPA (depending on the group to which the participantwas allocated) and next completed the post-task survey.During the second run, the participants that used OOmFPin the first run, now used FPA and the participants thatused FPA in the first run, now used OOmFP. Next, thepost-task survey was completed for the second time. Sincethe same system was measured in both runs, participantsdid not receive feedback on their task performance beforecompleting the post-task survey. Giving the students feed-back after their first-run measurement task could haveimpacted their behavior and subsequent performance inthe second run of the experiment. To maintain the compa-rability of the data collected during both runs, no feedbackwas given after the second-run measurement task either.

The consequence of giving students no feedback on theirperformance before completing the post-task surveys isthat the perceptions of ease of use, usefulness and intention

Table 1Descriptive statistics for measurement time and results

Descriptive statistics Measurement time (h) Function pointvalue (FP)

FPA OOmFP FPA OOmFP

Mean 2.4105 3.0070 132.40 159.55Standard deviation 0.3334 0.6689 17.443 7.964Minimum 1.85 2.00 114 148Maximum 3.33 4.23 174 173

S. Abrahao, G. Poels / Information and Software Technology 49 (2007) 366–380 373

to use are based on perceptions of task performance (sub-jective), rather than actual task performance in terms ofreproducibility and accuracy of the measurement results(objective). As this is true for both treatments, the lack offeedback before answering the questionnaire does notthreaten internal validity. Furthermore, when completingthe post-task survey for the second time, participants canbase their answers on an implicit comparison of their per-ceptions of task performance for the two measurement pro-cedures used (e.g. did I find the second procedure easier touse or more useful than the first procedure?). Answeringthe questionnaire in a relative, comparative sense doesnot threaten construct validity because we do not interpretthe participant ratings in an absolute sense (i.e., as absolutevalue judgments).

To avoid a possible ceiling effect, there was no time limitfor the experimental tasks. We also controlled that nointeraction whatsoever between participants occurred.

The performance-based variables were collected usingdata collection forms. These forms were used to recordthe outputs of the measurement procedures and the timespent (controlled by us). Once the data were collected, weverified whether the forms were complete. As two studentsdid not record their FPA results, we took into account onlythe responses of the twenty remaining students.

4. Data analysis and interpretation of results

4.1. First research question

Descriptive statistics for measurement time (in hours)and measurement results (function point values) are pre-sented in Table 1. Both mean measurement time and func-tion point value are higher for OOmFP than for FPA. Thefull data set is shown in Table B-1 of Appendix B.

We first tested hypothesis HTime related to the efficiencyof the measurement procedures in terms of measurementtime. The Kolmogorov–Smirnov test for normality wasapplied to the differences of the paired observations. Asthis distribution was normal, we used the Paired t-test tocheck for a difference in mean measurement time betweenOOmFP and FPA.

The result of the test9 does not allow us to reject the nullhypothesis, meaning that we cannot empirically corrobo-rate that OOmFP takes less time to apply than FPA. Infact, the test shows exactly the opposite. The mean mea-surement time for FPA is significantly lower than that forOOmFP (see Table 2).

Next, we tested hypothesis HRep related to the effective-ness of the measurement procedures in terms of the repro-ducibility of the measurement results. To assessreproducibility, we used a practical statistic similar to that

9 All the collected data were analyzed according to the following levelsof significance: not significant (a > 0.1), low significance (a < 0.1), mediumsignificance (a < 0.05), high significance (a < 0.01) and very high signifi-cance (a < 0.001).

proposed by Kemerer [40]. This statistic is calculated as thedifference in absolute value between the function point val-ue produced by a participant i and the average functionpoint value produced by the other n � 1 participants inthe sample (using the same measurement procedure), rela-tive to this average value. Reproducibility scores (REP)were thus obtained for each observation by applying thefollowing equation:

REP i ¼

Pn

k¼1;k 6¼i

FPValuesk

n�1� FPValuei

Pn

k¼1;k 6¼i

FPValuesk

n�1

���������

���������

: ð1Þ

The lower the REPi score for a participant, the closerthe function point value obtained by that participant isto the values obtained by the other participants. Theobtained REPi scores for FPA and OOmFP are presentedin Table B-2 of Appendix B.

The Kolmogorov–Smirnov test indicated a non-normaldistribution of the differences in paired reproducibilityscores. Therefore, we used the non-parametric Wilcoxon

signed rank test for the difference in median reproducibilityscores. The result of the one-tailed test (see Table 3) allowsempirically corroborating that OOmFP (median value0.04) results in more reproducible function point measure-ments than FPA (median value 0.09).

Next, we tested hypothesis HAcc related to the effective-ness of the measurement procedures in terms of accuracy.The values obtained by the four CFPSs were 178, 142,156 and 153 FPs. Therefore, we used the mean CFPS value(157 FPs) to measure the subjects accuracy. To comparethe function point value produced by a participant againstthe mean value obtained by the CFPS, the Magnitude of

Relative Error (MRE) was used [41–44]. Thus, accuracyscores were obtained by applying the following equationfor each observation:

MREi ¼CFPS FPvalue� FPValuei

CFPS FPValue

����

����: ð2Þ

The lower the MREi score is for a participant, the closerthe function point value obtained by the participant is tothe mean function point value obtained by the CFPSs,which is considered as the ‘true value’ of functional size.The obtained MREi for FPA and OOmFP are presentedin Table B-2 of Appendix B.

Table 4Descriptive statistics for perception-based variables

Descriptivestatistics

Perceived ease ofuse

Perceivedusefulness

Intention to use

FPA OOmFP FPA OOmFP FPA OOmFP

Mean 2.7100 2.90 2.4800 3.5250 2.6333 3.6667Standard

deviation0.4517 0.7033 0.4606 0.4920 0.6657 0.6839

Minimum 1.80 1.60 1.60 2.80 1.33 2.33Maximum 3.60 4.40 3.60 4.60 3.67 4.67

Table 2Paired samples t-test rank for differences in mean measurement times (FPA versus OOmFP; a = 0.05)

Dependentvariable

Mean difference Standard deviation Standard error mean 95% conf. interval of the difference t One-tailed p-value

MeasurementTime

�0.596 0.7716 .17254 �0.957 (lower)�0.235 (upper)

�3.457 0.0015

Table 3One-tailed Wilcoxon signed rank test for differences in median reproduc-ibility and accuracy scores (FPA versus OOmFP; a = 0.05)

Dependentvariable

Meanrank

Sum ofranks

z One-tailedp-value

Reproducibility 5.25 10.50 �3.409 0.001Accuracy 3.50 7.00 �3.662 0.001

374 S. Abrahao, G. Poels / Information and Software Technology 49 (2007) 366–380

Because the Kolmogorov–Smirnov test indicated a non-normal distribution of the paired differences in MREibetween OOmFP and FPA, we again used the Wilcoxon

signed rank test for the difference in median accuracy mea-surements. The result of the test (see Table 3) allows us toempirically corroborate that OOmFP results in more accu-rate measurements than FPA. The median magnitude ofrelative error was 18% for FPA, whereas it was only 4%for OOmFP.

4.2. Second research question

The second research question concerned the partici-pants’ perception of OOmFP’s efficacy and intention touse OOmFP, as compared to FPA. The perception/inten-tion-based dependent variables were measured accordingto subject ratings, using a measurement instrument basedon the literature. The construct validity of this instrumentwas evaluated using an inter-item correlation analysis [45].

The evaluation is based on two criteria: convergent valid-ity and discriminant validity. Convergent validity (CV)refers to the convergence among the different items thatare meant to measure a same construct. The correlationbetween such items should be high. The CV value of an itemis calculated as the average correlation between the scoresfor that item and the scores for the other items that areintended to measure the same construct. Discriminantvalidity (DV) refers to the divergence of the items used tomeasure different constructs. The correlation between itemsused to measure different constructs should be low. The DVvalue of an item is calculated as the average correlationbetween the scores for that item and the scores for the itemsthat are intended to measure another construct. Low DVvalues indicate high discriminant validity.

According to [45] an item’s CV value must be higherthan its DV value; otherwise the scores on the item shouldnot be used in the data analysis.

An inter-item correlation analysis (see Table B-3 ofAppendix B) shows that for all PEOU and ITU items theCV value was higher than the DV value (with a minimumdifference of 0.20). The average convergent validity of allitems was 0.57 for PEOU and 0.62 for ITU. For PU, five

out of six items had a CV value higher than the DV value.The remaining item, Q11, has a same value for CV andDV. For this reason, the Q11 item was excluded from thedata analysis. Removing Q11 increased the average conver-gent validity of the PU items from 0.40 to 0.47.

The use of multiple items to measure a same constructalso requires the examination of the reliability or internalconsistency among the measurements [46]. We evaluatedreliability using Chronbach’s a. For this analysis, itemQ11 was excluded. All constructs have an alpha value equalto or greater than 0.7 (0.82 for PEOU, 0.70 for PU, and0.70 for ITU), a common reliability threshold for explor-atory research [47]. The overall Cronbach’s a obtainedfor the instrument (excluding item Q11) was 0.74.

A participant’s score for a perception/intention-basedvariable was calculated as the mean of the scores assignedto the different items of that variable. Descriptive statisticsfor the three variables of interest, perceived ease of use(PEOU), perceived usefulness (PU), and intention to use(ITU), are presented in Table 4. The full data set is shownin Table B-4 of Appendix B. Higher scores mean morepositive perceptions and intentions towards a measurementprocedure. Mean scores for all three variables are higherfor OOmFP than for FPA.

Before testing HPEOU, HPU and HITU, the Kolmogorov-Smirnov test for normality was applied to the differencesof the paired observations. As all distributions were nor-mal, we used the parametric Paired t-test to evaluate thestatistical significance of the observed differences in meanPEOU, PU and ITU.

The results of the one-tailed t-tests (see Table 5) do notallow us to empirically corroborate that OOmFP is per-ceived to be easier to use than FPA (HPEOU) given thatthe test result was not significant (a > 0.1). On the otherhand, the test results allow us to corroborate that OOmFPis perceived as being more useful than FPA and the partic-ipants’ intention to use it is higher than with FPA. The sta-tistical significance was found to be very high (a < 0.001)for HPU and HITU.

10 The use of student participants in empirical software engineeringstudies is relatively well accepted as they are the next generation ofprofessionals, see e.g. [48].

Table 5One-tailed paired samples t-tests for differences in mean PEOU, PU and ITU (FPA versus OOmFP; a = 0.05)

Dependent variable Mean difference Standard deviation Standard error mean 95% conf. interval of the difference t One-tailed p-value

Perceived ease of use 0.190 0.8620 0.1927 �0.213 (lower) 0.593 (upper) 0.986 0.1685Perceived usefulness 1.020 0.6582 0.1471 0.7119 (lower) 1.3281 (upper) 6.930 <0.001Intention to use 1.0333 0.8976 0.2007 0.6132 (lower) 1.4535 (upper) 5.148 <0.001

S. Abrahao, G. Poels / Information and Software Technology 49 (2007) 366–380 375

4.3. Summary and interpretation of the results

The results of the data analysis can be summarized asfollows:

• The function point measurements obtained usingOOmFP were more reproducible and accurate thanthose obtained using the IFPUG measurementprocedure;

• The IFPUG function point measurement proceduretook less time to apply than OOmFP;

• OOmFP was perceived to be more useful and morelikely to be adopted in practice than the IFPUG mea-surement procedure;

• There was no difference in perceived ease of use betweenthe OOmFP and IFPUG function point measurementprocedures.

For the performance-based variables, the results suggestthat OOmFP accomplishes its objectives, but takes moreeffort to apply than the IFPUG procedure. A plausible rea-son why the experiment participants spent more timeapplying OOmFP is that its measurement rules are verydetailed and were defined taking into account four dimen-sions of an object-oriented system (i.e. data, behavior, pro-cess and presentation). To apply each of the measurementrules, participants needed to verify the corresponding con-ceptual model specification. For example, to apply the rule‘‘counting 1 FTR for each new class referenced in the formu-

la of a precondition’’, they had to check all the precondi-tions defined in the Dynamic Model specification of thesystem being measured. It is thus likely that the highereffectiveness/lower efficiency of OOmFP versus the IFPUGprocedure is a result of the high degree of detail of theOOmFP measurement procedure, resulting in less interpre-tation freedom on behalf of the measurer, but also requir-ing a meticulous investigation of the conceptual schema.

From a practical point of view, the relative inefficiencyof OOmFP vis-a-vis the IFPUG procedure is not really aproblem, given that the use of OOmFP can be supportedby a tool called OlivaNova Function Point Counter [49].By the same token, the observed advantages of usingOOmFP with respect to effectiveness are less relevant whenfunction point measurement is tool-supported and thus lessreliant on human judgment.

With respect to the perception/intention-based variablesperceived usefulness and intention to use, the results suggestthat participants perceived OOmFP to be an improvementover the ‘regular’ IFPUG measurement procedure to sizeOO systems specified using OO-Method. The lack of empir-

ical support for a higher perceived ease of use is not surpris-ing given that OOmFP took more time to apply than theIFPUG procedure. But even then, the perceived ease of useof OOmFP was not lower than that of the IFPUG procedure.

Before we draw conclusions based on the experimentresults, the limitations of the study and their impact onthe study validity need to be discussed. The next sectionoutlines the lessons learned from conducting the experi-mental comparison of the OOmFP and IFPUG functionpoint measurement procedures, and suggests someimprovements to the experimental procedure applied.

5. Lessons learned

The main limitation of the study is the use of studentparticipants, in particular students with no prior experiencein function point analysis and having received only limitedtraining in function point counting (i.e. 4 h for each mea-surement procedure employed in the experiment). Theuse of student participants in our study does not presenta threat to the study’s internal validity. The students partic-ipating in our experiment were final year students, whichcan be considered as being close to the real populationunder study (i.e. software developers, project managers).10

Furthermore, the level of training received was the samefor both treatments, allowing us to control this variable.

The use of students does reduce the external validity andthus generalizability of the study results. The group of stu-dents participating in the experiment are only representa-tive of novice users of FSM. It is possible that noviceFSM users prefer measurement procedures in the form ofa ‘script’ to follow (as OOmFP presents) more than experi-enced FSM users do. In addition, we do not know if thesame performance differences as in the experiment wouldbe observed at higher levels of training. Therefore, the cur-rent study needs to be replicated using, preferably, practi-tioners experienced in function point analysis. If notavailable, replication studies can be performed with stu-dents, but at higher levels of function point counting train-ing in order to confirm or disconfirm the currently obtainedresults and to increase external validity.

A related problem and possible threat to the study’sinternal validity, though only with respect to the percep-tion-based variables, is experimenter bias. The student par-ticipants, well versed in OO-Method, might perceiveOOmFP as more useful than the IFPUG measurement

376 S. Abrahao, G. Poels / Information and Software Technology 49 (2007) 366–380

procedure because they associate it with the familiar OO-Method approach and related tools (all developed at theiruniversity), and not because of their perceived performanceon the experimental task. They might also indicate a higherintention to use OOmFP (instead of the standard IFPUGfunction point measurement procedure) just to please theexperimenters.

Although the students were informed that their participa-tion in the experiment would have no effect on their coursegrade, we acknowledge that to really eliminate experimenterbias, a replication using an ‘independent’ sample of studyparticipants would be ideal. An improvement to the experi-mental procedure we employed, could be the use of the dou-ble-blind procedure, where the identity of both participantsand experimenters is hidden. A difficulty would be that sucha group of participants would need to be trained in OO-Method as well, as OO-Method practitioners would proba-bly link OOmFP to the research group that developed OO-Method (making the double-blind procedure ineffective).An alternative change to our experimental procedure is toalso measure user preferences towards particular FSM mea-surement procedures independent of the performance ofexperimental tasks. With such additional measurements, itcan be investigated whether OOmFP’s higher perceived use-fulness and intention to use is due to (biased) subjective pref-erences or to perceived differences in task performance withthe IFPUG procedure (or to both reasons).

Another lesson we have learned from this study is thatthe FSM results reported by the participants need to besufficiently detailed to allow for an in-deep analysis of taskperformance. This issue is related to the construct validityof the accuracy measure we used. In our interpretation ofthe results, the higher accuracy with OOmFP was assumedto be a function of the measurement procedure offered,which, like a ‘script’, directs the user to the functionalityto include in the function point count. So, it might be thatnovice counters could find functionality with OOmFP thatthey would miss using the IFPUG procedure. An alterna-tive explanation could be that the OOmFP users foundfunctionality that did not really exist (i.e., which shouldnot be counted according to the FPA view on functionalsize) and that compensated for real but missed functional-ity. Although unlikely given the rigorous verification of theOOmFP rules performed before (see [20]), to rule out thisalternative explanation requires an analysis of the detailsof the function point counts (looking for ‘false positives’and ‘false negatives’ instead of just using the end resultsof the counting process). For our study, such a post-hocanalysis of the detailed FSM results was not possiblebecause of insufficient data recording.

A final limitation confining the generalizability of anyconclusions drawn from our experiment results is relatedto the experimental materials used. We tried to use a sys-tem that is representative of future uses of OOmFP. Con-sidering the function point values obtained by the CFPSs,the system is rather small (purposely though, to managethe feasibility of an experiment with students). It is at this

moment an open question whether our results scale up tolarger applications.

6. Conclusions and future work

A study was presented that evaluated OOmFP, a newfunction point measurement procedure for systems develop-ment using the OO-Method method. The goal was to verify ifand how much OOmFP improves the performance of func-tion point measurement of object-oriented systems specifiedusing OO-Method. The results indicate that, for novice usersof functional size measurement methods, OOmFP is moretime-consuming than the ‘standard’ IFPUG procedure forfunction point counting, but the measurement results aremore reproducible and accurate. The results also indicatethat OOmFP is perceived to be more useful and is more likelyto be adopted in practice than IFPUG-FPA in the context ofOO-Method systems development.

From a practical perspective, we are aware that thisstudy provides only first evaluation results. We have out-lined some critical factors and lessons learned from thisstudy that must be handled in future experiments. Animportant limitation to the generalizability of the resultswas the use of students as participants. These studentscan at most be seen as being representative of novice usersof functional size measurement methods. Nevertheless, thisexperiment has value as a pilot study for fixing the experi-mental procedure before trying it with industry practitio-ners. More experiments with a larger number ofparticipants (e.g. students that are better trained in FPAand/or experienced FPA users) must be carried out to con-firm the preliminary results obtained.

From a research perspective, our study adds newinsights into the problem of how to evaluate alternativefunctional size measurement methods or measurement pro-cedures. Our evaluation approach is original in the sensethat it combines objective measures of user task perfor-mance with measures of user perceptions and intentions.The basis of this approach is provided by the Method Eval-uation Model which was adapted to the functional sizemeasurement context to assess the different factors thatexplain the user intention to use a particular method.

Apart from conducting replication studies, in futurework we also plan to investigate the generality of OOmFPas a function point measurement procedure for object-ori-ented systems. Generality was a property suggested byMarch and Smith [22] that was not evaluated in this studyas the application context was restricted to OO-Method.We wish to find out which of the OOmFP measurementrules are specific to OO-Method and which are not.

Acknowledgement

We thank Claudia Hazan, David Cleary, RichardJohnston, and John Ogilvie for their valuable contribu-tion on the estimation of the requirements specificationdocument.

S. Abrahao, G. Poels / Information and Software Technology 49 (2007) 366–380 377

Appendix A. FSM method survey instrument

For each of the following paired statements, please mark a cross over the circle which most closely matches your opin-ion. There are no ‘‘right’’ answers to these questions – just give your honest opinion, based on your experience using thesizing method.

Please read each question carefully before giving your response

1.

TableMeas

Subjec

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

I found the procedure for applying the method complexand difficult to follow

B-1urement time and function point values, per treatment (FPA or

t First run

procedure

Second run

procedure

Measurement tim

in FPA (h)

FPA OOmFP 2.00

2.15

2.25

2.50

2.75

2.37

2.33

2.37

2.40

2.50

OOmFP FPA 3.33

2.67

2.50

2.54

2.85

2.25

1.85

2.45

2.15

2.00

O

OOm

e

O

FP)

O

Me

in

2.30

3.00

2.95

2.00

2.85

4.15

4.23

2.50

2.25

2.00

3.25

3.33

2.95

3.40

2.75

2.25

3.50

3.15

4.00

3.33

O

asure

OOm

O

ment

FP (

I found the procedure for applying the methodsimple and easy to follow

2.

I believe that this method would reduce the timerequired to measure object-oriented systems.

O

O O O O I believe that this method would increase the timerequired to measure object-oriented systems.

3.

Overall, I found the FSM method difficult to use O O O O O Overall, I found the FSM method easy to use 4. I found the measurement rules of the method clear

and easy to understand

O O O O O I found the measurement rules of the method

confusing and difficult to understand

5. Overall, I found the FSM method to be useful O O O O O Overall, I did NOT find the FSM method to be

useful

6. I found the FSM method difficult to learn O O O O O I found the FSM method easy to learn 7. I will use this method if I have to measure

object-oriented systems in the future

O O O O O I will NOT use this method if I have to measure

object-oriented systems in the future

8. I think that this method would NOT improve the

accuracy of estimates of object-oriented systems

O O O O O I think that this method would improve the accuracy

of estimates of object-oriented systems

9. I found it difficult to apply the FSM method to the

case study

O O O O O I found it easy to apply the FSM method to the case

study

10. Overall, I think this method does NOT provide an

effective way of measuring the functional size of OOsystems during the requirements phase.

O

O O O O Overall, I think this method provides an effectiveway of measuring the functional size of OO systemsduring the requirements phase.

11.

Using this method would improve my performance inmeasuring OO systems

O

O O O O Using this method would NOT improve myperformance in measuring OO systems

12.

I would be easy for me to become skilful in using thisFSM method

O

O O O O I would be difficult for me to become skilful in usingthis FSM method

13.

Overall, I think this method is an improvement to theFPA method

O

O O O O Overall, I think this method is NOT an improvementto the FPA method

14.

I intend to use this FSM method in the future O O O O O I do NOT intend to use this FSM method in thefuture

Appendix B. Measurement results

See Appendix Tables B-1–B-4.

time

h)

Size in FPA (FPS) Size in OOmFP (FPs)

129 150

125 158

115 172

124 155

143 151

128 160

114 158

118 148

145 160

159 165

121 151

122 170

165 163

174 171

132 163

114 158

147 173

121 148

125 163

127 154

Table B-2Reproducibility (REPi) and accuracy (MREi) scores, per treatment (FPA or OOmFP)

Subject REPiin FPA REPiin OOmFP MREi in FPA MREiin OOmFP

1 0.03 0.06 0.18 0.04

2 0.06 0.01 0.20 0.01

3 0.14 0.08 0.27 0.10

4 0.07 0.03 0.21 0.01

5 0.08 0.06 0.09 0.04

6 0.03 0.030 0.18 0.02

7 0.15 0.01 0.27 0.01

8 0.11 0.08 0.25 0.06

9 0.10 0.100 0.08 0.02

10 0.21 0.04 0.01 0.05

11 0.09 0.06 0.23 0.04

12 0.08 0.07 0.22 0.08

13 0.26 0.02 0.05 0.04

14 0.34 0.08 0.11 0.09

15 0.00 0.02 0.16 0.04

16 0.15 0.01 0.27 0.01

17 0.12 0.09 0.06 0.10

18 0.09 0.08 0.23 0.06

19 0.06 0.02 0.20 0.04

20 0.04 0.04 0.19 0.02

Table B-3Correlation between survey items (construct validity analysis)

Perceived ease of use Perceived usefulness Intention to Use Overall

Q1 Q3 Q4 Q6 Q9 Q2 Q5 Q8 Q10 Q11 Q13 Q7 Q12 Q14 CV DV Valid?

PEOU Q1 1,00 0,61 0,42 0,37 0,47 0,24 0,55 0,37 0,51 0,54 0,31 0,15 0,39 0,40 0,58 0,38 Yes

Q3 0,61 1,00 0,46 0,40 0,51 0,18 0,68 0,08 0,33 0,51 0,24 0,24 0,06 0,36 0,60 0,30 Yes

Q4 0,42 0,46 1,00 0,48 0,36 0,21 0,41 0,03 0,48 0,34 0,33 �0,24 0,07 �0,14 0,54 0,17 Yes

Q6 0,37 0,40 0,48 1,00 0,56 0,43 0,45 0,27 0,31 0,13 0,66 0,02 0,41 0,11 0,56 0,31 Yes

Q9 0,47 0,51 0,36 0,56 1,00 0,28 0,50 0,14 0,10 0,26 0,31 0,04 �0,12 0,19 0,58 0,19 Yes

PU Q2 0,24 0,18 0,21 0,43 0,28 1,00 0,57 0,22 0,00 �0,16 0,41 0,20 0,15 0,30 0,34 0,25 Yes

Q5 0,55 0,68 0,41 0,45 0,50 0,57 1,00 0,03 0,37 0,44 0,52 0,38 0,17 0,61 0,49 0,47 Yes

Q8 0,37 0,08 0,03 0,27 0,14 0,22 0,03 1,00 0,27 0,16 0,30 0,39 0,41 0,27 0,33 0,24 Yes

Q10 0,51 0,33 0,48 0,31 0,10 0,00 0,37 0,27 1,00 0,54 0,41 0,29 0,41 0,32 0,43 0,35 Yes

Q11 0,54 0,51 0,34 0,13 0,26 �0,16 0,44 0,16 0,54 1,00 0,13 0,33 0,13 0,55 0,35 0,35 No

Q13 0,31 0,24 0,33 0,66 0,31 0,41 0,52 0,30 0,41 0,13 1,00 0,14 0,43 0,20 0,46 0,33 Yes

ITU Q7 0,15 0,24 �0,24 0,02 0,04 0,20 0,38 0,39 0,29 0,33 0,14 1,00 0,14 0,85 0,66 0,18 Yes

Q12 0,39 0,06 0,07 0,41 �0,12 0,15 0,17 0,41 0,41 0,13 0,43 0,14 1,00 0,27 0,47 0,23 Yes

Q14 0,40 0,36 �0,14 0,11 0,19 0,30 0,61 0,27 0,32 0,55 0,20 0,85 0,27 1,00 0,71 0,29 Yes

Table B-4PEOU, PU and ITU scores, per treatment (FPA or OOmFP)

Subject PEOU in FPA PEOU in OOmFP PUin FPA PUin OOmFP ITU in FPA ITU in OOmFP

1 3.60 3.20 2.60 4.00 3.67 4.33

2 3.20 2.60 2.20 3.20 1.67 3.67

3 3.20 2.60 1.80 3.20 1.67 3.33

4 3.00 3.40 2.00 4.60 1.33 4.67

5 2.80 2.60 2.80 3.40 3.00 4.33

6 2.40 2.00 2.20 3.00 2.33 3.00

7 3.40 1.60 3.00 3.40 3.67 3.67

8 2.60 3.40 2.20 3.20 2.33 4.00

9 2.40 3.80 2.60 3.60 2.67 3.33

10 2.80 4.40 3.60 4.40 3.67 3.67

11 1.80 2.80 2.60 3.40 2.33 3.33

12 2.20 2.60 2.80 3.20 3.33 4.33

13 2.80 2.60 2.80 2.80 2.33 2.33

14 2.60 2.40 2.00 3.00 2.67 3.33

15 2.20 2.60 2.80 3.60 2.00 4.00

16 2.40 3.80 2.40 3.60 2.67 2.33

17 3.20 3.40 3.40 3.80 3.00 4.33

18 2.60 2.60 2.40 3.80 3.00 3.67

19 2.60 2.0 2.80 2.80 2.67 3.00

20 2.40 3.60 1.60 4.00 267 4.67

378 S. Abrahao, G. Poels / Information and Software Technology 49 (2007) 366–380

S. Abrahao, G. Poels / Information and Software Technology 49 (2007) 366–380 379

References

[1] ISO, ISO/IEC 14143-1-Information Technology – Software Mea-surement – Functional Size Measurement. Part 1: Definition ofConcepts, 1998.

[2] A.J. Albrecht, Measuring application development productivity, in:Proceedings of the IBM Application Development Symposium, 1979,pp. 83–92.

[3] IFPUG, Function Point Counting Practices Manual, Release 4.1,International Function Point Users Group, Westerville, Ohio, USA,1999.

[4] L. Laranjeira, Software size estimation of object-oriented systems,IEEE Transactions on Software Engineering 16 (1990) 510–522.

[5] R.D. Banker, R.J. Kauffman, R. Kumar, An empirical test of object-based output measurement metrics in a computer aided softwareengineering (CASE) environment, Journal of Management Informa-tion Systems 8 (Winter 1991–92), 127–150.

[6] E. Rains, Function points in an ADA object-oriented design, OOPSMessenger 2 (1991) 23–25.

[7] N. Thomson, R. Johnson, R. MacLeod, G. Miller, T. Hansen, Projectestimation using an adaptation of function points and use cases forOO projects, in: Proceedings of the OOPSLA’94 Workshop onPragmatic Theoretical Directions in Object-Oriented Software Met-rics, Position Paper, 1994.

[8] H. Zhao, T. Stockman, Software sizing for OO software development– object function point analysis, in: Proceedings of the GSEConference, Berlin, Germany, 1995.

[9] H.M. Sneed, Estimating the development costs of object-orientedsoftware, in: Proceedings of the Seventh European Software Controland Metrics Conference, Wilmslow, UK, 1996, pp. 135–152.

[10] F. Minkiewicz, Measuring object oriented software with predictiveobject points, in: A.C. Rob Kusters, Fred Heemstra, Erik vanVeenendaal (Eds.), Project Control for Software Quality, ShakerPublishing, 1999.

[11] ASMA, Sizing in Object-Oriented Environments, Australian SoftwareMetrics Association (ASMA), Victoria, Australia 1994.

[12] J. Kammelar, A Sizing Approach for OO-environments, Proceedingsof the 4th International ECOOP Workshop on QuantitativeApproaches in Object-Oriented Software Engineering, Cannes,2000, pp.

[13] G. Antoniol, F. Calzolari, Adapting function points to objectoriented information systems, Proceedings of the 10th Conferenceon Advanced Information Systems Engineering (CAiSE’98) (1998)59–76.

[14] D.J. Ram, S.V.G.K. Raju, Object oriented design function points, in:Proceedings of the First Asia-Pacific Conference on Quality Software,IEEE Press, 2000, pp. 121–126.

[15] O.A. Lehne, Experience report: function points counting of objectoriented analysis and design based on the OOram method, Proceed-ings of the Conference on Object-Oriented Programming Systems,Languages, and Applications (OOPSLA’97) (1997).

[16] T. Fetcke, A. Abran, T.H. Nguyen, Function point analysis for theOO-Jacobson method: a mapping approach, in: Proceedings of theFESMA’98, Antwerp, Belgium, 1998, pp. 395–410.

[17] T. Uemura, O. Kusumoto, I. Katsuro, Function-point analysis usingdesign specifications based on the unified modelling language, in:Journal of Software Maintenance and Evolution: Research andPractice, vol. 13, John Wiley & Sons, Ltd., 2001, pp. 223–243.

[18] ISO, International Vocabulary of Basic and General Terms inMetrology, second ed., 1993.

[19] ISBSG, International Software Benchmarking Standards Group,<http://www.isbsg.org>.

[20] S. Abrahao, G. Poels, O. Pastor, A functional size measurementmethod for object-oriented conceptual schemas: design and evalua-tion issues, Software and System Modeling (SoSyM) 5 (2006) 48–71.

[21] O. Pastor, J. Gomez, E. Insfran, V. Pelechano, The OO-methodapproach for information systems modelling: from object-oriented

conceptual modeling to automated programming, Information Sys-tems 26 (2001) 507–534.

[22] S.T. March, G.F. Smith, Design and natural science research oninformation technology, Decision Support Systems 15 (1995) 251–266.

[23] D.L. Moody, Dealing with complexity: a practical method forrepresenting large entity relationship models, in: Department ofInformation Systems, University of Melbourne, Melbourne, Austra-lia, 2001, p. 354.

[24] D. Garmus, D. Herron, Function Point Analysis: MeasurementPractices for Sucessful Software Projects, 2000.

[25] C.A. Dekkers, S. Cartwright, Can function points be used to size real-time, scientific, object-oriented, extreme programmed, or web-basedsoftware? IT Metrics Strategies, Cutter Information Corp 7 (2001).

[26] T. Fetcke, A. Abran, R. Dumke, A generalized representation forselected functional size measurement methods, in: Proceedings of the11th International Workshop on Software Measurement, ShakerVerlag, Montreal, Canada, 2001, pp. 1–25.

[27] C.R. Symons, Software Sizing and Estimating: Mk IIFPA (FunctionPoint Analysis), John Wiley & Sons, 1991.

[28] A. Abran, J.M. Desharnais, S. Oligny, D. St.Pierre, C. Symons,COSMIC-FFP Measurement Manual, The COSMIC Implementa-tion Guide for ISO/IEC 19761, Version 2.2, The Common SoftwareMeasurement International Consortium, January 2003.

[29] ISO, ISO/IEC 14143-2 – Information Technology – Softwaremeasurement – Functional Size Measurement. Part 2: ConformityEvaluation of Software Size Measurement Methods to ISO/IEC14143-1:1998,(2002).

[30] ISO, ISO/IEC 14143-3 – Information Technology – SoftwareMeasurement – Functional Size Measurement – Part 3: Verificationof Functional Size Measurement Methods, 2003.

[31] ISO, ISO/IEC TR 14143-4: Information Technology – SoftwareMeasurement – Functional Size Measurement – Part 4: ReferenceModel, 2002.

[32] ISO, ISO/IEC TR 14143-5: 2004, Information Technology – SoftwareMeasurement – Functional Size Measurement – Part 5: Determina-tion of Functional Domains for Use with Functional Size Measure-ment, 2004.

[33] J.P. Jacquet, A. Abran, Metrics validation proposals: a structuredanalysis, in: Proceedings of the Eighth International Workshop onSoftware Measurement, Magdeburg, Germany, 1998, pp. 43–60.

[34] C.K. Riemenschneider, B.C. Hardgrave, F.D. Davis, Explainingsoftware developer acceptance of methodologies: a comparison of fivetheoretical models, IEEE Transactions on Software Engineering 28(2002) 1135–1145.

[35] N. Rescher, The Primacy of Practice, Basil Blackwel, Oxford, 1973.[36] F.D. Davis, Perceived usefulness, perceived ease of use and user

acceptance of information technology, MIS Quarterly 13 (1989).[37] R. Van Solingen, E. Berghout, The Goal/Question/Metric Method: A

Practical Guide for Quality Improvement of Software Development,McGraw-Hill, 1999.

[38] C. Wohlin, P. Runeson, M. Host, M.C. Ohlsson, B. Regnell, A.Wesslen, Experimentation in Software Engineering: An Introduction,2000.

[39] ANSI/IEEE, Standard 830-1998, in: IEEE Recommended Practicefor Software Requirements Specifications: The Institute of Electricaland Electronics Engineers, Ed., IEEE Computer Society Press, NewYork, NY: 1998.

[40] C.F. Kemerer, Reliability of function points measurement, Commu-nications of the ACM 36 (1993) 85–97.

[41] S.D. Conte, H.E. Dunsmore, V.Y. Shen, Software EngineeringMetrics and Models, The Benjamin/Cummings Publishing Company,Inc., 1986.

[42] L.C. Briand, K. El Emam, D. Surmann, I. Wieczorek, K.D. Maxwell,An assessment and comparison of common software cost estimationmodeling techniques, in: Proceedings of the 21st InternationalConference on Software Engineering, 1999, pp. 313–322.

380 S. Abrahao, G. Poels / Information and Software Technology 49 (2007) 366–380

[43] L.C. Briand, J. Wust, The impact of design properties on develop-ment cost in object-oriented systems, in: Proceedings of the SeventhInternational Software Metrics Symposium (METRICS’2001), IEEEPress, London, England, 2001, pp. 260–271.

[44] R.K. Smith, J.E. Hale, A.S. Parrish, An empirical study using taskassignment patterns to improve the accuracy of software effort estima-tion, IEEE Transactions on Software Engineering 27 (2001) 264–271.

[45] D.T. Campbell, D.W. Fiske, Convergent and discriminant validationby the multitrait-multimethod matrix, in: Psychological Bulletin, vol.56, 1959, pp. 81–105.

[46] D.W. Straub, Validating instruments in MIS research, MIS Quarterly13 (1989) 147–169.

[47] J. Nunally, Psychometric Theory, second ed., McGraw-Hill, NewYork, NY, 1978.

[48] B.A. Kitchenham, S. Pfleeger, et al., Preliminary guidelines forempirical research in software engineering, IEEE Transactions onSoftware Engineering 28 (2002) 721–734.

[49] I. Torres, F. Calatrava, Function points counting on conceptualmodels: Whitepaper, CARE Technologies, <http://www.care-t.com/html/whitepapers.html>, 2003.