Defining ‘success’ for software projects: An exploratory revelation

INTERNATIONAL JOURNAL OF

www.elsevier.com/locate/ijproman

International Journal of Project Management 24 (2006) 358–370

PROJECTMANAGEMENT

Defining �success� for software projects: An exploratory revelation

Nitin Agarwal a,*, Urvashi Rathod b

a Indian Institute of Management Indore, Pigdamber, Rau, Tahsil: MHOW, Indore, MP 453 331, Indiab ICFAI Business School Indore, Rajiv Gandhi Chouraha, Piplyapala, Indore, MP 452 017, India

Received 3 May 2005; received in revised form 12 July 2005; accepted 8 November 2005

Abstract

Success is found relatively rare in the world of software projects. One possible reason may be the difference in the perception of themeaning of �success� in the minds of people who evaluate the project performance. Usually, stakeholders external to the project organi-zation use target cost and time for inferring �project success� while those internal to project agree that the attainment of �scope� of devel-opment decides the �project success�. Hence, project success criteria, as believed by different groups of stakeholders, do not match. In thisstudy, we examine the views of one such internal set of stakeholders: Programmer/Developers, Project Managers and Customer AccountManagers. We conducted an exploratory survey to determine their view of a successful project. We found surprising uniformity in dif-ferent constituents of this particular group of stakeholders and all of them overwhelmingly considered meeting the �scope� of softwareprojects, which comprises the functionality and quality of the project outcome, as the highest determinant of success. We believe that thisview of software project success in the eyes of software professionals should be studied further to build a comprehensive project evalu-ation framework and should be utilized effectively to achieve success in terms of external objectives like customer satisfaction and cus-tomer happiness.� 2005 Elsevier Ltd and IPMA. All rights reserved.

Keywords: Managing projects; Success and strategy; Software project success criteria

1. Introduction

Various authors have investigated the reasons for thepoor performance of Software projects [1–4]. Brooks [5]concluded way back in 1987 that there is no silver bulletto overcome this problem. Unfortunately these observa-tions are as true today as back then. Though the reasonsfor lack of success in Software Projects have been wellresearched, few have looked at the very notion of a ‘‘Suc-cessful Software Project’’. Performance of a software pro-ject is assessed in terms of its ability to attain the targetcost, time and the desired level of product�s quality [2,3]and therefore, a software project is considered to be suc-cessful if it delivers the product with pre-agreed level ofquality within the given time and cost.

0263-7863/$30.00 � 2005 Elsevier Ltd and IPMA. All rights reserved.

doi:10.1016/j.ijproman.2005.11.009

* Corresponding author. Tel.: +91 731 2399101–109x110; fax: +91 7312399115.

E-mail address: [email protected] (N. Agarwal).

Unfortunately, the field of software development lacksreliable models to estimate the project�s cost and time withprecision. Also, a framework to guide flawless delineationof desired features and functionality of software is notavailable to evaluate their presence in the software product.In the absence of dependable tools for deciding the targets,it appears to be unwise to use specific targets as the criteriato label a project as the �successful� or �failed�. Moreover,what happens if a project delivers the product in given timeand budget with lesser functionality or the project cost highbut developed product with desired functionality withingiven time, should it be considered as completely �failed�project? Thus, in practice, it may be very difficult to claimthat the project was really successful or not. We observethat while successful software projects are hard to defineand measure, the ‘‘not successful software projects’’ cate-gory is even more ambiguous.

A Standish Group study, popular as the Chaos Report[3], handled this issue by calling a software project �successful�

mailto:[email protected]

N. Agarwal, U. Rathod / International Journal of Project Management 24 (2006) 358–370 359

if it meets all three aforesaid project objectives and �chal-lenged� if it fails in achieving any or all three of the objec-tives. Instead of declaring a project a �failure�, this reportconsidered a project as �Canceled� if it is terminated beforecompletion. However, such short-term perspective of pro-ject performance ignores the fact that a successful projectmay deliver a software product that is not usable or doesnot satisfy the user�s functional needs effectively. One needsto accept that a project can be evaluated from several per-spectives and restricting the performance criteria to cost,time and quality limits the subjectivity, which is inherentto the software projects.

In this paper, we are not so much concerned with howexactly to determine or classify whether a project is success-ful or failed but our main concern is to know about thenotion of success of a project as held by individuals work-ing on it. Given that there is a chance that the meaning of�success� could be different for different stakeholders of asoftware project, whether that be the customer, or user,or developer, or project manager or the owners of the soft-ware development organization and this difference canhave implications on the very �success� of the projects, thelack of research in this direction is surprising. In this paper,we aim to explore the notion of �success of projects� accord-ing to various stakeholders, which are internal to the soft-ware development organization.

1.1. Literature review

The literature on Project Management often mentionsCost, Time and Quality as the project success criterionthough there are many skeptics [6–10]. Several of themhave suggested the need for consideration of additionalproject success criteria like profitability, business successand meeting expectation, which are suitable to all stake-holders of the project. Additionally, a few empirical studies[8,11–13] conducted in diverse industrial environment withdifferent objectives support the inclusion of stakeholders�views in determining the project success.

There are two empirical studies dealing specifically withsuccess criterion for Software Projects: One is by Wateridge[12] where he did a survey of project managers, system ana-lysts, sponsors and users to find the five most important cri-teria for success of Information Systems/InformationTechnology (IS/IT) projects. This study concluded that �tomeet user requirements� is the most important success crite-rion for IS/IT projects according to the �users� and the �pro-ject managers� though it appeared to mean differently toboth the groups. Users reckoned �happy users� as the secondimportant criterion while project managers ranked that atthe seventh place in importance. For project managers,�meets budget� and �meets time� followed �to meet userrequirements� respectively in the order of significance. Thisindicates that users associate �meeting their requirements�with �their happiness� while project managers focus on thebudget and schedule targets set by the senior managementinstead of delivering a system that makes the users happy.

The other study is due to Linberg [13], which was a casestudy based on a software project. This study revealed thedeviation in software professionals� perception of projectsuccess from the conventional notion of meeting budget,schedule and quality targets. Developers link the successof a completed project to the quality of the product whilefor a cancelled one it is tied with the learning from the pro-ject that can be used in the next project. They link the suc-cess of their development efforts with the work satisfactionthey get through creativity and learning while working on aproject.

We contend that the success criteria may focus on twodifferent aspects of a project: (1) the characteristics internalto the project organization like time, cost and scope attain-ment and (2) the characteristics external to the projectorganization like customer satisfaction with the projectoutcome [8,11]. The first one is useful from the point ofview of execution, monitoring and control of the projecthaving short-term significance while the second criterionis more important from the point of view of value of delive-rables to the users of the projects outcome leading to long-term impact. Project�s internal characteristics connectdirectly to its technical aspects while external characteris-tics deal with its commercial implications. An undeniablefact about the external aspect of project performance isits vulnerability to several hidden and unknown factorsof real world, like user-developer relationship, which maybe instrumental in deciding the project success for thestakeholders external to it. Nevertheless, an ideal choicewill be to strike a suitable balance and pursue both of themtogether in a manner so that the internal organization whilesatisfying the first set of criterion delivers outcome whichsatisfies the second set.

Clearly, findings by Wateridge and Lindberg suggestthat the software development organizations are moreinternally focused. Though it is possible that both internaland external objectives can be met when developers andmanagers pursue their creative zeal to satisfy customerneeds, ample room for possible conflicts remains whenthe developers (and others) pursue their creativity withoutregard to customer needs.

1.2. Research problem

In this study, we restrict ourselves to the consensusamongst the project team members on the performancecriteria. Is there a consensus within project�s internalorganization about the assessment of project success andfailure? Do they have the same view of a broad and oftenobscure milieu of software development? Software projectorganization does not comprise a unit entity like projectmanager or programmer but it is a thickly knitted meshof interacting people, usually labeled as Software Profes-sionals, involved in intellectual activities to attain the pro-ject goals. Therefore, it is necessary to understand theirview of project success and to examine whether the pro-fessionals working in different capacities agree on their

360 N. Agarwal, U. Rathod / International Journal of Project Management 24 (2006) 358–370

notion of project success or not. On this premise, weopted to explore the answers to the following researchquestions:

1. Which characteristics of software project dominate thenotion of project success in the minds of softwareprofessionals?

2. Do the software professionals with diverse responsibili-ties in software projects have similar or different notionof project success?

To address above questions, we asked three potentially dif-ferent interest groups of software professionals namelydevelopers, project managers and customer accounts man-agers, that what would be their priority, the time, cost andscope in case of (1) some urgency in project schedule, (2) ingeneral, if the objective is to be isolated and (3) in the orderof preference.

1.3. Research findings

Our exploratory survey results imply that

� Cost, Time, Functionality and Quality remain theimportant criteria for assessing the performance of soft-ware projects in the mind of professionals,� Irrespective of different responsibilities in a development

organization, all software professionals unanimouslyselected scope, and more specifically functionality withinscope, as the foremost criterion in defining the softwareproject success. Even urgency like market schedule maynot reduce the importance of functionality or scope asthe top success criterion.� Additionally, our findings suggest that, in contrast to

general belief, cost is considered to be the least impor-tant criterion for inferring a project�s success. This una-nimity among different sets of software professionals issurprising because one would expect Customer AccountManagers to adopt more market oriented view whileproject managers would be expected to be favoring busi-ness goals of the software development organization.

Based on these results, we submit following propositionsfor investigation in a future study:

� Proposition 1. Software professionals in Indian Industryconsider project�s cost, time and scope targets as majorsuccess criteria.� Proposition 2. Software professionals in Indian Industry

give precedence to project�s scope over other successcriteria.� Proposition 3. Software professionals in Indian Industry

consider that meeting project�s cost target is the leastimportant criterion for evaluating project�s performance.

We contend that the insistence on reporting cost andtime overruns [13,4,14] for claiming project�s success

diverts the attention from the crucial issue of completionof product�s functionality. Use of estimated budget andschedule for success evaluation assumes the efficiency ofestimation models, which is a debatable issue for years tocome. Moreover, as pointed out by Kitchenham et al.[15] that in this process, one cannot ignore the possibilitythat estimated budget and schedule may influence manag-ers to achieve those targets and thus, introduce a bias intheir project management approach. As a result, a projectthat could have been finished in 4 months might take 6months for completion because the project duration wasestimated as 6 months in the beginning. We observe at thispoint that the consideration of evolving estimation tech-niques as the building blocks for performance measure-ment of software projects is not a healthy practice andindustry must work to develop more inclusive criteriaframework before proclaiming the poor success in thisindustry.

In comparison to earlier works, this study does not differonly in terms of the research setting, i.e., the coverage of awider range of software professionals, but also evaluatesthe results on a wider band of influencing situations, i.e.,general and situation specific assertions from respondents.The depth on people dimension in the organization andbreadth on project expectations dimension make it morecomprehensive study. Its exploratory nature opens upmore avenues to further study.

We hope that our work proves to be a step towardsunderstanding and converging the needs of various stake-holders like users, customers, organization, managers anddevelopers so that the chances of success in software pro-ject can be increased. In the following section, we firstdescribe various constructs used in this research. Then,we describe the research methodology followed by analysisresults, inferences and discussion. We end with pointingout weaknesses in our results and scope for further work.

2. Research constructs

In general, a project brings in the picture of some groupactivity aided by tools and techniques to achieve certaingoals within given schedule and cost constraints. Thisimpression is echoed in one of the definitions of projectgiven by Turner [16],

A project is an endeavor in which human, material and

financial resources are organized in a novel way, to

undertake a unique scope of work, of given specification,

within constraints of cost and time, so as to achieve ben-

eficial change defined by quantitative and qualitative

objectives.

In the purview of given definition, we observe that thecomputer software is developed in projects since itencompasses a unique scope of work with given specifica-tion which need to be completed in a given time at givencost. Additionally, software is developed to facilitate thework for customers and users. Software projects also


provide an opportunity to enhance professional experi-ence and skills of the people worked on it [13]. The de-gree, to which such beneficial changes have beenachieved within the given constraints of cost and time, re-mains the matter of curiosity for all involved. Evaluationof the performance of software projects involves thequantification of the extent to which the given objectivesof a software project have been achieved.

We observed that the Iron Triangle of cost, time andscope is included in all definitions of project. Assertionsby B. Reiss [9] that a project is a human activity that

achieves a clear objective against a time scale and by Wright[17] that Time and Budget are the twin imperatives of a pro-

ject sponsor are the best way to communicate their signifi-cance. But a project cannot be fully successful or afailure if seen from the eyes of all the stakeholders involvedin it. The same outcome of a project may mean differentthings to different people involved in it [12]. Linberg [13]found that software developers considered a project suc-cessful albeit it get cancelled before arriving at a conclusiveoutcome because it resulted into substantial learning thatcould be applied to future projects. Baker et al. [6] hasdefined project success as follows:

The project is considered an overall success if the project

meets the technical performance specification and/or mis-

sion to be performed, and if there is a high level of satis-faction concerning the project outcome among key people

in the parent organization, key people in the project team

and key users or clientele of the project effort.

Project stakeholders, like project manager, top-manage-ment of parent organization of the project, customer-clientorganization, team members [8,9] government, local politi-cians and several other groups view project success differ-ently [6]. Therefore, project success becomes a matter ofthe filters used by different stakeholders. Shenhar and Levy[8] argued that different people in the same organizationwill view success in different ways and Wateridge [12] nar-rated the hypothetical cases when project is perceived asfailed if the expected benefit is not delivered to the clientorganization.

This takes our attention to the project characteristicslike attainment of schedule, budget and specification tar-gets, which are internal to the project organization [8]and the external characteristics like product performance,customer satisfaction, profitability, revenue growth etc.[12]. Internal characteristics represent the efficiency ofthe project process and are used to measure the perfor-mance of an ongoing project while the external character-istics describe the effectiveness of product characteristics.Assessment fabricated around The Iron Triangle reflectsthe success or failure of project implementation process

[11] while the perceived value of the project, which relateswith the project team�s perception of the value and use-fulness of the project deliverables to the customer andthe client satisfaction with the delivered project are exter-nal to the scope of project process. We also notice that

the internal characteristics can be measured when theproject is in delivery phase or immediately after thatwhile the external characteristics like impact on custom-ers can be measured in a couple of weeks after implemen-tation and business success in a couple of years after that[9]. Thus, success criterion is variable in time [8] thatturns a project failed at delivery stage into a success aftera few years [12].

Keeping the exploratory nature of this study, we lim-ited the scope of our endeavour to investigating softwareprofessionals� perception of project success in terms ofinternal characteristics and define the software project suc-cess as

A software project�s ability to meet the scope that encom-

passes the software specifications in terms of functionality

and quality, within budget and schedule, by adopting

proper process, tools and techniques.

As we have discussed in the beginning that the unavail-ability of right estimation tools leads to the prevailingskepticism about validity of success criteria, we contendthat failing in meeting one of the aforesaid objectivesmakes the project �challenged� and not �failed�. Also, theability of project management to identify and employthe right process, tools and techniques for controllingthe project to deliver the desired outcome in terms ofset norms of cost, time and quality is implicit in this def-inition. Eventually, the efficiency and effectiveness of pro-ject management is expected to handle the issues ofimproper estimations and changing requirements favor-ably. Further, consideration of internal characteristicsfor evaluation limits the scope of this study to theshort-term performance of projects.

In the given definition, we propose four constraints on asoftware projects, viz., budget, schedule, functionality andquality. IEEE (Institute of Electrical and Electronics Engi-neers) standard glossary on software engineering termspublished in 1990 defines quality as the degree to which

the system, component or the process meets specified require-ments. For a software project requirements are specified interms of software functionality, features, non-functionalrequirements of accuracy, speed, scale, reliability, main-tainability etc. General understanding of the term softwarequality coincides with the non-functional requirementswhile the desired functions and features constitute the soft-ware functionality. We observe that in the beginning ofsoftware projects, software specifications about functional-ity and quality are defined in broader arena of the software

scope. For eliminating any possible confusion betweenthese terms in the minds of software professionals we usescope for implying the combined duo of functionality andquality. Thus, we reduce the criteria internal to softwareprojects to budget, schedule and scope for the sake of clar-ity of terms. We split the scope in functionality and qualityonly when we need to find the difference in importancegiven to them individually for defining the project successcriteria.

1 National Association of Software and Service Companies of India.2 The Capability Maturity Model of Software Engineering Institute,

Carnegie Mellon University, USA.


3. Research methodology

Present work is a part of a larger research study relatedto software projects and therefore, the population andinstruments are shared with the parent research. Followingsub-sections describe these and the analysis techniquesused:

3.1. Population characteristics

Software professionals connote a vast segment of man-power that handles various aspects of the activities associ-ated with the creation of software. We consider here thatfrom the person who collects the requirements from the cli-ent to the data entry operator who converts client�s manualdata in electronic form for implementation and operation-alization of the software, all constitute the community ofsoftware professionals. For this research, we targeted threemajor segments of this community.

1. Software Developers. These are the technical people whoare involved in analysis of requirements, design of thesoftware, coding, testing and technical engineers whowork on the implementation and operationalization.

2. Project Managers. They manage the project activities.They are responsible to plan, organize, monitor andcontrol the resources and activities to attain the definedgoals of the project. Additionally, some organizationsrequire the project managers to interact with the clients.Usually, a project manager is responsible for more thanone project at a time.

3. Customer Account Manager. He/she is the crucial linkbetween the customer and the development organizationand is expected to anticipate, learn and then translatethe customer needs to the project team (may be directlyor through Project/Program Manager) in the languageunderstandable by the latter and convey the constraintsof the development organization to the customers.

Our selection of these groups of professionals wasinspired by the variety of work individual group isinvolved with during the project. Since software is invis-ible and highly malleable [5], software developers are in aunique position to influence the nature of software pro-jects outcome despite many organizational controls,which are typically put on them. They effectively havetheir own evaluation parameters for the work they doand hence it is important to understand their view of asuccessful software project. Project Managers representthe internal organization and often have to make choicesbetween various goals and allocate resources accordingly.As a result, their perception of project success is liable todrive the project activities and hence has importantimpact on its success. Customer Account Manager isthe link between the project organization, parent organi-zation and customer organization. Their perception ofproject success will help us to understand the conflict,

if any, among the project stakeholders on the issues,which are internal to the projects.

For the parent research, we contacted about 61 Indiansoftware organizations by sending e-mails, which werelisted in NASSCOM1 [18] directory as the companies thathad achieved high levels of CMM2 and hence, higher levelsof software process maturity. This ensured that the work-ing professionals had a knowledge of systematic conductand review of projects. We chose Indian software contractindustry for its proximity to researchers and the amount ofexposure to different types of software projects. Wereceived inclination of 40, i.e., 66% of the contacted orga-nizations to meet personally regarding the study. Around19 out of 40 organizations visited (48%) agreed for partic-ipation. We received filled questionnaires from 12 organi-zations i.e., 63% of agreed, 30% of visited and 20% ofcontacted organizations responded.

We requested 19 participating organizations to get theresponses from five personnel from each defined categoryof programmers, project managers and customer account/delivery managers. In this way, 15 questionnaire responseswere expected from each organization, making the expectedsample size equal to 19 * 15 = 285, from which only 105questionnaires were received, leading to the response rateof 37%. These organizations are developing telecommuni-cation, manufacturing, engineering, finance, banking,insurance and e-business solutions and involved in custom-ized as well as product development. This indicates that therespondents were aware of commercial software develop-ment projects.

As the organizations were selected on the basis of theprocess maturity levels scored by them and the profession-als from specific groups filled the questionnaires, the sam-ple is strictly purposive, random only in terms of thepeople who belong to different strata. More than 80% ofthe questionnaires were received by e-mails from the peoplewho were assigned the responsibility of data collection inthe participating organizations.

3.2. Data collection instrument

Three questions as a part of a larger questionnaire werededicated to the present research (refer to Appendix A).First two questions treat functionality and quality underthe flag of scope. Though the terms functionality and qual-ity are used separately in the first question, but their impor-tance and non-importance is handled together in the samedirection. In second question, we put them together as thescope for isolation of a single criterion pertaining to theprocess characteristics, i.e., cost and time and scope.

First question gives a scenario explaining the situationwhen the software product is meant to reach in the marketwithin stringent schedule constraint. Respondents were

Table 2Organizational characteristics

Feature Average Maximum Minimum

Age from the year ofestablishment (in years)

12.7 26 4

Employee size (numbers) 5993 27,000 140Client size (numbers) 110 300 15


asked to make their choice in a combination of factors likeincrease in cost, compromise in scope and delay in productshipment. We assumed that a software development unitopts for an increase in project time or cost for achievingthe predefined scope. If scope is attainable in estimatedtime and cost, there is no need to increase time or cost.An increase in cost and time along with a decrease in scopeis an exceptional situation caused by some unusual externalconstraints like lack of fund, change in problem domain orirrelevance of software product in the user domain due todelayed delivery. We do not deny the presence of the caseswhen an increase in time or cost or both is preferred withsome reduction in scope but that may not be a commonpractice. Our assumption helped us in keeping the questioncomprehensible to the respondents. We offered them theoptions of choosing one of the following situations:

1. Time and cost remain the same and scope is reduced.2. Time and scope remain the same with increased

resources.3. Cost and scope remain the same with increased time.4. Time and cost both increase for scope completion.

Second question of the questionnaire aims to isolate theindividual success criterion that has greater relative impor-tance to the respondent with respect to others. This ques-tion also gave a space to respondents for specifying thesuccess criterion that is most important to him/her otherthan budget, schedule and scope. Third question askedthe respondent to rank the time, cost, functionality andquality in order of their relevance and also, gave him/heran opportunity to mention two other success criteria thatappeal him for their importance.

Appendix A and Table 1 explain the way, in which, theresponses to these questions were recorded for analysis.Variables for the first question were Time_1, Cost_1 andScope_1 and assume only two values, 1 if they are impor-tant, and 0 if they are not. For second question, the vari-ables Time_2, Cost_2, and Scope_2 were given value 1 ifstated exclusively as important, value 0 if not stated asimportant and value 2 if some other factor has been men-tioned as the most important success criterion. The successcriteria other than time, cost and scope were specified intextual form and are used in qualitative analysis. If respon-dent considered that all three or a combination of the indi-cators time, cost and scope were equally important, thenthe specified important factors get the value 3 while theothers (i.e., not-important ones) get the value 2.

Table 1Response categories for Question 2

Time, Cost and Scope are fl Rating

Exclusively important 1Exclusively not-important 0More than one is important (with others) 3Not-important when more than one are important (with others) 2

For third question, simple ranking from 1 to 4 wasrecorded for variables Time_Fac, Cost_Fac, Func_Facand Qual_Fac. In question 2 and 3 we asked respondentsto mention additional success criteria on their own withoutprompting or suggesting them for anything. We did so pur-posefully to get the fresh and raw ideas that might be pres-ent in their minds on the issue of project success.

3.3. Analysis techniques

We chose Chi-Square Test to determine the frequenciesof occurrence of 1, 3 (important) and 0, 2 (not-important)for the variables assigned to the first and second questions.For ensuring that the frequencies of occurrence of differentchoices are not by chance, we determined p-values, i.e., thestatistical significance of the findings.

For testing the statistical significance of the ranking onnon-parametric scale, we determined Kendall�s coefficientof concordance for the given ranks. Choice of this testwas based on two issues, 1) the ranks were assigned onordinal scale and 2) there was a possibility of assignmentof the same rank (ties) to four factors. Kendall�s coefficientdeals with these issues effectively [19].

4. Analysis and results

The respondents are the professionals working in 12Indian software organizations having the characteristicsshown in Table 2.

Average experience of respondent in software industrywas of 7.93 years with minimum of 2 years and maximumof 21 years. Average experience of different groups ofrespondents in industry and designation was as shown inTable 3.

4.1. Urgency of project completion

The first question in the survey questionnaire wasaimed at finding the most important factor that defines

Table 3Respondents characteristics

Designation Average industryexperience in years(range)

Average experience at presentdesignation in years (range)

Programmers 4.8 (2–13 years) 1.5 (few months to 5 years)Project Managers 8.3 (3–14 years) 2.0 (few months to 5 years)Customer Account

Managers13.13 (5–21 years) 2.5 (few months to 6 years)


the software professionals� perceived success of a project ina situation when it is urgent to deliver the software to mar-ket for getting the first entrant advantage.

Fig. 1 exhibits the percentage of respondents who havepreferred a combination of project cost, schedule and scopefor consideration of its success.

One early finding is in terms of higher preference givento the scope in defining project success criterion at all levelsalbeit the urgency of deploying product in market at righttime was stated. Analysis result of next question will throwmore light on this finding.

4.2. Isolation of one single factor

As stated in previous section that second question in thequestionnaire was aimed at encouraging respondents toselect one single factor that, according to their views,defines the project success. The Chi-Square test result forthis question is shown in Table 4. We excluded theresponses that considered multiple factors important byassigning the value 3 to corresponding variables and value2 to the unimportant ones. Such responses make 67% ofthe total and have been taken separately for qualitativeanalysis discussed in the end of this section.

Fig. 2 illustrates the percentage of respondents whochose Time, Cost or Scope as the sole factor or as the fac-tor in association with others that define software projectsuccess. We observe that all groups of software profession-

Fig. 1. Importance given to th

als considered �scope� as the most important criterionexclusively and together with other criteria like time andcost. This is a very important indication that the focus ofattention of all levels of software developers is on achiev-ing the product functionality with quality. Results for�scope� are valid for the population of software profession-als as p-values for high frequencies of scope is negligible tozero for all groups.

4.3. Ordering as per preference

Third question to the respondents was to rank the time,cost, functionality and quality in terms of their importancein defining the project success. Table 5 consolidates theSPSS results of analysis including the mean ranks, Ken-dall�s coefficients of concordance, chi-square values andp-values for all and different groups of software profession-als. Ties were not included in the computations. Kendall�scoefficient of concordance W is a descriptive measure of

agreement among the rankings and can be used as the teststatistics for testing the null hypothesis of no associationalong with the p-value lying in the right tail of distributionof W when the number of raters (judges) is less than orequal to 8 [19]. In case of large sample, distribution of W

approximates Chi-Square and p-values can be used for test-ing the possibility of getting mean ranks by chance. We cansee that though the values of W do not show high congru-ence in ranks, average ranks are at p-values falling in the

e Cost, Time and Scope.

Table 4Exclusive choice of criterion

Respondents

Programmers Project Managers Customer AccountManagers

All

Number of Resp.

44 [Exclusive:16 (36%),With Others:28 (64%)]

38 [Exclusive: 10 (26%),With Others: 28(74%)]

23 [Exclusive: 9(39%) WithOthers: 14 (61%)]

105 [(Exclusive: 35 (33%),With Others: 70 (67%)]

Time_2 Cost_2 Scope_2 Time_2 Cost_2 Scope_2 Time_2 Cost_2 Scope_2 Time_2 Cost_2 Scope_2

Exclusively important 1 1 14 2 0 8 2 0 7 5 1 29

Exclusively not-important 15 15 2 8 10 2 7 9 2 30 34 6Important with others 20 17 26 21 18 26 10 7 12 51 42 64

Not-important with others 8 11 2 7 10 2 4 7 2 19 28 6Chi-Square(a) 18.27 13.81 30.00 20.73 3.36 40.73 6.391 0.348 11.95 43.0 36.14 85.81Df 3 3 3 3 2 3 3 2 3 3 3 3Asymp. Sig. 0.000 0.003 0.000 0.00 0.186 0.000 0.094 0.84 0.008 0.00 0.000 0.000

Fig. 2. Exclusive choice of Criterion.


range from 0.000 to 0.004 for all groups, hence, are signif-icant [19,20].

4.4. Qualitative data analysis

As stated before, two questions in the questionnaire pro-vided some space to respondents for mentioning other fac-tors that appeal them more than time, cost and scope forsignifying the project success. Question two was meant

for directly asserting on one factor other than the trio whilethird question requested them to add two more factors.

In total 70 out of 105 respondents opted for more thanone important criteria in response to question 2, 12 respon-dents mentioned one and only 1 respondent mentioned twofactors in addition to the cost, time, functionality and qual-ity in response to question 3. Ten responses are countedmore than once for the reason of having the views belong-ing to two different categories and thus, total count of

Table 5Ranked criteria

Programmers (38) Project Managers (35) Customer Account Managers (23) Overall (96)

Mean rank Relative rank Mean rank Relative rank Mean rank Relative rank Mean rank Relative rank

TIME_FAC 2.79 3 2.37 3 2.78 3 2.64 3COST_FAC 3.28 4 3.20 4 3.13 4 3.21 4FUNC_FAC 1.96 1 2.17 1 1.87 1 2.02 1

QUAL_FAC 1.97 2 2.26 2 2.22 2 2.14 2W 0.253 0.137 0.194 0.181Chi-Square 28.830 14.390 13.407 42.261Df 3 3 3 3p-Value 0.000 0.002 0.004 0.000

Table 6Qualitative analysis of subjective responses

Category Frequency Example statement

More than one criteria out of Cost,Time, Functionality and Quality

76 (82%) 1. With full functionality and desired quality (achievement of scope),with minor (negligible) changes in the cost & schedule

2. Where I make money and the customer gets what he wants at the right time

Happy customer or customersatisfaction (though undefined)

7 (8%) 1. Option 3 + customer satisfaction on end product and customer�s acceptanceon schedule variance and readiness to pay extra cost

2. Within Scope, Schedule, Cost, and a happy customer

Priorities of specific project 8 (8%) 1. It depends on some many factors. Mainly Scope, schedule, cost and dependson the client and importance of the project

2. Cannot be generalized, will differ from project to project. Achieving the objective(the very purpose for which the s/w is being developed) is most important

Non-affirming 2 (2%) 1. Success means a Win–Win situation2. None of the above


replies get to 93. Table 6 shows four major categories, inwhich all 83 + 10 responses are classified, with some exam-ple statements.

Only 15% of total respondents mentioned somethingdifferent from the Iron Triangle of success criteria whilegiving their subjective views. Percentage of programmers,project managers and customer account managers whoconsidered more than one criterion from the trio can beseen in Table 4 while their percentage for criteria like�happy customers� and �project priorities� are given inFig. 3. We should not ignore the originality of these factorsas respondents mentioned these criteria on their own with-out any suggestion or clue given.

This indicates that a small population of software pro-fessionals looks beyond the project�s internal characteris-tics and incorporates the external characteristics of theproject important for evaluating its success. We can makeout that negligible number of project managers think any-thing beyond the scope, cost and time constraints as com-pared to the programmers and customer accountmanagers. Also, concepts like customer satisfaction or pro-ject priorities were not clearly defined or discussed in theresponses of any respondent. We do not deny the effectof organizational culture and smaller size of the sampleon this finding but collective insensitiveness of projectorganization towards the interests of external stakeholderscannot be overlooked.

5. Inferences and discussion

Exploratory analysis results of three questions lead tothe following observations:

1. For software project�s internal organization, accom-plishment of the scope, i.e., functionality and qualityof software product appears to be the most importantsuccess criterion for judging its performance.

2. When it comes to choosing between functionality andquality, software professionals tend to give more impor-tance to functionality (refer to Table 5).

3. Software professionals with different responsibilitiesconcerning software projects appear to have a consensuson scope as the most important success criterion.

4. Between cost and time, software professionals tend torate the project time as more important factor for defin-ing success and the degree of importance of criterionmay vary with the project priorities.

5. Software professionals appear to consider the internalcharacteristics of the project, built around the cost, timeand scope, more appropriate for evaluating its successthan the external ones, albeit given a chance to statetheir independent views on success criteria.

Question one was aimed at investigating the inclinationof software professionals to choose among the competing

Fig. 3. Success criteria other than Cost, Time and Scope.


priorities at the time of some business urgency. Fig. 1reveals that more than 50% of respondents in all categorieswere concerned about the on-time delivery but accomplish-

ment of �scope� remains at the forefront of success criteria.Cost gets minimum preference giving us a message to deli-ver a complete product at reasonably right time. High p-value for findings about the �time� in case of customeraccount managers is indicative of the conflict that existsbetween the business interests and the �scope� fulfillmentobjective of project organization. Such conflict is obviousfor the people who are meant to take care of organization�sbusiness interests through customers. Similar dilemma isevident, though at lower scale, in case of programmersand project managers for whom the p-values for the factor�time� are 0.070 and 0.033 respectively. Interestingly, pro-ject managers appear to be more concerned about thetimely delivery (68%) of the complete product (84%) ascompared to customer account managers (57% and 96%).According to us, these findings are understandable in thecontext of professional commitments and satisfaction.For managers as well as developers, business needs areimportant but �scope� is given equal or more importance,may be, for its direct association with customer satisfaction.In our view, this is a non-trivial finding of this work.

A part of software professionals considers that theattainment of scope is an exclusive indicator of project suc-cess (Table 4) while others consider that the scope is moreimportant over budget and schedule but is not the exclusive

determinant. One striking finding is the negligible impor-tance given to the schedule (14%) and budget (3%) whileselecting the success criterion exclusively. This indicatesthat a segment of software professionals at all levels focusfirmly on the scope while working on a development pro-ject. Higher p-values for �time� and �cost� in case of projectmanagers and customer account managers are obtaineddue to the equal importance given to these factors whilechoosing multiple criteria and can be attributed to themanagers� professional dilemma.

Functionality, quality, time and cost have been given therespective ranks 1 though 4 equivocally by all profession-als. Though low values of W, Kendall coefficient of concor-dance, reflect minimal agreement in the ranks given by therespondents of the same category, low p-values make theaverage ranks significant. We see that functionality andquality go hand-in-hand except in the eyes of customeraccount managers. The reason may lie in the underlyingsupposition that a functional system is the most appropri-ate means to satisfy a customer. Thus, product characteris-tics are the most important issue to the people working ona software project and then only �time� and �cost� come inthe order of priority.

In qualitative analysis (Table 6), we see that even thoughthe software professionals were given the freedom to statesome other success criteria (Question 2 and 3), majorityof them (82%) remained stuck to the Iron Triangle. Thisshows that for the people who are internal to the project


organization, their commitment to serve the customer islimited to the user requirements stated in terms of function-ality, time and cost. There were only a few responsesrelated to the customer satisfaction, happy customers orother priorities specific to individual project. But, the verypresence of these external performance criteria assures thatthe project organization is not oblivious of the needs ofother stakeholders.

These observations lead to the following conjectures:

� Members of software project organization considerattainment of cost, time and quality targets instrumentalfor evaluation of project�s performance.� For them, attainment of functionality and quality of

software supersedes other criteria.� Also, they seem to give the least importance to the cost

target set for the project.

An inclination towards the functionality in project orga-nization may be indicative of the chronic problem of �GoldPlating� prevalent among the software professionals thatleads to the common budget and schedule sacrifices. �GoldPlating� refers to the enhancements in the software featuresas an addition to the required functionality for the sake ofsatisfaction of developer�s creativity. Programmers, due tonature of their work, are said to fall into the trap of featureenhancements frequently. We argue that the possibility ofhaving an undercurrent of �gold plating� in the exclusiveselection of scope as the most important success criterionis minimum due to equal contribution of programmersand managers in the survey sample. �Gold Plating� is thephenomenon operational at the level of programmers,while project managers and customer account managersare supposed to resist it for the sake of the parent and cus-tomer organizations� interests. Low p-values for theresponses from these groups should be able to eliminateany doubt of hidden effect of �Gold Plating� on the findings.

Our results are in accordance with and supplementWateridge�s [12] findings. Similar to the findings of [12],we also find that �meeting user requirements� with good�quality�, which we define as �scope� is the most importantcriterion to assess the project success for the people work-ing on it. However, we supplement findings in [12] in thefollowing two important ways:

1. We do not consider project organization to be monolithrepresented by project managers but take different con-stituents of the organization like software developers,project managers and customer account managers intoconsideration. We find that across the board, �scope� isthe most important success criterion.

2. We test the importance of �scope� as success criterion inthe face of pressures of conflicting goals of cost, qualityand scope and find that it is still the first priority.

Comparing our results with that of Linberg�s [13], webelieve that there is concurrence when pursuit of creativity

and subsequent learning is seen as essential to providingfunctionality and hence enhanced scope. However, sincewe have not collected data from the �failed� projects, whichsignify an important aspect of Linberg�s analysis, the scopefor comparison is limited.

This study, though not confirmatory, but is more inclu-sive than the other two in terms of respondents and projectobjectives and therefore, establishes that the split betweenthe perceptions of project success is profound. We see anurgency of efforts to bring the expectations of all stakehold-ers at the same horizon and link them to steer the projectfor more successful destination.

One possible reason for the �scope� ridden view of �pro-ject success� in the eyes of software professionals may be anassumed relationship between the project scope and thecustomer satisfaction. One cannot deny the possibility ofexistence of such relationship but that has not been estab-lished yet.

If it is attainment of project scope that the software cus-tomers look for then this is an encouraging revelation thatthe software functionality vis-a-vis quality and the externalcharacteristics like customer satisfaction and project prior-ities get a place in the project success agenda of softwareprofessionals. We observe an opportunity for softwareindustry to bring in a discipline where the internal objec-tives of development merge with the external interests ofthe stakeholders seamlessly by carefully weaving up theinclination of software professionals and interests of theusers and customers.

6. Research limitations and future directions

As this study is aimed to explore the notion of projectsuccess within the project organization, its results shouldbe viewed in the context of given research setting. A purpo-sive sample of software professionals from Indian organi-zations involved in commercial development of a widerange of software has been used in this work for exploringthe major theme prevailing in the complex cobweb of per-formance criteria for software projects. We do not claimfor outright generalization of the outcomes and emphasizethat the exploratory results of this study are emphasizingthe need for expanding the canvass of software projectevaluation criteria.

We tried to unearth the issues objectively through a fewquestions in the questionnaires and majority of the filledquestionnaires were received through e-mails. We replacethe interviews with the option of writing comments onthe questionnaire and this approach may not be sufficientto make some major findings. But the p-value being lessthan 0.01 for the frequency of importance given to scopeby all groups of software professionals makes our findingssubstantial. An issue with sample size can be raised for twogroups of professionals that resulted into high p-values(Fig. 1 and Table 4) but the findings were subsidiary andwe cannot claim any clarity about the degree of importancegiven by software professionals to the project �cost� and


�time�. A larger sample may improve the robustness of theseresults.

Additionally, our questionnaires were addressed only tosoftware companies operating in India. A more geograph-ically diverse approach may lead to more robust findings.Also, we looked at the companies at CMM Level 3 orabove due to the parent research requirements. Futurework may take more diverse base of companies.

We did not collect data about the qualification ofrespondents and therefore, possibility of engineering back-ground being a reason behind their inclination towardscompletion of �scope� cannot be ruled out.

Two additional success criteria mentioned by respon-dents were customer satisfaction and project priorities. Afew organizations have already started surveying the cus-tomers for their satisfaction. We suggest a two-foldresearch in this area as described below:

1. An attempt to relate the scope attainment with customersatisfaction will lead to our understanding about thedepth of their relationship.

2. A scrutiny of project documents supported withdetailed interview will determine the project objectives.A relationship between the perceived success criteria ofprofessionals worked on the project and correspondingproject objectives can be investigated to learn aboutthe gaps between the perceived and actual success ofthe projects.

Results of suggested research would be helpful in devel-oping a detailed framework for evaluating software projectsuccess. We emphasize the need for rigorous empiricalwork in different aspects of software development to sup-port the formulation of scientific theories in the realm ofsoftware development.

7. Conclusion

Major objective of this research was to explore the suc-cess indicators as perceived by software professionals basedon the characteristics internal to the project organization.We proposed that the Cost, Time and Scope (softwarefunctionality and quality combined) are the parameters sig-nifying the internal characteristics of the process adoptedin a software project. We found that scope is consideredto be the most important success criterion and functional-ity is rated over the software quality in defining the pro-ject�s success. A limited number of software professionalsalso considered customer happiness, satisfaction and pro-ject specific priorities as important criteria in addition tothe three core parameters.

This finding is useful for software industry in terms ofmanaging its resources in such a way that the focus of pro-fessionals on functionality can be channeled to meet theproject priorities and customer satisfaction. This researchmay be taken as an initial step in developing a detailedframework for evaluation of software project success that

covers a broad spectrum of success criteria with cost, timeand scope targets at one end to the project priorities andstakeholders interests spread over to another end. This willgive us more rigorous methods for project evaluation andhopefully, more satisfying and correct picture of projectperformance in software world.

Acknowledgment

Authors are grateful to the Research and PublicationCommittee of Indian Institute of Management Indore, In-dia for funding this research under grant number RP-15.

Appendix A

Q.1. ‘‘The software product under development has goodmarket potential if ready in next 6 months. Afterone year, it would be having other competitor prod-ucts entering the market and will loose the first comeradvantage. Available resources are not sufficient todevelop this product in 6 months and one has toreduce the functionality and compromise with qualityfor completing the development in given time andresources’’. Will you consider the project successful if1. Project finishes in 6 months with reduced function-

ality and compromised quality with the availableresources. Time_1 = 1, Cost_1 = 1, Scope_1 = 0

2. Project finishes in 6 months with increased resources(hence, cost) and full functionality and desired qual-ity. Time_1 = 1, Cost_1 = 0, Scope_1 = 1

3. Project delivers the product with full functionalityand desired quality after 1 year with the sameresources. Time_1 = 0, Cost_1 = 1, Scope_1 = 1

4. Project delivers the product with full functionalityand desired quality after 9 months with a littleincrease in resources. Time_1 = 0, Cost_1 = 0,

Scope_1 = 1

Enter only one choice from (1 / 2 / 3 / 4)—————————————————

Q.2. You will consider the successful completion of projectwhen it finishes1. On schedule, irrespective of cost and scope (func-

tionality and quality combined) achievements.2. Within cost, irrespective of schedule and scope

achievements.3. With full functionality and desired quality

(achievement of scope), irrespective of cost andschedule achievements.

4. Other, Specify ———————————————Enter only one choice from (1 / 2 / 3 / 4)—————————————————

Q.3. In your opinion, which among the following deter-mines the success of a Software Project? (Rank interms of relevance, 1 for most relevant and 5 for leastrelevant. Please avoid same rank to two factors)
h Completion of Project in estimated time as prom-
ised to the customer


h Completion of Project within estimated cost aspromised to the customer

h Completion of Project with required functionalityagreed upon by the customer before commence-ment of the project

h Completion of Project with Promised Qualityagreed upon by the customer before commence-ment of the projectOther, Specify
1. ________________________________2. ________________________________
References

[1] Lyytenin K. Expectation failure concept and systems analyst�s view ofinformation systems failures: results of an exploratory study. Infor-mat Manage 1988;14(1):45–56.

[2] Paulk MC, Weber CV, Curtis B, Chrissis MB. Capability MaturityModelSM: guidelines for improving the software process. Reading,MA: Addison-Wesley, Longman; 1994.

[3] Standish Group CHAOS Report, 1994. Available from: http://www.standishgroup.com/sample_research/chaos_1994_1.php (lastseen on April 19, 2004).

[4] Hoch DJ, Roeding CR, Purkert G, Lindner SK. Secrets ofsoftware success. Boston, MA: Harvard Business School Press;2000.

[5] Brooks FP. No silver bullet: essence and accidents of softwareengineering. IEEE Comput 1987;20(4):10–9.

[6] de Wit A. Measurement of project success. Int J Project Manage1988;6(3):164–70.

[7] Deane RH, Clark TB. Creating a learning project environment (coverstory). Informat Syst Manage 1997;14(3):54–61.

[8] Shenhar AJ, Levy O. Mapping the dimensions of project success.Project Manage J 1997;28(2):8756–9728.

[9] Atkinson R. Project management: cost, time and quality, two bestguesses and a phenomenon, its time to accept other success criteria.Int J Project Manage 1999;17(6):337–42.

[10] Turner JR. Project management: a profession based on knowledge orfaith. Int J Project Manage 1999;17(6):329–30.

[11] Pinto JK, Mantel Jr SJ. The causes of project failure. IEEE TransEng Manage 1990;37(4):269–76.

[12] Wateridge J. How can IS/IT projects be measured for success. Int JProject Manage 1998;16(1):59–63.

[13] Linberg KR. Software developer perceptions about software projectfailure: a case study. J Syst Software 1999;49:177–92.

[14] Jorgensen M, Sjoberg D. The impact of customer expectation onsoftware development effort estimates. Int J Project Manage2004;22:317–25.

[15] Kitchenham B, Pfleeger SL, Fenton N. Towards a framework forsoftware measurement validation. IEEE Trans. Software Eng.1995;21(12):929–44.

[16] Turner JR, Muller R. On the nature of the project as a temporaryorganization. Int J Project Manage 2003;21:1–8.

[17] Wright JN. Time and budget: the twin imperatives of a projectsponsor. Int J Project Manage 1997;15(3):181–6.

[18] Indian IT software and service directory 2002. New Delhi, India:National Association of Software and Service Companies of India; 2002.

[19] Gibbons JD. Nonparametric measures of association. Sage universitypaper series on quantitative applications in social sciences 07-091.Newbury Park, CA: Sage; 1993.

[20] Cox TF, Dunn RT. An analysis of decathlon data. The Statistician2002;51(2):179–87.

http://www.standishgroup.com/sample_research/chaos_1994_1.php

http://www.standishgroup.com/sample_research/chaos_1994_1.php

Defining ‘success’ for software projects: An exploratory revelation

Documents

Transcript of Defining ‘success’ for software projects: An exploratory revelation