A DECISION SUPPORT SYSTEM FOR THE SOFTWARE PROCESS

A DECISION SUPPORT SYSTEM

FOR THE SOFTWARE PROCESS

A Thesis

Submitted to the Faculty of Graduate Studies and Research

In Partial Fulfillment of the Requirements

for the Degree of

Master of Applied Science

In Electronic Systems Engineering

University of Regina

By

Mazdak Chinichian

Regina, Saskatchewan

August 2005

© 2005: M. Chinichian

Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.

A DECISION SUPPORT SYSTEM

FOR THE SOFTWARE PROCESS

A Thesis

Submitted to the Faculty of Graduate Studies and Research

In Partial Fulfillment of the Requirements

for the Degree of

Master of Applied Science

In Electronic Systems Engineering

University of Regina

By

Mazdak Chinichian

Regina, Saskatchewan

August 2005

© 2005: M. Chinichian


1+1 Library and Bibliotheque et Archives Canada Archives Canada

Published Heritage Direction du Branch Patrimoine de redition

395 Wellington Street Ottawa ON KlA ON4 Canada

395, rue Wellington Ottawa ON KlA ON4 Canada

NOTICE: The author has granted a non-exclusive license allowing Library and Archives Canada to reproduce, publish, archive, preserve, conserve, communicate to the public by telecommunication or on the Internet, loan, distribute and sell theses worldwide, for commercial or non-commercial purposes, in microform, paper, electronic and/or any other formats.

The author retains copyright ownership and moral rights in this thesis. Neither the thesis nor substantial extracts from it may be printed or otherwise reproduced without the author's permission.

Your file Votre reference ISBN: 0-494-05989-3 Our file Notre reference ISBN: 0-494-05989-3

AVIS: L'auteur a accord& une licence non exclusive permettant a la Bibliotheque et Archives Canada de reproduire, publier, archiver, sauvegarder, conserver, transmettre au public par telecommunication ou par ('Internet, preter, distribuer et vendre des theses partout dans le monde, a des fins commerciales ou autres, sur support microforme, papier, electronique et/ou autres formats.

L'auteur conserve la propriete du droit d'auteur et des droits moraux qui protege cette these. Ni la these ni des extraits substantiels de celle-ci ne doivent etre imprimes ou autrement reproduits sans son autorisation.

In compliance with the Canadian Privacy Act some supporting forms may have been removed from this thesis.

While these forms may be included in the document page count, their removal does not represent any loss of content from the thesis.

1*1

Canada

Conformement a la loi canadienne sur la protection de la vie privee, quelques formulaires secondaires ont ete enleves de cette these.

Bien que ces formulaires aient inclus dans la pagination, it n'y aura aucun contenu manquant.


1*1 Library and Archives Canada

Published Heritage Branch

395 Wellington Street Ottawa ON K1A 0N4 Canada

Bibliotheque et Archives Canada

Direction du Patrimoine de I'edition

395, rue Wellington Ottawa ON K1A 0N4 Canada

Your file Votre reference ISBN: 0-494-05989-3 Our file Notre reference ISBN: 0-494-05989-3

NOTICE:The author has granted a nonexclusive license allowing Library and Archives Canada to reproduce, publish, archive, preserve, conserve, communicate to the public by telecommunication or on the Internet, loan, distribute and sell theses worldwide, for commercial or noncommercial purposes, in microform, paper, electronic and/or any other formats.

AVIS:L'auteur a accorde une licence non exclusive permettant a la Bibliotheque et Archives Canada de reproduire, publier, archiver, sauvegarder, conserver, transmettre au public par telecommunication ou par I'lnternet, preter, distribuer et vendre des theses partout dans le monde, a des fins commerciales ou autres, sur support microforme, papier, electronique et/ou autres formats.

The author retains copyright ownership and moral rights in this thesis. Neither the thesis nor substantial extracts from it may be printed or otherwise reproduced without the author's permission.

L'auteur conserve la propriete du droit d'auteur et des droits moraux qui protege cette these.Ni la these ni des extraits substantiels de celle-ci ne doivent etre imprimes ou autrement reproduits sans son autorisation.

In compliance with the Canadian Privacy Act some supporting forms may have been removed from this thesis.

While these forms may be included in the document page count, their removal does not represent any loss of content from the thesis.

Conformement a la loi canadienne sur la protection de la vie privee, quelques formulaires secondaires ont ete enleves de cette these.

Bien que ces formulaires aient inclus dans la pagination, il n'y aura aucun contenu manquant.

i * i

CanadaReproduced with permission of the copyright owner. Further reproduction prohibited without permission.

UNIVERSITY OF REGINA

FACULTY OF GRADUATE STUDIES AND RESEARCH

SUPERVISORY AND EXAMINING COMMITTEE

Mazdak Chinichian, candidate for the degree of Master of Applied Science, has presented

a thesis titled, A Decision Support System for the Software Process, in an oral

examination held on July 22, 2005. The following committee members have found the

thesis acceptable in form and content, and that the candidate demonstrated satisfactory

knowledge of the subject material.

External Examiner: Dr. Mehran Mehrandezh, Faculty of Engineering

Supervisor: Dr. Luigi Benedicenti, Faculty of Engineering

Committee Member: Dr. Stephen O'Leary, Faculty of Engineering

Committee Member: Professor Ken Runtz, Faculty of Engineering

Chair of Defense: Dr. Allan Cahoon, /VP Research and International/

Faculty of Business Administration


UNIVERSITY OF REGINA

FACULTY OF GRADUATE STUDIES AND RESEARCH

SUPERVISORY AND EXAMINING COMMITTEE

Mazdak Chinichian, candidate for the degree of Master of Applied Science, has presented

a thesis titled, A Decision Support System fo r the Software Process, in an oral

examination held on July 22, 2005. The following committee members have found the

thesis acceptable in form and content, and that the candidate demonstrated satisfactory

knowledge of the subject material.

External Examiner: Dr. Mehran Mehrandezh, Faculty of Engineering

Supervisor: Dr. Luigi Benedicenti, Faculty of Engineering

Committee Member: Dr. Stephen O'Leary, Faculty of Engineering

Committee Member: Professor Ken Runtz, Faculty of Engineering

Chair of Defense: Dr. Allan Cahoon, /VP Research and International/

Faculty of Business Administration


Abstract

Recent software development revolves around two main families of methods:

structured plan-driven methods and light agile methods. This thesis describes a model to

help determine which family of methods should be adopted by a company. The model is

the result of a validation of previous work and the integration of an interpretive layer that

allows project managers and other technical personnel to supply information that is then

used to determine the value of key properties for the optimal process to be followed. This

approach allows a company to perform an initial investigation into its process and the

methods used to develop software, and gain valuable insight before developing an action

plan that may require external consulting which is, in general, very costly.

The two models proposed ladder to a unified software process decision method.

Validation of the model was achieved only through the re-interpretation of existing

experimental evidence, so future work involves an experiment specifically conceived for

the models created. However, the application of the model to the two main methods for

software development showed that the model can accurately identify to which category

they belong. This provides a good initial level of confidence for the model presented.

ii


Abstract

Recent software development revolves around two main families o f methods:

structured plan-driven methods and light agile methods. This thesis describes a model to

help determine which family of methods should be adopted by a company. The model is

the result o f a validation o f previous work and the integration o f an interpretive layer that

allows project managers and other technical personnel to supply information that is then

used to determine the value of key properties for the optimal process to be followed. This

approach allows a company to perform an initial investigation into its process and the

methods used to develop software, and gain valuable insight before developing an action

plan that may require external consulting which is, in general, very costly.

The two models proposed ladder to a unified software process decision method.

Validation o f the model was achieved only through the re-interpretation of existing

experimental evidence, so future work involves an experiment specifically conceived for

the models created. However, the application of the model to the two main methods for

software development showed that the model can accurately identify to which category

they belong. This provides a good initial level o f confidence for the model presented.

ii


Acknowledgements

I would like to utterly thank my supervisor, Dr. Benedicenti, to whom the success

of this thesis is chiefly owed. I am profoundly grateful for his constant support, diligent

attention, and more than knowledgeable guidance at each and every point of this study.

There is no doubt that his vision and insightful suggestions greatly contributed to the

success of this thesis. I would also like to kindly thank the members of the supervisory

committee, Professor Runtz and Dr. O'Leary, for their support during my work.

In addition, I gratefully acknowledge the financial support from the Faculty of

Graduate Studies and Research, the Faculty of Engineering, the Faculty of Science, and

the TRLabs.

I would like to thank the following people for their help and advice during the

course of this study: Dr. Christine Chan, Craig Gelowitz, Peter Kort, and Robert

Harrison.

iii


Acknowledgements

I would like to utterly thank my supervisor, Dr. Benedicenti, to whom the success

o f this thesis is chiefly owed. I am profoundly grateful for his constant support, diligent

attention, and more than knowledgeable guidance at each and every point o f this study.

There is no doubt that his vision and insightful suggestions greatly contributed to the

success of this thesis. I would also like to kindly thank the members o f the supervisory

committee, Professor Runtz and Dr. O’Leary, for their support during my work.

In addition, I gratefully acknowledge the financial support from the Faculty of

Graduate Studies and Research, the Faculty of Engineering, the Faculty o f Science, and

the TRLabs.

I would like to thank the following people for their help and advice during the

course of this study: Dr. Christine Chan, Craig Gelowitz, Peter Kort, and Robert

Harrison.

111


Post Defence Acknowledgements

I would like to also profoundly thank the external examiner Dr. Mehrandezh for

his diligence and highly valued comments on the thesis. In addition I am thankful to the

chair of the defence Dr. Cahoon for agreeing to be present at my defence on such short

notice. He was perfectly in command of the defence process and his insightful

contribution is greatly treasured.

iv


Post Defence Acknowledgements

I would like to also profoundly thank the external examiner Dr. Mehrandezh for

his diligence and highly valued comments on the thesis. In addition I am thankful to the

chair of the defence Dr. Cahoon for agreeing to be present at my defence on such short

notice. He was perfectly in command of the defence process and his insightful

contribution is greatly treasured.

iv


Dedication

I would like to dedicate this thesis to my parents Mahkameh and Morteza. The

success of this work would have not been possible without their continuous moral

support. I hope that this achievement brings them joy and happiness. I would also like to

dedicate this work to the nice people of Saskatchewan who accepted me as one of their

own and gave me the opportunity to work and study in this beautiful province. I hope

that this accomplishment would further enrich the science of software process

management in this province and in all of Canada.


Dedication

I would like to dedicate this thesis to my parents Mahkameh and Morteza. The

success of this work would have not been possible without their continuous moral

support. I hope that this achievement brings them joy and happiness. I would also like to

dedicate this work to the nice people of Saskatchewan who accepted me as one of their

own and gave me the opportunity to work and study in this beautiful province. I hope

that this accomplishment would further enrich the science of software process

management in this province and in all of Canada.

v


Table of Contents

Abstract ii

Acknowledgements iii

Post Defence Acknowledgements iv

Dedication

List of Tables ix

List of Figures

List of Acronyms xi

List of Appendices xii

Chapter 1 Introduction 1

1.1 Overview and problem domain 1

1.2 Thesis contribution 3

1.3 Organization of the thesis 4

Chapter 2 Background Information 6

2.1 Software process definition 7

2.2 Why is it important to have a software process 7

2.3 Plan-driven methods 8

2.4 Agile methods 12

vi


Table of Contents

Abstract ii

Acknowledgements iii

Post Defence Acknowledgements iv

Dedication v

List o f Tables ix

List o f Figures x

List o f Acronyms xi

List o f Appendices xii

Chapter 1 Introduction 1

1.1 Overview and problem d o m ain ............................................................ 1

1.2 Thesis contribution................................................................................. 3

1.3 Organization o f the thesis.......................................................................4

Chapter 2 Background Information 6

2.1 Software process definition....................................................................7

2.2 Why is it important to have a software process....................................7

2.3 Plan-driven m ethods...............................................................................8

2.4 Agile m ethods........................................................................................12

vi


2.5 Hybrid / tailored methods 17

2.6 Decision support systems 17

2.7 Problem domain 19

Chapter 3 Methodology Design 21

3.1 Feasibility study 22

3.2 Knowledge acquisition 24

3.2.1 Agile and plan-driven methods' characteristics and

contrast (twelve factors) 25

3.2.2 Validating Boehm's work 38

3.3 Model implementation 47

3.3.1 Input criteria 50

3.3.2 Questionnaire 53

3.3.3 Models 55

Chapter 4 Results and Technology Transfer 61

4.1 Validation and testing 61

4.1.1 Validating the models using the data used in validating

Boehm's work 61

4.1.2 Validating the models using the practices and concepts

of CMMI and XP 62

4.1.3 Validating the models using real data 67

4.1.4 Validating the models dividing a company into two

process groups 68

vii


2.5 Hybrid / tai lored m ethods.......................................................................17

2.6 Decision support systems.......................................................................17

2.7 Problem dom ain......................................................................................19

Chapter 3 Methodology Design 21

3.1 Feasibility study..................................................................................... 22

3.2 Knowledge acquisition..........................................................................24

3.2.1 Agile and plan-driven methods’ characteristics and

contrast (twelve fac to rs).......................................................... 25

3.2.2 Validating Boehm’s w o rk ........................................................38

3.3 Model implementation......................................................................... 47

3.3.1 Input criteria.............................................................................. 50

3.3.2 Questionnaire............................................................................ 53

3.3.3 M odels.......................................................................................55

Chapter 4 Results and Technology Transfer 61

4.1 Validation and testing...........................................................................61

4.1.1 Validating the models using the data used in validating

Boehm’s w ork ........................................................................ 61

4.1.2 Validating the models using the practices and concepts

of CMMI and X P ..................................................................... 62

4.1.3 Validating the models using real d a ta ....................................67

4.1.4 Validating the models dividing a company into two

process groups....................................................................... 68

vii


4.2 Additional development 68

4.2.1 Future proposed models 69

4.2.2 Automated system structure design 72

Chapter 5 Conclusions 75

5.1 Summary 75

5.2 Conclusions 76

5.3 Future work 77

References 79

Appendices 86

viii


4.2 Additional development.......................................................................... 68

4.2.1 Future proposed m odels.............................................................69

4.2.2 Automated system structure design .......................................... 72

Chapter 5 Conclusions 75

5.1 Summary....................................................................................................75

5.2 Conclusions............................................................................................... 76

5.3 Future w ork ............................................................................................... 77

References 79

Appendices 86

viii


List of Tables

1 Some plan-driven concepts 10

2 Examples of plan-driven approaches 11

3 Agile manifesto 13

4 Some agile methods concepts 15

5 Example of agile methods in the market 16

6 Levels of software method understanding and use 35

7 Agile and plan-driven methods contrasts 36

8 Misconceptions and realities about agile and plan-driven

methods 37

9 Validating Boehm's work using actual case studies 40

10 Criteria matrix 44

11 The five critical factors 48

12 Sub-factors 51

13 Results test 1 64


A CMMI staged vs. continuous 90

B XP principals and XP practices (implementation) 93

ix


List of Tables

1 Some plan-driven concepts 10

2 Examples o f plan-driven approaches 11

3 Agile manifesto 13

4 Some agile methods concepts 15

5 Example o f agile methods in the market 16

6 Levels o f software method understanding and use 35

7 Agile and plan-driven methods contrasts 36

8 Misconceptions and realities about agile and plan-driven

methods 37

9 Validating Boehm’s work using actual case studies 40

10 Criteria matrix 44

11 The five critical factors 48

12 Sub-factors 51



A CMMI staged vs. continuous 90

B XP principals and XP practices (implementation) 93

ix


List of Figures

1 Graphical representation of the Boehm's five critical factors 49

2 Radar graph 71

3 Bar graph 71

4 System structure 72

5 Decision block 74

x


List of Figures

1 Graphical representation o f the Boehm’s five critical factors 49

2 Radar graph 71

3 Bar graph 71

4 System structure 72

5 Decision block 74

x


List of Acronyms

DoD Department of Defense

DSS Decision Support System

CMM Capability Maturity Model (SM)

CMMI Capability Maturity Model Integration (SM)

EIA Electronic Industries Alliance

IEEE Institute of Electrical and Electronic Engineering

ISO International Standard Organization

NDIA National Defense Industrial Association

PSP Personal Software Process

SEI Software Engineering Institute

TSP Team Software Process

XP eXtreme Programming

xi


List of Acronyms

DoD Department o f Defense

DSS Decision Support System

CMM Capability Maturity Model (SM)

CMMI Capability Maturity Model Integration (SM)

EIA Electronic Industries Alliance

IEEE Institute o f Electrical and Electronic Engineering

ISO International Standard Organization

NDIA National Defense Industrial Association

PSP Personal Software Process

SEI Software Engineering Institute

TSP Team Software Process

XP eXtreme Programming

xi


List of Appendices

A Overview of CMMI

B Overview of extreme programming

C Abstracts and references of the case studies

D Analysis, questions and weights of 12 factors and their sub-

factors

xii

86

91

96

106


List of Appendices

A Overview o f CMMI 86

B Overview o f extreme programming 91

C Abstracts and references o f the case studies 96

D Analysis, questions and weights o f 12 factors and their sub

factors 106

xii


Chapter 1

Introduction

1.1 Overview and the problem domain

In recent times, software development has become more and more structured in

order to deal with the complexity of corporate software development. Two main

paradigms have recently emerged: agile development and plan-driven development. The

two have advantages and disadvantages, but it is difficult for a company to decide which

paradigm is more appropriate for their software development. This thesis attempts to

present a model that allows a company to decide between agile methods and plan-driven

methods in their software engineering practices.

Before proceeding with the core of this thesis, it is appropriate to present some

definitions and conventions that will be used throughout this thesis. A software product is

a complex mix of tangible products (e.g., program code), user manuals, and services,

(e.g. maintenance) [Benedicenti]. Software engineering is "the application of a

systematic, disciplined, quantifiable approach to the development, operation, and

maintenance of software; that is, the application of engineering to software" [IEEE-STD].

This definition for software engineering covers a wide range of activities and disciplines.

Some of these disciplines come from the mainline practices of computer science,

engineering, and project management. One of the areas in which software engineering is

applied is through software process management. A software process is defined as a

1


Chapter 1

Introduction

1.1 Overview and the problem domain

In recent times, software development has become more and more structured in

order to deal with the complexity of corporate software development. Two main

paradigms have recently emerged: agile development and plan-driven development. The

two have advantages and disadvantages, but it is difficult for a company to decide which

paradigm is more appropriate for their software development. This thesis attempts to

present a model that allows a company to decide between agile methods and plan-driven

methods in their software engineering practices.

Before proceeding with the core of this thesis, it is appropriate to present some

definitions and conventions that will be used throughout this thesis. A software product is

a complex mix of tangible products (e.g., program code), user manuals, and services,

(e.g. maintenance) [Benedicenti]. Software engineering is “the application of a

systematic, disciplined, quantifiable approach to the development, operation, and

maintenance of software; that is, the application of engineering to software” [IEEE-STD].

This definition for software engineering covers a wide range of activities and disciplines.

Some of these disciplines come from the mainline practices of computer science,

engineering, and project management. One o f the areas in which software engineering is

applied is through software process management. A software process is defined as a

1


sequence of steps performed for the development of software [IEEE-STD]. This thesis

focuses on issues regarding the software process management.

The background presented in Chapter Two justifies the assumption that, in order

to deliver a quality software product, satisfy customer demands, and achieve

organisational excellence, a structured, disciplined, and well-managed process must be

adopted [Benedicenti]. However, there are many software development practices,

models and standards available. Herein we refer to such practices, models, and standards

as software processes.

An analysis of most of the software process methods available determines that

there are two broad and distinct groups of software process methodologies that are

fundamentally different. These two groups of software process methodologies are

referred to as agile and plan-driven methods. The contrast and distinguishing factors

between these two groups have yet to be decided upon. Experts in the domain of process

management, with a vast knowledge of software process methodologies, are required to

carry out such a task. Thus, software developers or managers of software development

companies are unable to decide which type of software processes they should adopt for

their corresponding organizations. It is also ascertained that the organizations, based on

several factors, are categorically different and there are serious implications in adopting

an unsuitable software process for a given company. In many cases, an incompatible

software process contributes to devastating outcomes for the companies adopting them

[Boehm]. The importance of selecting a suitable software process cannot be exaggerated.

However, access to experts who are able to provide such expertise in this field is costly

and limited.

2


sequence of steps performed for the development o f software [IEEE-STD]. This thesis

focuses on issues regarding the software process management.

The background presented in Chapter Two justifies the assumption that, in order

to deliver a quality software product, satisfy customer demands, and achieve

organisational excellence, a structured, disciplined, and well-managed process must be

adopted [Benedicenti]. However, there are many software development practices,

models and standards available. Herein we refer to such practices, models, and standards

as software processes.

An analysis o f most o f the software process methods available determines that

there are two broad and distinct groups of software process methodologies that are

fundamentally different. These two groups of software process methodologies are

referred to as agile and plan-driven methods. The contrast and distinguishing factors

between these two groups have yet to be decided upon. Experts in the domain o f process

management, with a vast knowledge of software process methodologies, are required to

carry out such a task. Thus, software developers or managers of software development

companies are unable to decide which type of software processes they should adopt for

their corresponding organizations. It is also ascertained that the organizations, based on

several factors, are categorically different and there are serious implications in adopting

an unsuitable software process for a given company. In many cases, an incompatible

software process contributes to devastating outcomes for the companies adopting them

[Boehm]. The importance o f selecting a suitable software process cannot be exaggerated.

However, access to experts who are able to provide such expertise in this field is costly

and limited.

2


This study proposes a decision support system to provide innovative but validated

models to help managers and software developers with an overview of their

organisation's characteristics as well as providing an expert opinion on the type of

software process they may adopt. The primary application of the models presented in

this study is to facilitate the decision making process for the managers and software

developers to select a software process methodology for their company. The models

provide a concise overview of the organization and will show the compatibility of some

of the company's characteristics with the two main groups of process methodologies.

They also give an impartial decision on what process methodologies the overall company

should adopt. In addition, the proposed system offers some unbiased advice and links to

further information so that the managers would be able to find more information in this

regard.

The models presented and the system proposed are validated with available

literature and have been tested independently. However, future experiments should be

devised to further validate these models. The models are a good base for inspiring future

empirical work in the field of software process management.

1.2 Thesis contribution

The primary objective of this research is in the domain of software process

management. Given our problem domain, our contributions to this field of study are as

follows:

First, a feasibility study was conducted in order to demonstrate the fact that a

decision support system (expert system) is one of the techniques that can provide

3


This study proposes a decision support system to provide innovative but validated

models to help managers and software developers with an overview of their

organisation’s characteristics as well as providing an expert opinion on the type of

software process they may adopt. The primary application o f the models presented in

this study is to facilitate the decision making process for the managers and software

developers to select a software process methodology for their company. The models

provide a concise overview of the organization and will show the compatibility of some

of the company’s characteristics with the two main groups o f process methodologies.

They also give an impartial decision on what process methodologies the overall company

should adopt. In addition, the proposed system offers some unbiased advice and links to

further information so that the managers would be able to find more information in this

regard.

The models presented and the system proposed are validated with available

literature and have been tested independently. However, future experiments should be

devised to further validate these models. The models are a good base for inspiring future

empirical work in the field o f software process management.

1.2 Thesis contribution

The primary objective of this research is in the domain o f software process

management. Given our problem domain, our contributions to this field o f study are as

follows:

First, a feasibility study was conducted in order to demonstrate the fact that a

decision support system (expert system) is one o f the techniques that can provide

3


solutions to our problem domain. Second, we independently validated the distinguishing

factors between agile and plan-driven methods. Third, we discovered that we could not

readily employ those factors in order to decide on a best choice of process for an

organization. As a result, we divided the factors into single entities called sub-factors.

Fourth, we created a questionnaire to attest to the presence of such sub-factors. Fifth, we

assigned weights to the each question, sub-factor, and finally to each factor. Having

these weights, we presented two models that allow one to make a decision regarding the

final selection of a process methodology for a given organisation. These two models

were partially validated using the data available. Sixth, we proposed two other models to

be studied in the future ventures. Seventh, we proposed a structure for the future decision

support system.

1.3 Organization of the thesis

This thesis is arranged in to five chapters as follows:

Chapter 1 Introduction

Chapter 2 Background information. In this chapter we define a software product,

then we provide a definition for the software process. After that, we justify the

importance of using a software process and the downfalls of not employing it. An

overview of different software processes in the market is given. Two distinct categories

of software processes are discussed at length and instances of such processes are given.

The application of a decision support system to the problem domain is also discussed.

Chapter 3 Methodology Design. In this chapter, the steps taken to approach the

problem domain are explained. The results of a feasibly study and knowledge acquisition

4


solutions to our problem domain. Second, we independently validated the distinguishing

factors between agile and plan-driven methods. Third, we discovered that we could not

readily employ those factors in order to decide on a best choice of process for an

organization. As a result, we divided the factors into single entities called sub-factors.

Fourth, we created a questionnaire to attest to the presence of such sub-factors. Fifth, we

assigned weights to the each question, sub-factor, and finally to each factor. Having

these weights, we presented two models that allow one to make a decision regarding the

final selection of a process methodology for a given organisation. These two models

were partially validated using the data available. Sixth, we proposed two other models to

be studied in the future ventures. Seventh, we proposed a structure for the future decision

support system.

1.3 Organization of the thesis

This thesis is arranged in to five chapters as follows:

Chapter 1 Introduction

Chapter 2 Background information. In this chapter we define a software product,

then we provide a definition for the software process. After that, we justify the

importance of using a software process and the downfalls o f not employing it. An

overview of different software processes in the market is given. Two distinct categories

of software processes are discussed at length and instances o f such processes are given.

The application of a decision support system to the problem domain is also discussed.

Chapter 3 Methodology Design. In this chapter, the steps taken to approach the

problem domain are explained. The results of a feasibly study and knowledge acquisition

4


are given. Boehm's twelve factors are discussed in details. These twelve factors allow

us to distinguish between agile and plan-driven methods. Boehm's work is

independently validated using case studies and empirical software articles. A criteria

matrix is created, and an in-depth analysis is made. After that, the inputs to the system are

defined and a questionnaire is produced. The inputs are given weights. Then, two

models for making a final decision are presented. The 12-factor weight and single weight

models are the presented models.

Chapter 4 Results and Technology Transfer. In this chapter, the structure of the

future system is proposed. In addition more models for the final selection are proposed.

The proposed system structure and proposed models can be used in future work.

Chapter 5 Conclusions.

5


are given. Boehm’s twelve factors are discussed in details. These twelve factors allow

us to distinguish between agile and plan-driven methods. Boehm’s work is

independently validated using case studies and empirical software articles. A criteria

matrix is created, and an in-depth analysis is made. After that, the inputs to the system are

defined and a questionnaire is produced. The inputs are given weights. Then, two

models for making a final decision are presented. The 12-factor weight and single weight

models are the presented models.

Chapter 4 Results and Technology Transfer. In this chapter, the structure o f the

future system is proposed. In addition more models for the final selection are proposed.

The proposed system structure and proposed models can be used in future work.

Chapter 5 Conclusions.

5


Chapter 2

Background Information

A software product is a complex mix of tangible products and services. Thus, a

software product can comprise of the application program (code), plus all of the support

services, manuals, warranty, etc. Typically, during software development, client

requirements are vague and unclear; therefore it is quite difficult to satisfy them. Most

small businesses use a singleton software development approach. In the singleton

approach, individuals are in charge of all development aspects. The success or failure of

the project is solely dependent on the efforts, drive, and innovation of the individuals

involved. Ad-hoc methods are mainly used, the practises are rarely reused, and the

quality of the final product is both unknown and highly variable. However, larger

companies usually use a corporate software development approach. This approach is

team-based and customer-oriented and the practices adopted in this approach are cost

efficient, reusable, and repeatable. The larger the company is, the bigger the project

becomes, and the more employees are needed. This makes the project more complex, so

the singleton approach becomes impossible to adopt, and the need for a standard way to

develop software becomes imperative. In these cases, a structured method is often used

to build a software product. This method is often called a Software Process

[Benedicenti]. It is important to mention that any organization, no matter what size,

would benefit from the adoption of a software process [Boehm].

6


Chapter 2

Background Information

A software product is a complex mix of tangible products and services. Thus, a

software product can comprise of the application program (code), plus all of the support

services, manuals, warranty, etc. Typically, during software development, client

requirements are vague and unclear; therefore it is quite difficult to satisfy them. Most

small businesses use a singleton software development approach. In the singleton

approach, individuals are in charge of all development aspects. The success or failure of

the project is solely dependent on the efforts, drive, and innovation of the individuals

involved. Ad-hoc methods are mainly used, the practises are rarely reused, and the

quality of the final product is both unknown and highly variable. However, larger

companies usually use a corporate software development approach. This approach is

team-based and customer-oriented and the practices adopted in this approach are cost

efficient, reusable, and repeatable. The larger the company is, the bigger the project

becomes, and the more employees are needed. This makes the project more complex, so

the singleton approach becomes impossible to adopt, and the need for a standard way to

develop software becomes imperative. In these cases, a structured method is often used

to build a software product. This method is often called a Software Process

[Benedicenti]. It is important to mention that any organization, no matter what size,

would benefit from the adoption of a software process [Boehm].

6


2.1 Software process definition

The IEEE Standard 610 defines a Process as "a sequence of steps performed for a

given purpose" [IEEE-STD]. Expanding this definition using the ISO 15504 terminology

yields a Software Process as the methodology and tools used by an organization or

project to plan, manage, execute, monitor, control, and improve software-related

activities. This is a broad description that covers a wide range of activities in a project or

an organization [ISO15504].

2.2 Why is it important to have a software process?

Complex software is very difficult to maintain. It is extremely difficult for anyone

other than the original software developer to understand partially completed work. Work

in progress usually is not documented. A new employee assigned to complete an

ongoing project would probably end up starting from scratch, because no matter how

good the previous work is, the new employee would have no organized information on

where to start. A singleton process may be of superb quality, but it is an ad-hoc process,

not a defined software process [Tyrrell].

Ad-hoc processes also make it very difficult to accurately gauge the quality of the

finished product through independent assessment. Two developers, each working

according to their own processes, defining their own tests along the way, have no

objective method of comparing their work either with each other, or, more importantly,

with customers' quality criteria [Tyrrell]. Also, there is no way to accurately predict the

cost or the timeline of an ad-hoc software project [Brooks].

7


2.1 Software process definition

The IEEE Standard 610 defines a Process as “a sequence of steps performed for a

given purpose” [IEEE-STD], Expanding this definition using the ISO 15504 terminology

yields a Software Process as the methodology and tools used by an organization or

project to plan, manage, execute, monitor, control, and improve software-related

activities. This is a broad description that covers a wide range of activities in a project or

an organization [ISO15504],

2.2 Why is it important to have a software process?

Complex software is very difficult to maintain. It is extremely difficult for anyone

other than the original software developer to understand partially completed work. Work

in progress usually is not documented. A new employee assigned to complete an

ongoing project would probably end up starting from scratch, because no matter how

good the previous work is, the new employee would have no organized information on

where to start. A singleton process may be of superb quality, but it is an ad-hoc process,

not a defined software process [Tyrrell],

Ad-hoc processes also make it very difficult to accurately gauge the quality o f the

finished product through independent assessment. Two developers, each working

according to their own processes, defining their own tests along the way, have no

objective method of comparing their work either with each other, or, more importantly,

with customers' quality criteria [Tyrrell]. Also, there is no way to accurately predict the

cost or the timeline of an ad-hoc software project [Brooks].

7


Finally, there is an overhead as individuals determine their own practices in

isolation. So it is important for each organization to define a process for the project

[Tyrrell].

As a result, many organizations came to the realization that in order to deliver

quality software to their clients and to reduce costs, they need a process in place.

However, determining what type of process they should adopt seems to be a very

complicated issue since there are many processes available and there is little informed

and unbiased advice. Many proponents of specific software processes and their

corresponding institutes try to justify the use of their proposed models. It is hard to find

any general-purpose evaluation criteria for the adoption of any software process in a

company. Therefore, organizations will often choose processes that are not suited for

them, with often catastrophic results: customer dissatisfaction, flawed software products,

over budget, and overdue projects to name a few [Boehm].

In the field of software process development, there are over a dozen well-known

and functional methods available. Each has its own unique application and

characteristics. However, they can be divided into two main categories of process

development. One is called plan-driven and the other is called agile [Boehm].

2.3 Plan-driven methods

Plan-driven methods are considered the "traditional" way to develop software.

They are based on concepts taken from mainline engineering fields, such as industrial

engineering, engineering management, systems engineering, and so on. The plan-driven

8


Finally, there is an overhead as individuals determine their own practices in

isolation. So it is important for each organization to define a process for the project

[Tyrrell],

As a result, many organizations came to the realization that in order to deliver

quality software to their clients and to reduce costs, they need a process in place.

However, determining what type of process they should adopt seems to be a very

complicated issue since there are many processes available and there is little informed

and unbiased advice. Many proponents of specific software processes and their

corresponding institutes try to justify the use of their proposed models. It is hard to find

any general-purpose evaluation criteria for the adoption o f any software process in a

company. Therefore, organizations will often choose processes that are not suited for

them, with often catastrophic results: customer dissatisfaction, flawed software products,

over budget, and overdue projects to name a few [Boehm].

In the field of software process development, there are over a dozen well-known

and functional methods available. Each has its own unique application and

characteristics. However, they can be divided into two main categories o f process

development. One is called plan-driven and the other is called agile [Boehm].

2.3 Plan-driven methods

Plan-driven methods are considered the “traditional” way to develop software.

They are based on concepts taken from mainline engineering fields, such as industrial

engineering, engineering management, systems engineering, and so on. The plan-driven

8


methods are regarded as a systematic engineering approach to software that adheres to

specific (defined) processes in moving software through series of phases from initial

requirements to finished product. The software development is approached in a

requirement => design => build paradigm. The organization's processes are well defined

and are improved continuously. The need for implementing such process methodologies

arose during the age of aerospace, satellites, and ballistic missiles in the mid 1960's. At

that time, software was not well understood and generally undisciplined (ad-hoc)

methods were used [Boehm]. The result was defective software that was often late and

over budget, and unfit for sensitive applications such as aerospace. To offset such

problems, the US Department of Defense (DoD) started to develop a series of instructions

which eventually became standards. These methods were the first instances of

standardised software processes [Boehm]. Other companies, such as IBM and Hitachi,

further developed their own versions of software process methods. As software

development became more important, more methods were developed to regulate software

development processes [Boehm] [Hitachi] [Mills].

Table 1 shows some of the plan-driven concepts [Boehm]

Table 2 shows some of the examples of Plan-Driven Approaches in the market. [Boehm]

9


methods are regarded as a systematic engineering approach to software that adheres to

specific (defined) processes in moving software through series o f phases from initial

requirements to finished product. The software development is approached in a

requirement => design => build paradigm. The organization’s processes are well defined

and are improved continuously. The need for implementing such process methodologies

arose during the age o f aerospace, satellites, and ballistic missiles in the mid 1960’s. At

that time, software was not well understood and generally undisciplined (ad-hoc)

methods were used [Boehm]. The result was defective software that was often late and

over budget, and unfit for sensitive applications such as aerospace. To offset such

problems, the US Department of Defense (DoD) started to develop a series o f instructions

which eventually became standards. These methods were the first instances of

standardised software processes [Boehm]. Other companies, such as IBM and Hitachi,

further developed their own versions of software process methods. As software

development became more important, more methods were developed to regulate software

development processes [Boehm][Hitachi][Mills].

Table 1 shows some of the plan-driven concepts [Boehm]

Table 2 shows some of the examples of Plan-Driven Approaches in the market. [Boehm]

9


Table 1. Some Plan-Driven concepts. Table from [Boehm].

Process Improvement A program of activities designed to improve the performance and maturity of the organization's processes, and the results of such program.

Process Capability The inherent ability of a process to produce planned results. As the capability of each process is improved, it becomes predictable and measurable, and the most significant causes of poor quality and productivity are controlled or eliminated. This narrows the range of expected and actual results that can be achieved by following a process.

Organizational Maturity By steadily improving its process capability, an organization is said to mature. Maturity encompasses not only individual project capability, but the common application of standard processes across an organization. Common processes are maintained and personnel trained in their application. Projects tailor the common process assets to meet their needs. Once common assets are deployed, the organization can begin to measure their effectiveness and improve them based on the measures.

Process group A collection of specialists that facilitate the definition, maintenance, and improvement of the processes used by an organization.

Risk management An organized, analytic process to identify uncertainties that might cause harm or loss (identify risks), assess and quantify the identified risks, and develop and apply risk management plans to prevent or handle risk causes that could result in significant harm or loss.

Verification Verification confirms that work produces (e.g. specifications, design, models) properly reflect the requirements specified for them (building the product right)

Validation Validation confirms the fitness or worth of a work product for its operational mission (building the right product)

Software System Architecture

A software system architecture defines a collection of system stakeholders' need statements; and rationale which demonstrates that the components, connectors, and constraints define a system that, if implemented, would satisfy the collection of system stakeholders' need statements.

10


Table 1. Some Plan-Driven concepts. Table from [Boehm].

Process Improvement A program of activities designed to improve the performance and maturity of the organization’s processes, and the results of such program.

Process Capability The inherent ability of a process to produce planned results. As the capability of each process is improved, it becomes predictable and measurable, and the most significant causes of poor quality and productivity are controlled or eliminated. This narrows the range of expected and actual results that can be achieved by following a process.

Organizational Maturity By steadily improving its process capability, an organization is said to mature. Maturity encompasses not only individual project capability, but the common application of standard processes across an organization. Common processes are maintained and personnel trained in their application. Projects tailor the common process assets to meet their needs. Once common assets are deployed, the organization can begin to measure their effectiveness and improve them based on the measures.

Process group A collection o f specialists that facilitate the definition, maintenance, and improvement o f the processes used by an organization.

Risk management An organized, analytic process to identify uncertainties that might cause harm or loss (identify risks), assess and quantify the identified risks, and develop and apply risk management plans to prevent or handle risk causes that could result in significant harm or loss.

Verification Verification confirms that work produces (e.g. specifications, design, models) properly reflect the requirements specified for them (building the product right)

Validation Validation confirms the fitness or worth of a work product for its operational mission (building the right product)

Software System Architecture

A software system architecture defines a collection of system stakeholders’ need statements; and rationale which demonstrates that the components, connectors, and constraints define a system that, if implemented, would satisfy the collection of system stakeholders’ need statements.

10


Table 2. Examples of Plan-driven Approaches. (After [Boehm])

Method Organization Reference

Military Standards DoD [DoD-STD]

General Process Standards ISO EIA

IEEE

[ISO 15504] [EIA/IEEE]

Software factories Hitachi General Electric

others

[Hitachi] [GE]

CleanRoom Harlan Mills IBM

[Mills]

Capability Maturity Model Software (SW-CMM)

SEI Air Force

Watts Humphrey Mark Paulk

[SEI] [SW-CMM]

CMM Integration (CMMI)

SEI DoD

NDIA Roger Bate

Jack Ferguson Mike Phillips

[SEI] [CMMI]

Personal Software Process (PSP) Team Software Process (TSP)

Watts Humphrey SEI

[SEI] [Humphrey 97]

[Humphrey 2000]

11


Table 2. Examples of Plan-driven Approaches. (After [Boehm])

Method Organization Reference

Military Standards DoD [DoD-STD]

General Process Standards ISOEIA

IEEE

[ISO 15504] [EIA/IEEE]

Software factories Hitachi General Electric

others

[Hitachi][GE]

CleanRoom Harlan Mills IBM [Mills]

Capability Maturity ModelSoftware(SW-CMM)

SEI Air Force

Watts Humphrey Mark Paulk

[SEI][SW-CMM]

CMM Integration (CMMI)

SEI DoD

NDIA Roger Bate

Jack Ferguson Mike Phillips

[SEI][CMMI]

Personal Software Process (PSP)Team Software Process (TSP)

Watts Humphrey SEI [SEI]

[Humphrey 97] [Humphrey 2000]

11


The CMMI (the Capability Maturity Model Integration) is the most

comprehensive instance of a plan-driven approach. The CMMI is a descendant of the

SW-CMM and has placed the software process in the larger context of systems

engineering. Appendix A shows an overview of the CMMI. The complete document for

CMMI can be found at [CMMI].

2.4 Agile Methods

Agile methods emerged in the late 1990's, which is also referred to as the days of

the "dot corn" build-up. Agile development is an outgrowth of rapid prototyping and

rapid development experiences. Agile development is the resurgence of the philosophy

that programming is a craft rather than an industrial process depicted thorough a plan-

driven approach. Agile development was created as an answer to the rapidly changing

nature of the Internet-based economy. In such an environment, flexibility and speed will

provide a competitive advantage for small to medium size companies. The agile

manifesto is a document developed and signed by a group of advocates for the agile

approach, known as the Agile Alliance. This document presents the obvious disposition

of agile development and the outlook of its proponents [Boehm].

12


The CMMI (the Capability Maturity Model Integration) is the most

comprehensive instance of a plan-driven approach. The CMMI is a descendant of the

SW-CMM and has placed the software process in the larger context of systems

engineering. Appendix A shows an overview of the CMMI. The complete document for

CMMI can be found at [CMMI].

2.4 Agile Methods

Agile methods emerged in the late 1990’s, which is also referred to as the days of

the “dot com” build-up. Agile development is an outgrowth of rapid prototyping and

rapid development experiences. Agile development is the resurgence of the philosophy

that programming is a craft rather than an industrial process depicted thorough a plan-

driven approach. Agile development was created as an answer to the rapidly changing

nature o f the Internet-based economy. In such an environment, flexibility and speed will

provide a competitive advantage for small to medium size companies. The agile

manifesto is a document developed and signed by a group of advocates for the agile

approach, known as the Agile Alliance. This document presents the obvious disposition

of agile development and the outlook o f its proponents [Boehm].

12


Agile Manifesto

Table 3 shows the proclamation of the agile manifesto.

Table 3. Agile Manifesto. From [Manifesto]

Agile Manifesto

Individuals and interactions over processes and tools

Working software over comprehensive documentation

Customer collaboration over contract negotiation

Responding to change over following a plan

Kent Beck

Mike Beedle

Arievan Bennekum

Alistair Cockburn

Ward Cunningham

Martin Fowler

Brian Marick

Jeff Sutherland

Dave Thomas

James Grenning

Jim Highsmith

Andrew Hunt

Ron Jeffries

Jon Kern

Robert C. Martin

Steve Mellor

Ken Schwaber

That is, while there is value in the items on the right, we value the items on the left more.

13


Agile Manifesto

Table 3 shows the proclamation of the agile manifesto.

Table 3. Agile Manifesto. From [Manifesto]

Agile Manifesto

Individuals and interactions over processes and tools

Working software over comprehensive documentation

Customer collaboration over contract negotiation

Responding to change over following a plan

Kent Beck James Grenning

Mike Beedle Jim Highsmith

Arievan Bennekum Andrew Hunt

Alistair Cockbum Ron Jeffries

Ward Cunningham Jon Kern

Martin Fowler Robert C. Martin

Brian Marick Steve Mellor

Jeff Sutherland Ken Schwaber

Dave Thomas

That is, while there is value in the items on the right, we value the items on the left more.

13


The following principles, found on the manifesto's website, further refines the

concepts proposed above [Manifesto].

1. Our highest priority is to satisfy the customer through early and continuous

delivery of valuable software.

2. Welcome changing requirements, even late in development. Agile processes

harness change for the customer's competitive advantage.

3. Deliver working software frequently, from a couple of weeks to a couple of

months, with a preference to the shorter timescale.

4. Business people and developers must work together daily throughout the project.

5. Build projects around motivated individuals. Give them the environment and

support they need, and trust them to get the job done.

6. The most efficient and effective method of conveying information to and within a

development team is face-to-face conversation.

7. Working software is the primary measure of progress.

8. Agile processes promote sustainable development. The sponsors, developers, and

users should be able to maintain a constant pace indefinitely.

9. Continuous attention to technical excellence and good design enhances agility.

10. Simplicity--the art of maximizing the amount of work not done--is essential.

11. The best architectures, requirements, and designs emerge from self-organizing

teams.

12. At regular intervals, the team reflects on how to become more effective, then

tunes and adjusts its behaviour accordingly.

14


The following principles, found on the manifesto’s website, further refines the

concepts proposed above [Manifesto].

1. Our highest priority is to satisfy the customer through early and continuous

delivery of valuable software.

2. Welcome changing requirements, even late in development. Agile processes

harness change for the customer's competitive advantage.

3. Deliver working software frequently, from a couple o f weeks to a couple of

months, with a preference to the shorter timescale.

4. Business people and developers must work together daily throughout the project.

5. Build projects around motivated individuals. Give them the environment and

support they need, and trust them to get the job done.

6. The most efficient and effective method of conveying information to and within a

development team is face-to-face conversation.

7. Working software is the primary measure o f progress.

8. Agile processes promote sustainable development. The sponsors, developers, and

users should be able to maintain a constant pace indefinitely.

9. Continuous attention to technical excellence and good design enhances agility.

10. Simplicity—the art o f maximizing the amount o f work not done—is essential.

11. The best architectures, requirements, and designs emerge from self-organizing

teams.

12. At regular intervals, the team reflects on how to become more effective, then

tunes and adjusts its behaviour accordingly.

14


Table 4 shows some of the agile methods concepts. In general agile methods are

lightweight processes that employ short iterative cycles. They rely on tacit knowledge

[Boehm].

Table 5 shows some of the examples of the agile methods in the market [Boehm].

eXtreme Programming (XP) is the most famous agile method. XP is fairly rigorous

and initially expects all of the practices defined in the method to be followed. See

Appendix B for an overview of XP.

Table 4. Some agile methods concepts. Table from [Boehm].

Embracing Change Seeing change as an ally rather than an enemy. Change allows for more creativity and quicker value to the customer

Fact cycle/frequent delivery Scheduling many releases with short time spans between them; forces implementation of only the highest priority functions, delivers value to the customer quickly, and speeds requirements emergence

Simple design Designing for the battle, not the war. The motto is YAGNI ("You Aren't Going to Need It"). Strips designs down to cover just what is currently being developed. Since change is inevitable, planning for future functions is a waste of effort

Refactoring The restructuring of software to remove duplication, improve communication, simplify, or add flexibility without changing its behaviour. Just in-time redesign.

Pair programming A style of programming in which two programmers work side by side at one computer, continually collaborating on the same design, algorithm, code, or test

Retrospective A post-iteration review of the effectiveness of the work performed, methods used, and estimates. The review supports team learning and estimation for furture iterations. Sometimes called "reflection"

Tacit Knowledge Agility is achieved by establishing and updating project knowledge in the participants' head rather than in documents (explicit knowledge)

Test Driven developments Module or method tests are incrementally written by the developers and customers before and during coding. Supports and encourages very short iteration cycles.

15


Table 4 shows some of the agile methods concepts. In general agile methods are

lightweight processes that employ short iterative cycles. They rely on tacit knowledge

[Boehm].

Table 5 shows some of the examples of the agile methods in the market [Boehm].

eXtreme Programming (XP) is the most famous agile method. XP is fairly rigorous

and initially expects all of the practices defined in the method to be followed. See

Appendix B for an overview of XP.

Table 4. Some agile methods concepts. Table from [Boehm].

Embracing Change Seeing change as an ally rather than an enemy. Change allows for more creativity and quicker value to the customer

Fact cycle/frequent delivery Scheduling many releases with short time spans between them; forces implementation o f only the highest priority functions, delivers value to the customer quickly, and speeds requirements emergence

Simple design Designing for the battle, not the war. The motto is YAGNI (“You Aren’t Going to Need It”). Strips designs down to cover just what is currently being developed. Since change is inevitable, planning for future functions is a waste of effort

Refactoring The restructuring of software to remove duplication, improve communication, simplify, or add flexibility without changing its behaviour. Just in-time redesign.

Pair programming A style of programming in which two programmers work side by side at one computer, continually collaborating on the same design, algorithm, code, or test

Retrospective A post-iteration review o f the effectiveness o f the work performed, methods used, and estimates. The review supports team learning and estimation for furture iterations. Sometimes called “reflection”

Tacit Knowledge Agility is achieved by establishing and updating project knowledge in the participants’ head rather than in documents (explicit knowledge)

Test Driven developments Module or method tests are incrementally written by the developers and customers before and during coding. Supports and encourages very short iteration cycles.

15


Table 5. Example of agile methods in the market (after [Boehm]).

Method People/Organizations Reference

eXtreme Programming (XP)

Kent Beck Cunningham Ron Jeffries

Daimler Chrysler

[Beck 2000]

Adaptive Software Development (ASD)

Jim Highsmith [Highsmith 02 B]

Crystal Alistair Cockburn [Cockburn]

Scrum Ken Schwaber Jeff Sutherland Mike Beedle

[Schwaber]

Feature-Driven Development (FDD)

Jeff DeLuca Peter Coad [Coad]

16


Table 5. Example of agile methods in the market (after [Boehm]).

Method People/Organizations Reference

eXtreme Programming (XP)

Kent Beck Cunningham Ron Jeffries

Daimler Chrysler

[Beck 2000]

Adaptive SoftwareDevelopment(ASD)

Jim Highsmith [Highsmith 02 B]

Crystal Alistair Cockbum [Cockbum]

Scrum Ken Schwaber Jeff Sutherland Mike Beedle

[Schwaber]

Feature-DrivenDevelopment(FDD)

Jeff DeLuca Peter Coad [Coad]

16


2.5 Hybrid / Tailored methods

The facts arising from the experiences with software development suggest that

there are cases when neither of the process methodologies available provides a suitable

solution for the success of an organization. In many instances, a combination of the

practices of agile and plan-driven development may be needed to respond to the unique

characteristics of a company. A methodology that incorporates practices and principles

of both agile and plan-driven methods is called a Hybrid method [Boehm]. The Rational

process introduced by Philippe Krutchen and Crystal Orange created by Alistair

Cockburn are two examples of hybrid methodologies [IBM], [Cockburn]. In the domain

of software process, the concept of the Hybrid method is very new, but there is some

recent research being conducted to explore and invent new hybrid methods.

There are some companies whose characteristics not only do not match the

practices of agile and plan-driven, but are also incompatible with a hybrid method. In

such cases, a team of software process experts may be deployed to construct a method for

such a company [Boehm]. The experts may use the concepts and the practices of agile

and plan-driven methods to exclusively undertake such a task. A method that is

exclusively made or invented for a given company to serve its unique organizational

concerns is often referred to as a Tailored method [Boehm].

2.6 Decision support systems (Expert systems)

As it has been established in the previous sections, an expert with a broad

knowledge in the field of Software Process is required to be able to evaluate a company

based on its characteristics and then be able to make a decision in regards to the selection

17


2.5 Hybrid / Tailored methods

The facts arising from the experiences with software development suggest that

there are cases when neither of the process methodologies available provides a suitable

solution for the success of an organization. In many instances, a combination of the

practices o f agile and plan-driven development may be needed to respond to the unique

characteristics of a company. A methodology that incorporates practices and principles

o f both agile and plan-driven methods is called a Hybrid method [Boehm]. The Rational

process introduced by Philippe Krutchen and Crystal Orange created by Alistair

Cockbum are two examples of hybrid methodologies [IBM], [Cockbum]. In the domain

o f software process, the concept of the Hybrid method is very new, but there is some

recent research being conducted to explore and invent new hybrid methods.

There are some companies whose characteristics not only do not match the

practices of agile and plan-driven, but are also incompatible with a hybrid method. In

such cases, a team of software process experts may be deployed to construct a method for

such a company [Boehm]. The experts may use the concepts and the practices o f agile

and plan-driven methods to exclusively undertake such a task. A method that is

exclusively made or invented for a given company to serve its unique organizational

concerns is often referred to as a Tailored method [Boehm].

2.6 Decision support systems (Expert systems)

As it has been established in the previous sections, an expert with a broad

knowledge in the field of Software Process is required to be able to evaluate a company

based on its characteristics and then be able to make a decision in regards to the selection

17


of a suitable process methodology for that company. However, access to such expertise

is usually very expensive and often limited. In this study, we will try to find an automated

solution that can potentially replace an expert.

A study of different automated systems suggests that a decision support system

would be a good candidate. An expert system is a program that relies on a body of

knowledge to perform a somewhat difficult task usually performed only by human

experts [Luger]. The principal power of an expert system is derived from the knowledge

that the system embodies rather than from search algorithms and specific reasoning

methods. An expert system successfully deals with problems for which a clear

algorithmic solution does not exist [Luger].

The knowledge is extracted from training, reading and experience and includes:

theories, facts, rules, cases, examples, etc. [Luger].

Currently, an expert system is not able to replace an expert since there is always a

given portion of the problem domain that the system cannot solve. Therefore, expert

systems can become good decision support systems [Luger]. Decision support systems

(DSS) can be divided according to the tasks they perform. These tasks are: Interpretation,

Diagnosis, Monitoring, Prediction, Planning, Design, Control, Configuration, and

Selection [Luger]. In this study, the task of selecting a suitable process methodology is

required; as a result, the proposed decision support system is of selection type.

Typically, a decision support system operates in the following manner:

Users consult the DSS => DSS poses questions => Users answer questions => DSS gives

the decision => Users ask for explanation or information regarding the decision from

other sources [Luger].

18


of a suitable process methodology for that company. However, access to such expertise

is usually very expensive and often limited. In this study, we will try to find an automated

solution that can potentially replace an expert.

A study o f different automated systems suggests that a decision support system

would be a good candidate. An expert system is a program that relies on a body of

knowledge to perform a somewhat difficult task usually performed only by human

experts [Luger]. The principal power of an expert system is derived from the knowledge

that the system embodies rather than from search algorithms and specific reasoning

methods. An expert system successfully deals with problems for which a clear

algorithmic solution does not exist [Luger],

The knowledge is extracted from training, reading and experience and includes:

theories, facts, rules, cases, examples, etc. [Luger].

Currently, an expert system is not able to replace an expert since there is always a

given portion of the problem domain that the system cannot solve. Therefore, expert

systems can become good decision support systems [Luger]. Decision support systems

(DSS) can be divided according to the tasks they perform. These tasks are: Interpretation,

Diagnosis, Monitoring, Prediction, Planning, Design, Control, Configuration, and

Selection [Luger]. In this study, the task of selecting a suitable process methodology is

required; as a result, the proposed decision support system is o f selection type.

Typically, a decision support system operates in the following manner:

Users consult the DSS => DSS poses questions => Users answer questions => DSS gives

the decision => Users ask for explanation or information regarding the decision from

other sources [Luger].

18


The advantages of employing a DSS include: it provides access to expertise while

the availability of the expert might be limited or costly. It captures and documents the

expertise while usually the knowledge of the experts is tacit. It improves reliability

since, unlike the expert, it does not become sick or take holidays and thus reduces the

down time due to the expert unavailability. It increases the quality since it is free of

human errors. Finally, the knowledge transfer can be easily done to the different areas or

countries [Luger].

However, a DSS can have the following disadvantages. A DSS, based on how

robust it is developed, can potentially fail. The soundness of the outcome is based on the

knowledge base (see chapter 4), and the comprehensiveness of the knowledge base

depends on the experts who have developed it; as a result, the expertise and the

development of the system can determine the reliability of the DSS. The process of

knowledge extraction in developing a DSS is extremely important, and can be very

difficult and potentially faulty. A DSS only works well in very narrow domains, so

scalability and conciseness is a problem. The conclusions provided by a DSS are very

difficult to validate; as a result, they cannot be very dependable. Also, there are some

sociological problems arising from an incorrect conclusion that the DSS supports, e.g., a

DSS cannot be sued [Luger].

2.7 Problem domain

Experts in the domain of process management with a broad knowledge of

software process management are required to be able to first evaluate an organization

based on their characteristics and second be able to recommend a process methodology

19


The advantages of employing a DSS include: it provides access to expertise while

the availability o f the expert might be limited or costly. It captures and documents the

expertise while usually the knowledge of the experts is tacit. It improves reliability

since, unlike the expert, it does not become sick or take holidays and thus reduces the

down time due to the expert unavailability. It increases the quality since it is free of

human errors. Finally, the knowledge transfer can be easily done to the different areas or

countries [Luger].

However, a DSS can have the following disadvantages. A DSS, based on how

robust it is developed, can potentially fail. The soundness of the outcome is based on the

knowledge base (see chapter 4), and the comprehensiveness of the knowledge base

depends on the experts who have developed it; as a result, the expertise and the

development o f the system can determine the reliability o f the DSS. The process of

knowledge extraction in developing a DSS is extremely important, and can be very

difficult and potentially faulty. A DSS only works well in very narrow domains, so

scalability and conciseness is a problem. The conclusions provided by a DSS are very

difficult to validate; as a result, they cannot be very dependable. Also, there are some

sociological problems arising from an incorrect conclusion that the DSS supports, e.g., a

DSS cannot be sued [Luger].

2.7 Problem domain

Experts in the domain o f process management with a broad knowledge of

software process management are required to be able to first evaluate an organization

based on their characteristics and second be able to recommend a process methodology

19


suitable to such organizations. Software developers, or managers of software

development companies, are often unable to decide which type of software process they

should adopt for their respective organizations. There can be severe consequences for

adopting an unsuitable software process for a given company. In many cases, an

incompatible software process may contribute to the failure or poor outcome for the

companies that adopted it [Boehm]. Nevertheless, the expertise of such experts who are

able to provide such expertise in this field are costly and are not readily available.

As mentioned in the previous section, a DSS can provide preliminary but reliable

models in order to help the managers and software developers with an overview of their

organization's characteristics, as well as providing expert opinions on the type of

software process that they may adopt. Chapter Three presents the models, and Chapter

Four proposes the system. The primary application of the models presented in this study

is to facilitate the decision making process for the managers and software developers in

order to select an appropriate software process methodology for their company. The

models provide a concise overview of the organization and show the compatibility of

some of the company's characteristics with the two main groups of process

methodologies. They also provide an impartial decision on what process methodology

the overall company should adopt. In addition, the proposed system offers some

unbiased advice and links to further recourses so that the managers would have the option

to find more information in this regard.

20


suitable to such organizations. Software developers, or managers of software

development companies, are often unable to decide which type o f software process they

should adopt for their respective organizations. There can be severe consequences for

adopting an unsuitable software process for a given company. In many cases, an

incompatible software process may contribute to the failure or poor outcome for the

companies that adopted it [Boehm], Nevertheless, the expertise o f such experts who are

able to provide such expertise in this field are costly and are not readily available.

As mentioned in the previous section, a DSS can provide preliminary but reliable

models in order to help the managers and software developers with an overview of their

organization’s characteristics, as well as providing expert opinions on the type of

software process that they may adopt. Chapter Three presents the models, and Chapter

Four proposes the system. The primary application of the models presented in this study

is to facilitate the decision making process for the managers and software developers in

order to select an appropriate software process methodology for their company. The

models provide a concise overview of the organization and show the compatibility of

some of the company’s characteristics with the two main groups of process

methodologies. They also provide an impartial decision on what process methodology

the overall company should adopt. In addition, the proposed system offers some

unbiased advice and links to further recourses so that the managers would have the option

to find more information in this regard.

20


Chapter 3

Methodology Design

The purpose of this study is to find a model, and eventually an automated system,

to solve our problem domain. Currently, there are no models or systems available to

address the issues concerning our problem domain. Existing knowledge in this field is

not clear and well defined. It is very difficult for a non-expert to be able to distinguish

the differences between different process methodologies. The boundary which separates

the two main categories of process methodologies is fuzzy. In many cases, a software

process expert is needed to be able to draw any conclusions. The skills that the experts

possess are in the form of tacit knowledge and he/she can only make a final selection

based on the information given to him/her. However, the availability of such experts is

very limited and even if an expert is found, it is very costly to employ them. Creating an

automated system in the form of an expert system can help people with limited expertise

to be able to make the selection. In the study of different expert systems, the concept of

using decision support systems comes as a solution to this problem. [Luger]. As a result,

it was decided that for this project an expert system in the form of a DSS be created.

Conventionally an expert system follows a specific life cycle in its development.

We decided to follow such a life cycle in order to develop our project. The Expert

system's life cycle has the following phases: Feasibly study, Knowledge acquisition

21


Chapter 3

Methodology Design

The purpose o f this study is to find a model, and eventually an automated system,

to solve our problem domain. Currently, there are no models or systems available to

address the issues concerning our problem domain. Existing knowledge in this field is

not clear and well defined. It is very difficult for a non-expert to be able to distinguish

the differences between different process methodologies. The boundary which separates

the two main categories of process methodologies is fuzzy. In many cases, a software

process expert is needed to be able to draw any conclusions. The skills that the experts

possess are in the form of tacit knowledge and he/she can only make a final selection

based on the information given to him/her. However, the availability of such experts is

very limited and even if an expert is found, it is very costly to employ them. Creating an

automated system in the form of an expert system can help people with limited expertise

to be able to make the selection. In the study o f different expert systems, the concept of

using decision support systems comes as a solution to this problem. [Luger]. As a result,

it was decided that for this project an expert system in the form of a DSS be created.

Conventionally an expert system follows a specific life cycle in its development.

We decided to follow such a life cycle in order to develop our project. The Expert

system’s life cycle has the following phases: Feasibly study, Knowledge acquisition

21


(conceptual design), System Implementation, Testing and Validation, and Technology

Transfer (maintenance) [Luger].

3.1 Feasibility study

There are several criteria that may be employed in order to assess the feasibility of a

project such as this one. Since this project is an instance of an expert system and has a

specified problem domain we should evaluate the feasibility criteria for expert systems

technology. There are a few questions and issues that should be asked and considered in

order to evaluate the likelihood of a successful and feasible solution to a problem domain

[Luger].

• Is the problem domain too easy or too difficult? A problem domain is too difficult

to be feasible for an expert system if the problem domain needs innovation or

creativity to be solved. In our case, the innovation and the creativity issues are

solved by incorporating them in our model. As a result, the automated system

(expert system) does not need to be innovative or creative, thus it can be built.

Also the problem domain cannot be too easy. An easy problem domain relies on

common knowledge, thus there is no need for an expert system. In our case, given

our problem domain it is acknowledged that the project does not rely on common

knowledge and thus is not too easy.

• Does the task have a narrow focus? A feasible expert system should have a

focused and narrowed down problem domain since it is impossible to make an

automated solution for a vague and complex problem domain. Consequently, in

our plans we decided to narrow down the focus of our problem domain and

22


(conceptual design), System Implementation, Testing and Validation, and Technology

Transfer (maintenance) [Luger].

3.1 Feasibility study

There are several criteria that may be employed in order to assess the feasibility of a

project such as this one. Since this project is an instance o f an expert system and has a

specified problem domain we should evaluate the feasibility criteria for expert systems

technology. There are a few questions and issues that should be asked and considered in

order to evaluate the likelihood of a successful and feasible solution to a problem domain

[Luger],

• Is the problem domain too easy or too difficult? A problem domain is too difficult

to be feasible for an expert system if the problem domain needs innovation or

creativity to be solved. In our case, the innovation and the creativity issues are

solved by incorporating them in our model. As a result, the automated system

(expert system) does not need to be innovative or creative, thus it can be built.

Also the problem domain cannot be too easy. An easy problem domain relies on

common knowledge, thus there is no need for an expert system. In our case, given

our problem domain it is acknowledged that the project does not rely on common

knowledge and thus is not too easy.

• Does the task have a narrow focus? A feasible expert system should have a

focused and narrowed down problem domain since it is impossible to make an

automated solution for a vague and complex problem domain. Consequently, in

our plans we decided to narrow down the focus o f our problem domain and

22


identify a specific problem domain as mentioned in the previous chapter; that is,

to select a process methodology that best matches the characteristics and

operations of a given organization.

• Is there an automated solution to the problem domain? There must be a model

available that is comprehensible or transferable for automation, (for example, a

math equation or a qualitative model). As far as we have investigated, there was

no model readily available; however, it was determined that a model should be

created as a component of this project, and that it would become the integral part

of this study.

• Do the conventions in which the problem domain resides change frequently? If

there are frequent changes to the criteria / rules in which a solution to a problem

domain is classified, it might or probably would make the expert system obsolete

in a short matter of time. Software Process management is not an area that

changes rapidly. Hence, the criteria that deal with this problem domain would not

change rapidly. Nonetheless, we envision a mechanism in which the rules of the

system could be altered without compromising its operation. However, this

mechanism is only sketched in this thesis for future work since the formation of

such mechanism was beyond the scope of our project.

• Is there an expert available? The presence of an expert is a necessity in order to

make an expert system. I was to be the expert in this case. However, would the

expert perform demonstrably better that an amateur? Given the complexity of our

problem domain it would be hard for an amateur to perform better than the expert.

Could the skills of the expert be learned / documented? The answer was

23


identify a specific problem domain as mentioned in the previous chapter; that is,

to select a process methodology that best matches the characteristics and

operations o f a given organization.

• Is there an automated solution to the problem domain? There must be a model

available that is comprehensible or transferable for automation, (for example, a

math equation or a qualitative model). As far as we have investigated, there was

no model readily available; however, it was determined that a model should be

created as a component of this project, and that it would become the integral part

o f this study.

• Do the conventions in which the problem domain resides change frequently? If

there are frequent changes to the criteria / rules in which a solution to a problem

domain is classified, it might or probably would make the expert system obsolete

in a short matter of time. Software Process management is not an area that

changes rapidly. Hence, the criteria that deal with this problem domain would not

change rapidly. Nonetheless, we envision a mechanism in which the rules of the

system could be altered without compromising its operation. However, this

mechanism is only sketched in this thesis for future work since the formation of

such mechanism was beyond the scope of our project.

• Is there an expert available? The presence of an expert is a necessity in order to

make an expert system. I was to be the expert in this case. However, would the

expert perform demonstrably better that an amateur? Given the complexity o f our

problem domain it would be hard for an amateur to perform better than the expert.

Could the skills o f the expert be learned / documented? The answer was

23


affirmative and many articles and resources were to be consulted in order to learn

and document the skills needed for the expert system

A plan document was produced in order to outline the schedule and procedures of

the system's development. This document cited all of the life cycle phases. Each phase

include activities, deliverables, and time lines. There are four distinct phases comprised

in this plan document: Literature review, Knowledge acquisition, Development, and

Thesis write up.

3.2 Knowledge acquisition (Conceptual design)

Knowledge Extraction is the most important task in any expert system project.

Acquiring knowledge and expertise that can help design a model and subsequently make

a system is essential to expert systems and usually proves to be a major hurdle.

Knowledge acquisition stands to the system like the brain stands to the body. In this

project, knowledge extraction proved to be complicated and posed serious challenges.

One of the challenges faced was the lack of formalized knowledge. Most of the

time, knowledge existed in the form of tacit expertise, and although it has been presented

in the form of formal papers, it was extremely difficult to interpret it to suit our purpose.

Another challenge was the mismatch between the two sides of the argument in regards to

the process management. The expertise on either side made claims that their related

speciality was "the best".

In our project we lacked an expert that was knowledgeable and easily accessible

so that we could readily use his/her expert opinion. As a result, it was decided that I

24


affirmative and many articles and resources were to be consulted in order to leam

and document the skills needed for the expert system

A plan document was produced in order to outline the schedule and procedures of

the system’s development. This document cited all o f the life cycle phases. Each phase

include activities, deliverables, and time lines. There are four distinct phases comprised

in this plan document: Literature review, Knowledge acquisition, Development, and

Thesis write up.

3.2 Knowledge acquisition (Conceptual design)

Knowledge Extraction is the most important task in any expert system project.

Acquiring knowledge and expertise that can help design a model and subsequently make

a system is essential to expert systems and usually proves to be a major hurdle.

Knowledge acquisition stands to the system like the brain stands to the body. In this

project, knowledge extraction proved to be complicated and posed serious challenges.

One o f the challenges faced was the lack of formalized knowledge. Most of the

time, knowledge existed in the form of tacit expertise, and although it has been presented

in the form of formal papers, it was extremely difficult to interpret it to suit our purpose.

Another challenge was the mismatch between the two sides o f the argument in regards to

the process management. The expertise on either side made claims that their related

speciality was “the best”.

In our project we lacked an expert that was knowledgeable and easily accessible

so that we could readily use his/her expert opinion. As a result, it was decided that I

24


would acquire the knowledge and expertise that would enable me to act as an expert for

our project. This knowledge is available in the literature, but it needs to be rationalized

and objectified.

Conceptual design

Conceptual design specifies ways in which an expert system will carry out its

tasks. In such design tasks, objectives and goals are formalized and the relationships and

constraints on these tasks are defined. Finally, these aspects are structured together in the

form of a conceptual design [Luger].

The following sections contain a detailed analysis of the factors contributing to the

decision of adopting an agile or plan-driven method. The analysis is substantiated by

relevant references to existing literature as appropriate.

3.2.1 Agile and plan-driven methods' characteristics and contrast (twelve factors)

In order to find a solution applicable to our problem domain, we should look into

characteristics that are exclusive to the agile and plan-driven methods, and then find some

distinguishing factors. To do so, many references were consulted that provided

information on the differences between agile and plan-driven methodologies. This task is

complex since there has not been a lot of work done in this field and so it is difficult to

determine a clear border between agile and plan-driven methods. Barry Boehm is one of

the few experts who attempted to present an unbiased comparison between the two

methods. A great deal of knowledge and study in this field is needed in order to enable

one to determine such distinguishing factors. It is also noteworthy to mention that

25


would acquire the knowledge and expertise that would enable me to act as an expert for

our project. This knowledge is available in the literature, but it needs to be rationalized

and objectified.

Conceptual design

Conceptual design specifies ways in which an expert system will carry out its

tasks. In such design tasks, objectives and goals are formalized and the relationships and

constraints on these tasks are defined. Finally, these aspects are structured together in the

form of a conceptual design [Luger].

The following sections contain a detailed analysis o f the factors contributing to the

decision o f adopting an agile or plan-driven method. The analysis is substantiated by

relevant references to existing literature as appropriate.

3.2.1 Agile and plan-driven methods’ characteristics and contrast (twelve factors)

In order to find a solution applicable to our problem domain, we should look into

characteristics that are exclusive to the agile and plan-driven methods, and then find some

distinguishing factors. To do so, many references were consulted that provided

information on the differences between agile and plan-driven methodologies. This task is

complex since there has not been a lot of work done in this field and so it is difficult to

determine a clear border between agile and plan-driven methods. Barry Boehm is one of

the few experts who attempted to present an unbiased comparison between the two

methods. A great deal o f knowledge and study in this field is needed in order to enable

one to determine such distinguishing factors. It is also noteworthy to mention that

25


personal judgment based on such knowledge still has to be applied to draw any

meaningful conclusions.

Nevertheless, according to Boehm there are clear differences between agile and plan-

driven methods that can be categorised in four well-defined characteristics. These are

• Application Characteristics, including Primary project goals, project size, and

application environment.

• Management Characteristics, including customer relations, planning and control,

and project communications.

• Technical characteristics, including approaches to requirements definition,

development, and testing.

• Personnel characteristics, including customer characteristics, developer

characteristics, and organizational culture.

A summary of these four characteristics, and their twelve corresponding factors, is

presented here. Boehm discusses these factors at great length in chapter two of his book

Balancing Agility and discipline A guide for the perplexed [Boehm].

Application Characteristics

Primary Goals

Agile development has a primary goal that is rapid value and responsive to

change, in other words, the priority is to satisfy the customer through early and

continuous delivery. There is no return on investment analysis produced. Agile

development software is produced in rapid iterations. A reactive posture procedure -

responding to change rather than following the plan — is adopted in order to respond to an

environment distressed by rapid changes in market place and technology. [Manifesto]

26


personal judgment based on such knowledge still has to be applied to draw any

meaningful conclusions.

Nevertheless, according to Boehm there are clear differences between agile and plan-

driven methods that can be categorised in four well-defined characteristics. These are

• Application Characteristics, including Primary project goals, project size, and

application environment.

• Management Characteristics, including customer relations, planning and control,

and project communications.

• Technical characteristics, including approaches to requirements definition,

development, and testing.

• Personnel characteristics, including customer characteristics, developer

characteristics, and organizational culture.

A summary of these four characteristics, and their twelve corresponding factors, is

presented here. Boehm discusses these factors at great length in chapter two of his book

Balancing Agility and discipline A guide fo r the perplexed [Boehm],


Primary Goals

Agile development has a primary goal that is rapid value and responsive to

change, in other words, the priority is to satisfy the customer through early and

continuous delivery. There is no return on investment analysis produced. Agile

development software is produced in rapid iterations. A reactive posture procedure -

responding to change rather than following the plan - is adopted in order to respond to an

environment distressed by rapid changes in market place and technology. [Manifesto]

26


The goals of plan-driven development are predictability, stability, and high-

assurance. All of the practices of the plan-driven methods are in place to support such

goals. Process improvement is focused on predictability and stability and is achieved

through increasing process capability. Using standardization, measurement, and control

can increase process capability. The Plan-driven approach is optimal for relatively stable

projects but it is not useful for unstable projects with high rates of unforeseen changes.

In case of safety-critical projects and projects with high-assurance, it is imperative to

follow the plans and use standards. It is of the essence to use the plan-driven approach

when dealing with such projects.

Size

Agile development works best for small to medium sized companies or teams.

Any number bigger than 40 people seems to prevent agile development from working

[Beck 2000]. Some sources say ten is the optimal number of people per team

[Constantine] [McBreen]. Scaling up is exceptionally difficult if it is doable at all.

Plan-driven development works better for larger organizations and thus it is not

efficient for small projects. Plan-driven methods tend to create a bureaucratic

environment that can be detrimental to smaller entities. However, if a company has more

than 150 people working on a project, it should definitely use plan-driven methods. In

addition, using plan-driven development makes it easy to expand the number of people

working on a project in later times.

27


The goals o f plan-driven development are predictability, stability, and high-

assurance. All o f the practices of the plan-driven methods are in place to support such

goals. Process improvement is focused on predictability and stability and is achieved

through increasing process capability. Using standardization, measurement, and control

can increase process capability. The Plan-driven approach is optimal for relatively stable

projects but it is not useful for unstable projects with high rates o f unforeseen changes.

In case o f safety-critical projects and projects with high-assurance, it is imperative to

follow the plans and use standards. It is of the essence to use the plan-driven approach

when dealing with such projects.

Size

Agile development works best for small to medium sized companies or teams.

Any number bigger than 40 people seems to prevent agile development from working

[Beck 2000], Some sources say ten is the optimal number of people per team

[Constantine] [McBreen], Scaling up is exceptionally difficult if it is doable at all.

Plan-driven development works better for larger organizations and thus it is not

efficient for small projects. Plan-driven methods tend to create a bureaucratic

environment that can be detrimental to smaller entities. However, if a company has more

than 150 people working on a project, it should definitely use plan-driven methods. In

addition, using plan-driven development makes it easy to expand the number of people

working on a project in later times.

27


Environment

Agile development is most applicable to turbulent, high change, and high-risk

environments. Agile development focuses on the product at hand and ignores problems

that might occur later. Agile development does not have any concern with the

organization beyond the project. This type of approach is not good for concepts such as

code integration, overall infrastructure optimization, scale in production, distributed

development, and cross continental users. Nevertheless, agile development is ideal for in

house development for local users or customers. Agile development assumes flexible

users to accommodate system evolution; as a result, agile development will not

accommodate the following scenarios.

• Where several independently evolved applications must subsequently be closely

integrated - "Stovepipes"

• Where temporary procedural workarounds harden into unchangeable constraints

on systems evolution and cause unnecessary rework - "information-sclerosis"

• Where the new software is incrementally replacing the old one - "Bridging

situations"

• Where a product cannot be delivered in modules, e.g., 20% of an aircraft flight

control module - "Monolithic requirements"

• Where familiarity with the system across a large, mission-critical user base must

be maintained. e.g., monthly changes to air traffic controller's operational

procedure - "Continuity requirements".

28


Environment

Agile development is most applicable to turbulent, high change, and high-risk

environments. Agile development focuses on the product at hand and ignores problems

that might occur later. Agile development does not have any concern with the

organization beyond the project. This type of approach is not good for concepts such as

code integration, overall infrastructure optimization, scale in production, distributed

development, and cross continental users. Nevertheless, agile development is ideal for in

house development for local users or customers. Agile development assumes flexible

users to accommodate system evolution; as a result, agile development will not

accommodate the following scenarios.

• Where several independently evolved applications must subsequently be closely

integrated - “Stovepipes”

• Where temporary procedural workarounds harden into unchangeable constraints

on systems evolution and cause unnecessary rework - “information-sclerosis”

• Where the new software is incrementally replacing the old one - “Bridging

situations”

• Where a product cannot be delivered in modules, e.g., 20% of an aircraft flight

control module - “Monolithic requirements”

• Where familiarity with the system across a large, mission-critical user base must

be maintained, e.g., monthly changes to air traffic controller’s operational

procedure - “Continuity requirements”.

28


Plan-driven development has requirements that are determinable in advance and

are relatively stable, i.e., the rate of change is 1% or less of the requirements per month.

Plan-driven development can address important issues such as product lines,

organizational concerns, and enterprise concerns, including outsourcing that spans

multiple projects. Using Plan-driven methodologies, some capabilities in areas related to

design, architecture, systems engineering and human factors will be developed. Plan

driven developments exhibit strong, quantitative process improvements for the duration

of implementation in an organization.

Management Characteristics

Customer relations

Agile development desires dedicated and collocated (on-site) customers.

Customers must be in sync with users and developers. Customers should know what they

want; their absence or lack of knowledge can be catastrophic for the project.

Plan-driven development assumes that the projects are normally on a

contract/specification base. All of the discussions are in the form of formalizing

solutions and are compiled in the document agreement. Imprecise contracts can be a

major source of stress, incompatible expectations, and a lack of trust. In order to achieve

a successful project, the stakeholders must trust the developing organization. In these

cases, process maturity as is defined and rated by most plan-driven approaches can be a

source of trust and confidence.

29


Plan-driven development has requirements that are determinable in advance and

are relatively stable, i.e., the rate o f change is 1% or less of the requirements per month.

Plan-driven development can address important issues such as product lines,

organizational concerns, and enterprise concerns, including outsourcing that spans

multiple projects. Using Plan-driven methodologies, some capabilities in areas related to

design, architecture, systems engineering and human factors will be developed. Plan

driven developments exhibit strong, quantitative process improvements for the duration

of implementation in an organization.


Customer relations

Agile development desires dedicated and collocated (on-site) customers.

Customers must be in sync with users and developers. Customers should know what they

want; their absence or lack of knowledge can be catastrophic for the project.

Plan-driven development assumes that the projects are normally on a

contract/specification base. All of the discussions are in the form of formalizing

solutions and are compiled in the document agreement. Imprecise contracts can be a

major source o f stress, incompatible expectations, and a lack o f trust. In order to achieve

a successful project, the stakeholders must trust the developing organization. In these

cases, process maturity as is defined and rated by most plan-driven approaches can be a

source o f trust and confidence.

29


Planning and Control

Agile development treats planning as a means to an end. Approximately 20% of

the time it is used for planning/re-planning in agile approaches. The speed comes from

the tacit interpersonal knowledge rather than the explicit documented knowledge as

represented in plans. Agile development assumes pair-programming, stand-up meetings,

all-hands meetings, shared code ownership, and on site development, as instances of

planning. Planning is more about developing tacit knowledge than anything else [Beck].

Plan-driven development uses planning to anchor the process and provides broad-

spectrum communications within the organization. Plans take a large portion of the

requirement document and rely heavily on schedules, milestones, procedures,

requirements, architecture, and standards. Historical data is preserved for accurate future

projections. Plans make explicit expectations and form relationships between various

project efforts.

Project Communication

Agile development mainly relies on tacit and interpersonal knowledge.

Knowledge is shared across the organization as experienced people work on more tasks

with less experienced people. Communication is frequent and person-to-person.

However, the assumption that everybody's tacit knowledge is consistent across the

organization is extremely risky. For a team with n members, there are n(n-1)/2 different

interpersonal communication paths, which shows how difficult and complex a project can

become once the number of people on a project increases (scalability problems).

30



Agile development treats planning as a means to an end. Approximately 20% of

the time it is used for planning/re-planning in agile approaches. The speed comes from

the tacit interpersonal knowledge rather than the explicit documented knowledge as

represented in plans. Agile development assumes pair-programming, stand-up meetings,

all-hands meetings, shared code ownership, and on site development, as instances of

planning. Planning is more about developing tacit knowledge than anything else [Beck],

Plan-driven development uses planning to anchor the process and provides broad-

spectrum communications within the organization. Plans take a large portion of the

requirement document and rely heavily on schedules, milestones, procedures,

requirements, architecture, and standards. Historical data is preserved for accurate future

projections. Plans make explicit expectations and form relationships between various

project efforts.


Agile development mainly relies on tacit and interpersonal knowledge.

Knowledge is shared across the organization as experienced people work on more tasks

with less experienced people. Communication is frequent and person-to-person.

However, the assumption that everybody’s tacit knowledge is consistent across the

organization is extremely risky. For a team with n members, there are n(n-l)/2 different

interpersonal communication paths, which shows how difficult and complex a project can

become once the number of people on a project increases (scalability problems).

30


Plan-driven development relies on explicit documented knowledge. Most of the

communication happens one way in the form of progress reports, process descriptions

etc.

It is important to note that both approaches use both types of knowledge (explicit

and implicit), for instance, agile development's source code and test cases are explicit

knowledge. In organizations using plan-driven methodologies, people would not be able

to get along with each other without some type of interpersonal communication. Also, it

is noteworthy to mention that agile development adds documents when they are needed

whereas plan-driven development subtracts documents when they are not needed. In

addition, it is observed that less experienced teams tend to create more documentation as

a security blanket, where it will generally drive the project to create more documentation

than the actual work.

Technical Characteristics

Requirements

Agile development's requirements are in the form of adjustable and informal

stories. Requirements go through rapid iterations and are prioritized by users.

Negotiations between developers and users or customers determine the contents of the

next iteration that normally happens monthly.

Plan-driven methods' requirements are specific, formalised, complete, consistent,

traceable, and testable. For the most part, such requirements lack the concept of user

prioritization. CMMI for example, included integrated teaming, requirements

development, and risk management concepts in their approach. Plan-driven development

31


Plan-driven development relies on explicit documented knowledge. Most of the

communication happens one way in the form of progress reports, process descriptions

etc.

It is important to note that both approaches use both types of knowledge (explicit

and implicit), for instance, agile development’s source code and test cases are explicit

knowledge. In organizations using plan-driven methodologies, people would not be able

to get along with each other without some type o f interpersonal communication. Also, it

is noteworthy to mention that agile development adds documents when they are needed

whereas plan-driven development subtracts documents when they are not needed. In

addition, it is observed that less experienced teams tend to create more documentation as

a security blanket, where it will generally drive the project to create more documentation

than the actual work.


Requirements

Agile development’s requirements are in the form of adjustable and informal

stories. Requirements go through rapid iterations and are prioritized by users.

Negotiations between developers and users or customers determine the contents of the

next iteration that normally happens monthly.

Plan-driven methods’ requirements are specific, formalised, complete, consistent,

traceable, and testable. For the most part, such requirements lack the concept o f user

prioritization. CMMI for example, included integrated teaming, requirements

development, and risk management concepts in their approach. Plan-driven development

31


handles non-functional requirements, such as reliability, throughput, real-time deadline

satisfaction, and scalability in their practices. For plan-driven methods, quality is not an

"add on" concept; rather, it is a necessity.

Development

Agile development uses simple design; design elements emerge as functionality is

implemented. In agile development, extra features are not designed unless needed. Agile

development proponents assumes that the cost of refactoring is low and they believe that

the code added for new features will never be used due to changes in the requirements.

Such an outlook with regards to design is interpreted as program efficiency by agile

development advocates. They commonly use the famous phrase of "You aren't going to

need it (YAGNI)". However, the assumption that rework can be achieved with a low

cost is not always guaranteed. The following are some examples of problems that can

occur when early and simple design decisions result in foreseeable changes that would

cause a breakage in design ("architecture breakers"):

• When the application and the user base become larger

• Escalating reliability, throughput, response time, or other quality attributes during

the development within tightly specified real-time control loops

• Multinational or multilingual operations

Plan-driven methodology by nature incorporates planning and architecture-based

designs. Such architecture will enable software reuse across product lines, and therefore

will result in rapid development and efficiency. However, architecture can waste time in

32


handles non-functional requirements, such as reliability, throughput, real-time deadline

satisfaction, and scalability in their practices. For plan-driven methods, quality is not an

“add on” concept; rather, it is a necessity.

Development

Agile development uses simple design; design elements emerge as functionality is

implemented. In agile development, extra features are not designed unless needed. Agile

development proponents assumes that the cost of refactoring is low and they believe that

the code added for new features will never be used due to changes in the requirements.

Such an outlook with regards to design is interpreted as program efficiency by agile

development advocates. They commonly use the famous phrase o f “You aren’t going to

need it (YAGNI)”. However, the assumption that rework can be achieved with a low

cost is not always guaranteed. The following are some examples of problems that can

occur when early and simple design decisions result in foreseeable changes that would

cause a breakage in design (“architecture breakers”):

• When the application and the user base become larger

• Escalating reliability, throughput, response time, or other quality attributes during

the development within tightly specified real-time control loops

• Multinational or multilingual operations

Plan-driven methodology by nature incorporates planning and architecture-based

designs. Such architecture will enable software reuse across product lines, and therefore

will result in rapid development and efficiency. However, architecture can waste time in

32


a rapidly changing environment. In such cases the "Big Design Up Front (BDUF)"

practice is ineffective.

Testing

Agile development performs tests in short increments. The practice of pair

programming theoretically would remove more code defects as the code is being

generated. The following are the advantages of continuous regression testing performed

in agile methods:

• Requirements are testable

• Avoids documentation for requirements

• Enables incremental build-and-test

• Helps modularize the application

• Results in explicit working knowledge of the application

However, such practices have the following disadvantages:

• Rapid changes causes breakage in the tests

• Mismatches and race problems between code and test

• Lack of testing expertise can cause inadequate test coverage

Plan-driven development's testing strategies mainly fix problems by developing

and checking requirements and architectural specifications. Usually organizations

adopting plan-driven methodologies would invest in automated test suits. The practice of

testing in plan-driven development would create numerous documents that may or may

33


a rapidly changing environment. In such cases the “Big Design Up Front (BDUF)”

practice is ineffective.

Testing

Agile development performs tests in short increments. The practice of pair

programming theoretically would remove more code defects as the code is being

generated. The following are the advantages of continuous regression testing performed

in agile methods:

• Requirements are testable

• Avoids documentation for requirements

• Enables incremental build-and-test

• Helps modularize the application

• Results in explicit working knowledge of the application

However, such practices have the following disadvantages:

• Rapid changes causes breakage in the tests

• Mismatches and race problems between code and test

• Lack of testing expertise can cause inadequate test coverage

Plan-driven development’s testing strategies mainly fix problems by developing

and checking requirements and architectural specifications. Usually organizations

adopting plan-driven methodologies would invest in automated test suits. The practice of

testing in plan-driven development would create numerous documents that may or may

33


not be used in other projects. Such practices tend to create an independent testing

bureaucracy that is separate from customers or developers.

Personnel Characteristics

Customers

Agile development needs a Collaborative, Representative, Authorized,

Committed, and Knowledgeable (CRACK) customer. A customer that lacks one or more

of the mentioned traits can impose a significant risk to the success of an agile

development project.

Plan-driven customers do not need to be fully Collaborative, Representative,

Authorized, Committed, and Knowledgeable. They can manage with partially

Collaborative, Representative, Authorized, Committed, and Knowledgeable customers.

However, managers who prioritize contract compliance above project success have to be

careful. In some plan-driven methodologies, the Customer-Manager relationship can

become bureaucratic.

Developers

Agile methods developers thrive on amicability, talent, skill, and interpersonal

communication. However, agile development advocates assume that all of the people

working in their team possess those premium qualities.

Plan-driven development can perform better with less experienced people. Alistair

Cockburn classifies the people working on a software development team based on the

level of software method understanding and use. Table 6 explains the 5 levels of software

developers according to Cockburn [Cockburn].

34


not be used in other projects. Such practices tend to create an independent testing

bureaucracy that is separate from customers or developers.

Personnel Characteristics

Customers

Agile development needs a Collaborative, Representative, Authorized,

Committed, and Knowledgeable (CRACK) customer. A customer that lacks one or more

of the mentioned traits can impose a significant risk to the success of an agile

development project.

Plan-driven customers do not need to be fully Collaborative, Representative,

Authorized, Committed, and Knowledgeable. They can manage with partially

Collaborative, Representative, Authorized, Committed, and Knowledgeable customers.

However, managers who prioritize contract compliance above project success have to be

careful. In some plan-driven methodologies, the Customer-Manager relationship can

become bureaucratic.

Developers

Agile methods developers thrive on amicability, talent, skill, and interpersonal

communication. However, agile development advocates assume that all o f the people

working in their team possess those premium qualities.

Plan-driven development can perform better with less experienced people. Alistair

Cockbum classifies the people working on a software development team based on the

level of software method understanding and use. Table 6 explains the 5 levels of software

developers according to Cockbum [Cockbum],

34


Table 6. Levels of Software Method Understanding and Use (after [Cockburn]).

Level Characteristics 3 Able to revise a method (break its rules) to fit an unprecedented

new situation 2 Able to tailor a method to fit a precedented new situation

1A With training, able to perform discretionary method steps (e.g. sizing stories to fit increments, composing patterns, compound refactoring, complex Code integration). With experience, can become level 2

1B With training, able to perform procedural method steps (e.g. coding a simple method, simple refactoring, following coding standards and CM procedures, running tests). With experience, can master some level lA skills.

-1 May have technical skills, but unable or unwilling to collaborate or follow shared methods

Culture

The goal of Agile development is to make people feel comfortable and to

empower them. In this approach, people are empowered when they are trusted to do

whatever work is necessary. This concept is referred to a culture with many degrees of

freedom.

In Plan-driven methods, people feel comfortable and empowered when their roles

are defined and they can follow a clear policy and procedures. This is referred to as a

production line environment.

It is noteworthy to mention that once the organization's culture is well

established, it is difficult to change it. The change in culture that comes with the

implementation of agile methodologies is radical. In plan-driven methodologies,

especially CMM types of methods, culture change should happen in the very early stages.

However, the adoption of CMMI is a culture change in of itself.

For a snapshot of the Agile and plan-driven methods contrast, see table 7.

35


Table 6. Levels of Software Method Understanding and Use (after [Cockbum]).

Level Characteristics3 Able to revise a method (break its rules) to fit an unprecedented

new situation2 Able to tailor a method to fit a precedented new situation

1A With training, able to perform discretionary method steps (e.g. sizing stories to fit increments, composing patterns, compound refactoring, complex Code integration). With experience, can become level 2

IB With training, able to perform procedural method steps (e.g. coding a simple method, simple refactoring, following coding standards and CM procedures, running tests). With experience, can master some level 1A skills.

-1 May have technical skills, but unable or unwilling to collaborate or follow shared methods

Culture

The goal of Agile development is to make people feel comfortable and to

empower them. In this approach, people are empowered when they are trusted to do

whatever work is necessary. This concept is referred to a culture with many degrees of

freedom.

In Plan-driven methods, people feel comfortable and empowered when their roles

are defined and they can follow a clear policy and procedures. This is referred to as a

production line environment.

It is noteworthy to mention that once the organization’s culture is well

established, it is difficult to change it. The change in culture that comes with the

implementation of agile methodologies is radical. In plan-driven methodologies,

especially CMM types of methods, culture change should happen in the very early stages.

However, the adoption of CMMI is a culture change in of itself.

For a snapshot of the Agile and plan-driven methods contrast, see table 7.

35


Table 8 illustrates some of the realities and misconceptions about agile and plan-driven

methods.

Table 7. Agile and Plan-driven Methods Contrasts. (after [Boehm]).

Characteristics Agile Plan-driven Application Primary Goals Rapid Value; Responding to

change Predictability; High Assurance

Size Smaller teams and projects Larger teams and projects Environment Turbulent; High change;

Project-focused Stable; Low-change; project/organization focused

Management Customer relations

Dedicated on-site customers; focused on prioritised increments

As-needed customer interactions; focused on contract provisions


Internalized plans; qualitative control

Documented plans; quantitative control

Communications Tacit interpersonal knowledge Explicit documented knowledge Technical Requirements Prioritized informal stories and

test cases; undergoing unforeseen change

Formalized project; capability; interface, quality, foreseeable evolution requirements

Development Simple design; short increments; refactoring assumed inexpensive

Extensive design; longer increments; refactoring assumed expensive

Test Executable test cases define requirements; testing

Documented test plans and procedures

Personnel Customers Dedicated, collocated CRACK*

performers CRACK* performers, not always collocated

Developers At least 30% full-time Cockburn level 2 and 3 experts; no Level 1B or —1 personnel

50% Cockburn Level 1 and 3s early; 10% throughout; 30% Level lB's workable; no level —1s**

Culture Comfort and empowerment via many degrees of freedom (thriving on chaos)

Comfort and empowerment via framework of policies (thriving on order)

* Collaborative, Responsive, Authorized, Committed, Knowledgeable ** These numbers will particularly vary with the complexity of the application

36


Table 8 illustrates some o f the realities and misconceptions about agile and plan-driven

methods.

Table 7. Agile and Plan-driven Methods Contrasts, (after [Boehm]).

Characteristics Agile Plan-drivenApplicationPrimary Goals Rapid Value; Responding to

changePredictability; High Assurance

Size Smaller teams and projects Larger teams and projectsEnvironment Turbulent; High change;

Project-focusedStable; Low-change; project/organization focused

ManagementCustomerrelations

Dedicated on-site customers; focused on prioritised increments

As-needed customer interactions; focused on contract provisions


Internalized plans; qualitative control

Documented plans; quantitative control

Communications Tacit interpersonal knowledge Explicit documented knowledgeTechnicalRequirements Prioritized informal stories and

test cases; undergoing unforeseen change

Formalized project; capability; interface, quality, foreseeable evolution requirements

Development Simple design; short increments; refactoring assumed inexpensive

Extensive design; longer increments; refactoring assumed expensive

Test Executable test cases define requirements; testing

Documented test plans and procedures

PersonnelCustomers Dedicated, collocated CRACK*

performersCRACK* performers, not always collocated

Developers At least 30% full-time Cockbum level 2 and 3 experts; no Level IB o r-1 personnel

50% Cockbum Level 1 and 3 s early; 10% throughout; 30% Level IB ’s workable; no level -Is**

Culture Comfort and empowerment via many degrees of freedom (thriving on chaos)

Comfort and empowerment via framework o f policies (thriving on order)

* Collaborative, Responsive, Authorized, Committed, ** These numbers will particularly vary with the comp

Knowledgeable lexity o f the application

36


Table 8. Misconceptions and Realities about Agile and Plan-driven Methods (After

[Boehm]).

Misconceptions Realities Plan-driven Methods

Plan-driven methods are uniformly bureaucratic

Overly bureaucratic cultures and methods can stultify software development

Having documented plans guarantees compliance with plans

Not necessarily

Plan-Driven methods can succeed with a lack of talented people

Plan-driven methods can succeed with smaller percentage of talented people

High maturity guarantees success Explicit, documented plans provide more of a safety net than tacit plan

There are no penalties in applying plan- driven methods when change is unforeseeable

Plan-driven methods work best in accommodating foreseeable change

Agile Methods Agile methods don't plan Agile methods get much of their speed and

agility through creating and exploiting tacit knowledge

Agile methods require uniformly talented people

Agile methods work best when there is a critical mass of highly talented people involved

Agile methods can make the slope of cost- to-change vs. time curve uniformly flat

Agile methods can reduce the slope of the cost-to-change vs. time curve

YAGNI is a universally safe assumption, and won't alienate your customers

YAGNI helps handle unforeseeable change, but is risky when change is foreseeable

37


Table 8. Misconceptions and Realities about Agile and Plan-driven Methods (After

[Boehm]).

Misconceptions RealitiesPlan-driven Methods

Plan-driven methods are uniformly bureaucratic

Overly bureaucratic cultures and methods can stultify software development

Having documented plans guarantees compliance with plans

Not necessarily

Plan-Driven methods can succeed with a lack of talented people

Plan-driven methods can succeed with smaller percentage o f talented people

High maturity guarantees success Explicit, documented plans provide more of a safety net than tacit plan

There are no penalties in applying plan- driven methods when change is unforeseeable

Plan-driven methods work best in accommodating foreseeable change

Agile MethodsAgile methods don’t plan Agile methods get much of their speed and

agility through creating and exploiting tacit knowledge

Agile methods require uniformly talented people

Agile methods work best when there is a critical mass of highly talented people involved

Agile methods can make the slope of cost- to-change vs. time curve uniformly flat

Agile methods can reduce the slope o f the cost-to-change vs. time curve

YAGNI is a universally safe assumption, and won’t alienate your customers

YAGNI helps handle unforeseeable change, but is risky when change is foreseeable

37


3.2.2 Validating Boehm's work

In the last section, we discussed in detail the characteristics and factors that

differentiate agile and plan-driven methods. Boehm refers to these differences as home

grounds for agile and plan-driven methodologies. Boehm has come up with this set

mostly using intuition in looking at the workings of each method. He has also used some

real life examples to justify his claims. Consistent with one of the requirements of this

project, it was mandated that we would independently validate the soundness of Boehm's

ideas.

In order to validate Boehm's work, we needed to go through several case studies

in the literature. We discovered that many case studies are available to provide valuable

information that can be used in order to validate Boehm's work. Thirty articles were

chosen out of a pool of over 200 candidates, mostly obtained from the Agile Alliance

web site and the SEI publications.

The main criterion for choosing a case study was that it mainly had to deal with a

real life story of an organization or project that adopted a given process methodology. If

so, the other criterion used is whether any of the twelve factors mentioned by Boehm had

anything to do with the success or failure of the organization.

For example, one case study would describe a company with 2000 employees, a

critical primary goal, and no on-site customers, but they have chosen to use agile

development. According to Boehm, such a company would fail or do poorly. The article

pertaining to such a company would say that indeed the company assumed many losses

and this would actually validate Boehm's claims that those three mentioned factors were

significant in the selection of a process methodology. The thirty articles were examined

38


3.2.2 Validating Boehm’s work

In the last section, we discussed in detail the characteristics and factors that

differentiate agile and plan-driven methods. Boehm refers to these differences as home

grounds for agile and plan-driven methodologies. Boehm has come up with this set

mostly using intuition in looking at the workings of each method. He has also used some

real life examples to justify his claims. Consistent with one of the requirements of this

project, it was mandated that we would independently validate the soundness of Boehm’s

ideas.

In order to validate Boehm’s work, we needed to go through several case studies

in the literature. We discovered that many case studies are available to provide valuable

information that can be used in order to validate Boehm’s work. Thirty articles were

chosen out o f a pool of over 200 candidates, mostly obtained from the Agile Alliance

web site and the SEI publications.

The main criterion for choosing a case study was that it mainly had to deal with a

real life story o f an organization or project that adopted a given process methodology. If

so, the other criterion used is whether any of the twelve factors mentioned by Boehm had

anything to do with the success or failure o f the organization.

For example, one case study would describe a company with 2000 employees, a

critical primary goal, and no on-site customers, but they have chosen to use agile

development. According to Boehm, such a company would fail or do poorly. The article

pertaining to such a company would say that indeed the company assumed many losses

and this would actually validate Boehm’s claims that those three mentioned factors were

significant in the selection of a process methodology. The thirty articles were examined

38


comprehensively. Most of these articles were case studies or empirical software

engineering reports. It is worth mentioning that sifting through more than 200 articles

and choosing the ones that were to some extent relevant to our task required a

considerable amount of time and experience. In addition, trying to decipher the

characteristics and factors from plain unorganized text proved to be a challenge. In most

of the cases, the terminologies were different than the ones used by Boehm. Some of the

concepts were easy to decipher, (for example the size of the organization), but for the

most part, it was difficult to interpret the more complex factors. For example, factors

such as culture were almost impossible to determine.

Table 9 provides a snapshot of all of the thirty articles and their relevance in

validating Boehm's work. The article number, whether it is an agile or plan-driven

related article, and how relevant it was to our task is shown on the first column. The next

four columns show whether Boehm's four characteristics and their factors were

positively or negatively validated. A legend appears after the last page of table 9.

The references and an abstract for the thirty articles are provided in Appendix C

39


comprehensively. Most of these articles were case studies or empirical software

engineering reports. It is worth mentioning that sifting through more than 200 articles

and choosing the ones that were to some extent relevant to our task required a

considerable amount o f time and experience. In addition, trying to decipher the

characteristics and factors from plain unorganized text proved to be a challenge. In most

of the cases, the terminologies were different than the ones used by Boehm. Some of the

concepts were easy to decipher, (for example the size of the organization), but for the

most part, it was difficult to interpret the more complex factors. For example, factors

such as culture were almost impossible to determine.

Table 9 provides a snapshot of all of the thirty articles and their relevance in

validating Boehm’s work. The article number, whether it is an agile or plan-driven

related article, and how relevant it was to our task is shown on the first column. The next

four columns show whether Boehm’s four characteristics and their factors were

positively or negatively validated. A legend appears after the last page o f table 9.

The references and an abstract for the thirty articles are provided in Appendix C

39


Table 9. Validating Boehm's Work using actual Case studies

A r t i c 1 e

Application Characteristics - Primary goals - Size - Environment

Management Characteristics - Customer relations - Planning & Control - Project Communication

Technical Char. - Requirements - Development - Testing

Personnel Char. - Customers - Developers - Culture

[1] (+Environment, (+ Simple (+ Fast (+ A turbulent) planning, Development) Developers) ** (+Primary

goals, helpdesk)

helpdesk) (+ Customer

relations) [2] N/A N/A (+ N/A A Development)

[3] (+Size, 50) N/A N/A (+ culture, XP A (+Primary will fail * Goals, service) without it )

[4] (+ (+ Project (+ Development, (+ A Environment, communication, incremental) Developers) * high change) large room) (+Testing, as

code written)

[5] N/A 0 N/A 0 A [6] N/A (+Planning) (+ N/A A Development)

(+Testing)

[7] N/A (+ Customer N/A (+Customer, A relations) on-site)

(+ Project communication)

[8] (+Environment, (+Planning & 0 0 A turbulent) Control) * (+Primary

goals) (+ Project

communication)

[9] (+Primary (+ Project (+ (+ Customer, A goals) communication, Development) onsite)

*** (+Size,<100) pair programming)

(+Testing) (+Culture, years)

(+Developers) [10] N/A N/A N/A N/A

A

40


Table 9. Validating Boehm’s Work using actual Case studies

Artic1e

ApplicationCharacteristics- Primary goals- Size- Environment

ManagementCharacteristics- Customer relations- Planning & Control- Project Communication

TechnicalChar.- Requirements- Development- Testing

PersonnelChar.- Customers- Developers- Culture

[1] (+Environment, (+ Simple (+ Fast (+A**

turbulent)(+Primary

goals,helpdesk)

planning, helpdesk)

(+ Customer relations)

Development) Developers)

[2]A

N/A N/A (+Development)

N/A

[3]A*

(+Size, 50) (+Primary

Goals, service)

N/A N/A (+ culture, XP will fail

without i t )[4] (+ (+ Project (+ Development, (+A Environment, communication, incremental) Developers)* high change) large room) (+Testing, as

code written)[5]A

N/A 0 N/A 0

[6]A

N/A (+Planning) (+Development)

(+Testing)

N/A

[7]A

N/A (+ Customer relations)(+ Project

communication)

N/A (+Customer,on-site)

[8]A*

(-(-Environment,turbulent)(+Primary

goals)

(+Planning & Control)


0 0

[9] (+Primary (+ Project (+ (+ Customer,A goals) communication, Development) onsite)

*** (+Size,<100) pairprogramming)

(+Testing) (+Culture,years)

(+Developers)[10]At

N/A N/A N/A N/A

40


Table 9. Continued

[11] A *

(+1-Environment)

0 (+ Development)

(+Developers)

[12] N/A N/A (+ N/A A Development)

[13] 0 0 0 0 A

[14] (+Environment, 0 0 0 A turbulent)

(+Primary goals)

[15] N/A N/A (+ N/A A Development)

(+Testing) [16] N/A N/A (+ (+Developers)

A Development) [17] (+Environment, 0 (+ 0 A turbulent, less Development) * bureaucracy) (+Testing)

(+Size <25) [18] (+Environment, (+ Project (+ Development, 0

A turbulent) communication, rapid) * (+Primary sharing data)

Goals) (+ Customer relations)

[19] 0 (+ Project 0 (+Developers) A communication) *

[20] (+Environment, (+ Project (+ Development, (+Customer, A high risk) communication) continuous) on-site) * (+Testing)

[21] (+Primary (+Planning and (+Requirements) (+Developers) P Goals, process control) (+Development) ** improvement)

(+Environment, stable)

(+Project communication)

(+Customer (+Size, IRS) Relations)

[22] (+Environment (+Planning and 0 0 P ) control) * (+Primary

goals) (+Project

communication)

41


Table 9. Continued

[11]A*

(+/-Environment)

0 (+Development)

(+Developers)

[12]A


N/A

[13]A

0 0 0 0

[14]A

(+Environment,turbulent)(+Primary

goals)

0 0 0

[15]A


(+Testing)

N/A

[16]A


(+Developers)

[17]A*

(+Environment, turbulent, less bureaucracy) (+Size <25)

0 (+Development)

(+Testing)

0

[18]A*

(+Environment,turbulent)(+Primary

Goals)

(+ Project communication,

sharing data) (+ Customer

relations)

(+ Development, rapid)

0

[19]A*

0 (+ Project communication)

0 (+Developers)

[20]A*

(+Environment, high risk)


(+ Development, continuous) (+Testing)

(+Customer,on-site)

[21]P**

(+Primary Goals, process improvement)

(+Environment, stable)

(+Size, IRS)

(+Planning and control)


(+Customer Relations)

(+Requirements)(+Development)

(+Developers)

[22]P*

(+Environment)

(+Primarygoals)



0 0

41


Table 9. Continued

[23] P

(+Primary goals)

(+Customer relations)

(+Development) N/A

[24] (+Primary (+Planning and (+Development) 0 P goals) control) * (-Size) (+Customer

relations, return on investment)

[25] N/A (+Customer (+Testing) N/A P relations,

maintenance) [26] (+Environment (+Planning and 0 (+Developers)

P ) control) (+Customer Relations)

[27] (+Primary (+Customer (+Development, 0 P goals) Relations, less defects, * (+Size, GM,

Boeing) satisfaction, return

on investment) cheaper, faster,

quality) [28] (+Size) (+Customer (+Developers)

P (+Environment Relations, return * ) on investment)

[29] N/A N/A N/A N/A I

[30] (+Primary (+Customer (+Development, 0 P goals, process

improvement) relations, return on investment)

faster)

$ There was no mention of any factors in these articles however, reading them gave some ideas indirectly applicable to validating Boehm's work.

Table Legend: N/A : Not applicable, no mention of the factor

Confirms that the factor played a role Refutes that the factor played a role

+/- Neither confirmed nor refuted that the factor played any role but mentions the factor

o Not able to comprehend whether the factor played any role A Agile related case study / article P Plan-driven related case study / article

Good article (that was easy to validated Boehm's work) ** Very Good Article (that was easy to validated Boehm's work) *** Couldn't be better (that was easy to validated Boehm's work)

42


Table 9. Continued

[23]P

(+Primarygoals)

(+Customerrelations)

(+Development) N/A

[24]P*

(+Primarygoals)(-Size)


(+Customer relations, return on investment)

(+Development) 0

[25]P

N/A (+Customerrelations,

maintenance)

(+Testing) N/A

[26]P

(+Environment)


(+Customer Relations)

0 (+Developers)

[27]P*

(+Primary goals)

(+Size, GM, Boeing)

(+Customer Relations,

satisfaction, return on investment)

(+Development, less defects,

cheaper, faster, quality)

0

[28]P*

(+Size)(+Environment

)

(+Customer Relations, return on investment)

(+Developers)

[29]J

N/A N/A N/A N/A

[30]P

(+Primary goals, process improvement)

(+Customer relations, return on investment)

(+Development,faster)

0

| There was no mention of any factors in these articles however, reading them gave some ideas indirectly applicable to validating Boehm’s work.

Table Legend:N/A +

+ / -

oAP *

* *

42

Not applicable, no mention of the factor Confirms that the factor played a role Refutes that the factor played a roleNeither confirmed nor refuted that the factor played any role but mentions the factorNot able to comprehend whether the factor played any roleAgile related case study / articlePlan-driven related case study / articleGood article (that was easy to validated Boehm’s work)Very Good Article (that was easy to validated Boehm’s work)Couldn’t be better (that was easy to validated Boehm’s work)


Observations on validation of Boehm's work

Following the reading and analysis of the thirty relevant articles, some

observations and rulings were made. We categorised such observations into three sets:

Raw data observations, In depth analysis, and Additional remarks.

Raw data observations

The raw data observations provide the numeric value for the number of times a

factor was mentioned positively or negatively in order to support Boehm's claim. For

example, the factor environment was mentioned in eleven articles and in those articles it

played a positive role in confirming Boehm's work. We were not able to find any article

that would refute Boehm's claims and nineteen articles did not mention this factor as one

of the important issues to their success or failure.

Based on the comprehension arising from the articles, a decision was made. The

decision is in the form of "pass" or "fail" and indicates whether the articles validate that a

given Boehm factor is a relevant issue in selecting a software process. The decision also

has six levels of confidence that indicates how convincing the ruling is. These levels are:

very high confidence, high confidence, moderately high confidence, moderate

confidence, low confidence, and very low confidence. This data is summarized in table

10.

43


Observations on validation of Boehm’s work

Following the reading and analysis of the thirty relevant articles, some

observations and rulings were made. We categorised such observations into three sets:

Raw data observations, In depth analysis, and Additional remarks.

Raw data observations

The raw data observations provide the numeric value for the number of times a

factor was mentioned positively or negatively in order to support Boehm’s claim. For

example, the factor environment was mentioned in eleven articles and in those articles it

played a positive role in confirming Boehm’s work. We were not able to find any article

that would refute Boehm’s claims and nineteen articles did not mention this factor as one

o f the important issues to their success or failure.

Based on the comprehension arising from the articles, a decision was made. The

decision is in the form of “pass” or “fail” and indicates whether the articles validate that a

given Boehm factor is a relevant issue in selecting a software process. The decision also

has six levels o f confidence that indicates how convincing the ruling is. These levels are:

very high confidence, high confidence, moderately high confidence, moderate

confidence, low confidence, and very low confidence. This data is summarized in table

10.

43


Table 10. Criteria matrix.

No.

Positive

No.

Negative

No.

no mention

Confidence Decision

Primary goals 12 0 18 High Pass

Size 6 1 23 Moderate Pass

Environment 11 0 19 High Pass

Customer relations 11 0 19 High Pass

Planning and Control 7 0 23 Moderate Pass

Project

Communication

9 0 21 Moderately

High

Pass

Requirements 1 0 29 Low Pass

Development 17 0 13 Very High Pass

Testing 7 0 23 Moderate Pass

Customers 3 1 26 Low Pass

Developers 9 0 21 Moderate Pass

Culture 2 0 28 Very low Pass

In depth analysis

The following observations were made analysing the thirty articles:

• In dealing with Primary goals, if an organization treats "continuous process

improvement" as one of its primary goals, then they are constrained to adopt a

plan-driven approach. However, being responsive to change would be the

defining criteria for agile development.

44


Table 10. Criteria matrix.

No.

Positive

No.

Negative

No.

no mention

Confidence Decision

Primary goals 12 0 18 High Pass

Size 6 1 23 Moderate Pass

Environment 11 0 19 High Pass

Customer relations 11 0 19 High Pass

Planning and Control 7 0 23 Moderate Pass

Project

Communication

9 0 21 Moderately

High

Pass

Requirements 1 0 29 Low Pass

Development 17 0 13 Very High Pass

Testing 7 0 23 Moderate Pass

Customers 3 1 26 Low Pass

Developers 9 0 21 Moderate Pass

Culture 2 0 28 Very low Pass

In depth analysis

The following observations were made analysing the thirty articles:

• In dealing with Primary goals, if an organization treats “continuous process

improvement” as one of its primary goals, then they are constrained to adopt a

plan-driven approach. However, being responsive to change would be the

defining criteria for agile development.

44


• A turbulent environment can definitely be properly addressed through agile

development, whereas for plan-driven methods the task would be to reduce the

turbulence before doing anything else.

• Except for one successful experience, the plan-driven approach was successful if

the number of people on a project is more than 150. A project of 10-150 people is

the zone in which one might be able to use either methodologies. Agile

development is very efficient for teams with less than 50 employees.

• Planning and control is important with plan-driven methods where in agile

development people are empowered if they are not under direct scrutiny.

• Customer relations, especially the concept of customer satisfaction by means of

delivering on time, delivering quality software, and return on investment is

addressed properly by plan-driven methods.

• Project communication was very successful through practices such as pair

programming, stand up meetings, etc., that are addressed through agile

development. The communications in Plan-driven approaches were done mainly

through documentation.

• In the case of Requirements, we were not able to find conclusive evidence that

either of the models succeeded or failed.

• For Development, both agile development and plan-driven methods had a

substantial amount of evidence in regards to the success of their corresponding

development practices.

• Although testing was mentioned often in agile development, it can be argued that

testing while coding ignores other aspects of testing such as the acceptance test

45


• A turbulent environment can definitely be properly addressed through agile

development, whereas for plan-driven methods the task would be to reduce the

turbulence before doing anything else.

• Except for one successful experience, the plan-driven approach was successful if

the number of people on a project is more than 150. A project o f 10-150 people is

the zone in which one might be able to use either methodologies. Agile

development is very efficient for teams with less than 50 employees.

• Planning and control is important with plan-driven methods where in agile

development people are empowered if they are not under direct scrutiny.

• Customer relations, especially the concept of customer satisfaction by means of

delivering on time, delivering quality software, and return on investment is

addressed properly by plan-driven methods.

• Project communication was very successful through practices such as pair

programming, stand up meetings, etc., that are addressed through agile

development. The communications in Plan-driven approaches were done mainly

through documentation.

• In the case o f Requirements, we were not able to find conclusive evidence that

either o f the models succeeded or failed.

• For Development, both agile development and plan-driven methods had a

substantial amount o f evidence in regards to the success o f their corresponding

development practices.

• Although testing was mentioned often in agile development, it can be argued that

testing while coding ignores other aspects of testing such as the acceptance test

45


and the performance test. It should be noted that customers in agile development

perform the acceptance tests.

• Although one of the pillars of a successful agile development project lies in

having a Collaborative Responsive Authorized Committed Knowledgeable

(CRACK) customer, a survey showed that 50% of the time, a "CRACK"

customer was not available. Plan-driven methods can be performed well even

without a "CRACK" customer.

• If an organization does not have experienced developers, they should not consider

adopting agile development. Plan-driven methods through the practices of

training and documentation work better with less experienced developers.

• Culture was something hardly mentioned, but it could be interpreted as a

combination of all the other factors.

Additional observations

The following are some further observations arising from the examining the

articles.

• Agile development can be successfully used for help desk and customer service

tasks.

• The authors of agile articles were trying to present their claims more persuasively,

as if they had to prove that what they say is right, whereas the authors for Plan-

driven methods confidently assumed the correctness of their choice.

• Agile development articles were written mainly to be published in magazines and

as a result contained more wit, and compelling or shocking evidence was

46


and the performance test. It should be noted that customers in agile development

perform the acceptance tests.

• Although one o f the pillars of a successful agile development project lies in

having a Collaborative Responsive Authorized Committed Knowledgeable

(CRACK) customer, a survey showed that 50% of the time, a “CRACK”

customer was not available. Plan-driven methods can be performed well even

without a “CRACK” customer.

• If an organization does not have experienced developers, they should not consider

adopting agile development. Plan-driven methods through the practices of

training and documentation work better with less experienced developers.

• Culture was something hardly mentioned, but it could be interpreted as a

combination of all the other factors.

Additional observations

The following are some further observations arising from the examining the

articles.

• Agile development can be successfully used for help desk and customer service

tasks.

• The authors of agile articles were trying to present their claims more persuasively,

as if they had to prove that what they say is right, whereas the authors for Plan-

driven methods confidently assumed the correctness o f their choice.

• Agile development articles were written mainly to be published in magazines and

as a result contained more wit, and compelling or shocking evidence was

46


frequently presented. The articles for Plan-driven methods were merely

presenting the empirical data.

• Indirectly, the proponents of agile development were trying to market their idea to

Internet programmers, Java developers, and UML users. It could be argued that

they claim agile development would work better if the programming language

was Java or the design use was UML.

• Plan-driven methodologies claim that CMMI or other major models can be

tailored to any project regardless of size or character. All the other disagreeable

factors can be altered, e.g., to stabilize a turbulent environment and make it

suitable for a Plan-driven methodology.

• The culture of agile development thrives on the combination of experienced

developers, on site customers, and other traits that are detrimental to the success

of an organization; if even one is lacking, adoption of agile development can

become catastrophic.

• Proponents of agile development claim that it would save money. However,

there has been no evidence of this in any of the proposed calculations or methods

of prediction, whereas Plan-driven methods can successfully forecast and

calculate cost reduction. Plan-driven development also presents return on

investment estimates that would address issues such as outsourcing.

3.3 Model implementation

Currently there are no automated models or systems available to provide a

solution to our problem domain. However, Boehm proposes a technique that shows the

47


frequently presented. The articles for Plan-driven methods were merely

presenting the empirical data.

• Indirectly, the proponents of agile development were trying to market their idea to

Internet programmers, Java developers, and UML users. It could be argued that

they claim agile development would work better if the programming language

was Java or the design use was UML.

• Plan-driven methodologies claim that CMMI or other major models can be

tailored to any project regardless of size or character. All the other disagreeable

factors can be altered, e.g., to stabilize a turbulent environment and make it

suitable for a Plan-driven methodology.

• The culture of agile development thrives on the combination of experienced

developers, on site customers, and other traits that are detrimental to the success

o f an organization; if even one is lacking, adoption o f agile development can

become catastrophic.

• Proponents of agile development claim that it would save money. However,

there has been no evidence of this in any of the proposed calculations or methods

o f prediction, whereas Plan-driven methods can successfully forecast and

calculate cost reduction. Plan-driven development also presents return on

investment estimates that would address issues such as outsourcing.

3.3 Model implementation

Currently there are no automated models or systems available to provide a

solution to our problem domain. However, Boehm proposes a technique that shows the

47


compatibility of the companies based on five aspects that he refers to as Critical Factors.

These five critical factors and the reasoning behind their selection is presented in Table

11 [Boehm].

These five critical factors are size, criticality, dynamism, personnel, and culture.

Figure 1 summarises these factors graphically. The closer to the centre of an axis, the

more agile the factor is [Boehm].

Table 11. The five critical factors. Table from [Boehm].

Critical Factor

Agility Discriminator Plan-Driven Discriminator

Size Well-matched to small products and teams. Reliance on tacit knowledge limits scalability

Methods evolved to handle large products and teams. Hard to tailor down to small projects

Criticality Untested on safety-critical products. Potential difficulties with simple design and lack of documentation.

Methods evolved to handle highly critical products. hard to tailor down to low-criticality products.

Dynamism Simple design and continuous refactoring are excellent for highly dynamic environments, but a source of potentially expensive rework for highly stable environments

Detailed plans and Big design Up Front excellent for highly stable environment, but a source of expensive rework for highly dynamic environments.

Personnel Requires continuous presence of a critical mass of scarce Cockburn Level 2 or 3 experts. Risky to use non-agile Level 1B people

Needs a critical mass of scarce Cockburn Level 2 and 3 experts during project definition, but can work with fewer later in the project — unless the environment is highly dynamic. Can usually accommodate some level 1B people.

Culture Thrives in a culture where people feel comfortable and empowered by having many degrees of freedom. (Thriving on chaos)

Thrives in a culture where people feel comfortable and empowered by having their roles defined by clear policies and procedures. (Thriving on order)

48


compatibility of the companies based on five aspects that he refers to as Critical Factors.

These five critical factors and the reasoning behind their selection is presented in Table

11 [Boehm].

These five critical factors are size, criticality, dynamism, personnel, and culture.

Figure 1 summarises these factors graphically. The closer to the centre o f an axis, the

more agile the factor is [Boehm].

Table 11. The five critical factors. Table from [Boehm].

CriticalFactor

Agility Discriminator Plan-Driven Discriminator

Size Well-matched to small products and teams. Reliance on tacit knowledge limits scalability

Methods evolved to handle large products and teams. Hard to tailor down to small projects

Criticality Untested on safety-critical products. Potential difficulties with simple design and lack of documentation.

Methods evolved to handle highly critical products, hard to tailor down to low-criticality products.

Dynamism Simple design and continuous refactoring are excellent for highly dynamic environments, but a source o f potentially expensive rework for highly stable environments

Detailed plans and Big design Up Front excellent for highly stable environment, but a source of expensive rework for highly dynamic environments.

Personnel Requires continuous presence of a critical mass o f scarce Cockbum Level 2 or 3 experts. Risky to use non-agile Level IB people

Needs a critical mass o f scarce Cockbum Level 2 and 3 experts during project definition, but can work with fewer later in the project - unless the environment is highly dynamic. Can usually accommodate some level IB people.

Culture Thrives in a culture where people feel comfortable and empowered by having many degrees of freedom. (Thriving on chaos)

Thrives in a culture where people feel comfortable and empowered by having their roles defined by clear policies and procedures. (Thriving on order)

48


Criticality (Loss due to impact of defects)

Size (Number of personnel)

Personnel (% Level 1B) (% Level 2 and 3)

40 15

30 20

20 25

10 30

Many 0 35 Lives Single

Life Essential Funds Discretionary

Funds Comfort 50

Dynamism (% Requirements-change/month)

Culture (% Thriving on chaos vs. order)

Figure 1. graphical representation of Boehm's Five Critical factors. Image from [Boehm].

As it can be readily seen, the technique presented by Boehm is not an automated

solution, nor does it provide any understandable information for the users. The input

criteria are not defined. There are no weights or measures of the distances introduced.

Furthermore, there is no final conclusion or recommendation provided. In this technique,

the presence of an expert is still required. As a result, the managers and the software

developers are not able to use Boehm's technique because of the above-mentioned issues.

In this chapter we present two models that can be employed for a decision support

system and ultimately be readily used by managers and software developers without too

much difficulty.

49


Personnel(% Level 1B) (% Level 2 and 3)

40 15

30 -• 20

20 - - 25Criticality(Loss due to impact of defects)

Dynamism(% Requirements-change/month)10 -• 30

Many * Lives Single

Life Essential —■Funds Discretionary

Funds 'Comfort

100

300,

Size '(Number of personnel) C ulture

(% Thriving on chaos vs. order)

Figure 1. graphical representation of Boehm’s Five Critical factors. Image from [Boehm].

As it can be readily seen, the technique presented by Boehm is not an automated

solution, nor does it provide any understandable information for the users. The input

criteria are not defined. There are no weights or measures o f the distances introduced.

Furthermore, there is no final conclusion or recommendation provided. In this technique,

the presence of an expert is still required. As a result, the managers and the software

developers are not able to use Boehm’s technique because of the above-mentioned issues.

In this chapter we present two models that can be employed for a decision support

system and ultimately be readily used by managers and software developers without too

much difficulty.

49


3.3.1 Input Criteria

Based on the findings in the knowledge acquisition section, we ascertained that

there are four major distinctive characteristics in agile and plan-driven development.

These characteristics are: Application Characteristics, Management Characteristics,

Technical Characteristics, and Personnel Characteristics. We also divided these

characteristics into twelve factors, and we discussed each of them extensively.

So it would be expected that these twelve factors be the criteria for our inputs. However,

as seen in previous sections, each of these twelve factors encompasses more detailed

concepts, practices, or documents. We decided to break up each factor further into sub-

factors. The reasoning behind creating sub-factors from factors is that the sub-factors can

be measured relatively easily and used as the input of the DSS, and thus render a solution

to our problem.

In order to find these sub-factors (single input entities) we looked into each one of

the twelve factors and broke them into any individual concept, practice, or routine, and

any related documents that can be independently investigated. The principle behind

those concepts, practices, and documents were either found in Boehm's book or

discovered through the course of studying the thirty articles for validating Boehm's work.

Table 12 illustrates these sub-factors.

50


3.3.1 Input Criteria

Based on the findings in the knowledge acquisition section, we ascertained that

there are four major distinctive characteristics in agile and plan-driven development.

These characteristics are: Application Characteristics, Management Characteristics,

Technical Characteristics, and Personnel Characteristics. We also divided these

characteristics into twelve factors, and we discussed each of them extensively.

So it would be expected that these twelve factors be the criteria for our inputs. However,

as seen in previous sections, each of these twelve factors encompasses more detailed

concepts, practices, or documents. We decided to break up each factor further into sub

factors. The reasoning behind creating sub-factors from factors is that the sub-factors can

be measured relatively easily and used as the input of the DSS, and thus render a solution

to our problem.

In order to find these sub-factors (single input entities) we looked into each one of

the twelve factors and broke them into any individual concept, practice, or routine, and

any related documents that can be independently investigated. The principle behind

those concepts, practices, and documents were either found in Boehm’s book or

discovered through the course of studying the thirty articles for validating Boehm’s work.

Table 12 illustrates these sub-factors.

50


Table 12. Sub-factors

Factor Sub-factor


Primary Goals

- Continuous delivery - Responsiveness - Return on investment analysis - Quick build - Predictability - Stability - High-assurance

Size - Number of people - Scaling up

Environment

- Requirements rate of change - Organizational concerns - Stovepipes - Information sclerosis - Bridging situations - Monolithic requirements - Continuity requirements - Process improvement


Customer relations - Collocated customers - Contract specification - Process maturity


- Tacit knowledge vs. explicit knowledge - Pair programming - Meetings - Shared code ownership - On-site development - Developing tacit knowledge - Developing explicit knowledge

Project Communication - Person-to-person communication - Reports


Requirements

- Formal vs. informal - Prioritized requirements - Adjustability - Quality control / risk management

Development

- Refactoring - New features - Quality attributes - Multinational operations - Program efficiency - Architecture based design - Software reuse

Testing - Continuous regression testing - Automated test suits - Separate testing administration

51


Table 12. Sub-factors

Factor Sub-factor

ApplicationCharacteristics

Primary Goals

- Continuous delivery- Responsiveness- Return on investment analysis- Quick build- Predictability- Stability- High-assurance

Size - Number o f people- Scaling up

Environment

- Requirements rate of change- Organizational concerns- Stovepipes- Information sclerosis- Bridging situations- Monolithic requirements- Continuity requirements- Process improvement

ManagementCharacteristics

Customer relations- Collocated customers- Contract specification- Process maturity


- Tacit knowledge vs. explicit knowledge- Pair programming- Meetings- Shared code ownership- On-site development- Developing tacit knowledge- Developing explicit knowledge

Project Communication - Person-to-person communication- Reports

TechnicalCharacteristics

Requirements

- Formal vs. informal- Prioritized requirements- Adjustability- Quality control / risk management

Development

- Refactoring- New features- Quality attributes- Multinational operations- Program efficiency- Architecture based design- Software reuse

Testing- Continuous regression testing- Automated test suits- Separate testing administration

51


Table 12. Continued

Personnel characteristics

Customers - CRACK customer

Developers - Interpersonal skills - Levels of software developers (Cockburn levels)

Culture - Employee degrees of freedom

In order to validate the sub-factors chosen for each factor, we may need to go

back to our resources including Boehm's book and the thirty articles previously analysed.

However, an expert is required to determine the sub-factors from the available literature.

The articles provide even less obvious mentioning of the sub-factors, so expert advice is

needed to independently validate that the chosen sub-factors are actually acceptable

entities defining a factor. The following are some of the examples from the available

literature that ascertain the role of a sub-factor in the existence of a factor.

In this example, the sub-factor number of people is stated to play a role in the

factor size:

"...We see no way to avoid an activity of this nature (150-person, dealing with 34

highly complex systems elements) on extremely large systems of systems, and absolutely

no way to handle the problem with agile stand up meetings and tacit knowledge

propagation" [Boehm].

In this example the sub-factor stovepipes plays a role in the presence of the

environment factor:

" ...Early experience with elements of XP on departmental applications produced

code that didn't integrate well with the company's overall infrastructure or scale in

production" [Sliwa].

52


Table 12. Continued

Personnelcharacteristics

Customers - CRACK customer

Developers- Interpersonal skills- Levels of software developers (Cockbum levels)

Culture - Employee degrees of freedom

In order to validate the sub-factors chosen for each factor, we may need to go

back to our resources including Boehm’s book and the thirty articles previously analysed.

However, an expert is required to determine the sub-factors from the available literature.

The articles provide even less obvious mentioning o f the sub-factors, so expert advice is

needed to independently validate that the chosen sub-factors are actually acceptable

entities defining a factor. The following are some o f the examples from the available

literature that ascertain the role of a sub-factor in the existence of a factor.

In this example, the sub-factor number o f people is stated to play a role in the

factor size\

“...W e see no way to avoid an activity o f this nature (150-person, dealing with 34

highly complex systems elements) on extremely large systems o f systems, and absolutely

no way to handle the problem with agile stand up meetings and tacit knowledge

propagation’’ [Boehm].

In this example the sub-factor stovepipes plays a role in the presence of the

environment factor:

“ ...Early experience with elements of XP on departmental applications produced

code that didn’t integrate well with the company’s overall infrastructure or scale in

production” [Sliwa],

52


3.3.2 Questionnaire

A question is created in order to investigate the presence of a sub-factor. This

question is called a sample question (SQ). Sample questions will either inquire about the

presence of a sub-factor or will rate the efficiency, capability, and other attributes of the

existing sub-factor. The answer to the question is multiple choice, and the user has the

ability to quantitatively reply to the question. These sample questions are then refined

and developed into a questionnaire. By refining the questions to be easily understandable

to the users (who are in our case the software developers or managers) into a simple

sentence, the understanding of the question should be unproblematic. The choice of

terminology was the same as the ones used currently by managers in software developing

companies, thus there should not be any problems in understanding what a question is

asking.

The assigned weight of the question

As mentioned above, each question has a set of choices attached to it. Each

answer or option carries a weight called the assigned weight of the question. We decided

to use percentages to represent the weight of each choice. 100% signifies a perfect mark

for the choice of a plan-driven method, whereas 0% indicates a perfect mark for the

choice of agile development. 50% indicates either a mixed option (hybrid) or an option

that is neither plan-driven nor agile methodology. We have also added an extra choice

of, "Not sure," with no weight attached to it. Once this choice has been selected, the

question will be discarded since the user is unable to answer the question appropriately.

53


3.3.2 Questionnaire

A question is created in order to investigate the presence o f a sub-factor. This

question is called a sample question (SQ). Sample questions will either inquire about the

presence o f a sub-factor or will rate the efficiency, capability, and other attributes of the

existing sub-factor. The answer to the question is multiple choice, and the user has the

ability to quantitatively reply to the question. These sample questions are then refined

and developed into a questionnaire. By refining the questions to be easily understandable

to the users (who are in our case the software developers or managers) into a simple

sentence, the understanding of the question should be unproblematic. The choice of

terminology was the same as the ones used currently by managers in software developing

companies, thus there should not be any problems in understanding what a question is

asking.

The assigned weight of the question

As mentioned above, each question has a set of choices attached to it. Each

answer or option carries a weight called the assigned weight o f the question. We decided

to use percentages to represent the weight of each choice. 100% signifies a perfect mark

for the choice o f a plan-driven method, whereas 0% indicates a perfect mark for the

choice o f agile development. 50% indicates either a mixed option (hybrid) or an option

that is neither plan-driven nor agile methodology. We have also added an extra choice

of, “Not sure,” with no weight attached to it. Once this choice has been selected, the

question will be discarded since the user is unable to answer the question appropriately.

53


The general weight of the sub-factor

Each sub-factor will also have a General Weight attached to it. This weight is

different than the assigned weight of the question. This weight is used to determine how

much the sub-factor is important to the selection of the final decision. Six values are used

for the general weight of a sub-factor. The reasoning behind each of these general

weights comes from the observations recorded in Boehm's work and the readings of the

thirty articles. Our own experience was used in order to justify such assignments. In

some cases, an arbitrary but sensible weight has been assigned, since there was no

sufficiently accurate information to determine the given weight. The scale of the general

weight of the sub-factor is as follows:

5: Decisive

This means that this sub-factor is a critical one, and an answer to a question in this sub-

factor would indeed determine the outcome of the final decision.

4: Heavy

This means that there is a substantial weight associated with the factors with this General

weight. The sub-factor is highly critical to the decision.

3: Moderate

This means that the factor is fairly important and can indeed change the outcome of the

decision.

2: Light

This means that the corresponding factor is not as important and can be ignored.

1: Very Light

It means that most of the times the importance of the factor is negligible.

54


The general weight of the sub-factor

Each sub-factor will also have a General Weight attached to it. This weight is

different than the assigned weight of the question. This weight is used to determine how

much the sub-factor is important to the selection of the final decision. Six values are used

for the general weight of a sub-factor. The reasoning behind each o f these general

weights comes from the observations recorded in Boehm’s work and the readings of the

thirty articles. Our own experience was used in order to justify such assignments. In

some cases, an arbitrary but sensible weight has been assigned, since there was no

sufficiently accurate information to determine the given weight. The scale of the general

weight of the sub-factor is as follows:

5: Decisive

This means that this sub-factor is a critical one, and an answer to a question in this sub

factor would indeed determine the outcome of the final decision.

4: Heavy

This means that there is a substantial weight associated with the factors with this General

weight. The sub-factor is highly critical to the decision.

3: Moderate

This means that the factor is fairly important and can indeed change the outcome o f the

decision.

2: Light

This means that the corresponding factor is not as important and can be ignored.

1: Very Light

It means that most of the times the importance of the factor is negligible.

54


0: Non-Issue

Since the twelve factors and their sub-factors have been validated as necessary, we do not

have any factor / sub-factor with a general weight of zero. However, in future work, one

might want to take the importance of a given factor completely away, using a zero as the

general weight.

Appendix D shows the complete list of the sub-factors, their questions, and the

assigned and general weights.

3.3.3 Models

In the feasibility study, we mentioned that in order to be able to have an

automated solution to a problem domain, there should exist a conceptual model. This

model as discussed is a formal (structured) design that accepts the input and processes to

make a decision or to give a suggestion [Luger].

Currently, there are no models available that can help in choosing the most

suitable development paradigm for a given organization. However, many techniques are

available and have been used for expert systems dealing with the selection of a choice.

Two models are presented here and more are proposed for the future work section.

12-factor weight model

In this model, we evaluate each and every one of the twelve factors to see whether

they reside in the plan-driven or agile methods domain. Users (managers or software

developers) are asked to answer a questionnaire. Using the weight assigned to each

answer, and the general weight of the sub-factors, we can independently determine the

55


0: Non-Issue

Since the twelve factors and their sub-factors have been validated as necessary, we do not

have any factor / sub-factor with a general weight of zero. However, in future work, one

might want to take the importance of a given factor completely away, using a zero as the

general weight.

Appendix D shows the complete list o f the sub-factors, their questions, and the

assigned and general weights.

3.3.3 Models

In the feasibility study, we mentioned that in order to be able to have an

automated solution to a problem domain, there should exist a conceptual model. This

model as discussed is a formal (structured) design that accepts the input and processes to

make a decision or to give a suggestion [Luger],

Currently, there are no models available that can help in choosing the most

suitable development paradigm for a given organization. However, many techniques are

available and have been used for expert systems dealing with the selection of a choice.

Two models are presented here and more are proposed for the future work section.

12-factor weight model

In this model, we evaluate each and every one of the twelve factors to see whether

they reside in the plan-driven or agile methods domain. Users (managers or software

developers) are asked to answer a questionnaire. Using the weight assigned to each

answer, and the general weight of the sub-factors, we can independently determine the

55


compatibility of each of the twelve factors to the agile and plan-driven approaches. This

model shows a detailed picture of the twelve factors in an organization. As a result, the

users can see which areas in their organization are more or less compatible. With either

development methods, the sub-factors of all of the twelve factors are necessary to reach

an outcome for deciding whether that specific factor is in the agile or the plan-driven

domain.

This example will further illustrate the 12-factor weight model.

Example 1. We want to find out what is the outcome for the factor "Customer relations"

in a given organization. See Appendix D for a complete listing of the questions and their

weights.

First, we ask all of the questions regarding that factor. Suppose that these are the answers

to the questions:

1. Are your customers willing / able to be on-site and devote themselves to the

development of the project?

NO

2. Are your Customers knowledgeable enough to make project related decisions?

MAYBE

3. Does your organization depend upon formalized contract specification?

YES

4. Is the contract specification a source of trust (or lack of it a source of stress)

between your organization and your customers?

NOT SURE

56


compatibility o f each o f the twelve factors to the agile and plan-driven approaches. This

model shows a detailed picture of the twelve factors in an organization. As a result, the

users can see which areas in their organization are more or less compatible. With either

development methods, the sub-factors o f all of the twelve factors are necessary to reach

an outcome for deciding whether that specific factor is in the agile or the plan-driven

domain.

This example will further illustrate the 12-factor weight model.

Example 1. We want to find out what is the outcome for the factor “Customer relations”

in a given organization. See Appendix D for a complete listing of the questions and their

weights.

First, we ask all o f the questions regarding that factor. Suppose that these are the answers

to the questions:

1. Are your customers willing / able to be on-site and devote themselves to the

development o f the project?

NO

2. Are your Customers knowledgeable enough to make project related decisions?

MAYBE

3. Does your organization depend upon formalized contract specification?

YES

4. Is the contract specification a source of trust (or lack o f it a source o f stress)

between your organization and your customers?

NOT SURE

56


5. Has your customers asked you about the maturity of your processes? (i.e. Do they

expect your organization's processes be compatible with some well established

standards such as CMM or CMMI?)

NO

Based on Appendix D, each answer will correspond to these assigned weights (in

percent) and general weight of the sub-factor:

1. 75% 4

2. 50% 4

3. 100% 3

4. Discard

5. 25% 1

In order to calculate the weight of the factor each question's assigned weight will

be multiplied by the general weight of the sub-factor and then added together, and finally

divided by the sum of the general weights. The result will be a percentage value.

(75% x 4) + (50% x 4) + (100% x 3) + (25% x 1) / 4 + 4 + 3 + 1

= 68.75 %

According to the way the percentages are defined (100% completely plan-driven,

50% Hybrid, 0% completely agile), the 68% will somewhat lie under the Plan-driven

domain but not very comfortably. As a result, in this instance, this organization's

"Customer relations" factor is somewhat compatible with a plan-driven approach.

The advantages of the 12-factor weight model are that it gives a snapshot of the

organization and shows all the twelve factors (areas). This enables the managers and the

software developers to see each of the factors and whether they are compatible to the two

57


5. Has your customers asked you about the maturity of your processes? (i.e. Do they

expect your organization’s processes be compatible with some well established

standards such as CMM or CMMI?)

NO

Based on Appendix D, each answer will correspond to these assigned weights (in

percent) and general weight o f the sub-factor:

1. 75% 4

2. 50% 4

3. 100% 3

4. Discard

5. 25% 1

In order to calculate the weight of the factor each question’s assigned weight will

be multiplied by the general weight o f the sub-factor and then added together, and finally

divided by the sum of the general weights. The result will be a percentage value.

(75% x 4) + (50% x 4) + (100% x 3) + (25% x l ) / 4 + 4 + 3 + l

= 68.75 %

According to the way the percentages are defined (100% completely plan-driven,

50% Hybrid, 0% completely agile), the 68% will somewhat lie under the Plan-driven

domain but not very comfortably. As a result, in this instance, this organization’s

“Customer relations” factor is somewhat compatible with a plan-driven approach.

The advantages of the 12-factor weight model are that it gives a snapshot of the

organization and shows all the twelve factors (areas). This enables the managers and the

software developers to see each of the factors and whether they are compatible to the two

57


main methodologies. This way they would discover which areas or departments they

need to consider for compatibility issues.

However the disadvantage of the 12-factor weight is that the model does not

enable anyone to make a decision on what process methodology to use, as it ignores the

connection among the factors.

The results of the 12-factor weight model can be used in other models to reach a

final decision related to the selection of a proper process methodology.

Single-weight model

In the single weight model, we determine a single result that indicates what type

of process a given organization should use. The single weight answers the plain question

"What should my organization use between agile and plan-driven methodologies?"

The single weight model is based on the results of the 12-factor weight model applied to

every factor. Each of the twelve are computed using the 12-factor weight model, and

then the results are combined together according to their factors. This retraces our

understanding that not all of the twelve factors are equality important (critical) in making

the final decision.

The assignment of the criticality value is based on the interpretations arising from

Boehm's assessments and the experience gained from the analysis and validation of

Boehm's work. Again, some of these measures are somewhat arbitrary, based on our

understanding.

The criticality of the twelve factors we selected are as follows:

Primary goals 8

58


main methodologies. This way they would discover which areas or departments they

need to consider for compatibility issues.

However the disadvantage of the 12-factor weight is that the model does not

enable anyone to make a decision on what process methodology to use, as it ignores the

connection among the factors.

The results o f the 12-factor weight model can be used in other models to reach a

final decision related to the selection of a proper process methodology.

Single-weight model

In the single weight model, we determine a single result that indicates what type

of process a given organization should use. The single weight answers the plain question

“What should my organization use between agile and plan-driven methodologies?”

The single weight model is based on the results o f the 12-factor weight model applied to

every factor. Each of the twelve are computed using the 12-factor weight model, and

then the results are combined together according to their factors. This retraces our

understanding that not all o f the twelve factors are equality important (critical) in making

the final decision.

The assignment o f the criticality value is based on the interpretations arising from

Boehm’s assessments and the experience gained from the analysis and validation of

Boehm’s work. Again, some o f these measures are somewhat arbitrary, based on our

understanding.

The criticality o f the twelve factors we selected are as follows:

Primary goals 8

58


Size 8

Environment 8

Developers 8

Customer relations 1

Planning and control 1

Project Communication 1

Requirements 1

Development 1

Testing 1

Customers 1

Culture 1

This example further illustrates the single-weight model.

Example 2. The following are the results of a 12-weight model for a given

organization:

Primary goals 76%

Size 65%

Environment 71%

Developers 68%

Customer relations 52%

Planning and control 65%

Project Communication 69%

Requirements 25%

Development 17%

59


Size

Environment

Developers

Customer relations

Planning and control


Requirements

Development

Testing

Customers

Culture

This example further illustrates the single-weight model.

Example 2. The following are the results of a 12-weight model for a given

organization:

76%

65%

Primary goals

Size

Environment 71%

Developers 68%

Customer relations 52%

Planning and control 65%

Project Communication 69%

Requirements 25%

Development 17%

59


Testing 61%

Customers 75%

Culture 49%

The decision value is obtained through a weighted average of the results shown

above, using the criticality of each factor as a weight, as follows:

{[(Primary goal) x 8] + [(Size) x 8] + [(Environment) x 8] + [(Developers) x 8 +

Customer relations + Planning and Control + Project Communication +

Requirements + Development + Testing + Customers + Culture} /

8+8+8+8+1+1+1+1+1+1+1+1

= 67.54%

The value obtained is a percentage and shows that the organization is better

served through the plan-driven approach, the performance, however, is weak.

The advantage of the single weight model is that it would provide a simple and yet

important suggestion with regards to the selection of a process methodology.

The disadvantage of this model is that since the criticality factors are chosen arbitrarily,

the final decision is not an absolute verdict and, depending upon different organizations,

the accuracy level of the final decision can vary.

60


Testing 61%

Customers 75%

Culture 49%

The decision value is obtained through a weighted average o f the results shown

above, using the criticality of each factor as a weight, as follows:

{[(Primary goal) x 8] + [(Size) x 8] + [(Environment) x 8] + [(Developers)x 8 +

Customer relations + Planning and Control + Project Communication +

Requirements + Development + Testing + Customers + Culture} /

8+8+ 8+ 8+ 1+ 1+ 1+ 1+ 1+ 1+ 1+1

= 67.54%

The value obtained is a percentage and shows that the organization is better

served through the plan-driven approach, the performance, however, is weak.

The advantage of the single weight model is that it would provide a simple and yet

important suggestion with regards to the selection of a process methodology.

The disadvantage o f this model is that since the criticality factors are chosen arbitrarily,

the final decision is not an absolute verdict and, depending upon different organizations,

the accuracy level o f the final decision can vary.

60


Chapter 4

Results and Technology Transfer

4.1 Validation and testing

In order to discover whether the models presented are justifiable, we need to

validate them. There are different ways to validate these models. However, given our

constraints, only a few of the validation techniques may be applied and the rest is

proposed for future work.

We used two approaches to validate our models: the first validation method

employs data and concepts used in validating Boehm's work, while the second validation

method employs the key practices of CMMI and XP. We also suggest two other

validation approaches for future work. The first one is validating the model by feeding

empirical data extracted from a company. The second one is dividing a company in two

process groups.

4.1.1 Validating the model using the data collected to validate Boehm's work

In this approach, we used the data and concepts drawn from the thirty articles

selected to validate Boehm's work, and we applied our questionnaire to them and then

used the models. It was nearly impossible to determine all the data from the articles,

since most of the articles only provide conceptual information; however, based on the

concepts provided it can be said that the 12-factor-weight model and, consequently, the

single-weight model are both valid. There was no direct way to feed the data to the

questionnaire, so we decided to answer the questions based solely on my understanding

61


Chapter 4

Results and Technology Transfer

4.1 Validation and testing

In order to discover whether the models presented are justifiable, we need to

validate them. There are different ways to validate these models. However, given our

constraints, only a few of the validation techniques may be applied and the rest is

proposed for future work.

We used two approaches to validate our models: the first validation method

employs data and concepts used in validating Boehm’s work, while the second validation

method employs the key practices of CMMI and XP. We also suggest two other

validation approaches for future work. The first one is validating the model by feeding

empirical data extracted from a company. The second one is dividing a company in two

process groups.

4.1.1 Validating the model using the data collected to validate Boehm’s work

In this approach, we used the data and concepts drawn from the thirty articles

selected to validate Boehm’s work, and we applied our questionnaire to them and then

used the models. It was nearly impossible to determine all the data from the articles,

since most of the articles only provide conceptual information; however, based on the

concepts provided it can be said that the 12-factor-weight model and, consequently, the

single-weight model are both valid. There was no direct way to feed the data to the

questionnaire, so we decided to answer the questions based solely on my understanding

61


of the articles. Then we tried to apply the models based on such answers. Many of the

questions do not have any answers, since each given article did not talk about all of the

concepts or sub-factors. It was expected that the same articles that validated Boehm's

work would indeed confirm the validity of the models. It is essential to mention that

there is no convincing way of validating the model based on concepts alone, and no

empirical evidence was provided. Nonetheless, the data from the articles agrees with the

presented models.

4.1.2 Validating the models using the practices and concepts of CMMI and XP

The second validation employed in this thesis works by feeding the known and

defined practices of CMMI and XP into our models. Our models, as mentioned before,

are mainly based on the distinguishing characteristics of plan-driven and agile methods.

As a result, one will expect that these models should confirm that CMMI is a plan-driven

method and that XP is an agile method. This means that if we test the models with the

practices of CMMI, which is one of the best working examples of the plan-driven

methodologies, the result should also indicate a preference for plan-driven methods.

Similarly, since XP is one of the best examples of the agile methodologies, the result

should indicate a preference for agile methods.

Test 1: We assume that the practices of CMMI are employed (see Appendix A for

an overview of CMMI). Now we answer the questionnaire using only the practices of

CMMI. We ignore all the questions that are specific to the organization. Here, we are

only interested to see whether the practices of CMMI used in our models are able to

select a plan-driven decision, since CMMI is the best example of a plan-driven

62


of the articles. Then we tried to apply the models based on such answers. Many of the

questions do not have any answers, since each given article did not talk about all of the

concepts or sub-factors. It was expected that the same articles that validated Boehm’s

work would indeed confirm the validity of the models. It is essential to mention that

there is no convincing way of validating the model based on concepts alone, and no

empirical evidence was provided. Nonetheless, the data from the articles agrees with the

presented models.

4.1.2 Validating the models using the practices and concepts of CMMI and XP

The second validation employed in this thesis works by feeding the known and

defined practices o f CMMI and XP into our models. Our models, as mentioned before,

are mainly based on the distinguishing characteristics o f plan-driven and agile methods.

As a result, one will expect that these models should confirm that CMMI is a plan-driven

method and that XP is an agile method. This means that if we test the models with the

practices o f CMMI, which is one of the best working examples of the plan-driven

methodologies, the result should also indicate a preference for plan-driven methods.

Similarly, since XP is one of the best examples o f the agile methodologies, the result

should indicate a preference for agile methods.

Test 1: We assume that the practices of CMMI are employed (see Appendix A for

an overview o f CMMI). Now we answer the questionnaire using only the practices of

CMMI. We ignore all the questions that are specific to the organization. Here, we are

only interested to see whether the practices of CMMI used in our models are able to

select a plan-driven decision, since CMMI is the best example of a plan-driven

62


methodology. A degree of uncertainty is expected, since answering all of the questions of

the questionnaire is extremely important, and is determinant in the making of the final

decisions; however, there should be adequate information to justify the validation of our

models.

Table 13 shows the results of the 12-factor weight model. As can be seen, some

of the questions cannot be answered since they are organization-specific. However, just

the assumption that CMMI practices have been used is sufficient to provide high scores

for 8 of the twelve factors. Thus, the 12-factor weight model strongly suggests a plan-

driven approach for 8 of the twelve factors. The 12-factor weight can be the input to the

single-weight model.

The factors of size, customers, developers, and culture are ignored, since there is

no information about them. This reduces the effectiveness of the models. Still, the

decision value is 89% which means that CMMI belongs to the plan-driven domain. This

is in compliance with our predictions and serves a part of the model validation.

63


methodology. A degree of uncertainty is expected, since answering all of the questions of

the questionnaire is extremely important, and is determinant in the making of the final

decisions; however, there should be adequate information to justify the validation of our

models.

Table 13 shows the results of the 12-factor weight model. As can be seen, some

of the questions cannot be answered since they are organization-specific. However, just

the assumption that CMMI practices have been used is sufficient to provide high scores

for 8 o f the twelve factors. Thus, the 12-factor weight model strongly suggests a plan-

driven approach for 8 o f the twelve factors. The 12-factor weight can be the input to the

single-weight model.

The factors o f size, customers, developers, and culture are ignored, since there is

no information about them. This reduces the effectiveness of the models. Still, the

decision value is 89% which means that CMMI belongs to the plan-driven domain. This

is in compliance with our predictions and serves a part o f the model validation.

63


Table 13. Results Test 1 Factor Sub-Factor % Weight Factor

Primary Goals

- Continuous delivery - Responsiveness - Return on investment analysis - Quick build - Predictability - Stability - High assurance

--75% 3 - 100% 3 --

87.5%


- -

-

Environment


-75% 5 -- - --100% 3

84%

Customer relations

- Collocated customer - Contract specification - Process maturity

- 100% 3 100% 1

100%



100% 3 - -- -75% 2 100% 2

93%


- Person-to-person communication - Reports

- 100% 2

100%

Requirements

- Formal vs. informal - Prioritized requirements - Adjustability - Quality control / Risk management

100% 4 - -100% 4

100%

Development


75% 2 100% 4 100% 4 - 100% 3 75% 2 100% 3

94.5%


-100% 1 100% 2

100%

64


Table 13. Results Test 1Factor Sub-Factor % Weight Factor

Primary Goals

Continuous delivery Responsiveness Return on investment analysis Quick build Predictability Stability High assurance

75% 3

100% 387.5%

Size Number o f peopleScaling up________________Requirements rate of change Organizational concerns

■ Stovepipes■ Information sclerosis • Bridging situations■ Monolithic requirements■ Continuity requirements■ Process improvement______

75%

Environment

100%

84%

Customerrelations

Collocated customer Contract specification Process maturity

100%100%

100%


Tacit knowledge vs. explicit knowledgePair programmingMeetingsShared code ownership On-site development Developing tacit knowledge Developing explicit knowledge

100%

75% 2100% 2

93%

ProjectCommunication

Person-to-person communication Reports 100% 2

• Formal vs. informal• Prioritized requirements ■ Adjustability• Quality control / Risk management

100%

Requirements

100% 4

100% 4

100%

Development

Refactoring New features Quality attributes Multinational operations Program efficiency Architecture based design Software reuse

75% 2100% 4100% 4

100% 375% 2100% 3

94.5%

TestingContinuous regression testing Automated test suits Separate testing administration

100% 1 100% 2

100%

64


Table 13. Continued

Customers - CRACK customer - -


-- -

Culture - Employee degrees of freedom - -

Now we assume that the twelve practices of XP are employed (see Appendix B

for an overview of XP). We answer the questionnaire using only the twelve practices of

XP. Again, we ignore all of the questions that are organizational in nature. Here, we are

only interested to see, given the twelve practices and principles of XP, whether our

models are able to select an agile approach decision since XP is the best-known agile

development methodology. Again, not all of the questions on the questionnaire can be

answered. Table 14 shows the results of the 12-factor weight model.

Even if some of the questions cannot be answered since they are organization

specific, a low score is given to Seven out of twelve factors.

Applying these results to the single-weight model, we have a result of 0.39%

which is completely in the agile domain. This further confirms our hypothesis.

65


Table 13. Continued


- Interpersonal skills -

Developers - Levels o f software developers - -(Cockbum levels)


Now we assume that the twelve practices of XP are employed (see Appendix B

for an overview o f XP). We answer the questionnaire using only the twelve practices of

XP. Again, we ignore all o f the questions that are organizational in nature. Here, we are

only interested to see, given the twelve practices and principles o f XP, whether our

models are able to select an agile approach decision since XP is the best-known agile

development methodology. Again, not all of the questions on the questionnaire can be

answered. Table 14 shows the results of the 12-factor weight model.

Even if some o f the questions cannot be answered since they are organization

specific, a low score is given to Seven out o f twelve factors.

Applying these results to the single-weight model, we have a result o f 0.39%

which is completely in the agile domain. This further confirms our hypothesis.

65


Table 14. Results Test 2 Factor Sub-Factor % Weight

Primary Goals

- Continuous delivery - Responsiveness - Return on investment analysis - Quick build - Predictability - Stability - High assurance

0% 3 0% 3 -0% 3 ---

0%


- -

-

Environment


--- - ----

-

Customer relations

- Collocated customer - Contract specification - Process maturity

0% 4 - -

0%



0% 3 0% 3 25% 3 0% 3 -0% 2 0% 2

4.5%


- Person-to-person communication - Reports

0% 2 -

0 %

Requirements

- Formal vs. informal - Prioritized requirements - Adjustability - Quality control / Risk management

- - 0% 4 -

0%

Development


0% 2 --- 0% 3 --

0%


0% 1 - -

0%

66


Table 14. Results Test 2Factor Sub-Factor % Weight

Primary Goals

Continuous deliveryResponsivenessReturn on investment analysisQuick buildPredictabilityStabilityHigh assurance_____________

0%0%

0%

33

0%

Size Number of people Scaling up_______

Environment

Requirements rate of change Organizational concerns Stovepipes Information sclerosis Bridging situations Monolithic requirements Continuity requirements Process improvement______

Customerrelations

Collocated customer Contract specification Process maturity

0%0%


Tacit knowledge vs. explicit knowledgePair programmingMeetingsShared code ownership On-site development Developing tacit knowledge Developing explicit knowledge

0% 30% 325% 30% 3

0% 20% 2

4.5%

ProjectCommunication

■ Person-to-person communication Reports

0% 0%

Requirements

■ Formal vs. informal■ Prioritized requirements■ Adjustability• Quality control / Risk management

0%0%

Development

Refactoring New features Quality attributes Multinational operations Program efficiency Architecture based design Software reuse

0%

0%0%

TestingContinuous regression testing Automated test suits Separate testing administration

0%0%

66


Table 14. Continued



-- -


4.1.3 Validating the models using real data

In this proposed method, one should find a company that develops software and

does not have a process in place and would want to implement one. The company would

fill in the questionnaire and the results would be used in our models. The models would

select a process methodology for the company. The company would then adopt the

suggested methodology and then it would be evaluated to see whether the selection was

the correct one. For example, if the company becomes more successful, that would mean

that the decision was correct and this would ultimately validate our models. However, if

the company becomes less successful, then that would be an indication of the failure for

our models.

Alternatively, if the company is already using a given process methodology, the

company may want to know if their current methodology is appropriate for them. The

company would fill in the questionnaire and the results would be used in our models.

These are very unlikely approaches to the validation of these models, since they are very

difficult to administer. There are not many companies that I know of that would be

willing to put themselves at risk just to validate our models. However, this type of

validation is the ultimate way to thoroughly validate our models. If the models confirm

that the methodology they employ is appropriate for them, then this result could be used

as statistical evidence for the validation purposes. However, if the models suggest a

67


Table 14. Continued


- Interpersonal skills -Developers - Levels o f software developers

(Cockbum levels)- -


4.1.3 Validating the models using real data

In this proposed method, one should find a company that develops software and

does not have a process in place and would want to implement one. The company would

fill in the questionnaire and the results would be used in our models. The models would

select a process methodology for the company. The company would then adopt the

suggested methodology and then it would be evaluated to see whether the selection was

the correct one. For example, if the company becomes more successful, that would mean

that the decision was correct and this would ultimately validate our models. However, if

the company becomes less successful, then that would be an indication of the failure for

our models.

Alternatively, if the company is already using a given process methodology, the

company may want to know if their current methodology is appropriate for them. The

company would fill in the questionnaire and the results would be used in our models.

These are very unlikely approaches to the validation of these models, since they are very

difficult to administer. There are not many companies that I know of that would be

willing to put themselves at risk just to validate our models. However, this type of

validation is the ultimate way to thoroughly validate our models. If the models confirm

that the methodology they employ is appropriate for them, then this result could be used

as statistical evidence for the validation purposes. However, if the models suggest a

67


change, we would be able to monitor their change and the resulting increase in success.

This would prove or disprove the validity of our models.

In short, these are very unlikely approaches to validation, since they put

companies at risk.

4.1.4 Validating the models dividing a company into two process groups

This validation approach is very similar to the previous one. However, in this

approach, we first feed the data from the questionnaire to our models, and then the

models will select a process methodology. At this point, we divide the company into two

process groups. These two groups should be similar, i.e., same size, type of environment,

projects, personnel, etc.

Group 1 would adopt the methodology that the model ,determined and the other

group would use the opposing methodology. To validate our model Group 1 should be

more successful than Group 2.

We believe that this can be a very thorough and relatively safe validation of our

models; however, finding companies that are willing to do undertake such an experiment

is not within the scope of this thesis.

4.2 Additional developments

In this section, we propose two additional models and we propose the design of an

automated system.

68


change, we would be able to monitor their change and the resulting increase in success.

This would prove or disprove the validity of our models.

In short, these are very unlikely approaches to validation, since they put

companies at risk.

4.1.4 Validating the models dividing a company into two process groups

This validation approach is very similar to the previous one. However, in this

approach, we first feed the data from the questionnaire to our models, and then the

models will select a process methodology. At this point, we divide the company into two

process groups. These two groups should be similar, i.e., same size, type of environment,

projects, personnel, etc.

Group 1 would adopt the methodology that the model ,determined and the other

group would use the opposing methodology. To validate our model Group 1 should be

more successful than Group 2.

We believe that this can be a very thorough and relatively safe validation of our

models; however, finding companies that are willing to do undertake such an experiment

is not within the scope o f this thesis.

4.2 Additional developments

In this section, we propose two additional models and we propose the design of an

automated system.

68


4.2.1 Future proposed models

As mentioned in the methodology design section, we presented two models (12-

factor weight and single-weight) to process the answers of the questionnaire and draw a

final decision. Here, we propose some other models and techniques that can be

potentially used in the future.

Two-weight model

This model is very similar to the single weight model; however, it offers a

percentage weight value for agile development and a percentage weight value for a plan-

driven approach. In the single weight model, each question was constructed in a way that

the multiple-choice answer would be a percentage. A result near to 100% would indicate

a plan-driven approach, while a result near to 0% would indicate an agile development.

This form of questioning would put agile and plan-driven methods in opposition.

Although in most cases, that may be true, there is evidence that, in certain cases, plan-

driven and agile development use the same practises. As a result, in the two-weight

model we propose that the questionnaire be altered and two percentages arise from each

question: one to indicate how close the sub-factor (answer) is to plan-driven development

and the other to indicate how close the sub-factor (answer) is to agile development. In

this model, the final result comes in the form of two percentages: one for agile methods,

and the other for plan-driven methods. This method would provide a more realistic view

of the final result.

69


4.2.1 Future proposed models

As mentioned in the methodology design section, we presented two models (12-

factor weight and single-weight) to process the answers of the questionnaire and draw a

final decision. Here, we propose some other models and techniques that can be

potentially used in the future.

Two-weight model

This model is very similar to the single weight model; however, it offers a

percentage weight value for agile development and a percentage weight value for a plan-

driven approach. In the single weight model, each question was constructed in a way that

the multiple-choice answer would be a percentage. A result near to 100% would indicate

a plan-driven approach, while a result near to 0% would indicate an agile development.

This form of questioning would put agile and plan-driven methods in opposition.

Although in most cases, that may be true, there is evidence that, in certain cases, plan-

driven and agile development use the same practises. As a result, in the two-weight

model we propose that the questionnaire be altered and two percentages arise from each

question: one to indicate how close the sub-factor (answer) is to plan-driven development

and the other to indicate how close the sub-factor (answer) is to agile development. In

this model, the final result comes in the form of two percentages: one for agile methods,

and the other for plan-driven methods. This method would provide a more realistic view

of the final result.

69


Graphical models

In these models, one can use a graphical approach to present the final decision.

One way is to place each of the percentages of the twelve factors on an axis. The closer

to the middle, the more an axis is in agile domain, and the farther from plan-driven

domain. This scheme can use the 12-factor weight and give them a graphical

representation to further illustrate the company in a snapshot. This approach is very

similar to Boehm's proposed technique; however, in our proposed model, we use all

twelve factors. The graph for such image is conventionally called a radar graph. Figures

2 shows the radar graph.

In addition to the radar representation, a bar representation can also be used. In

this graph the twelve factors are represented as bars. The shorter the height of the bar,

the more in the agile domain it is, and the longer the bar, the more in the plan-driven

domain it is. Figures 3 shows the bar graphs.

Sample data for the graphs:

Factor 1 17% Factor 7 10%






70


Graphical models

In these models, one can use a graphical approach to present the final decision.

One way is to place each of the percentages of the twelve factors on an axis. The closer

to the middle, the more an axis is in agile domain, and the farther from plan-driven

domain. This scheme can use the 12-factor weight and give them a graphical

representation to further illustrate the company in a snapshot. This approach is very

similar to Boehm’s proposed technique; however, in our proposed model, we use all

twelve factors. The graph for such image is conventionally called a radar graph. Figures

2 shows the radar graph.

In addition to the radar representation, a bar representation can also be used. In

this graph the twelve factors are represented as bars. The shorter the height o f the bar,

the more in the agile domain it is, and the longer the bar, the more in the plan-driven

domain it is. Figures 3 shows the bar graphs.

Sample data for the graphs:







70


Figures 2. Radar graph.

Figure 3. Bar graph.

71


Figures 2. Radar graph.

100

80

60

40

20

01 2 3 4 5 6 7 8 9 10 11 12

Figure 3. Bar graph.

71


4.2.2 Automated System Structure Design

A proposed structure of an automated system consists of three blocks: the Input

Block, the Decision Block, and the Output Block. Figure 4 illustrates the system

structure.

Input Block Decision

Block

1** User

Figure 4. System structure.

Output Block

Input block

The user interface should be developed in this block. The user interface provides

the communication between the user and the system [Luger]. The interface initially

provides users with some basic information regarding the operation of the system, system

applications, and some background knowledge of the software process management. The

usability of the user interface should be emphasised, since it is important to be able to

easily interact with the users.

Then the system should administer the questionnaire. A help mechanism would

further illustrate each question. The data entered (the answers of the questions) are sent to

the decision block for processing.

72


4.2.2 Automated System Structure Design

A proposed structure of an automated system consists of three blocks: the Input

Block, the Decision Block, and the Output Block. Figure 4 illustrates the system

structure.

User

DecisionBlock

OutputBlockInput Block

Figure 4. System structure.

Input block

The user interface should be developed in this block. The user interface provides

the communication between the user and the system [Luger]. The interface initially

provides users with some basic information regarding the operation of the system, system

applications, and some background knowledge of the software process management. The

usability of the user interface should be emphasised, since it is important to be able to

easily interact with the users.

Then the system should administer the questionnaire. A help mechanism would

further illustrate each question. The data entered (the answers o f the questions) are sent to

the decision block for processing.

72


Decision block

A typical DSS would have the following components in its decision block:

Knowledge base (Rule base), Inference engine, and Working memory (Database). The

Rule base contains the following information: information for solving the problem

domain, additional material to tutor the user, and facts about the domain. The Inference

engine processes the knowledge in the Rule base. The model (reasoning) is implemented

in the Inference engine in order to make a judgment and arrive at the conclusion. The

Working memory, or the Database, contains all the information plus any extra material

that can be used by the Inference engine to process the rule base [Luger].

In our proposed system, the 12-factor weight and single model would be

implemented in the inference engine. The information regarding these factors and their

sub-factors would be placed in the Rule base. Also, the rules regarding decisive sub-

factors would be put in the Rule base. Such rules would allow drawing conclusions

without asking any more questions. For example, if a user enters a number larger than

150 for the number of people working on a project, the rule in the Rule base determines a

plan-driven method decision, without asking any more questions, since this sub-factor is

a decisive sub-factor.

All the supportive and complementary information, including feedback, can be

placed in the working memory (Database).

73


Decision block

A typical DSS would have the following components in its decision block:

Knowledge base (Rule base), Inference engine, and Working memory (Database). The

Rule base contains the following information: information for solving the problem

domain, additional material to tutor the user, and facts about the domain. The Inference

engine processes the knowledge in the Rule base. The model (reasoning) is implemented

in the Inference engine in order to make a judgment and arrive at the conclusion. The

Working memory, or the Database, contains all the information plus any extra material

that can be used by the Inference engine to process the rule base [Luger].

In our proposed system, the 12-factor weight and single model would be

implemented in the inference engine. The information regarding these factors and their

sub-factors would be placed in the Rule base. Also, the rules regarding decisive sub

factors would be put in the Rule base. Such rules would allow drawing conclusions

without asking any more questions. For example, if a user enters a number larger than

150 for the number of people working on a project, the rule in the Rule base determines a

plan-driven method decision, without asking any more questions, since this sub-factor is

a decisive sub-factor.

All the supportive and complementary information, including feedback, can be

placed in the working memory (Database).

73


Figure 5 illustrates the schematic of the decision block.

Inference Engine

Knowledge Base

(Rule Base)

Figure 5.

Working Memory

(Data Base) } Decision Block. (Figure from [Luger]).

Output block

The ruling of the decision is presented to the users by this block. Here, the user

discovers the result of the selection. The system declares whether the user is better off

adopting agile or plan-driven methods. If the result is in the middle, i.e. no clear decision

can be made, then the system suggests that the user either uses a hybrid method or

recruits someone to tailor a process specifically for their organization. In addition to this

decision, the output can include some recommendations and feedback regarding different

aspects and practices for their organization. Also, some on-line resources and support

could be available to the users in case they wish to discuss the results of this selection

with other professionals.

74


Figure 5 illustrates the schematic o f the decision block.

InferenceEngine

Working Memory

(Data Base)

Knowledge Base

(Rule Base)

Figure 5. Decision Block. (Figure from [Luger]).

Output block

The ruling of the decision is presented to the users by this block. Here, the user

discovers the result of the selection. The system declares whether the user is better off

adopting agile or plan-driven methods. If the result is in the middle, i.e. no clear decision

can be made, then the system suggests that the user either uses a hybrid method or

recruits someone to tailor a process specifically for their organization. In addition to this

decision, the output can include some recommendations and feedback regarding different

aspects and practices for their organization. Also, some on-line resources and support

could be available to the users in case they wish to discuss the results of this selection

with other professionals.

74


Chapter 5

Conclusions

5.1 Summary

The topics studied in this thesis were related to the principles of software

engineering in general and software process management in particular. We made the

assumption that in order to deliver a quality software product, satisfy customer demands,

and achieve organizational excellence, a structured, disciplined, and well-managed

process should be adopted [Benedicenti]. Thereafter, a study of different categories of

software processes was presented. It was discovered that there are two broad and distinct

groups of software process methodologies that are fundamentally different. Agile and

plan-driven approaches are the two types of software process methodologies. Using the

literature available to us, we determined the contrasts and organizational differences

between these two groups. These organizational differences were divided into four

groups: Application Characteristics, Management Characteristics, Technical

Characteristics, and Personnel Characteristics. Each of these characteristics were further

divided into 4 factors for a total of twelve distinguishing factors, referred to as (twelve

factors) [Boehm]. After this, we independently validated the soundness of the twelve

factors using case studies. Once the twelve factors were validated, we divided them into

single verifiable entities called sub-factors. The existence of each sub-factor was

confirmed by asking sample questions (SQ). By compiling the sample questions, a

questionnaire was produced to be given to managers and software developers.

75


Chapter 5

Conclusions

5.1 Summary

The topics studied in this thesis were related to the principles of software

engineering in general and software process management in particular. We made the

assumption that in order to deliver a quality software product, satisfy customer demands,

and achieve organizational excellence, a structured, disciplined, and well-managed

process should be adopted [Benedicenti]. Thereafter, a study o f different categories of

software processes was presented. It was discovered that there are two broad and distinct

groups of software process methodologies that are fundamentally different. Agile and

plan-driven approaches are the two types o f software process methodologies. Using the

literature available to us, we determined the contrasts and organizational differences

between these two groups. These organizational differences were divided into four

groups: Application Characteristics, Management Characteristics, Technical

Characteristics, and Personnel Characteristics. Each of these characteristics were further

divided into 4 factors for a total of twelve distinguishing factors, referred to as (twelve

factors) [Boehm]. After this, we independently validated the soundness of the twelve

factors using case studies. Once the twelve factors were validated, we divided them into

single verifiable entities called sub-factors. The existence o f each sub-factor was

confirmed by asking sample questions (SQ). By compiling the sample questions, a

questionnaire was produced to be given to managers and software developers.

75


An assigned weight was given to each answer of a question on the questionnaire,

which was called Assigned weight of a question; this weight was tabulated using a

percentage value. A value of 100% was to be a perfect match to the plan-driven approach

domain and a 0% a perfect match to the agile development domain. Each of the sub-

factors also was given a general weight. Once the weights were assigned, two models for

processing the answers were presented. The 12-factor weight model analyzed each of the

factors using the sub-factor's general weight and the question's assigned weight. A

percentage for each factor is the result of the 12-factor weight. This value is obtained by

weighted average. The single-weight model used the results of the 12-factor weight and

added a criticality value to each of the factors, which will produce an overall percentage

that indicates in what process methodology domain a specific organization resides. These

two models were validated using the data available in the literature and the concepts of

CMMI and XP.

5.2 Conclusions

The software developers or managers of software development companies are

often unable to decide which type of software processes they should adopt for their

corresponding organizations as much expertise and knowledge is needed to draw such

decisions. It is extremely important for the success of an organization to select a suitable

software process. The implications for adopting an incompatible process for any

organization can be catastrophic [Boehm]. Nevertheless, access to experts who are able

to provide such expertise is expensive and limited. As a result, this study embarked to

propose a DSS to replace such expertise. This DSS is to provide preliminary but reliable

76


An assigned weight was given to each answer o f a question on the questionnaire,

which was called Assigned weight o f a question; this weight was tabulated using a

percentage value. A value o f 100% was to be a perfect match to the plan-driven approach

domain and a 0% a perfect match to the agile development domain. Each of the sub

factors also was given a general weight. Once the weights were assigned, two models for

processing the answers were presented. The 12-factor weight model analyzed each of the

factors using the sub-factor’s general weight and the question’s assigned weight. A

percentage for each factor is the result o f the 12-factor weight. This value is obtained by

weighted average. The single-weight model used the results o f the 12-factor weight and

added a criticality value to each of the factors, which will produce an overall percentage

that indicates in what process methodology domain a specific organization resides. These

two models were validated using the data available in the literature and the concepts of

CMMI and XP.

5.2 Conclusions

The software developers or managers of software development companies are

often unable to decide which type of software processes they should adopt for their

corresponding organizations as much expertise and knowledge is needed to draw such

decisions. It is extremely important for the success of an organization to select a suitable

software process. The implications for adopting an incompatible process for any

organization can be catastrophic [Boehm], Nevertheless, access to experts who are able

to provide such expertise is expensive and limited. As a result, this study embarked to

propose a DSS to replace such expertise. This DSS is to provide preliminary but reliable

76


means to help managers and software developers with an overview of their organization's

characteristics as well as providing expert opinion on the type of software process they

may adopt. However, by no means is this system able to replace an expert. The primary

function of the system is to only to show the users where their organizations lie in regards

with the compatibility with different process methodologies. Secondly, it facilitates the

decision making process for the users for the selection of a software process

methodology. The proposed system, in addition, offers some unbiased advice and links

to further resources so that the managers can find more information in this regard. It can

also encourage the users to find more information and educate themselves in the areas of

software process management.

In short, it is difficult and expensive to select a suitable process methodology for a

given company. The DSS proposed in this study can definitely aid in selecting a suitable

software process at a reasonable cost, but it cannot completely replace an expert.

5.2 Future work

In the future, it is essential to further validate the two models presented in this

study (12-factor weight and single weight). There are two more validation techniques

presented in chapter four. However, more validation techniques can be devised to verify

the presented models. These models can inspire a good base for any future empirical

work with regards to software process management.

Two additional models have also been proposed (two-weight model and graphical

model). In the future, these models may be further developed and validated.

77


means to help managers and software developers with an overview of their organization’s

characteristics as well as providing expert opinion on the type o f software process they

may adopt. However, by no means is this system able to replace an expert. The primary

function of the system is to only to show the users where their organizations lie in regards

with the compatibility with different process methodologies. Secondly, it facilitates the

decision making process for the users for the selection of a software process

methodology. The proposed system, in addition, offers some unbiased advice and links

to further resources so that the managers can find more information in this regard. It can

also encourage the users to find more information and educate themselves in the areas of

software process management.

In short, it is difficult and expensive to select a suitable process methodology for a

given company. The DSS proposed in this study can definitely aid in selecting a suitable

software process at a reasonable cost, but it cannot completely replace an expert.

5.2 Future work

In the future, it is essential to further validate the two models presented in this

study (12-factor weight and single weight). There are two more validation techniques

presented in chapter four. However, more validation techniques can be devised to verify

the presented models. These models can inspire a good base for any future empirical

work with regards to software process management.

Two additional models have also been proposed (two-weight model and graphical

model). In the future, these models may be further developed and validated.

77


This study also proposes a design structure for the development of a decision

support system. This structure is divided into three blocks: the input block, the decision

block, and the output block. Also a proposed design for the decision block is given using

a typical expert system scheme. Using the concepts proposed, one should be able to

create the DSS itself. It is also noteworthy to mention that such a system can easily be

made into a web application for online clients.

78


This study also proposes a design structure for the development o f a decision

support system. This structure is divided into three blocks: the input block, the decision

block, and the output block. Also a proposed design for the decision block is given using

a typical expert system scheme. Using the concepts proposed, one should be able to

create the DSS itself. It is also noteworthy to mention that such a system can easily be

made into a web application for online clients.

78


References

[ADC]

[Agiledata]

[Allen]

[AM]

[Ambler]

[Ambysoft]

[Andreou]

[Beck]

[Beck 2000]

[Benedicenti]

Agile Development Conference website

http://www.agiledevelopmentconference.com/2003/schedule/researchpape

rs.html Accessed October 2004

Agile Data website, Scott Ambler, http://www.agiledata.org/ Accessed

June 24, 2005

XP Explained, Paul Allen, The Cutter Edge, June 5, 2001

Agile Modeling website, Scott Ambler, http://www.agilemodeling.com/

Accessed June 24, 2005

Agile Modeling: Effective Practices for Extreme Programming and the

Unified Process, Scott Ambler, Ron Jeffries, John Wiley and Sons Ltd,

April 2002

AmbySoft website, Scott Ambler http://www.ambysoft.com/ Accessed

June 24, 2005

Introducing a process framework for implementing models of large-scale

real-world systems in software, Andreas Andreou, Wiley Interscience,

Vol. 9 No. 3 (p 133-155)

Extreme Programming Pros and Cons: What Questions Remain?, Kent

Beck, IEEE Computer Society Dynabook, September 2001

Extreme Programming Explained, Kent Beck, Addison-Wesley, 2000

The process Modeling Handbook, L. Benedicenti, University of Regina,

2005

79


References

[ADC]

[Agiledata]

[Allen]

[AM]

[Ambler]

[Ambysoft]

[Andreou]

[Beck]

[Beck 2000]

[Benedicenti]

Agile Development Conference website


rs.html Accessed October 2004

Agile Data website, Scott Ambler, http://www.agiledata.org/ Accessed

June 24, 2005

XP Explained, Paul Allen, The Cutter Edge, June 5, 2001

Agile Modeling website, Scott Ambler, http://www.agilemodeling.com/


Agile Modeling: Effective Practices for Extreme Programming and the

Unified Process, Scott Ambler, Ron Jeffries, John Wiley and Sons Ltd,

April 2002

AmbySoft website, Scott Ambler http://www.ambysoft.com/ Accessed

June 24, 2005

Introducing a process framework for implementing models of large-scale

real-world systems in software, Andreas Andreou, Wiley Interscience,

Vol. 9 No. 3 (p 133-155)

Extreme Programming Pros and Cons: What Questions Remain?, Kent

Beck, IEEE Computer Society Dynabook, September 2001

Extreme Programming Explained, Kent Beck, Addison-Wesley, 2000

The process Modeling Handbook, L. Benedicenti, University o f Regina,

2005

79



http://www.agiledata.org/

http://www.agilemodeling.com/

http://www.ambysoft.com/

[Brooks] The Mythical Man-Month: Essays on Software Engineering, 20th

Anniversary Edition, Frederick Brooks, Addison-Wesley, 1995

[Brooks 04] Results of rapid bottom-up software process modeling, Andrew Brooks,

Wiley Interscience, Vol. 9 No. 4 (p 265-278)

[Boehm] Balancing Agility and Discipline: A guide for the Perplexed, Barry

Boehm, Richard Turner, Addison-Wesley, 2002

[Boehm 02 A] CMMI and the Balance of Discipline and Agility, Barry Boehm, USC

CMMI Technology Conference '02, November 13, 2002

[Boehm 02 B] Rebalancing Your Organization's Agility and Discipline, Barry Boehm,

Richard Turner, 2002

[Boehm 03] Value-Based Software Engineering: A Case Study, Barry Boehm, Li Guo

Huang, IEEE Computer Society, March 2003

[Boehm 04] Observations on Balancing Discipline and Agility, Barry Boehm,

University of Southern California, Richard Turner, George Washington

University

[Casey] A practical Application of the IDEAL (SM) Model, Valentine Casey, Ita

Richardson, University of Limerick, Ireland, Software Process

Improvement and Practice, 9 (p123-132), 2004

[CBS] COTS-Based Systems (CBS) Initiative website

http://www.sei.cmu.edu/cbs Accessed June 2005

[CeBASE] Centre for empirically based Software Engineering website,

www.cebase.org Accessed July 2005

80


[Brooks] The Mythical Man-Month: Essays on Software Engineering, 20th

Anniversary Edition, Frederick Brooks, Addison-Wesley, 1995

[Brooks 04] Results o f rapid bottom-up software process modeling, Andrew Brooks,

Wiley Interscience, Vol. 9 No. 4 (p 265-278)

[Boehm] Balancing Agility and Discipline: A guide for the Perplexed, Barry

Boehm, Richard Turner, Addison-Wesley, 2002

[Boehm 02 A] CMMI and the Balance of Discipline and Agility, Barry Boehm, USC

CMMI Technology Conference ’02, November 13, 2002

[Boehm 02 B] Rebalancing Your Organization’s Agility and Discipline, Barry Boehm,

Richard Turner, 2002

[Boehm 03] Value-Based Software Engineering: A Case Study, Barry Boehm, Li Guo

Huang, IEEE Computer Society, March 2003

[Boehm 04] Observations on Balancing Discipline and Agility, Barry Boehm,

University o f Southern California, Richard Turner, George Washington

University

[Casey] A practical Application of the IDEAL (SM) Model, Valentine Casey, Ita

Richardson, University of Limerick, Ireland, Software Process

Improvement and Practice, 9 (pl23-132), 2004

[CBS] COTS-Based Systems (CBS) Initiative website

http://www.sei.cmu.edu/cbs Accessed June 2005

[CeBASE] Centre for empirically based Software Engineering website,

www.cebase.org Accessed July 2005

80


http://www.sei.cmu.edu/cbs

http://www.cebase.org

[Chan] An Expert System Architectural Framework for Engineering Selection,

Christine Chan, Patrick Lau, University of Regina, Engineering

Applications Artificial Intelligence, Vol. 10. No. 4. (p 357-367), 1997

[Chrissis] CMMI ® Interpretive Guidance Project: What We Learned, Mary Beth

Chrissis, Mike Konrad, Sandy Shrum, Kenneth Smith, Gian Wemyss

http://www.sei.cmu.edu/pub/documents/04.reports/pdf/04sr008.pdf

Accessed October 2004

[CMMI] Capability Maturity Model 0 Integration (CMMI SM), Version 1.1, SEI,

2002, continuous representation

[Coad] Java Modeling in colour with UML, Peter Coad, Eric LeFebvre, and Eric

DeLuca, Prentice-Hall 2000

[Cockburn] Agile Software Development, Alistair Cockburn, Addison-Wesley, 2002

[Cockburn 03] Agile Software Development: It's about Feedback and Change, Laurie

Williams, North Carolina State University; and Alistair Cockburn,

Humans and Technology, Computer Magazine, June 2003

[Constantine] Methodological Agility, Larry Constantine, Software development on-

line, (p 67-69), June 2001

[Dion] Process Academy's White Paper, What is the CMMI?, Version: 1.00,

Francis Dion, July 30, 2003

http://www.dovico.com/documents/What-is-the-CMMI-process-

improvement-v1-00-20030730.pdf

[DoD-STD] http://www.software.org/Quagmire/descriptions/dod-std-2167a.asp

Accessed July 5, 2005

81


[Chan] An Expert System Architectural Framework for Engineering Selection,

Christine Chan, Patrick Lau, University o f Regina, Engineering

Applications Artificial Intelligence, Vol. 10. No. 4. (p 357-367), 1997

[Chrissis] CMMI ® Interpretive Guidance Project: What We Learned, Mary Beth

Chrissis, Mike Konrad, Sandy Shrum, Kenneth Smith, Gian Wemyss


Accessed October 2004

[CMMI] Capability Maturity Model ® Integration (CMMI SM), Version 1.1, SEI,

2002, continuous representation

[Coad] Java Modeling in colour with UML, Peter Coad, Eric LeFebvre, and Eric

DeLuca, Prentice-Hall 2000

[Cockbum] Agile Software Development, Alistair Cockbum, Addison-Wesley, 2002

[Cockbum 03] Agile Software Development: It's about Feedback and Change, Laurie

Williams, North Carolina State University; and Alistair Cockbum,

Humans and Technology, Computer Magazine, June 2003

[Constantine] Methodological Agility, Larry Constantine, Software development on

line, (p 67-69), June 2001

[Dion] Process Academy's White Paper, What is the CMMI?, Version: 1.00,

Franks Dion, July 30, 2003


improvement-v 1 -00-2003 073 O.pdf

[DoD-STD] http://www.software.org/Quagmire/descriptions/dod-std-2167a.asp

Accessed July 5, 2005

81




http://www.software.org/Quagmire/descriptions/dod-std-2167a.asp

[EIA/IEEE] http://standards.ieee.org/reading/ieee/std public/description/se/12207.0-

1996 desc.html Accessed July 5, 2005

[Emery] The Dangers of Extreme Programming, Patrick Emery, 2002,

http://members.cox.net/pemery/ Accessed March 2005

[Evans] A Business Case for Software Process Improvement, Paul Evans, Black

Diamond Software, 2003

[Fowler 2000] Planning Extreme Programming, Kent Beck, Martin Fowler, Addison-

Wesley, October 2000

[Fowler 03] The New Methodology, Martin Fowler, April 2003

http://martinfowler.com/articles/newMethodology.html Accessed June 24,

2005

[GE] General electric website, http://www.ge.com Accessed July 5, 2005

[Giese] Data Collection in a Process-Sensitive Software Engineering

Environment, P. Giese, B. Hoisly, C. M. Lottz, and H. D. Rombach

Research Group for Software Engineering Department of Computer

Science University of Kaiserslautern, Position Paper for ISPW9, 5-7

October 1994

[Highsmith 02 A]

Agile Software Development Ecosystems, Jim Highsmith, Addison-

Wesley, 2002

[Highsmith 02 B]

What Is Agile Software Development?, Jim Highsmith, CrossTalk

magazine, October 2002

82


[EIA/IEEE] http://standards.ieee.org/reading/ieee/std public/description/se/12207.0-

1996 desc.html Accessed July 5, 2005

[Emery] The Dangers of Extreme Programming, Patrick Emery, 2002,

http://members.cox.net/pemery/ Accessed March 2005

[Evans] A Business Case for Software Process Improvement, Paul Evans, Black

Diamond Software, 2003

[Fowler 2000] Planning Extreme Programming, Kent Beck, Martin Fowler, Addison-

Wesley, October 2000

[Fowler 03] The New Methodology, Martin Fowler, April 2003

http://martinfowler.com/articles/newMethodology.html Accessed June 24,

2005

[GE] General electric website, http://www.ge.com Accessed July 5, 2005

[Giese] Data Collection in a Process-Sensitive Software Engineering

Environment, P. Giese, B. Hoisly, C. M. Lottz, and H. D. Rombach

Research Group for Software Engineering Department o f Computer

Science University o f Kaiserslautern, Position Paper for ISPW9, 5-7

October 1994

[Highsmith 02 A]

Agile Software Development Ecosystems, Jim Highsmith, Addison-

Wesley, 2002

[Highsmith 02 B]

What Is Agile Software Development?, Jim Highsmith, CrossTalk

magazine, October 2002

82


http://standards.ieee.org/reading/ieee/std

http://members.cox.net/pemery/

http://martinfowler.com/articles/newMethodology.html

http://www.ge.com

http://www.adaptivesd.com/articles/cross oct02.pdf Accessed November

2004

[Highsmith 04]

Agile Project Management: Creating Innovative Products, Jim Highsmith,

Addison-Wesley, 2004

[Hitachi] Hitachi website, http://www.hitachi-soft.com/ Accessed July 4, 2005

[Humphrey 97]

Introduction to the Personal Software Process, Watts Humphrey, Addison-

Wesley, 1997

[Hyde] Intangible benefits of CMM-based software process improvement, Kevin

Hyde, David Wilson, Wiley Interscience Vol. 9 No. 4 (p 217-228)

[IBM] IBM website, http://www.rational.com Accessed July 5, 2005

[IEEE-STD] IEEE standards website, http://standards.ieee.org/ Accessed June 20, 2005

[ISO 15504] ISO 15504: Information Technology - Software Process Assessment

Guide To Qualification Of Assessors, ISO/IEC JTC1/SC7 Software

Engineering Canada (SCC), 1998

[Jeffries] Extreme Programming and the Capability Maturity Model, Ron Jeffries,

2000, http://www.xprogramming.com/xpmag/xp and cmm.htm Accessed

January 2005

[Lee] Multi-project software engineering analysis using systems thinking,

Bengee Lee, James Miller, Wiley Interscience Vol. 9 No. 3(p 173-214)

[Luger] Artificial Intelligence: Structures and Strategies for Complex Problem-

Solving, George Luger, The Benjamin/Cummings Pub. Co., Inc. 2004

83


http://www.adaptivesd.com/articles/cross oct02.pdf Accessed November

2004

[Highsmith 04]

Agile Project Management: Creating Innovative Products, Jim Highsmith,

Addison-Wesley, 2004

[Hitachi] Hitachi website, http://www.hitachi-soft.com/ Accessed July 4, 2005

[Humphrey 97]

Introduction to the Personal Software Process, Watts Humphrey, Addison-

Wesley, 1997

[Hyde] Intangible benefits of CMM-based software process improvement, Kevin

Hyde, David Wilson, Wiley Interscience Vol. 9 No. 4 (p 217-228)

[IBM] IBM website, http://www.rational.com Accessed July 5, 2005

[IEEE-STD] IEEE standards website, http://standards.ieee.org/ Accessed June 20, 2005

[ISO 15504] ISO 15504: Information Technology - Software Process Assessment

Guide To Qualification O f Assessors, ISO/IEC JTC1/SC7 Software

Engineering Canada (SCC), 1998

[Jeffries] Extreme Programming and the Capability Maturity Model, Ron Jeffries,

2000, http://www.xprogramming.com/xpmag/xp and cmm.htm Accessed

January 2005

[Lee] Multi-project software engineering analysis using systems thinking,

Bengee Lee, James Miller, Wiley Interscience Vol. 9 No. 3(p 173-214)

[Luger] Artificial Intelligence: Structures and Strategies for Complex Problem-

Solving, George Luger, The Benjamin/Cummings Pub. Co., Inc. 2004

83


http://www.adaptivesd.com/articles/cross

http://www.hitachi-soft.com/

http://www.rational.com

http://standards.ieee.org/

http://www.xprogramming.com/xpmag/xp

[Manifesto]

[McBreen]

[Mills]

[Nawrocki]

[Paulk]

[Pressman]

[Refactoring]

[Rico]

[Rumpe]

Manifesto website, http://www.agilemanifesto.org/ Accessed on June 6,

2005

Questioning Extreme Programming, Pete McBreen, Addison-Wesley,

2003

Cleanroom Software Engineering, Harlan Mills, Michael Dyer, Richard

Linger, IEEE Software 2, No. 9, (p19-24), 1984

Comparison of CMM Level 2 and Extreme Programming, Jerzy

Nawrocki, Poznan University of Technology, Poland

Extreme Programming from a CMM Perspective, Mark Paulk, Software

Engineering Institute, 2001

Pressman website, http://www.rspa.com/spi/ Accessed June 24, 2005

Refactoring website, Martin Fowler, http://www.refactoring.com/


Using Cost Benefit Analyses to Develop Software Process Improvement

(SPI) Strategies, David Rico, Data &Analysis Center for Software

(DACS), 2000

Quantitative Survey on Extreme Programming Projects, Bernhard Rumpe,

Astrid Schroder, Software & Systems Engineering Munich University of

Technology,

http://www.agilealliance.org/articles/articles/QuantitativeSurvey.pdf


84


[Manifesto]

[McBreen]

[Mills]

[Nawrocki]

[Paulk]

[Pressman]

[Refactoring]

[Rico]

[Rumpe]

Manifesto website, http://www.agilemanifesto.org/ Accessed on June 6,

2005

Questioning Extreme Programming, Pete McBreen, Addison-Wesley,

2003

Cleanroom Software Engineering, Harlan Mills, Michael Dyer, Richard

Linger,, IEEE Software 2, No. 9, (p i9-24), 1984

Comparison o f CMM Level 2 and Extreme Programming, Jerzy

Nawrocki, Poznan University of Technology, Poland

Extreme Programming from a CMM Perspective, Mark Paulk, Software

Engineering Institute, 2001

Pressman website, http://www.rspa.com/spi/ Accessed June 24, 2005

Refactoring website, Martin Fowler, http://www.refactoring.com/


Using Cost Benefit Analyses to Develop Software Process Improvement

(SPI) Strategies, David Rico, Data &Analysis Center for Software

(DACS), 2000

Quantitative Survey on Extreme Programming Projects, Bernhard Rumpe,

Astrid Schroder, Software & Systems Engineering Munich University of

Technology,

http://www.agilealliance.org/articles/articles/OuantitativeSurvey.pdf


84


http://www.agilemanifesto.org/

http://www.rspa.com/spi/

http://www.refactoring.com/

http://www.agilealliance.org/articles/articles/OuantitativeSurvey.pdf

[SCAMPI] Standard CMMI SM Appraisal Method for Process Improvement

(SCAMPI SM), Version 1.1, Members of the Assessment Method

Integrated Team, Method Definition Document, SEI, 2001

[Scacchi] Comparative Case Analysis for Understanding Software Processes, Walt

Scacchi, USC ATRIUM Laboratory, Los Angeles, CA

http://www.usc.edu/dept/ATRIUM/Papers/New/CCA-Draft.html


[Schwaber] Agile Software development with Scrum, Ken Schwaber and Mike

Beedle, Prentice-Hall, 2002

[SEI] http://www.sei.cmu.edu/ Accessed July 5, 2005

[Shull] Knowledge-Sharing Issues in Experimental Software Engineering, Forrest

Shull

http://www.cs.umd.edu/users/basili/cfBasilipapers/Knowledge%20Sharing

%20Issues%2082.91.pdf Accessed June 24, 2005

[Sliwa] Users warm up to agile programming, Carole Sliwa, ComputerWorld p.8,

March 18, 2002

[SW-CMM] Key Practices of the Capability Maturity Model SM, Mark Paulk, Version

1.1, SEI, 1993

[Tyrrell] The Many Dimensions of the Software Process, Sebastian Tyrrell, ACM

Crossroads, Summer 2000

[Williams] Strengthening the Case for Pair Programming, Williams, L., R.R. Kessler,

et al. IEEE Software, 17, 4 (July/August 2000) 19 — 25

[XP] http://www.extremeprogramming.org/ Accessed June 2005

85


[SCAMPI]

[Scacchi]

[Schwaber]

[SEI]

[Shull]

[Sliwa]

[SW-CMM]

[Tyrrell]

[Williams]

[XP]

Standard CMMI SM Appraisal Method for Process Improvement

(SCAMPI SM), Version 1.1, Members of the Assessment Method

Integrated Team, Method Definition Document, SEI, 2001

Comparative Case Analysis for Understanding Software Processes, Walt

Scacchi, USC ATRIUM Laboratory, Los Angeles, CA



Agile Software development with Scrum, Ken Schwaber and Mike

Beedle, Prentice-Hall, 2002

http://www.sei.emu.edu/ Accessed July 5, 2005

Knowledge-Sharing Issues in Experimental Software Engineering, Forrest

Shull


%20Issues%2082.91 .pdf Accessed June 24,2005

Users warm up to agile programming, Carole Sliwa, ComputerWorld p.8,

March 18, 2002

Key Practices of the Capability Maturity Model SM, Mark Paulk, Version

1.1, SEI, 1993

The Many Dimensions o f the Software Process, Sebastian Tyrrell, ACM

Crossroads, Summer 2000

Strengthening the Case for Pair Programming, Williams, L., R.R. Kessler,

et al. IEEE Software, 17, 4 (July/August 2000) 1 9 - 2 5

http://www.extremeprogramming.org/ Accessed June 2005

85



http://www.sei.emu.edu/


http://www.extremeprogramming.org/

Appendix A

Overview of the CMMI

The following paragraphs are directly taken from the literature for the readers' convenience. Readers are encouraged to consult the literature for more in depth information.

Capability Maturity Model Integration (CMMI)

The purpose of CMM Integration is to provide guidance for improving organizations' processes and the ability to manage the development, acquisition, and maintenance of products or services. CMM Integration places proven approaches into a structure that helps organizations appraise its organizational maturity or process area capability, establish priorities for improvement, and implement these improvements. The organizations can use a CMMI model to help set process-improvement objectives and priorities, improve processes, and provide guidance for ensuring stable, capable, and mature processes. A selected CMMI model can serve as a guide for improvement of organizational processes. [CMMI]

Process Maturity

In CMMI, software process maturity is the extent to which a specific process is explicitly defined, managed, measured, controlled, and effective. Maturity implies a potential for growth in capability and indicates both the richness of an organization's software process and the consistency with which it is applied in projects throughout the organization. The software process is well understood throughout a mature organization, usually through documentation and training, and the process is continually being monitored and improved by its users. The capability of a mature software process is known. Software process maturity implies that the productivity and quality resulting from an organization's software process can be improved over time through consistent gains in the discipline achieved by using its software process. [SW-CMM]

Maturity Levels

A maturity level is a well-defined evolutionary plateau toward achieving a mature software process. Each maturity level provides a layer in the foundation for continuous process improvement. Each level comprises a set of process goals that, when satisfied,

86


Appendix A

Overview of the CMMI

The following paragraphs are directly taken from the literature for the readers’ convenience. Readers are encouraged to consult the literature for more in depth information.

Capability Maturity Model Integration (CMMI)

The purpose of CMM Integration is to provide guidance for improving organizations’ processes and the ability to manage the development, acquisition, and maintenance of products or services. CMM Integration places proven approaches into a structure that helps organizations appraise its organizational maturity or process area capability, establish priorities for improvement, and implement these improvements. The organizations can use a CMMI model to help set process-improvement objectives and priorities, improve processes, and provide guidance for ensuring stable, capable, and mature processes. A selected CMMI model can serve as a guide for improvement of organizational processes. [CMMI]

Process Maturity

In CMMI, software process maturity is the extent to which a specific process is explicitly defined, managed, measured, controlled, and effective. Maturity implies a potential for growth in capability and indicates both the richness of an organization's software process and the consistency with which it is applied in projects throughout the organization. The software process is well understood throughout a mature organization, usually through documentation and training, and the process is continually being monitored and improved by its users. The capability of a mature software process is known. Software process maturity implies that the productivity and quality resulting from an organization’s software process can be improved over time through consistent gains in the discipline achieved by using its software process. [SW-CMM]

Maturity Levels

A maturity level is a well-defined evolutionary plateau toward achieving a mature software process. Each maturity level provides a layer in the foundation for continuous process improvement. Each level comprises a set of process goals that, when satisfied,

86


stabilize an important component of the software process. Achieving each level of the maturity framework establishes a different component in the software process, resulting in an increase in the process capability of the organization. There are five maturity levels, numbered 1 through 5. Each maturity level comprises a predefined set of process areas. [SW-CMM]

1. Initial 2. Managed 3. Defined 4. Quantitatively Managed 5. Optimizing

Initial

The software process is characterized as ad hoc, and occasionally even chaotic. Few processes are defined, and success depends on individual effort. There is not a defined process that is consistently followed by different project teams. Activity is chaotic and success depends on heroic efforts of individuals. Measurement and reporting, if they take place at all, vary from project to project. [Evans]

Repeatable

Basic project management processes are established to track cost, schedule, and functionality. The necessary process discipline is in place to repeat earlier successes on projects with similar applications. This level focuses on several practices that bring project under management control and make it possible for an organization to achieve repeatable results. The main areas addressed are project management, requirements, configuration management, and quality assurance. [Evans]

Defined

The software process for both management and engineering activities is documented, standardized, and integrated into a standard software process for the organization. All projects use an approved, tailored version of the organization's standard software process for developing and maintaining software. This level is about institutionalizing the good practices of level two that is these practices need to be documented and teams need to be trained in them. [Evans]

Managed

Detailed measures of the software process and product quality are collected. Both the software process and products are quantitatively understood and controlled. The

87


stabilize an important component of the software process. Achieving each level of the maturity framework establishes a different component in the software process, resulting in an increase in the process capability of the organization. There are five maturity levels, numbered 1 through 5. Each maturity level comprises a predefined set of process areas. [SW-CMM]

1. Initial2. Managed3. Defined4. Quantitatively Managed5. Optimizing

Initial

The software process is characterized as ad hoc, and occasionally even chaotic. Few processes are defined, and success depends on individual effort. There is not a defined process that is consistently followed by different project teams. Activity is chaotic and success depends on heroic efforts o f individuals. Measurement and reporting, if they take place at all, vary from project to project. [Evans]

Repeatable

Basic project management processes are established to track cost, schedule, and functionality. The necessary process discipline is in place to repeat earlier successes on projects with similar applications. This level focuses on several practices that bring project under management control and make it possible for an organization to achieve repeatable results. The main areas addressed are project management, requirements, configuration management, and quality assurance. [Evans]

Defined

The software process for both management and engineering activities is documented, standardized, and integrated into a standard software process for the organization. All projects use an approved, tailored version of the organization's standard software process for developing and maintaining software. This level is about institutionalizing the good practices o f level two that is these practices need to be documented and teams need to be trained in them. [Evans]

Managed

Detailed measures o f the software process and product quality are collected. Both the software process and products are quantitatively understood and controlled. The

87


organization institutes a quality and metrics management program. The software process as well as the products produced are quantitatively managed and controlled. [Evans]

Optimizing

Continuous process improvement is enabled by quantitative feedback from the process and from piloting innovative ideas and technologies. [SW-CMM]

Capability Levels

Capability levels focus on growing the organization's ability to perform, control, and improve its performance in a process area. Capability levels enable you to track, evaluate, and demonstrate your organization's progress as you improve processes associated with a process area. Capability levels build on each other, providing a recommended order for approaching process improvement. [CMMI] There are six capability levels, designated by the numbers 0 through 5: 0. Incomplete 1. Performed 2. Managed 3. Defined 4. Quantitatively Managed 5. Optimizing

CMMI Model Components

Process Areas: A process area is a cluster of related practices in an area that, when performed collectively, satisfy a set of goals considered important for making significant improvement in that area. All CMMI process areas are common to both continuous and staged representations. In the continuous representation, process areas are organized by process area categories. [CMMI]

Specific Goals: Specific goals apply to a process area and address the unique characteristics that describe what must be implemented to satisfy the process area. Specific goals are required model components and are used in appraisals to help determine whether a process area is satisfied. There can be specific practices at different capability levels mapped to the same goal. However, every goal has at least one capability level 1 practice mapped to it. [CMMI]

Specific Practices: A specific practice is an activity that is considered important in achieving the associated specific goal. The specific practices describe the activities expected to result in achievement of the specific goals of a process area. Every specific

88


organization institutes a quality and metrics management program. The software process as well as the products produced are quantitatively managed and controlled. [Evans]

Optimizing

Continuous process improvement is enabled by quantitative feedback from the process and from piloting innovative ideas and technologies. [SW-CMM]

Capability Levels

Capability levels focus on growing the organization’s ability to perform, control, and improve its performance in a process area. Capability levels enable you to track, evaluate, and demonstrate your organization’s progress as you improve processes associated with a process area. Capability levels build on each other, providing a recommended order for approaching process improvement. [CMMI]There are six capability levels, designated by the numbers 0 through 5:0. Incomplete1. Performed2. Managed3. Defined4. Quantitatively Managed5. Optimizing

CMMI Model Components

Process Areas: A process area is a cluster of related practices in an area that, when performed collectively, satisfy a set of goals considered important for making significant improvement in that area. All CMMI process areas are common to both continuous and staged representations. In the continuous representation, process areas are organized by process area categories. [CMMI]

Specific Goals: Specific goals apply to a process area and address the unique characteristics that describe what must be implemented to satisfy the process area. Specific goals are required model components and are used in appraisals to help determine whether a process area is satisfied. There can be specific practices at different capability levels mapped to the same goal. However, every goal has at least one capability level 1 practice mapped to it. [CMMI]

Specific Practices: A specific practice is an activity that is considered important in achieving the associated specific goal. The specific practices describe the activities expected to result in achievement of the specific goals of a process area. Every specific

88


practice is associated with a capability level. Specific practices are expected model components. [CMMI]

Typical Work Products: Typical work products are an informative model component that provides example outputs from a specific or generic practice. These examples are called "typical work products" because there are often other work products that are just as effective, but are not listed. [CMMI]

Subpractices: Subpractices are detailed descriptions that provide guidance for interpreting specific or generic practices. Subpractices may be worded as if prescriptive, but are actually an informative component in CMMI models meant only to provide ideas that may be useful for process improvement. [CMMI]

Generic Goals: Each capability level (1-5) has only one generic goal that describes the institutionalization that the organization must achieve at that capability level. Thus, there are five generic goals; each appears in every process area. Achievement of a generic goal in a process area signifies improved control in planning and implementing the processes associated with that process area thus indicating whether these processes are likely to be effective, repeatable, and lasting. Generic goals are required model components and are used in appraisals to determine whether a process area is satisfied. (Only the generic goal title and statement appear in the process areas.)

Generic Practices: Generic practices provide institutionalization to ensure that the processes associated with the process area will be effective, repeatable, and lasting. Generic practices are categorized by capability level and are expected components in CMMI models. [CMMI]

CMMI Staged Vs. Continuous

The continuous representation uses capability levels to measure process improvement, while the staged representation uses maturity levels. The main difference between maturity levels and capability levels is the representation they belong to and how they are applied. [CMMI]

• Capability levels, which belong to the continuous representation, apply to an organization's process-improvement achievement for each process area. There are six capability levels, numbered 0 through 5. Each capability level corresponds to a generic goal and a set of generic and specific practices. [CMMI]

• Maturity levels, which belong to the staged representation, apply to an organization's overall maturity. There are five maturity levels, numbered 1 through 5. Each maturity level comprises a predefined set of process areas. [CMMI]

89


practice is associated with a capability level. Specific practices are expected model components. [CMMI]

Typical Work Products: Typical work products are an informative model component that provides example outputs from a specific or generic practice. These examples are called “typical work products” because there are often other work products that are just as effective, but are not listed. [CMMI]

Subpractices: Subpractices are detailed descriptions that provide guidance for interpreting specific or generic practices. Subpractices may be worded as if prescriptive, but are actually an informative component in CMMI models meant only to provide ideas that may be useful for process improvement. [CMMI]

Generic Goals: Each capability level (1-5) has only one generic goal that describes the institutionalization that the organization must achieve at that capability level. Thus, there are five generic goals; each appears in every process area. Achievement of a generic goal in a process area signifies improved control in planning and implementing the processes associated with that process area thus indicating whether these processes are likely to be effective, repeatable, and lasting. Generic goals are required model components and are used in appraisals to determine whether a process area is satisfied. (Only the generic goal title and statement appear in the process areas.)

Generic Practices: Generic practices provide institutionalization to ensure that the processes associated with the process area will be effective, repeatable, and lasting. Generic practices are categorized by capability level and are expected components in CMMI models. [CMMI]

CMMI Staged Vs. Continuous

The continuous representation uses capability levels to measure process improvement, while the staged representation uses maturity levels. The main difference between maturity levels and capability levels is the representation they belong to and how they are applied. [CMMI]

• Capability levels, which belong to the continuous representation, apply to anorganization’s process-improvement achievement for each process area. There are six capability levels, numbered 0 through 5. Each capability level corresponds to a generic goal and a set o f generic and specific practices. [CMMI]

• Maturity levels, which belong to the staged representation, apply to anorganization’s overall maturity. There are five maturity levels, numbered 1through 5. Each maturity level comprises a predefined set o f process areas. [CMMI]

89


The continuous representation has more specific practices than the staged representation because the continuous representation has two types of specific practices, base and advanced, whereas the staged representation has only one type of specific practice. In the continuous representation, generic practices exist for capability levels 1-5, whereas, in the staged representation, only the generic practices from capability levels 2 and 3 appear; there are no generic practices from capability levels 1, 4, and 5. [CMMI]

Table A. CMMI Staged Vs. Continuous [Emery]

Continuous Representation Staged Representation

Process areas are organized by process area categories. !Process areas are organized by maturity level.

Improvement is measured using capability levels that are used to measure maturity or a particular process across an organization.

Improvement is measured using maturity levels that are used to measure maturity of a set of processes across an organization.

There are six capability levels 0-5. There are five maturity levels 1-5.

There are N + advanced practices because there are two types of specific practices: base and advanced.

There are N practices because there is only onetype of specific practice. The concept of advanced practices is not used.

Capability levels are used to organize the generic practices.

Common features are used to organize generic practices.

All generic practices are listed in each of the process areas.

Only the generic practices that are applicable tothat maturity level are listed in the process areas at that level.

Generic practices exist for capability levels 1-5. Generic practices exist for maturity levels 1-3.

Overview text is written to describe the continuous representation.

Overview text is written to describe the staged representation.

An additional appendix describing equivalent staging is included, which allows comparison between the continuous and staged representations.

There is no equivalence concept back to the continuous representation.

90


The continuous representation has more specific practices than the staged representation because the continuous representation has two types of specific practices, base and advanced, whereas the staged representation has only one type of specific practice.In the continuous representation, generic practices exist for capability levels 1-5, whereas, in the staged representation, only the generic practices from capability levels 2 and 3 appear; there are no generic practices from capability levels 1, 4, and 5. [CMMI]

Table A. CMMI Staged Vs. Continuous [Emery]

Continuous R epresentation Staged Representation

Process areas are organized by process area categories. Process areas are organized by maturity level.

Improvement is measured using capability levels that are used to measure maturity or a particular process across an organization.

Improvement is measured using maturity levels that are used to measure maturity o f a set of processes across an organization.

There are six capability levels 0-5. There are five maturity levels 1-5.

There are N + advanced practices because there are two types of specific practices: base and advanced.

There are N practices because there is only one type o f specific practice. The concept of advanced practices is not used.

Capability levels are used to organize the generic practices.

Common features are used to organize generic practices.

All generic practices are listed in each o f the process areas.

Only the generic practices that are applicable to that maturity level are listed in the process areas at that level.

Generic practices exist for capability levels 1-5. Generic practices exist for maturity levels 1-3.Overview text is written to describe the continuous representation.

Overview text is written to describe the staged representation.

An additional appendix describing equivalent staging is included, which allows comparison between the continuous and staged representations.

There is no equivalence concept back to the continuous representation.

90


Appendix B

Overview of the XP

The following paragraphs are directly taken from the literature. Readers are encouraged to consult with the literature for more in depth information.

Extreme Programming

Extreme Programming (XP) is a deliberate and disciplined approach to software development. It is a system of practices that software developers use to address the problem of quickly delivering quality software. XP breaks long projects into a sequence of self-contained 1-3 week mini projects (iterations), which deliver functional, tested software (releases). All programming in XP is done by pairs of programmers i.e. one writes, one reviews, tests are written before code. Everyone on the project owns code. Tested, running code is available at all times XP emphasizes teamwork; customers, managers, and programmers are all part of a team dedicated to delivering quality software. The Customer defines business value, decides what to do and what to defer, and defines the test to show that the system works. The manager coordinates, reports, rewards, and removes obstacles. The programmer analyzes, designs, tests, programs, and integrates the system. XP improves a software project in four essential ways: communication, simplicity, feedback, and courage. [Benedicenti]

These are the most distinct features of XP:

• The software development is split into a number of small increments (each one about 2-3 weeks long), giving the developers fast feedback from the customer; planning is performed just-in-time, only for the forthcoming increment.

• The customer role is crucial; he provides the programmers with user stories (informal, customer-written description of the functionality) and he is capable of changing them at any time. They also specify the acceptance tests.

• Quality assurance is test—centered: writing every piece of code should be preceded by preparation of a set of test cases for the implemented function. If a defect is detected, the programmer writes new test cases that detect the defect if it reappears.

• Programming is done in pairs. This is a technique thought as substitution for reviews and inspections — a pair of programmers work on a single piece of code, so they can review it continuously.

• The code is co-owned by all team members, so that everyone can change anything. It requires a defined coding standard. [Nawrocki]

91


Appendix B

Overview of the XP

The following paragraphs are directly taken from the literature. Readers are encouraged to consult with the literature for more in depth information.

Extreme Programming

Extreme Programming (XP) is a deliberate and disciplined approach to software development. It is a system of practices that software developers use to address the problem of quickly delivering quality software. XP breaks long projects into a sequence of self-contained 1-3 week mini projects (iterations), which deliver functional, tested software (releases). All programming in XP is done by pairs o f programmers i.e. one writes, one reviews, tests are written before code. Everyone on the project owns code. Tested, running code is available at all times XP emphasizes teamwork; customers, managers, and programmers are all part of a team dedicated to delivering quality software. The Customer defines business value, decides what to do and what to defer, and defines the test to show that the system works. The manager coordinates, reports, rewards, and removes obstacles. The programmer analyzes, designs, tests, programs, and integrates the system. XP improves a software project in four essential ways: communication, simplicity, feedback, and courage. [Benedicenti]

These are the most distinct features of XP:

• The software development is split into a number o f small increments (each one about 2-3 weeks long), giving the developers fast feedback from the customer; planning is performed just-in-time, only for the forthcoming increment.

• The customer role is crucial; he provides the programmers with user stories (informal, customer-written description o f the functionality) and he is capable of changing them at any time. They also specify the acceptance tests.

• Quality assurance is test-centered: writing every piece of code should be preceded by preparation of a set of test cases for the implemented function. If a defect is detected, the programmer writes new test cases that detect the defect if it reappears.

• Programming is done in pairs. This is a technique thought as substitution for reviews and inspections - a pair of programmers work on a single piece of code, so they can review it continuously.

• The code is co-owned by all team members, so that everyone can change anything. It requires a defined coding standard. [Nawrocki]

91


The Rules and Practices of Extreme Programming

The rules and practices of XP are: • Planning • Designing • Coding • Testing

In planning user stories are written, release-planning meeting creates the schedule. Frequent small releases are made, the project velocity is measured, and project is divided into iterations. Iteration planning starts at each iteration. People are moved around. Stand-up meeting starts each day. Fixes are applied when XP breaks. In designing emphasis is on simplicity, metaphors are chosen, Class Responsibility Collaboration (CRC) cards are used for design sessions. Spikes are applied to the solutions to reduce risk. No functionality is added early on. The Practice of refactoring is used whenever and wherever possible. [Benedicenti]

In coding the following rules are followed: • The customer is always available. • Code must be written to agreed standards. • Code the unit test first. • All code is pair programmed. • Only one pair integrates code at a time. • Integrate often. • Use collective code ownership. • Leave optimization until the last. • No overtime.

In testing all the codes must have unit tests; they must pass all unit tests before it can be released. When a bug is found tests are created. Acceptance tests are run often and the score is published. [Benedicenti]

Table B. illustrates the principals and practices of XP in a snap shot.

92


The Rules and Practices of Extreme Programming

The rules and practices o f XP are:• Planning• Designing• Coding• Testing

In planning user stories are written, release-planning meeting creates the schedule. Frequent small releases are made, the project velocity is measured, and project is divided into iterations. Iteration planning starts at each iteration. People are moved around. Stand-up meeting starts each day. Fixes are applied when XP breaks. In designing emphasis is on simplicity, metaphors are chosen, Class Responsibility Collaboration (CRC) cards are used for design sessions. Spikes are applied to the solutions to reduce risk. No functionality is added early on. The Practice o f refactoring is used whenever and wherever possible. [Benedicenti]

In coding the following rules are followed:• The customer is always available.• Code must be written to agreed standards.• Code the unit test first.• All code is pair programmed.• Only one pair integrates code at a time.• Integrate often.• Use collective code ownership.• Leave optimization until the last.• No overtime.

In testing all the codes must have unit tests; they must pass all unit tests before it can be released. When a bug is found tests are created. Acceptance tests are run often and the score is published. [Benedicenti]

Table B. illustrates the principals and practices of XP in a snap shot.

92


Table B. XP principals and XP practices (implementation) (After [Paulk])

Common Sense

Practice

XP Extreme XP Implementation

Code reviews Review code all the time Pair programming Testing Test all the time, even by the customers Unit testing,

functional testing Design Make design part of everybody's daily business Refactoring

Simplicity Always leave the system with the simplest design that supports its current functionality

The simplest thing that could possibly

work Architecture Everybody will work to refine the architecture Metaphor Integration

testing Integrate and test several times a day Continuous

integration Short

Iterations Make the iterations really, really short —

Seconds and minutes and hours not weeks and months

Planning game

Key Practices of Extreme Programming

According to Beck there are twelve key practices in XP. The following are the twelve practices of XP and a brief description of them [Beck]

Planning game. Customers decide the scope and timing of releases based on estimates provided by programmers. Programmers implement only the functionality demanded by the stories in this iteration.

Small releases. The system is put into production in a few months, before solving the whole problem. New releases are made often anywhere from daily to monthly.

Metaphor. The shape of the system is defined by a metaphor or set of metaphors shared between the customer and programmers.

Simple design. At every moment, the design runs all the tests, communicates everything the programmers want to communicate, contains no duplicate code, and has the fewest possible classes and methods. This rule can be summarized as, "Say everything once and only once."

Tests. Programmers write unit tests minute by minute. These tests are collected and they must all run correctly. Customers write functional tests for the stories in an iteration. These tests should also all run, although practically speaking, sometimes a business decision must be made comparing the cost of shipping a known defect and the cost of delay.

93


Table B. XP principals and XP practices (implementation) (After [Paulk])

CommonSense

Practice

XP Extreme XPImplementation

Code reviews Review code all the time Pair programmingTesting Test all the time, even by the customers Unit testing,

functional testingDesign Make design part of everybody’s daily business Refactoring

Simplicity Always leave the system with the simplest design that supports its current functionality

The simplest thing that could possibly

workArchitecture Everybody will work to refine the architecture MetaphorIntegration

testingIntegrate and test several times a day Continuous

integrationShort

IterationsMake the iterations really, really short -

Seconds and minutes and hours not weeks and months

Planning game

Key Practices of Extreme Programming

According to Beck there are twelve key practices in XP. The following are the twelve practices o f XP and a brief description of them [Beck]

Planning game. Customers decide the scope and timing of releases based on estimates provided by programmers. Programmers implement only the functionality demanded by the stories in this iteration.

Small releases. The system is put into production in a few months, before solving the whole problem. New releases are made often anywhere from daily to monthly.

Metaphor. The shape o f the system is defined by a metaphor or set of metaphors shared between the customer and programmers.

Simple design. At every moment, the design runs all the tests, communicates everything the programmers want to communicate, contains no duplicate code, and has the fewest possible classes and methods. This rule can be summarized as, "Say everything once and only once."

Tests. Programmers write unit tests minute by minute. These tests are collected and they must all run correctly. Customers write functional tests for the stories in an iteration. These tests should also all run, although practically speaking, sometimes a business decision must be made comparing the cost of shipping a known defect and the cost of delay.

93


Refactoring. The design of the system is evolved through transformations of the existing design that keep all the tests running.

Pair programming. All production code is written by two people at one screen / keyboard / mouse.

Continuous integration. New code is integrated with the current system after no more than a few hours. When integrating, the system is built from scratch and all tests must pass or the changes are discarded.

Collective ownership. Every programmer improves any code anywhere in the system at any time if they see the opportunity.

On-site customer. A customer sits with the team full-time.

40-hour weeks. No one can work a second consecutive week of overtime. Even isolated overtime used too frequently is a sign of deeper problems that must be addressed.

The planning game and small releases depend on the customer providing a set of "stories," or short descriptions of features, that characterize the work to be performed in each release. Releases are two weeks apart, and the team and customer must come to agreement on which stories (simple use cases) will be implemented within a two-week period. A pool of stories characterizes the full functionality desired by the customer, but only the subset identified as those features most desired by the customer for next two-week release are being implemented at any time. New stories can be added to the pool at any time, thus the requirements can be highly volatile, but implementation proceeds in two-week chunks based on the most desired functions currently in the pool, thus the volatility is managed. An on-site customer is needed to support this style of iterative life cycle. [Paulk]

"Metaphor" provides the overarching vision for the project. This could be considered a high-level architecture, but XP emphasizes design while at the same time minimizing design documentation. Some have characterized XP as not allowing documentation outside code [Allen], but it is probably more accurate to say that, since XP emphasizes continual redesign (via refactoring whenever necessary), there is little value to detailed design documentation. Maintainers rarely trust anything other than the code anyway. Design documentation is typically thrown away after the code is written. The only time design documentation is kept is when the customer can no longer come up with any new stories. Then it is time to put the system in mothballs and write a five to ten page "mothball tour" of the system. A natural corollary of the emphasis on refactoring is to always implement the simplest solution to satisfy the immediate need. Changes in the requirements are likely to supersede "general solutions" anyway. Pair programming is one of the more controversial practices in XP since it has resource consequences for the managers who decide whether or not the project will use XP. Although it may appear that pair programming will consume twice the resources, research has shown that pair programming leads to higher quality and decreased cycle time [Williams]. For an experienced team the effort increase may be as little as 15%, while the reduction in cycle

94


Refactoring. The design of the system is evolved through transformations of the existing design that keep all the tests running.

Pair programming. All production code is written by two people at one screen / keyboard / mouse.

Continuous integration. New code is integrated with the current system after no more than a few hours. When integrating, the system is built from scratch and all tests must pass or the changes are discarded.

Collective ownership. Every programmer improves any code anywhere in the system at any time if they see the opportunity.

On-site customer. A customer sits with the team full-time.

40-hour weeks. No one can work a second consecutive week of overtime. Even isolated overtime used too frequently is a sign o f deeper problems that must be addressed.

The planning game and small releases depend on the customer providing a set of “stories," or short descriptions of features, that characterize the work to be performed in each release. Releases are two weeks apart, and the team and customer must come to agreement on which stories (simple use cases) will be implemented within a two-week period. A pool o f stories characterizes the full functionality desired by the customer, but only the subset identified as those features most desired by the customer for next two- week release are being implemented at any time. New stories can be added to the pool at any time, thus the requirements can be highly volatile, but implementation proceeds in two-week chunks based on the most desired functions currently in the pool, thus the volatility is managed. An on-site customer is needed to support this style of iterative life cycle. [Paulk]

"Metaphor" provides the overarching vision for the project. This could be considered a high-level architecture, but XP emphasizes design while at the same time minimizing design documentation. Some have characterized XP as not allowing documentation outside code [Allen], but it is probably more accurate to say that, since XP emphasizes continual redesign (via refactoring whenever necessary), there is little value to detailed design documentation. Maintainers rarely trust anything other than the code anyway. Design documentation is typically thrown away after the code is written. The only time design documentation is kept is when the customer can no longer come up with any new stories. Then it is time to put the system in mothballs and write a five to ten page "mothball tour" of the system. A natural corollary of the emphasis on refactoring is to always implement the simplest solution to satisfy the immediate need. Changes in the requirements are likely to supersede "general solutions" anyway. Pair programming is one o f the more controversial practices in XP since it has resource consequences for the managers who decide whether or not the project will use XP. Although it may appear that pair programming will consume twice the resources, research has shown that pair programming leads to higher quality and decreased cycle time [Williams], For an experienced team the effort increase may be as little as 15%, while the reduction in cycle

94


time may be 40- 50%. For interne-time environments, the increased speed to market may be well worth the increment in effort. Collaboration improves the problem-solving process, and the increase in quality will also have a significant impact on maintenance costs, which appears likely to more than pay for any added resource costs over the total life cycle. Collective ownership means that anyone can change any piece of code in the system at any time. The XP emphasis on continuous integration, continual regression testing, and pair programming are intended as protections against problems here. "Test then code" is the phrase used to express XP's emphasis on testing. It captures the principle that testing should be planned early and test cases developed in parallel with requirements analysis, although the traditional emphasis is on black box testing. Thinking about testing early in the life cycle is a well-known good software engineering practice, even if too infrequently practiced. The basic XP management tool is the metric, and the medium of the metric is the "big visible chart." In the XP style, three or four measures are typically all a team can stand at one time, and those should be actively used and visible to the team. "Project velocity," the number of stories of a given size that can be done in iteration, is one recommended XP metric. [Paulk]

95


time may be 40- 50%. For internet-time environments, the increased speed to market may be well worth the increment in effort. Collaboration improves the problem-solving process, and the increase in quality will also have a significant impact on maintenance costs, which appears likely to more than pay for any added resource costs over the total life cycle. Collective ownership means that anyone can change any piece of code in the system at any time. The XP emphasis on continuous integration, continual regression testing, and pair programming are intended as protections against problems here. “Test then code” is the phrase used to express XP's emphasis on testing. It captures the principle that testing should be planned early and test cases developed in parallel with requirements analysis, although the traditional emphasis is on black box testing. Thinking about testing early in the life cycle is a well-known good software engineering practice, even if too infrequently practiced. The basic XP management tool is the metric, and the medium of the metric is the "big visible chart." In the XP style, three or four measures are typically all a team can stand at one time, and those should be actively used and visible to the team. "Project velocity," the number o f stories of a given size that can be done in iteration, is one recommended XP metric. [Paulk]

95


Appendix C

Bibliography of the case studies

[1]

[2]

[3]

[4]

Business Rules: Managing Released Software and IT Problem Resolutions Using Scrum and Agile, Michael Dwyer, CSM October 6 2004 http ://www.agilealliance.org/articles/articles/BizRulesAndITOps.pdf Accessed March 2005

This paper describes the empirically generated set of rules and tools that allowed an existing IT operation to adjust their workflow in maintaining, enhancing, and supporting business demands — and to do this in a proactive, agile, manner. A Help desk operation.

Framework XP —Building Frameworks using XP, Gerard Meszaros, Jennitta Andrea, Shaun Smith, 2002 http ://www. agi I eallianc e. org/articles/articles/FrameworkXP.p d f Accessed March 2005

Frameworks are different from other kinds of software because they need special care. Learn how a little re-tailoring makes XP suit able for building frameworks.

The Unbearable Lightness of Programming: A Tale of Two Cultures, Laurent Bossavit, 2002 http://www.agilealliance.org/articles/articles/UnbearableLightness.pdf Accessed March 2005 Excellent article summarizing the author's experiences with XP at two different companies with two very different cultures. Both failed due to lack of proper XP culture

Best Practices in Scrum Project Management and XP Agile Software Development, Ken Schwaber, 2004 http://www.obj ectm entor. com/resourc es/arti cl es/Primavera Accessed March 2005 Description of a highly successful agile project at Primavera.

96


Appendix C

Bibliography of the case studies

[1] Business Rules: Managing Released Software and IT ProblemResolutions Using Scrum and Agile, Michael Dwyer, CSM October 6 2004http://www.agilealliance.org/articles/articles/BizRulesAndITOps.pdf Accessed March 2005

This paper describes the empirically generated set of rules and tools that allowed an existing IT operation to adjust their workflow in maintaining, enhancing, and supporting business demands - and to do this in a proactive, agile, manner. A Help desk operation.

[2] Framework XP -Building Frameworks using XP, Gerard Meszaros,Jennitta Andrea, Shaun Smith, 2002http://www.agilealliance.org/articles/articles/FrameworkXP.pdf Accessed March 2005

Frameworks are different from other kinds o f software because they need special care. Learn how a little re-tailoring makes XP suit able for building frameworks.

[3] The Unbearable Lightness of Programming: A Tale of Two Cultures,Laurent Bossavit, 2002http://www.agilealliance.org/articles/articles/UnbearableLightness.pdf Accessed March 2005Excellent article summarizing the author's experiences with XP at two different companies with two very different cultures.Both failed due to lack of proper XP culture

[4] Best Practices in Scrum Project Management and XP Agile SoftwareDevelopment, Ken Schwaber, 2004 http://www.obiectmentor.com/resources/articles/Primavera Accessed March 2005Description of a highly successful agile project at Primavera.

96


http://www.agilealliance.org/articles/articles/BizRulesAndITOps.pdf

http://www.agilealliance.org/articles/articles/FrameworkXP.pdf

http://www.agilealliance.org/articles/articles/UnbearableLightness.pdf

http://www.obiectmentor.com/resources/articles/Primavera

[5]

[6]

[7]

Agility, Uncertainty, and Software Project Estimation, Todd Little, 2004 http://www.macs.ece.mcgill.ca/—radu/304428W03/AgilityUncertaintyAnd Estimation.pdf Accessed March 2005

This paper presents the results of studies, estimates and actual data for 120 commercial projects.

A Metric Leading to Agility, Ron Jeffries, 2004 http://www.x_programming.com/xpmag/jatRtsMetric.htm Accessed March 2005

Describes a new metric for the number of Running Tested Features delivered in a product and how this leads a team to develop in an agile manner.

Extreme Programming in a Customer Services Organization, Pulugurtha, Srinivasa Neveu, Jean-Noel Lynch, Francis, 2002 http://www.agilealliance.org/articles/articles/XPinCustSvcs.pdf Accessed March 2005

A case study that shows how the adoption of XP practices helps bring a Customer Services organization up to speed. Not much information!

[8] Agile insurance, Thomas Ailgum, 2004 http://www.cio.com/archive/081504/et article.html Accessed March 2005

[9]

Case study of how Pinnacol Assurance had to respond quickly when insurance industry regulators wanted to shut down one of their products. Because of their agile development process, the company was able to quickly change their product to satisfy the regulators.

Quantitative Survey on Extreme Programming Projects, Bernhard Rumpe, Astrid Schroder, 2002 http://www.agilealliance.orearticles/articles/QuantitativeSurvey.pdf Accessed March 2005

Survey with a small sample (49 entries) on where and how XP is used in IT development projects.

97


[5] Agility, Uncertainty, and Software Project Estimation, Todd Little,2004http://www.macs.ece.mcgill.ca/~radu/304428W03/AgilitvUncertaintyAnd Estimation.pdf Accessed March 2005

This paper presents the results of studies, estimates and actual data for 120 commercial projects.

[6] A Metric Leading to Agility, Ron Jeffries, 2004http://www.xprogramming.com/xpmag/iatRtsMetric.htm Accessed March 2005

Describes a new metric for the number o f Running Tested Features delivered in a product and how this leads a team to develop in an agile manner.

[7] Extreme Programming in a Customer Services Organization,Pulugurtha, Srinivasa Neveu, Jean-Noel Lynch, Francis, 2002 http://www.agilealliance.org/articles/articles/XPinCustSvcs.pdf Accessed March 2005

A case study that shows how the adoption of XP practices helps bring a Customer Services organization up to speed. Not much information!

[8] Agile insurance, Thomas Ailgum, 2004http://www.cio.com/archive/081504/et article.html Accessed March 2005

Case study o f how Pinnacol Assurance had to respond quickly when insurance industry regulators wanted to shut down one o f their products. Because of their agile development process, the company was able to quickly change their product to satisfy the regulators.

[9] Quantitative Survey on Extreme Programming Projects, BernhardRumpe, Astrid Schroder, 2002http://www.agilealliance.org/articles/articles/QuantitativeSurvev.pdf Accessed March 2005

Survey with a small sample (49 entries) on where and how XP is used in IT development projects.

97


http://www.macs.ece.mcgill.ca/~radu/304428W03/AgilitvUncertaintyAnd

http://www.xprogramming.com/xpmag/iatRtsMetric.htm

http://www.agilealliance.org/articles/articles/XPinCustSvcs.pdf

http://www.cio.com/archive/081504/et

http://www.agilealliance.org/articles/articles/QuantitativeSurvev.pdf

[10] The Challenge: Cost-Cutting, Megan Santosus, 2004 http://www.cio.com/archive/081504/challenge cost.html Accessed March 2005

Short case study about one of CIO Magazine's "Agile 100" organizations that responded in an agile manner to a cost-cutting initiative.

An Essential Distinction of Agile Software Development Processes Based on Systems Thinking in Software Engineering Management, Peter Wendorff, 2002 http://www.agilealliance.orglarticles/articles/EssentialDistinction.pdf Accessed March 2005

An interesting approach which considers Agile Software Development as seen from the viewpoint of General Systems Theory. As a result, the author identifies two sources of project delay and explains how Agile Methods address them.

[12] The Challenge Globalization, Mindy Blodgett, 2004 http://www.cio.com/archive/081504/challenge globalization.html Accessed March 2005

Short case study about one of CIO Magazine's "Agile 100" that had to quickly deal with becoming a global company

[13] FDD Implementations, Jeff De Luca, 2002 http://www.nebulon.com/articles/fdd/fddimplementations.html Accessed March 2005

This article describes some of the key Feature-Driven Development projects through the years.

[14] Inside an Agile Transformation, Stephanie Overby, 2004 http://www.cio.com/archive/081504/profile.html Accessed March 2005

This article describes the transformation of the United States Defence Logistics Agency (DLA).

[15] Testing in the Fast Lane, Lisa Crispin, Tip House, 2002 http://www.agilealliance.org/articles/articles/TestingInFastLane.pdf Accessed March 2005

98


[10] The Challenge: Cost-Cutting, Megan Santosus, 2004http://www.cio.com/archive/081504/challenge cost.html Accessed March 2005

Short case study about one of CIO Magazine's "Agile 100" organizations that responded in an agile manner to a cost-cutting initiative.

[11] An Essential Distinction of Agile Software Development ProcessesBased on Systems Thinking in Software Engineering Management,Peter Wendorff, 2002http://www.agilealliance.org/articles/articles/EssentialDistinction.pdf Accessed March 2005

An interesting approach which considers Agile Software Development as seen from the viewpoint o f General Systems Theory. As a result, the author identifies two sources o f project delay and explains how Agile Methods address them.

[12] The Challenge Globalization, Mindy Blodgett, 2004http://www.cio.com/archive/081504/challenge globalization.html Accessed March 2005

Short case study about one of CIO Magazine's "Agile 100" that had to quickly deal with becoming a global company

[13] FDD Implementations, Jeff De Luca, 2002http://www.nebulon.com/articles/fdd/fddimplementations.html Accessed March 2005

This article describes some of the key Feature-Driven Development projects through the years.

[14] Inside an Agile Transformation, Stephanie Overby, 2004http://www.cio.com/archive/081504/profile.html Accessed March 2005

This article describes the transformation of the United States Defence Logistics Agency (DLA).

[15] Testing in the Fast Lane, Lisa Crispin, Tip House, 2002http://www.agilealliance.org/articles/articles/TestingInFastLane.pdf Accessed March 2005

98


http://www.cio.com/archive/081504/challenge

http://www.agilealliance.org/articles/articles/EssentialDistinction.pdf

http://www.cio.com/archive/081504/challenge

http://www.nebulon.com/articles/fdd/fddimplementations.html

http://www.cio.com/archive/081504/profile.html

http://www.agilealliance.org/articles/articles/TestingInFastLane.pdf

Continuing Kent Beck's metaphor of XP as driving a car, the authors argue that the tester is the navigator and plays an important role in XP projects.

[16] Come Together Right Now, Kim Girard, 2004 http://www.cio.com/archive/081504/merge.html Accessed March 2005

The benefits of a merger or acquisition can be lost if IT and business leaders do not act quickly to make staffing and organizational decisions, integrate systems and provide training. These CIO Magazine "Agile 100" honourees describe what it takes to accomplish agile acquisitions.

[17] Lessons in Agility from Internet-Based Development, Scott Ambler http://www.agilealliance.orgiarticles/articles/LessonsInAgility.pdf Accessed March 2005

Describes two Internet start-ups that adopted effective and efficient modeling and documentation practices, one taking a communal, team-based approach and the other a chief-programmer approach.

[18] The Supple Supply Chain, Tracy Mayor, 2004 http://www.cio.com/archive/081504/supply.html Accessed March 2005

Agile companies can adjust their operations to cope with unexpected events, whether it's a surge in new business or a backlog in the warehouse. Four CIO Magazine "Agile 100" honourees tell how IT makes them and their trading partners flexible.

[19] Patterns of Productive Software Organizations, Neil Harrison, James Coplien, 1996 http://www.agilealliance.org/articles/articles/PatternsOfProductiveSoftwar eOrganizations.pdf Accessed March 2005

This paper attempts to isolate important factors that contribute to software productivity. Patterns of organization and process that are characteristic of highly productive software projects are uncovered with visual and quantitative data. Also presented are guiding principles for dramatically increasing productivity.

[20] Extreme Programming by Example, Moacir Pedroso, Marcos Visoli, Joao Antunes, F. G., 2002

99


Continuing Kent Beck's metaphor o f XP as driving a car, the authors argue that the tester is the navigator and plays an important role in XP projects.

[16] Come Together Right Now, Kim Girard, 2004http://www.cio.com/archive/081504/merge.html Accessed March 2005

The benefits of a merger or acquisition can be lost if IT and business leaders do not act quickly to make staffing and organizational decisions, integrate systems and provide training. These CIO Magazine "Agile 100" honourees describe what it takes to accomplish agile acquisitions.

[17] Lessons in Agility from Internet-Based Development, Scott Amblerhttp://www.agilealliance.org/articles/articles/LessonsInAgilitv.ndf Accessed March 2005

Describes two Internet start-ups that adopted effective and efficient modeling and documentation practices, one taking a communal, team- based approach and the other a chief-programmer approach.

[18] The Supple Supply Chain, Tracy Mayor, 2004http://www.cio.com/archive/081504/supply.html Accessed March 2005

Agile companies can adjust their operations to cope with unexpected events, whether it's a surge in new business or a backlog in the warehouse. Four CIO Magazine "Agile 100" honourees tell how IT makes them and their trading partners flexible.

[19] Patterns of Productive Software Organizations, Neil Harrison, JamesCoplien, 1996http://www.agilealliance.org/articles/articles/PattemsOfProductiveSoftwar eOrganizations.pdf Accessed March 2005

This paper attempts to isolate important factors that contribute to software productivity. Patterns of organization and process that are characteristic of highly productive software projects are uncovered with visual and quantitative data. Also presented are guiding principles for dramatically increasing productivity.

[20] Extreme Programming by Example, Moacir Pedroso, Marcos Visoli,Joao Antunes, F. G., 2002

99


http://www.cio.com/archive/081504/merge.html

http://www.agilealliance.org/articles/articles/LessonsInAgilitv.ndf

http://www.cio.com/archive/081504/supply.html

http://www.agilealliance.org/articles/articles/PattemsOfProductiveSoftwar

http://www.agilealliance.org/articles/articles/Pedroso-Marcos--ExtremeProgrammingbyExample.pdf Accessed March 2005

Describes how XP was adapted for the successful development of a risky project. As a side effect, XP spread throughout the company.

The abstracts of the following articles are provided:

[21] Case Study: IRS Business System Modernization Process Improvement, L. Anderson, M. Fisher, and J. Gross, 2004 http://www.sei.cmu.edu/publications/documents/04.reports/04tr002.html Accessed March 2005

Recognizing problems with its legacy information technology systems, the U.S. Internal Revenue Service (IRS) embarked on a modernization effort over a decade ago, with limited success. In 1998, the IRS embarked on a new approach and awarded a contract to the PRIME Alliance to assume the development and integration role for the systems that were expected to modernize the IRS way of doing business. The IRS Business Systems Modernization Office (BSMO) was established to manage this program.

However, as with past modernization efforts, the BSMO had difficulties in developing the discipline to efficiently and effectively manage the acquisition aspects of this modernization effort. The General Accounting Office suggested that the BSM program in still this discipline by improving a number of management process areas, including its acquisition processes. This suggestion included application of the Software Acquisition Capability Maturity Model® (SA-CMM®) as guidance on how to improve. This paper provides an overview of applying the SA-CMM to the IRS modernization effort to establish and implement more effective acquisition management processes and practices. The experience includes the process improvement planning stages of first selecting the SA-CMM as a framework for process improvement, through to completion of the final assessment where a maturity level 2 rating was achieved against the SA-CMM.

[22] CMMI Acquisition Module (CMMI-AM) Version 1.0 Bernard, T.; Gallagher, B.; Bate, R.; Wilson, H., 2004

http://www.sei.cmu.edu/publications/documents/04.reports/04tr00 Lhtml Accessed March 2005

Building on relevant best practices extracted from the Capability Maturity Model Integration (CMMI) Framework, this report defines effective and

100


http://www.agilealliance.org/articles/articles/Pedroso-Marcos-- ExtremeProgrammingbyExample.pdf Accessed March 2005

Describes how XP was adapted for the successful development o f a risky project. As a side effect, XP spread throughout the company.

The abstracts o f the following articles are provided:

[21] Case Study: IRS Business System M odernization ProcessImprovem ent, L. Anderson, M. Fisher, and J. Gross, 2004 http://www.sei.cmu.edu/publications/documents/04.reports/04tr002.html Accessed March 2005

Recognizing problems with its legacy information technology systems, the U.S. Internal Revenue Service (IRS) embarked on a modernization effort over a decade ago, with limited success. In 1998, the IRS embarked on a new approach and awarded a contract to the PRIME Alliance to assume the development and integration role for the systems that were expected to modernize the IRS way of doing business. The IRS Business Systems Modernization Office (BSMO) was established to manage this program.

However, as with past modernization efforts, the BSMO had difficulties indeveloping the discipline to efficiently and effectively manage the acquisition aspects of this modernization effort. The General Accounting Office suggested that the BSM program in still this discipline by improving a number of management process areas, including its acquisition processes. This suggestion included application o f the Software Acquisition Capability Maturity Model® (SA-CMM®) as guidance on how to improve. This paper provides an overview of applying the SA-CMM to the IRS modernization effort to establish and implement more effective acquisition management processes and practices. The experience includes the process improvement planning stages o f first selecting the SA-CMM as a framework for process improvement, through to completion of the final assessment where a maturity level 2 rating was achieved against the SA-CMM.

[22] CMMI Acquisition Module (CMMI-AM) Version 1.0 Bernard, T.;Gallagher, B.; Bate, R.; Wilson, H., 2004

http ://www. sei .emu. edu/publications/documents/04.reports/04tr001 .html Accessed March 2005

Building on relevant best practices extracted from the Capability Maturity Model Integration (CMMI) Framework, this report defines effective and

100


http://www.agilealliance.org/articles/articles/Pedroso-Marcos--

http://www.sei.cmu.edu/publications/documents/04.reports/04tr002.html

efficient practices for government acquisition organizations. Acquisition best practices are focused inside the acquisition organization to ensure the acquisition is conducted effectively, and outside the acquisition organization as it conducts project monitoring and control of its suppliers. These best practices provide a foundation for acquisition process discipline and rigor that enables product and service development to be repeatedly executed with high levels of ultimate acquisition success. This report contains the acquisition practices that should be performed by government acquisition organizations acquiring systems and/or services. These practices, however, can also be used by non-government organizations to improve their acquisition practices. This report does not contain prescribed implementation approaches for achieving acquisition best practices. Instead, the proven content of the CMMI Framework is used as a base and amplifications specific to the acquisition process are added. Questions related to CMMI process areas are provided in the appendix to help managers and executives understand the acquisition organization's documented acquisition practices and the consistent application of those practices. Descriptions of implementation details can be found in the source documents listed in the bibliography.

[23] Integrated Approach to Software Process Improvement at Wipro Technologies: V. Subramanyam, Sambuddha Deb, Priya Krishnaswamy, Rituparna Ghosh, 2004

http://www.sei.cmu.edu/publications/documents/04.reports/04tr006.html Accessed March 2005

This report captures the details of Wipro's quality journey through continuous process improvement. This journey towards excellence has led to the prestigious Institute of Electrical and Electronics Engineers (IEEE) Computer Society Award for Software Process Achievement in 2003. This award is for achieving high soft-ware process capability and establishing a basis for moving to a broad improvement program that concerns people and products, rather than just the processes. This report details the process improvement activities and the evolution of processes and systems over a period of time. Wipro's integrated approach to quality-veloci-Q-is detailed with an emphasis on the process improvement initiatives, supporting technology, and people capability that bring positive business results and enable Wipro to meet its customers' ever-increasing demands. The challenges and benefits accrued from Wipro's total quality approach are outlined, including the quantitative results of measuring the impact of Wipro's process improvement model.

[24] CMMI® Interpretive Guidance Project: What We Learned Chrissis, M.; Konrad, M.; Shrum, S.; Smith, K.; & Wemyss, G., 2004

101


efficient practices for government acquisition organizations. Acquisition best practices are focused inside the acquisition organization to ensure the acquisition is conducted effectively, and outside the acquisition organization as it conducts project monitoring and control of its suppliers. These best practices provide a foundation for acquisition process discipline and rigor that enables product and service development to be repeatedly executed with high levels o f ultimate acquisition success. This report contains the acquisition practices that should be performed by government acquisition organizations acquiring systems and/or services. These practices, however, can also be used by non-government organizations to improve their acquisition practices. This report does not contain prescribed implementation approaches for achieving acquisition best practices. Instead, the proven content o f the CMMI Framework is used as a base and amplifications specific to the acquisition process are added. Questions related to CMMI process areas are provided in the appendix to help managers and executives understand the acquisition organization's documented acquisition practices and the consistent application of those practices. Descriptions of implementation details can be found in the source documents listed in the bibliography.

[23] Integrated Approach to Software Process Improvement at WiproTechnologies: V. Subramanyam, Sambuddha Deb, Priya Krishnaswamy, Ritupama Ghosh, 2004

http://www.sei.cmu.edu/publications/documents/04.reports/04tr006.html Accessed March 2005

This report captures the details of Wipro's quality journey through continuous process improvement. This journey towards excellence has led to the prestigious Institute of Electrical and Electronics Engineers (IEEE) Computer Society Award for Software Process Achievement in 2003. This award is for achieving high soft-ware process capability and establishing a basis for moving to a broad improvement program that concerns people and products, rather than just the processes. This report details the process improvement activities and the evolution of processes and systems over a period of time. Wipro's integrated approach to quality-veloci-Q-is detailed with an emphasis on the process improvement initiatives, supporting technology, and people capability that bring positive business results and enable Wipro to meet its customers' ever-increasing demands. The challenges and benefits accrued from Wipro's total quality approach are outlined, including the quantitative results o f measuring the impact of Wipro's process improvement model.

[24] CMMI® Interpretive Guidance Project: What We Learned Chrissis, M.;Konrad, M.; Shrum, S.; Smith, K.; & Wemyss, G., 2004

101


http://www.sei.cmu.edu/publications/documents/04.reports/04tr006.html

http://www.sei.cmu.edu/publications/documents/04.reports/04sr008.html Accessed March 2005

This report summarizes the results of the Capability Maturity Model Integration (CMMI) Interpretive Guidance Project. It summarizes and analyzes the 7500 comments received regarding CMMI adoption that were reported by CMMI users and potential users. It also describes the actions being taken by the Software Engineering Institute (SEI) to address the issues identified by Interpretive Guidance Project participants. Although the initial goal of the project was to develop interpretive guidance, after data gathering and analysis the team realized that most respondents' input did not require interpretive guidance. Based on a relatively small number of comments, interpretive guidance was planned, including papers, frequently asked questions (FAQs), and new CMMI courses. However, participant comments clearly showed CMMI interpretive guidance to be less of an adoption issue than suspected. Some comments covered issues already being addressed as part of SEI activities, including the development of Standard CMMI Appraisal Method for Process Improvement (SCAMPIsm) Class B and C methods, the collection of cost and benefit information, and the creation and improvement of CMMI training courses. The majority (approximately 80%) of the actionable comments received were best handled as change requests to help guide the ongoing improvement of the CMMI Product Suite.

[25] CMMIR1-Based Process Improvement and Schedule Deviation in Software Maintenance, Jung, H. & Goldenson, D. , 2003 http ://www.sei . cmu. edu/pub lications/docum ents/03 .reports/03 tn015 .htm I Accessed March 2005

The objective of this study is to evaluate the predictive validity of the Capability Maturity Model ERI(CMMERI) for Software (SW-CMM) as applied to software maintenance. The SW-CMM is intended to apply to both software development and maintenance. A basic premise (hypothesis) of the SW-CMM is that improving process maturity will result in better project performance and product quality. The extent to which that hypothesis is supported empirically is called a test of its predictive validity. No previous evaluation exists of the predictive validity of the SW-CMM in a maintenance context. The extent to which schedule estimates differ from reality is one important measure of project performance. But is higher maturity in fact correlated with a reduction in schedule deviation? Data from 752 maintenance projects drawn from 441 SW-CMM assessments are analyzed using a zero inflated Poisson (ZIP) regression model, and the results are validated using a bootstrap estimation method. Projects from higher maturity organizations typically

102



This report summarizes the results o f the Capability Maturity Model Integration (CMMI) Interpretive Guidance Project. It summarizes and analyzes the 7500 comments received regarding CMMI adoption that were reported by CMMI users and potential users. It also describes the actions being taken by the Software Engineering Institute (SEI) to address the issues identified by Interpretive Guidance Project participants. Although the initial goal of the project was to develop interpretive guidance, after data gathering and analysis the team realized that most respondents' input did not require interpretive guidance. Based on a relatively small number o f comments, interpretive guidance was planned, including papers, frequently asked questions (FAQs), and new CMMI courses. However, participant comments clearly showed CMMI interpretive guidance to be less of an adoption issue than suspected. Some comments covered issues already being addressed as part of SEI activities, including the development of Standard CMMI Appraisal Method for Process Improvement (SCAMPIsm) Class B and C methods, the collection o f cost and benefit information, and the creation and improvement o f CMMI training courses. The majority (approximately 80%) of the actionable comments received were best handled as change requests to help guide the ongoing improvement of the CMMI Product Suite.

[25] CMM,R,-Based Process Improvement and Schedule Deviation inSoftware Maintenance, Jung, H. & Goldenson, D . , 2003 http://www.sei.cmu.edu/publications/documents/03.reports/03tn015.html Accessed March 2005

The objective of this study is to evaluate the predictive validity o f the Capability Maturity Model [R](CMM[R]) for Software (SW-CMM) as applied to software maintenance. The SW-CMM is intended to apply to both software development and maintenance. A basic premise(hypothesis) o f the SW-CMM is that improving process maturity willresult in better project performance and product quality. The extent to which that hypothesis is supported empirically is called a test o f its predictive validity. No previous evaluation exists of the predictive validity o f the SW-CMM in a maintenance context. The extent to which schedule estimates differ from reality is one important measure of project performance. But is higher maturity in fact correlated with a reduction in schedule deviation? Data from 752 maintenance projects drawn from 441 SW-CMM assessments are analyzed using a zero inflated Poisson (ZIP) regression model, and the results are validated using a bootstrap estimation method. Projects from higher maturity organizations typically

102


http://www.sei.cmu.edu/publications/documents/04.reports/04sr008.html

http://www.sei.cmu.edu/publications/documents/03.reports/03tn015.html

report less schedule deviation than those from organizations assessed at lower maturity levels.

[26] Interpreting Capability Maturity Modef ill Integration (CMMIIRI) for Service Organizations-a Systems Engineering and Integration Services Example, Herndon, M.; Moore, R.; Phillips, M.; Walker, J.; & West, L. 2003 http ://www. sei. cmu. edu/pub lications/docum ents/03 .reports/03tn005 .html Accessed March 2005

Capability Maturity Modellin Integration (CMMI[R]) provides a framework for improving the processes organizations use to develop, deliver, and maintain products and services. This technical note presents one organization's interpretation of CMMI best practices for organizations that primarily provide services. Service organizations can use this example interpretation of CMMI practices to inform management and staff about how CMMI practices apply to their work. The interpretation will also help appraisal team members ensure that implemented practices provide the business value necessary to satisfy the goals for quality process improvement that are stated in the CMMI models.

[27] Demonstrating the Impact and Benefits of CMMI: An Update and Preliminary Results, Goldenson, D. & Gibson, D. , 2003

http ://www.sei.cmu. edu/publications/documents/03.reports/03sr009.html Accessed March 2005

There is a widespread demand for evidence about the impact and benefits of process improvement based on Capability Maturity ModeltRi Integration (CMMIrRI) models. Much has been documented about the practice of CMMERI-based process improvement and its value for the development and maintenance of software and software-intensive systems; however, the existing information is sometimes outdated and there are increasing calls for evidence directly based on CMMI experience. This special report presents selected results from twelve case studies drawn from 11 organizations. While still limited, the case studies provide credible evidence that CMMI-based process improvement can help organizations achieve better project performance and produce higher quality products. The report also describes plans for gathering further evidence from organizations using CMMI models.

103


report less schedule deviation than those from organizations assessed at lower maturity levels.

[26] Interpreting Capability Maturity ModelIR| Integration (CMMI,R|) forService Organizations-a Systems Engineering and Integration Services Example, Herndon, M.; Moore, R.; Phillips, M.; Walker, J.; & West, L. 2003http://www.sei.cmu.edu/publications/documents/03.reports/03tn005.html Accessed March 2005

Capability Maturity Model[R] Integration (CMMI[R]) provides a framework for improving the processes organizations use to develop, deliver, and maintain products and services. This technical note presents one organization's interpretation of CMMI best practices for organizations that primarily provide services. Service organizations can use this example interpretation of CMMI practices to inform management and staff about how CMMI practices apply to their work. The interpretation will also help appraisal team members ensure that implemented practices provide the business value necessary to satisfy the goals for quality process improvement that are stated in the CMMI models.

[27] Demonstrating the Impact and Benefits of CMMI: An Update andPreliminary Results, Goldenson, D. & Gibson, D ., 2003


There is a widespread demand for evidence about the impact and benefits o f process improvement based on Capability Maturity Model[R] Integration (CM M I^) models. Much has been documented about the practice of CMM[R]-based process improvement and its value for the development and maintenance of software and software-intensive systems; however, the existing information is sometimes outdated and there are increasing calls for evidence directly based on CMMI experience. This special report presents selected results from twelve case studies drawn from 11 organizations. While still limited, the case studies provide credible evidence that CMMI-based process improvement can help organizations achieve better project performance and produce higher quality products. The report also describes plans for gathering further evidence from organizations using CMMI models.

103


http://www.sei.cmu.edu/publications/documents/03.reports/03tn005.html


[28] CMMI Interpretive Guidance Project: Preliminary Report, Chrissis, M.; Wemyss, G.; Goldenson, D.; Konrad, M.; Smith, K.; & Svolou, A. , 2003 http://www.sei.cmu.edu/publications/documents/03.reports/03sr007.html Accessed March 2005

The CMMI[RI (Capability Maturity Model[R1 Integration) Interpretive Guidance project was formed to help commercial software, information technology (IT), and information systems (IS) organizations adopt CMMI. Project members collected data to learn more about how CMMI is being accepted by these organizations. This report describes the data- collection activities and includes summaries of the data collected through August 2003. The project received both positive and negative comments that lead to some interesting and surprising observations. Overall, the positive comments greatly outnumbered the negative. Input provided by commercial software, IT, and IS organizations was similar to input from organizations from other disciplines. Organizations reported that CMMI is adequate for guiding their process improvement activities and that CMMI training courses and appraisal methods are suitable for their needs, although there are specific opportunities for improvement. Having two representations caused concern and confusion for some but was a benefit for others, so the project will investigate these comments further to see what can be done to address these concerns. The cost of CMMI is an issue that affected adoption decisions for some but not for others. Finally, return-on-investment information is usually helpful to organizations when making the business case to adopt CMMI.

[29] CMM®-Based Process Improvement and Schedule Deviation in Software Maintenance, Ho-Won Jung Dennis R. Goldenson, 2003 http://www.sei.cmu.edu/pub/documents/03.reports/pdf/03tn015.pdf Accessed March 2005

The objective of this stud is to evaluate the predictive validity of the Capability Maturity Model (CMM®) for Software (SW-CMM) as applied to software maintenance. The SW-CMM is intended to apply to both software development and maintenance. A basic premise (hypothesis) of the SW-CMM is that improving process maturity will result in better project performance and product quality. The extent to which that hypothesis is supported empirically is called a test of its predictive validity. No previous evaluation exists of the predictive validity of the SW-CMM in a maintenance context. The extent to which schedule estimates differ from reality is one important measure of project performance. But is higher maturity in fact correlated with a reduction in schedule deviation? Data from 752 maintenance projects drawn from 441 SW-CMM assessments are analyzed using a zero inflated Poisson (ZIP)

104


[28] CMMI Interpretive Guidance Project: Preliminary Report, Chrissis,M.; Wemyss, G.; Goldenson, D.; Konrad, M.; Smith, K.; & Svolou, A. , 2003http://www.sei.cmu.edu/publications/documents/03.reports/03sr007.html Accessed March 2005

The CMMI[R] (Capability Maturity Model[R] Integration) Interpretive Guidance project was formed to help commercial software, information technology (IT), and information systems (IS) organizations adopt CMMI. Project members collected data to learn more about how CMMI is being accepted by these organizations. This report describes the data- collection activities and includes summaries o f the data collected through August 2003. The project received both positive and negative comments that lead to some interesting and surprising observations. Overall, the positive comments greatly outnumbered the negative. Input provided by commercial software, IT, and IS organizations was similar to input from organizations from other disciplines. Organizations reported that CMMI is adequate for guiding their process improvement activities and that CMMI training courses and appraisal methods are suitable for their needs, although there are specific opportunities for improvement. Having two representations caused concern and confusion for some but was a benefit for others, so the project will investigate these comments further to see what can be done to address these concerns. The cost o f CMMI is an issue that affected adoption decisions for some but not for others. Finally, retum-on-investment information is usually helpful to organizations when making the business case to adopt CMMI.

[29] CMM®-Based Process Improvement and Schedule Deviation inSoftware Maintenance, Ho-Won Jung Dennis R. Goldenson, 2003 http://www.sei.cmu.edu/pub/documents/03.reports/pdf/03tn015.pdf Accessed March 2005

The objective o f this study is to evaluate the predictive validity of the Capability Maturity Model (CMM®) for Software (SW-CMM) as applied to software maintenance. The SW-CMM is intended to apply to both software development and maintenance. A basic premise (hypothesis) of the SW-CMM is that improving process maturity will result in better project performance and product quality. The extent to which that hypothesis is supported empirically is called a test of its predictive validity. No previous evaluation exists of the predictive validity of the SW-CMM in a maintenance context. The extent to which schedule estimates differ from reality is one important measure o f project performance. But is higher maturity in fact correlated with a reduction in schedule deviation? Data from 752 maintenance projects drawn from 441 SW-CMM assessments are analyzed using a zero inflated Poisson (ZIP)

104



http://www.sei.cmu.edu/pub/documents/03.reports/pdf/03tn015.pdf

regression model, and the results are validated using a bootstrap estimation method. Projects from higher maturity organizations typically report less schedule deviation than those from organizations assessed at lower maturity levels.

[30] Benefits of CMM-Based Software Process Improvement: Executive Summary of Initial Results, James Herbsleb, Anita Carleton, James Rozum, Jane Siegel, David Zubrow 1994 http://www.sei.cmu.edu/publications/documents/94.reports/ 94.sr.013.html Accessed March 2005

Data from 13 organizations were collected and analyzed to obtain information on the results of software process improvement efforts that were based on the capability maturity model (CMM). We report the cost and business value of improvement efforts, as well as the yearly improvement in productivity, early defect detection, time to market, and post-release defect reports. Case studies of improvement efforts and results in 5 organizations are summarized. We end with conclusions about the results of software process improvement (SPI) efforts.

105


regression model, and the results are validated using a bootstrap estimation method. Projects from higher maturity organizations typically report less schedule deviation than those from organizations assessed at lower maturity levels.

[30] Benefits of CMM-Based Software Process Improvement: ExecutiveSummary of Initial Results, James Herbsleb, Anita Carleton, James Rozum, Jane Siegel, David Zubrow 1994http://www.sei.cmu.edu/publications/documents/94.reports/ 94.sr.013.html Accessed March 2005

Data from 13 organizations were collected and analyzed to obtain information on the results o f software process improvement efforts that were based on the capability maturity model (CMM). We report the cost and business value of improvement efforts, as well as the yearly improvement in productivity, early defect detection, time to market, and post-release defect reports. Case studies o f improvement efforts and results in 5 organizations are summarized. We end with conclusions about the results o f software process improvement (SPI) efforts.

105


http://www.sei.cmu.edu/publications/documents/94.reports/

Appendix D

Analysis, questions and weights of twelve factors and their sub-factors

LEGEND:

SQ: Sample question

Comfortably Agile 0% Somewhat Agile: 25% Mixed, Hybrid, Tailored: 50% Somewhat Plan-Driven: 75% Comfortably Plan-Driven: 100%

General weight scheme: 5: Decisive 4: Heavy 3: Moderate 2: Light 1: Very light 0: Non-issue (Can be ignored comfortably)


Primary Goals

Continuous delivery The need to see things as they are built SQ: Do you or your client need the working software delivered in phases? YES: Agile 0% No: Plan-Driven 100% May be: Hybrid / Tailored 50% Not Sure: Follow up question, Disregard

General weight: 3

SQ: Do you or your client need to physically see the progress of the SW? YES: Agile 0% No: Plan-Driven 100%

106


Appendix D

Analysis, questions and weights of twelve factors and their sub-factors

LEGEND:

SQ: Sample question

Comfortably Agile 0%Somewhat Agile: 25%Mixed, Hybrid, Tailored: 50%Somewhat Plan-Driven: 75%Comfortably Plan-Driven: 100%

General weight scheme:5: Decisive 4: Heavy 3: Moderate 2: Light 1: Very light0: Non-issue (Can be ignored comfortably)


Primary Goals

Continuous deliveryThe need to see things as they are builtSQ: Do you or your client need the working software delivered in phases?YES: Agile 0%No: Plan-Driven 100%May be: Hybrid / Tailored 50%Not Sure: Follow up question, Disregard

General weight: 3

SQ: Do you or your client need to physically see the progress o f the SW?YES: Agile 0%No: Plan-Driven 100%

106


May be: Hybrid / Tailored 50% Not Sure: Follow up question, Disregard

General weight: 3

Responsiveness The flexibility to changing environment, requirements, marketplace, technology SQ: How responsive your organization is or wants to be? Very much: Agile 0% Fairly: Agile 25% Somewhat: Tailored / Hybrid 50% Not much: 75% Plan-Driven Not at all: 100% Plan-Driven

General weight 3

Return-on Investment Analysis SQ: Does a Return-on Investment document exist? YES: 50% NO: Agile 0%

General weight 3

SQ: The importance of following such analysis (return-on investment)? Very Important: Plan-driven 100 % Important: Plan-Driven 75% Not Very Important: Agile 25% Not Important / NA: Agile 0% Not sure: Follow up question / discard

General weight 3

Quick Build: Whether there is a need for a quick fix or quick delivery of product SQ: Do your products have to be delivered quickly? YES: Agile 0% Somewhat: Hybrid /Tailored 50% NO: Plan-Driven 100% Not sure: Follow up question / discard

General weight 3

Short Term optimization: Whether there is a need for a quick fix of the process SQ: Are you looking for a short-term solution to improve your process? YES: Agile 0%

107


May be: Hybrid / Tailored 50%Not Sure: Follow up question, Disregard

General weight: 3

ResponsivenessThe flexibility to changing environment, requirements, marketplace, technology SQ: How responsive your organization is or wants to be?Very much: Agile 0%Fairly: Agile 25%Somewhat: Tailored / Hybrid 50%Not much: 75% Plan-Driven Not at all: 100% Plan-Driven

General weight 3

Return-on Investment AnalysisSQ: Does a Retum-on Investment document exist?YES: 50%NO: Agile 0%

General weight 3

SQ: The importance of following such analysis (retum-on investment)?Very Important: Plan-driven 100 %Important: Plan-Driven 75%Not Very Important: Agile 25%Not Important / NA: Agile 0%Not sure: Follow up question / discard

General weight 3

Quick Build:Whether there is a need for a quick fix or quick delivery of product SQ: Do your products have to be delivered quickly?YES: Agile 0%Somewhat: Hybrid /Tailored 50%NO: Plan-Driven 100%Not sure: Follow up question / discard

General weight 3

Short Term optimization:Whether there is a need for a quick fix of the processSQ: Are you looking for a short-term solution to improve your process?YES: Agile 0%

107


NO: 50% Not sure: Follow up question / discard

General Weight 2

Predictability: Prediction is based on the measurements of prior standard activities SQ: How foreseeable the process and the end product shall be (Time lines, Product quality) Very much: Plan-driven 100% Fairly: Plan-Driven 75% Somewhat: Tailored / Hybrid 50% Not much: Agile 25% No need: agile 0% Not sure: Follow up question / discard

General weight 3

SQ: Do your organization want to have Work Products, Verification / Validation strategies? YES: Plan-Driven 100% NO: Agile 0% Maybe: 50% Not sure: Follow up question / discard

General weight 3

Stability: The existence of unprecedented projects with high rate of unforeseeable change SQ: Do your projects have high rate of unforeseeable change? YES: Agile 0% To Some Extent: Agile 25% Somewhat: Tailored / Hybrid 50% Not very often: Plan-Driven 75% NO / NA: Plan-Driven 100% Not sure: Follow up question / discard

General Weight: 4

High Assurance: Safety-critical Projects SQ: How critical is the quality / reliability of the SW? (E.g. Game / MRI SW) Highly critical: Plan-Driven 100% * Decisive Moderately critical: Plan-Driven 75% Somewhat Critical: 50% Not very critical: Agile 25%

108


NO: 50%Not sure: Follow up question / discard

General Weight 2

Predictability:Prediction is based on the measurements o f prior standard activitiesSQ: How foreseeable the process and the end product shall be (Time lines, Productquality)Very much: Plan-driven 100%Fairly: Plan-Driven 75%Somewhat: Tailored / Hybrid 50%Not much: Agile 25%No need: agile 0%Not sure: Follow up question / discard

General weight 3

SQ: Do your organization want to have Work Products, Verification / Validation strategies?YES: Plan-Driven 100%NO: Agile 0%Maybe: 50%Not sure: Follow up question / discard

General weight 3

Stability:The existence o f unprecedented projects with high rate o f unforeseeable change SQ: Do your projects have high rate of unforeseeable change?YES: Agile 0%To Some Extent: Agile 25%Somewhat: Tailored / Hybrid 50%Not very often: Plan-Driven 75%NO / NA: Plan-Driven 100%Not sure: Follow up question / discard

General Weight: 4

High Assurance:Safety-critical ProjectsSQ: How critical is the quality / reliability o f the SW? (E.g. Game / MRI SW) Highly critical: Plan-Driven 100% * DecisiveModerately critical: Plan-Driven 75%Somewhat Critical: 50%Not very critical: Agile 25%

108


Not critical at all: Agile 0% Not sure: Follow up Question

General Weight: 5 for High critical projects make this factor final to be Plan-Driven

Size

Number: The Number of employees working on a given project including managers, support, clients on site, engineers, developer, programmers/analysts 0-10 0 Agile * Decisive 10-40 Agile25% 40-150 Plan-Driven75% 150+ Plan-Driven 100% *Decisive

General Weight: 5 * Decisive

Scaling up: SQ: Is your organization going to grow (number of employees) in near future? By how many? (Including your current) 0-11 0 Agile * Decisive 10-41 Agile25% 40-151 Plan-Driven75% 150+ Plan-Driven 100% *Decisive


Note: Scaling up is impossible with agile

Environment

Requirements Rate of Change: Whether the requirements change or new requirements emerge SQ: Are the requirements determinable in advance? YES: Plan-Driven 100% No: Agile 0% Somewhat: 50% Not Sure: Follow up questions/ discard

General weight: 4

109


Not critical at all: Agile 0%Not sure: Follow up Question

General Weight: 5 for High critical projects make this factor final to be Plan-Driven

Size

Number:The Number o f employees working on a given project including managers, support, clients on site, engineers, developer, programmers/analysts 0-10 0 Agile * Decisive10-40 Agile25%40-150 Plan-Dri ven7 5 %150+ Plan-Driven 100% *Decisive


Scaling up:SQ: Is your organization going to grow (number of employees) in near future? By how many? (Including your current)0-11 0 Agile * Decisive10-41 Agile25%40-151 Plan-Driven7 5 %150+ Plan-Driven 100% *Decisive


Note: Scaling up is impossible with agile

Environment

Requirements Rate of Change:Whether the requirements change or new requirements emerge SQ: Are the requirements determinable in advance?YES: Plan-Driven 100%No: Agile 0%Somewhat: 50%Not Sure: Follow up questions/ discard

General weight: 4

109


SQ: What percentage is the Requirements-Change per month? 1% and less: Plan-Driven 100% 1-5%: Plan-Driven 75% 5-10%: Hybrid / Tailored 50% 10-30%: Agile 25% 30%+: Agile 0% Not Sure: Follow up questions/ discard

General weight: 4

Organizational Concerns: Product line, Design, architecture, systems engineering, human factors, outsourcing, distributed development, far-off users SQ: Is your organization concerned / involved in the above-mentioned concepts? Yes Very much: Plan-driven 100% *Decisive To some degree: Plan-Driven 75% Somewhat: Tailored /Hybrid 50% Not really: Agile 25% Not at all / NA: Agile 0%

General Weight: 5 for YES make this factor final to be Plan-Driven

Stovepipes: SQ: Do several independently evolved applications have to subsequently be closely integrated? (Rephrase the question of course) YES: Plan-Driven 100% *Decisive No: Agile 0% Not sure: Follow up question / discard

General weight 5

Information Sclerosis: SQ: Do temporary procedural standards harden into unchangeable constraints on system development and cause unnecessary rework? YES: Plan-Driven 100% No: Agile 0% Not sure: Follow up question / discard

General weight 3

Bridging Situations: SQ: Does new software incrementally replace the old one? YES: Plan-Driven 100% No: Agile 0% Not sure: Follow up question / discard

110


SQ: What percentage is the Requirements-Change per month?1% and less: Plan-Driven 100%1-5%: Plan-Driven 75%5-10%: Hybrid / Tailored 50%10-30%: Agile 25%30%+: Agile 0%Not Sure: Follow up questions/ discard

General weight: 4

Organizational Concerns:Product line, Design, architecture, systems engineering, human factors, outsourcing, distributed development, far-off usersSQ: Is your organization concerned / involved in the above-mentioned concepts?Yes Very much: Plan-driven 100% *Decisive To some degree: Plan-Driven 75%Somewhat: Tailored /Hybrid 50%Not really: Agile 25%Not at all / NA: Agile 0%


Stovepipes:SQ: Do several independently evolved applications have to subsequently be closely integrated? (Rephrase the question of course)YES: Plan-Driven 100% *Decisive No: Agile 0%Not sure: Follow up question / discard

General weight 5

Information Sclerosis:SQ: Do temporary procedural standards harden into unchangeable constraints on system development and cause unnecessary rework?YES: Plan-Driven 100%No: Agile 0%Not sure: Follow up question / discard

General weight 3

Bridging Situations:SQ: Does new software incrementally replace the old one?YES: Plan-Driven 100%No: Agile 0%Not sure: Follow up question / discard

110


General weight 3

Monolithic Requirements: SQ: Can the product be delivered in modules? YES: Agile 0% No: Plan-Driven 100%*Decisive Not sure: follow up question / discard

General Weight: 5 for Yes make this factor final to be Plan-Driven

Continuity Requirements: Does the familiarity with the system across a large, mission-critical user base have to be maintained? (Rephrase the question of course) YES: Plan-Driven 100% *Decisive No: Agile 0% Not sure: Follow up question / discard


Process Improvement: The idea that the processes of the organization shall improve once the model is placed SQ: How Important is it for you to continuously improve the processes of your organization? Very much: Plan-Driven 100% Somewhat: 50% Not very much: Agile 25% Not at all, N/A: Agile 0% Not sure: Follow up question / discard

General weight 3


Customer relations

Collocated Customer: The need for the presence of an on-site customer SQ: Are your customers willing / able to be on-site and devote themselves to the development of the project? YES: Agile 25% No: Plan-Driven 75% May be: Hybrid / Tailored 50% Not Sure: Follow up question, Disregard General Weight: 4

111


General weight 3

Monolithic Requirements:SQ: Can the product be delivered in modules?YES: Agile 0%No: Plan-Driven 100%*Decisive Not sure: follow up question / discard

General Weight: 5 for Yes make this factor final to be Plan-Driven

Continuity Requirements:Does the familiarity with the system across a large, mission-critical user base have to be maintained? (Rephrase the question of course)YES: Plan-Driven 100% *Decisive No: Agile 0%Not sure: Follow up question / discard


Process Improvement:The idea that the processes o f the organization shall improve once the model is placed SQ: How Important is it for you to continuously improve the processes of your organization?Very much: Plan-Driven 100%Somewhat: 50%Not very much: Agile 25%Not at all, N/A: Agile 0%Not sure: Follow up question / discard

General weight 3


Customer relations

Collocated Customer:The need for the presence o f an on-site customerSQ: Are your customers willing / able to be on-site and devote themselves to the development o f the project?YES: Agile 25%No: Plan-Driven 75%May be: Hybrid / Tailored 50%Not Sure: Follow up question, Disregard General Weight: 4

111


SQ: Are your Customers knowledgeable enough to make project related decisions? YES: Agile 75% No: Plan-Driven 0% May be: Hybrid / Tailored 50% Not Sure: Follow up question, Disregard

General weight: 4

Contract Specification: Whether the organization is contract base SQ: Does your organization depend on formalised contract specification? SQ: Is the contract specification a source of trust (or lack of it a source of stress) between your organization and your customers? YES: Plan-Driven 100% NO: Agile 25% May be / Not sure: Follow up question, disregard

General Weight: 3

Process Maturity: SQ: Has your customers asked you about the maturity of your processes? (i.e. Do they except your organisation's processes be compatible with some well established standards such as CMM,CMMI?) YES: Plan-Driven 100% No: Agile 25%

General weight: 1


Tacit knowledge vs. explicit documented knowledge SQ: Does your organisation thrive on the tacit knowledge of its developers? YES: Agile 0% No: Plan Driven 100% A mix of both: Tailored/hybrid 50% Not sure: follow up question, disregard

General weight: 3

Pair programming: SQ: Does the interpersonal skill level of your developers allow the implementation of the "pair programming" idea? YES: Agile 0% NO: Plan-Driven 100% Mix: Tailored / Hybrid 50% Not sure: Follow up question, disregard

112


SQ: Are your Customers knowledgeable enough to make project related decisions?YES: Agile 75%No: Plan-Driven 0%May be: Hybrid / Tailored 50%Not Sure: Follow up question, Disregard

General weight: 4

Contract Specification:Whether the organization is contract baseSQ: Does your organization depend on formalised contract specification?SQ: Is the contract specification a source of trust (or lack of it a source o f stress) between your organization and your customers?YES: Plan-Driven 100%NO: Agile 25%May be / Not sure: Follow up question, disregard

General Weight: 3

Process Maturity:SQ: Has your customers asked you about the maturity of your processes? (i.e. Do they except your organisation’s processes be compatible with some well established standards such as CMM,CMMI?)YES: Plan-Driven 100%No: Agile 25%

General weight: 1


Tacit knowledge vs. explicit documented knowledgeSQ: Does your organisation thrive on the tacit knowledge of its developers?YES: Agile 0%No: Plan Driven 100%A mix of both: Tailored/hybrid 50%Not sure: follow up question, disregard

General weight: 3

Pair programming:SQ: Does the interpersonal skill level of your developers allow the implementation of the “pair programming” idea?YES: Agile 0%NO: Plan-Driven 100%Mix: Tailored / Hybrid 50%Not sure: Follow up question, disregard

112


General Weight: 3

Meetings SQ: Are you willing to specify a time slot everyday for a short informal meeting? (Shake hands meeting) YES: Agile 0% NO: Plan-driven 75% May be: Tailored / Hybrid 50% Not Sure: Disregard

General weight: 2

Shared code Ownership SQ: Is it feasible in your company to share the produced code across the organisation? YES: Agile 0% NO: Plan-driven 100% May be: Tailored / Hybrid 50% Not Sure: Disregard

General weight: 2

On-Site Development: SQ: Are the projects built for local (in house) applications? Yes: Agile 25% No: Plan-Driven 75% Mix: Tailored / Hybrid 50% No answer: disregard

General Weight: 2

Developing tacit knowledge: SQ: Are you willing to devote 20% of your time and resources in developing the tacit knowledge of your developers through practices mentioned above? (It is noteworthy to mention that the larger the organisation and the projects are, the longer this will take and the more expensive) YES: Agile 0% NO: Plan-Driven 75% Maybe: 50% Not sure: Follow up question, disregard

General weight: 2

Developing explicit knowledge: SQ: Does your organisation rely heavily on schedules, milestones, procedures, requirement, architecture, and standards?

113


General Weight: 3

MeetingsSQ: Are you willing to specify a time slot everyday for a short informal meeting? (Shake hands meeting)YES: Agile 0%NO: Plan-driven 75%May be: Tailored / Hybrid 50%Not Sure: Disregard

General weight: 2

Shared code OwnershipSQ: Is it feasible in your company to share the produced code across the organisation? YES: Agile 0%NO: Plan-driven 100%May be: Tailored / Hybrid 50%Not Sure: Disregard

General weight: 2

On-Site Development:SQ: Are the projects built for local (in house) applications?Yes: Agile 25%No: Plan-Driven 75%Mix: Tailored / Hybrid 50%No answer: disregard

General Weight: 2

Developing tacit knowledge:SQ: Are you willing to devote 20% of your time and resources in developing the tacit knowledge of your developers through practices mentioned above? (It is noteworthy to mention that the larger the organisation and the projects are, the longer this will take and the more expensive)YES: Agile 0%NO: Plan-Driven 75%May be: 50%Not sure: Follow up question, disregard

General weight: 2

Developing explicit knowledge:SQ: Does your organisation rely heavily on schedules, milestones, procedures, requirement, architecture, and standards?

113


YES: Plan-Driven 100% No: Agile 0% Somewhat: tailored / hybrid 50% Do not know / should we? : Follow up questions, disregard

General weight: 2

SQ: Are accurate future projections by the means of preserving historical data important to your organisation? (Does your organisation rely on future forecasts) YES: Plan-driven 100% NO: Agile 25% May be: Tailored / hybrid 50% Not sure: follow up question, disregard

General weight: 2


Person-to-person communication: Does your organisation encourage / is willing to share the knowledge across the organisation by person-to-person interactions (i.e. pair programming, stand-up meetings...) YES: Agile 0% NO: Plan-Driven 100% Somewhat: 50% Not sure: Follow up questions, disregard

General weight: 2

Note: scalability is an issue in this regard for n persons there are n(n-10/2 different means of communication.

Note: the assumption that everybody's tacit knowledge is consistent across an organisation is highly risky.

Reports: SQ: Do you use progress reports, process descriptions...? i.e. one way communication) for passing knowledge across? Or you have no other choice but to do so? YES: Plan-Driven 100% NO: Agile 0% Somewhat: 50% Not sure: Follow up questions, disregard

General weight: 2

114


YES: Plan-Driven 100%No: Agile 0%Somewhat: tailored / hybrid 50%Do not know / should we? : Follow up questions, disregard

General weight: 2

SQ: Are accurate future projections by the means of preserving historical data important to your organisation? (Does your organisation rely on future forecasts)YES: Plan-driven 100%NO: Agile 25%Maybe: Tailored / hybrid 50%Not sure: follow up question, disregard

General weight: 2


Person-to-person communication:Does your organisation encourage / is willing to share the knowledge across the organisation by person-to-person interactions (i.e. pair programming, stand-up meetings...)YES: Agile 0%NO: Plan-Driven 100%Somewhat: 50%Not sure: Follow up questions, disregard

General weight: 2

Note: scalability is an issue in this regard for n persons there are n(n-10/2 different means of communication.

Note: the assumption that everybody’s tacit knowledge is consistent across an organisation is highly risky.

Reports:SQ: Do you use progress reports, process descriptions...? i.e. one way communication) for passing knowledge across? Or you have no other choice but to do so?YES: Plan-Driven 100%NO: Agile 0%Somewhat: 50%Not sure: Follow up questions, disregard

General weight: 2

114


Note: both types of knowledge (explicit / implicit) / communication (person-to-person, one-way) should exist in both methods to be successful

Note: less experienced developers in plan-driven environment tend to take up most of the time creating unnecessary documentation for a security blanket.


Requirements

Formal Vs. Informal SQ: Are your requirements formalised (specific)? YES: Plan-Driven 100% No: Agile 0% Somewhat: Tailored / Hybrid 50% Not sure: Follow up questions, disregard

General weight: 4

Prioritized Requirements SQ: Do your customers need to prioritize the requirements? YES: Agile 0% NO: Plan-Driven 100% Maybe: Follow up questions, disregard

General weight: 3

Adjustability SQ: Do you prefer to negotiate the contents of the requirements on an on going basis (e.g. once a month) rather than having an established set of requirements? YES: Agile 0% NO: Plan-Driven 100% Somewhat: Tailored / hybrid 50% Not sure: Follow up questions, disregard

General weight: 4

Quality, Risk Management SQ: Does your organisation find the presence of the following concepts important? (Risk management, Quality handling, reliability, throughput, real-time deadline satisfaction, scalability) VERY MUCH: Plan-Driven 100% YES: Plan-Driven 75%

115


Note: both types o f knowledge (explicit / implicit) / communication (person-to-person, one-way) should exist in both methods to be successful

Note: less experienced developers in plan-driven environment tend to take up most of the time creating unnecessary documentation for a security blanket.


Requirements

Formal Vs. InformalSQ: Are your requirements formalised (specific)?YES: Plan-Driven 100%No: Agile 0%Somewhat: Tailored / Hybrid 50%Not sure: Follow up questions, disregard

General weight: 4

Prioritized RequirementsSQ: Do your customers need to prioritize the requirements?YES: Agile 0%NO: Plan-Driven 100%Maybe: Follow up questions, disregard

General weight: 3

AdjustabilitySQ: Do you prefer to negotiate the contents of the requirements on an on going basis (e.g. once a month) rather than having an established set o f requirements?YES: Agile 0%NO: Plan-Driven 100%Somewhat: Tailored / hybrid 50%Not sure: Follow up questions, disregard

General weight: 4

Quality, Risk ManagementSQ: Does your organisation find the presence of the following concepts important? (Risk management, Quality handling, reliability, throughput, real-time deadline satisfaction, scalability)VERY MUCH: Plan-Driven 100%YES: Plan-Driven 75%

115


Somewhat: Tailored / Hybrid 50% NO: Agile 25% NOT AT ALL: Agile 0% Not sure: Follow up questions, disregard

General Weight: 4

Development

Refactoring SQ: Do you posses the kind of developers that make the practice of refactoring (rework) inexpensive? YES: Agile 0% NO: Plan-Driven 75% Somewhat: 50% Not sure: Follow up questions, disregard

General weight : 2

New features (Scalability) SQ: Is there a possibility that the application (software) becomes larger (New features to be added)? YES: Plan-Driven 100% NO: Agile 0% Not Many: Plan-Driven 75% Somewhat: 50% Only a few: Agile 25% Not sure: Follow up questions, disregard

General weight: 4

Quality attributes SQ: Is there a possibility that the quality attributes such as reliability, throughput, and response time to increase during development? YES: Plan-Driven 100% NO: Agile 0% Somewhat: Tailored / Hybrid 50% Not sure: Follow up questions, disregard

General weight: 4

Multinational Operations SQ: Is your organisation's operation multinational that creates multilingual applications? YES: Plan-Driven 100% NO: Agile 0% Not sure: Follow up questions, disregard

116


Somewhat: Tailored / Hybrid 50%NO: Agile 25%NOT AT ALL: Agile 0%Not sure: Follow up questions, disregard

General Weight: 4

Development

RefactoringSQ: Do you posses the kind of developers that make the practice of refactoring (rework) inexpensive?YES: Agile 0%NO: Plan-Driven 75%Somewhat: 50%Not sure: Follow up questions, disregard

General w eight: 2

New features (Scalability)SQ: Is there a possibility that the application (software) becomes larger (New features to be added)?YES: Plan-Driven 100%NO: Agile 0%Not Many: Plan-Driven 75%Somewhat: 50%Only a few: Agile 25%Not sure: Follow up questions, disregard

General weight: 4

Quality attributesSQ: Is there a possibility that the quality attributes such as reliability, throughput, and response time to increase during development?YES: Plan-Driven 100%NO: Agile 0%Somewhat: Tailored / Hybrid 50%Not sure: Follow up questions, disregard

General weight: 4

Multinational OperationsSQ: Is your organisation’s operation multinational that creates multilingual applications? YES: Plan-Driven 100%NO: Agile 0%Not sure: Follow up questions, disregard

116


General weight 4

Program efficiency SQ: Do you prefer a very simple design for your applications and you are not concerned about extra features, quality attributes.. .(an efficient program at present and you are not going to need it YAGNI) YES: Agile 0% No: Plan-Driven 100% Somewhat: Tailored / Hybrid 50% Not sure: Follow up questions, disregard

General weight 3

Architecture based design SQ: Does your organisation conventionally (readily) use an architecture-based design to create software? YES: Plan-driven 75% NO: Agile 0% Somewhat: 50% Not sure: Follow up questions, disregard

General weight 2

Software reuse SQ: Is the concept of software reuse across product lines important to your operations? (I.e. you don't have a rapidly changing environment, which allows you to reuse your design architecture, product lines) YES: Plan-Driven 100% No: agile 0% Somewhat: 50% Not sure: Follow up questions, disregard

General weight 3

Note: Technically one could incorporate some agile practices in Plan-driven development

Testing

Continuous regression testing SQ: Do you want to be able to test the requirements? YES: Agile 0% NO: Plan-Driven 100% Not sure: Follow up questions, disregard

SQ: Do you want to avoid documentation for requirements?

117


General weight 4

Program efficiencySQ: Do you prefer a very simple design for your applications and you are not concerned about extra features, quality attributes.. .(an efficient program at present and you are not going to need it YAGNI)YES: Agile 0%No: Plan-Driven 100%Somewhat: Tailored / Hybrid 50%Not sure: Follow up questions, disregard

General weight 3

Architecture based designSQ: Does your organisation conventionally (readily) use an architecture-based design to create software?YES: Plan-driven 75%NO: Agile 0%Somewhat: 50%Not sure: Follow up questions, disregard

General weight 2

Software reuseSQ: Is the concept o f software reuse across product lines important to your operations? (I.e. you don’t have a rapidly changing environment, which allows you to reuse your design architecture, product lines)YES: Plan-Driven 100%No: agile 0%Somewhat: 50%Not sure: Follow up questions, disregard

General weight 3

Note: Technically one could incorporate some agile practices in Plan-driven development

Testing

Continuous regression testingSQ: Do you want to be able to test the requirements?YES: Agile 0%NO: Plan-Driven 100%Not sure: Follow up questions, disregard

SQ: Do you want to avoid documentation for requirements?

117


Customers Collaborative Responsive Authorized Committed Knowledgeable (CRACK) Customer SQ: Do you have customers that are Collaborative, Responsive, Authorized, Committed and Knowledgeable (CRACK)? YES: Agile 0% NO: Plan-Driven 100% Somewhat: 50% Not sure: Follow up questions, disregard

General Weight 4

Developers

Interpersonal Skills SQ: Do your developers possess high interpersonal skills (friendliness, communication skills)? YES: Agile 0% NO: Plan Driven 100% Somewhat: 50% Not sure: Follow up questions, disregard

Levels of software developers The percent presence of the following level personnel -1 may have technical skills, but unable or unwilling to collaborate or follow shared

methods 1B with training, able to perform procedural method steps (e.g. coding a simple

method, simple refactoring, following coding standards and CM procedures, running tests). With experience, can master some level 1A skills.

lA with training, able to perform discretionary method steps (e.g. sizing stories to fit increments, composing patterns, compound refactoring, complex COTS integration). With experience, can become level 2

2 Able to tailor a, method to fit a precedented new situation 3 Able to revise a method (break its rules) to fit an unprecedented new situation

SQ: Indicate the percentage of Level 1B and Level lA in your organisation 0-10: Agile 0% 10-20: Agile 25% 20-30: Hybrid / Tailored 50% 30-40: Plan-Driven 75% 40%+: Plan-Driven 100% Not sure: Follow up questions, disregard

General Weight 4

119


CustomersCollaborative Responsive Authorized Committed Knowledgeable (CRACK) CustomerSQ: Do you have customers that are Collaborative, Responsive, Authorized, Committed and Knowledgeable (CRACK)?YES: Agile 0%NO: Plan-Driven 100%Somewhat: 50%Not sure: Follow up questions, disregard

General Weight 4

Developers

Interpersonal SkillsSQ: Do your developers possess high interpersonal skills (friendliness, communication skills)?YES: Agile 0%NO: Plan Driven 100%Somewhat: 50%Not sure: Follow up questions, disregard

Levels of software developersThe percent presence of the following level personnel-1 may have technical skills, but unable or unwilling to collaborate or follow shared

methodsIB with training, able to perform procedural method steps (e.g. coding a simple

method, simple refactoring, following coding standards and CM procedures, running tests). With experience, can master some level 1A skills.

1A with training, able to perform discretionary method steps (e.g. sizing stories to fit increments, composing patterns, compound refactoring, complex COTS integration). With experience, can become level 2

2 Able to tailor a, method to fit a precedented new situation3 Able to revise a method (break its rules) to fit an unprecedented new situation

SQ: Indicate the percentage of Level IB and Level 1A in your organisation 0-10: Agile 0%10-20: Agile 25%20-30: Hybrid / Tailored 50%30-40: Plan-Driven 75%40%+: Plan-Driven 100%Not sure: Follow up questions, disregard

General Weight 4

119


SQ: Indicate the percentage of level 2 and 3 in your organisation 0-15: Plan-Driven 100% 15-20: Plan Driven 75% 20-25: Tailored / hybrid 50% 30-35: Agile 25% 35%+: Agile 0%

General weight 4

Culture

SQ: Do your people feel comfortable or empowered when they are trusted to do whatever work necessary? (Many degrees of freedom) YES: Agile 0% Somewhat: Tailored / Hybrid 50% NO: Plan-Driven 100% Not sure: Follow up questions, disregard

General weight 4

SQ: Do your people feel comfortable if their roles are defined by the means of policies and procedures? YES: Plan-Driven 100% Somewhat: Hybrid / Tailored 50% No: Agile 0%

General weight 4

120


SQ: Indicate the percentage o f level 2 and 3 in your organisation 0-15: Plan-Driven 100%15-20: Plan Driven 75%20-25: Tailored / hybrid 50%30-35: Agile 25%35%+: Agile 0%

General weight 4

Culture

SQ: Do your people feel comfortable or empowered when they are trusted to do whatever work necessary? (Many degrees o f freedom)YES: Agile 0%Somewhat: Tailored / Hybrid 50%NO: Plan-Driven 100%Not sure: Follow up questions, disregard

General weight 4

SQ: Do your people feel comfortable if their roles are defined by the means o f policies and procedures?YES: Plan-Driven 100%Somewhat: Hybrid / Tailored 50%No: Agile 0%

General weight 4

120


A DECISION SUPPORT SYSTEM FOR THE SOFTWARE PROCESS

Documents

Transcript of A DECISION SUPPORT SYSTEM FOR THE SOFTWARE PROCESS