Improving firmware development at Axis Communications
Lund University Faculty of Engineering Department of Computer Science
Authors: Sebastian Nordin
Christoffer Weidmar
Examiner: Dietmar Pfahl, LTH
Supervisors: Pontus Bergendahl, Axis Communications AB
Patrick Berling, Axis Communications AB
August 2011
III
Preface This master thesis was performed at the firmware department of Axis
Communications AB in Lund, in collaboration with the Department of Computer
Science at Lund Institute of Technology. The thesis marks the final examination
towards our Master of Science in Industrial Engineering and Management.
The thesis corresponds to 30 ECTS, which equals one semester of studies. The
thesis work was commenced in March and completed in August of 2011.
We found the work to be both challenging and inspirational; it has provided us
with knowledge that will surely be beneficial for us in our future professional
careers.
We would hereby like to extend our deepest gratitude to:
The firmware development department at Axis for their invaluable
feedback and cooperation in the progression of the thesis work. The
personnel at Axis Communications in Lund welcomed us with great
enthusiasm.
Pontus Bergendahl and Patrick Berling, our supervisors at Axis, for their
patience and support throughout the thesis. Without their input and
dedication this thesis would not have been possible.
Dietmar Pfahl, examiner and co-supervisor, for his observations,
comments and helpful guidance throughout the entirety of our thesis
work.
Martin Höst, Professor at Lund University Faculty of Engineering, for
providing us with important initial guidance and feedback.
Björn Carlsson, consultant at Jayway, for skillfully providing deepened
knowledge in areas related to the thesis.
Lund, August 2011
Sebastian Nordin and Christoffer Weidmar
IV
Abstrakt Titel: Analys och förbättring av firmware utvecklingen på Axis
Communications
Författare: Sebastian Nordin
Christoffer Weidmar
Handledare: Pontus Bergendahl, Manager på Axis Communications
Patrick Berling, Project Manager på Axis Communications
Examinator: Dietmar Pfahl, Docent, Institutionen för Datavetenskap på
Lunds Universitet, Lunds Tekniska Högskola
Problem: Hur kan firmware utvecklingen förbättras genom att använda
teorier inom mjukvaruutveckling och affärs-orienterade
metoder?
Syfte: Identifiera problem i befintliga processer och föreslå
förbättringar.
Begränsningar: Detta examensarbete fokuserar enbart på de aspekter av
programvaruteknik som relaterar till firmware utveckling.
Arbetsprocess: Kvalitativa data samlades in från intern dokumentation och
processobservationer. Litteraturstudier gav en teoretisk
bakgrund som användes för att framställa det
förbättringsförslag som presenteras i arbetet.
Slutsats: Denna avhandling understryker vikten av att beakta både
affärs-och mjukvarumässiga dimensioner inom moderna
teknikintensiva företag. Genom att kombinera teorier kan
organisationer utnyttja fördelarna från båda.
Arbetsuppdelning Christoffer Weidmar ansvarig för delar rörande Business
Process Management.
Sebastian Nordin ansvarig för delar rörande Software Product
Lines.
Nyckelbegrepp: Firmware utveckling, hantering av affärsprocesser,
produktlinjer, mjukvaruutveckling
V
Abstract Title: Improving Firmware development at Axis Communications
Authors: Sebastian Nordin
Christoffer Weidmar
Supervisors: Pontus Bergendahl, Manager at Axis Communications
Patrick Berling, Project Manager at Axis Communications
Examiner: Dietmar Pfahl, Associate Professor, Department of Computer
Science at Lund University, Faculty of Engineering
Problem: How can firmware development be improved by using
software engineering theories as well as business oriented
methodologies?
Purpose: Identify issues in current processes and suggest improvements.
Delimitations: This thesis focuses solely on the software engineering aspects
of firmware development.
Work process: Qualitative data was collected from internal documentation
and process observations. Literature studies provided a
theoretical background which was used as the source for the
improvement suggestion presented in the thesis.
Conclusion: This thesis underlines the importance of considering both
business and software engineering dimensions within modern,
technology intensive companies. By combining theories,
organizations may be able to utilize benefits from both
practices.
Division of work Christopher Weidmar is responsible for parts concerning
Business Process Management.
Sebastian Nordin is responsible for parts concerning Software
Product Lines.
Key Words: Firmware development, business process management,
software product lines, software engineering process.
VI
Table of contents 1 Introduction ...................................................................................................... 1
1.1 Background ............................................................................................... 1
1.2 Purpose and goals ..................................................................................... 2
1.3 Focus and limitations ................................................................................ 3
1.4 Report composition .................................................................................. 3
2 Methodology ..................................................................................................... 3
2.1 Induction and Deduction .......................................................................... 4
2.2 Scientific methodology ............................................................................. 5
2.3 Data format and sources ........................................................................... 5
2.4 Techniques for data acquisition ................................................................ 6
2.5 Validity, reliability and objectivity............................................................. 7
2.6 Redefining validity, reliability and objectivity into a qualitative context . 7
2.7 Trustworthiness and authenticity of this thesis........................................ 8
3 Theory ............................................................................................................. 10
3.1 Software Product Lines ........................................................................... 10
3.2 Business Process Management ............................................................... 22
3.3 Firmware and embedded software ........................................................ 30
3.4 Firmware Quality .................................................................................... 31
3.5 Software metrics and measurements ..................................................... 32
4 Case study, Axis Communications AB ............................................................. 35
4.1 Product Portfolio ..................................................................................... 35
4.2 Axis Firmware Development ................................................................... 35
4.3 Tollgates .................................................................................................. 38
4.4 Linux Firmware Platform ......................................................................... 42
4.5 Software Product Lines at Axis................................................................ 46
4.6 Global Information Tracker (GIT) ............................................................ 47
4.7 Trouble .................................................................................................... 48
VII
4.8 QACE ....................................................................................................... 48
4.9 Roles at Axis ............................................................................................ 49
5 Analysis ........................................................................................................... 52
5.1 Quality Assurance ................................................................................... 52
5.2 Software product lines ............................................................................ 52
5.3 Business Process Management ............................................................... 60
5.4 Packet distribution analysis .................................................................... 62
6 Solution proposal ............................................................................................ 65
6.1 Software Product Lines ........................................................................... 65
6.2 Business Process Management ............................................................... 66
7 Conclusions ..................................................................................................... 72
7.1 Implementation ...................................................................................... 72
7.2 Results ..................................................................................................... 74
7.3 Limitations............................................................................................... 75
7.4 Evaluation ............................................................................................... 75
7.5 Fulfillment of purpose ............................................................................. 76
8 Discussion ........................................................................................................ 78
8.1 Generalizability ....................................................................................... 78
8.2 Reflections............................................................................................... 78
8.3 Future work ............................................................................................. 80
9 References ...................................................................................................... 81
10 List of abbreviations .................................................................................... 83
1
1 Introduction
The purpose of this chapter is to present the project background, research
question, purpose and limitations. A report outline concludes the chapter.
After this chapter the reader should have an understanding of research purpose
and how the report is structured.
1.1 Background Founded in 1984, Axis Communications was the student project of business major
Mikael Karlsson and the engineer student Martin Green. A year later, sales
specialist Keith Bloodworth joined the duo to form the founding group of Axis.
Mikael Karlsson and Martin Green quickly gained a reputation as being a ‘dynamic
duo’ with the powerful combination of Mikael’s excellent business sense and
Martins keen eye for innovative technology. In the late 80’s, Axis had established
itself as one of the world’s leading companies within printer servers and protocol
converters technology. By the mid-nineties, printer protocols had become
increasingly standardized and simplified. With maturing markets in the protocol
converter domain, Axis started to branch out into new business areas using their
expertise in network technology. In 1996 Axis launched the world’s first
surveillance camera that uses IP technology to connect to local networks or the
internet. This technology would come to shape the future of the company. Today,
network cameras form the basis of the Axis product portfolio.
The Axis company structure is centered on global partnerships through
distributors, resellers and system integrators worldwide. Axis has a worldwide
presence in more than 30 countries with a global total of nearly a thousand
employees 1 [1]. With a steadily growing portfolio of network surveillance
equipment, Axis currently has a total of 45 products. Axis has pioneered the use of
Power over Ethernet (PoE). The PoE technology enables network attached
products to be powered via the ethernet cable by the network switch. This makes
for easier and less invasive installations and remains a big selling point for the
product portfolio. Axis was also one of the driving forces behind ONVIF (Open
Network Video Interface Forum). The initiative was launched as a cooperative
effort by Axis, Bosch and Sony in 2008. The purpose was to establish a global open
interface standard for network video products and to support the ongoing shift
1 March 2011
2
from analog to network-based surveillance in the security market. According to
market institute IMS Research, Axis Communications secured a second place
among suppliers of surveillance cameras and fourth place among suppliers of
surveillance equipment worldwide in 2010. The total market in 2009 is estimated
by IMS to have reached USD 8.2 billion and expected to grow to 14.5 billion USD
in 2014 [2].
Having a steadily expanding product portfolio in a rapid growing market creates a
necessity to look upon internal processes in order to retain efficiency and
effectiveness. All of Axis products contain both hardware and software
components. The embedded software is called Firmware. The focus of this thesis
will solely be on firmware development for Axis existing product portfolio. All
software development at Axis is based on a shared common platform named the
Linux Firmware Platform. This platform based approach will be extensively
investigated throughout the thesis.
1.2 Purpose and goals As time to market is absolutely crucial in the high-tech intensive market in which
Axis operate, the company has begun trying to shorten project roll-out times. The
firmware development process has in the past year been shortened from an
average of 18 months down to 12 months per project2. But even so, further
improvements are crucial to remain competitive as the number of products
increase. The use of a common software platform traditionally results in
substantially shortened roll-out times, which hasn’t been the case for Axis. The
development process is also overly complicated which leads to difficulties in the
management of resources and responsibilities. The overall purpose of this master
thesis is thus to address the following research questions:
RQ1: How should Axis firmware development organization be
restructured to better utilize the benefits from using a software platform?
RQ2: How can firmware development at Axis be measured and managed
using a process based management approach?
In order to fulfill this we identified certain goals to be reached:
G1: Compilation of current academic work in the area.
2 There are different types of projects. These are further discussed in chapter 4.2.
3
G2: Investigation of current firmware development processes at Axis
Communications.
G3: Suggest possible improvements based on academic studies.
G4: Evaluation of our suggested improvements and their validity.
1.3 Focus and delimitations This master thesis will address the software engineering process at the firmware
department of Axis. The involvement of product owners, clients and end
consumers will not be specifically addressed. Also, all processes regarding
hardware focused product research and development are considered to be
outside the scope of this thesis. The main focus will be on the common software
platform, and how it is utilized throughout the development process.
1.4 Report composition • Chapter 1: Introduction
The section presents project background, research question, purpose and imitations.
• Chapter 2: Methodology The section presents relevant scientific methods and describes and motivates the chosen methods.
• Chapter 3: Theory The section summarizes research and theories in areas relevant to the master thesis.
• Chapter 4: Case study, Axis Communications AB The section describes the current situation at Axis Communications.
• Chapter 5: Analysis This section concludes findings from the case study and highlights the main issues addressed in the solution proposal.
• Chapter 6: Solution Proposal This section presents the solution proposal for issues highlighted in the analysis.
• Chapter 7: Conclusions This section presents the most important experiences and findings of the thesis and concludes these.
• Chapter 8: Discussion This section concludes the thesis with a discussion of generalizability and suggestions for future work.
4
2 Methodology This chapter presents a brief explanation of scientific methods and concepts.
Selected methods are described and the selection of these motivated.
Leaving this chapter, the reader should be aware of methods used and how these
affect the end result.
The purpose of determining which set of methods to be used is to establish a base
set of principles and rules to ensure an efficient and scientific approach when
writing a thesis. Methodology functions as the quality assurance of the thesis,
ensuring a scientific and fact based approach. There are often multiple options
when selecting which methodology to use in reaching pre-determined goals.
2.1 Induction and Deduction When conducting a scientific study there are different options of going from
theory to empiricism. The two main alternatives are induction and deduction.
Induction
Using an inductive approach a researcher is not guided by preconceived
ideas. Empirical data is the starting point; the researcher induces data and
observations into more generalizable theory. The inductive logic works
well to discover connections and correlations in data but not to justify
general assumptions [3].
Deduction
Using deductive approach, the study’s starting point is established theory
within the field of research. From this knowledge base, hypotheses are
deducted [4].
This thesis used a hybrid approach. Related to the previously established research
questions RQ1 and RQ2, the following approach was chosen:
1. In order to suggest an organizational restructuring, a case study of the
current situation was conducted.
2. To ensure that existing theories within the relevant fields are
emphasized, a literature study was carried out.
The starting point was to study established theories in order to later apply them
to the current situation forming a solution proposal. Our project model was
5
influenced both by inductive and deductive approaches, sometimes referred to as
abductive reasoning.
2.2 Scientific methodology [5] describes the four most commonly used methods for applied science theses.
2.2.1 Survey
A method used to map a certain fact or phenomenon, often performed by
sampling a small part of a target population.
2.2.2 Case study
An in-depth study of one or many cases with minimal influence on the studied
object. Often the case studying researcher chooses qualitative methods, even
though the case study format doesn’t exclude quantitative methods.
2.2.3 Experiment
A comparative research method where variables are isolated and manipulated in
a controlled manner.
2.2.4 Action research
A carefully monitored and documented study of an activity, aimed at solving a
specific problem.
2.2.5 Methods used in this thesis
This thesis was performed in the format of a case study where we aimed at
depicting the actual situation in the most correct and objective nature [5]. This
case study intended to depict reality in an explanatory manner, focused on
understanding the actual situation at Axis. The case study and analysis were used
to produce an improvement suggestion aimed at specific perceived issues.
2.3 Data format and sources To perform a case study, it was necessary to collect a viable amount of subject
data. Such data can be of two types, qualitative or quantitative. Furthermore,
data can be obtained either from primary or secondary sources.
2.3.1 Quantitative and qualitative data
Quantitative data can be counted or classified; the idea of quantitative research is
to utilize mathematical models, theories and/or hypotheses to reach conclusions.
6
Qualitative data can be of a subjective, objective nature or both. Qualitative
research seeks to interpret such data into meaningful conclusions [7].
2.3.2 Primary and secondary data
Primary data is collected by a researcher for the purpose of a specific study [3].
Secondary data is data that already has been collected and compiled in a different
context [3].
2.3.3 Data used in this thesis
This thesis was conducted using mainly qualitative data from both primary and
secondary sources.
2.4 Techniques for data acquisition Different data acquisition methods can be used to establish a foundation of useful
data. Often there is a strong connection between the selected research method
and the data gathering technique being used [7]. A selection from methods
described in [5]:
2.4.1 Logbook
Documentation describing the progression of the scientific work from start to
finish.
2.4.2 Data compiled by others
Acquire statistical data published by independent organizations, unprocessed
register data, academic studies or archive data.
2.4.3 Observations
Study a phenomenon or act with the senses, with or without technical aids.
Observations can be conducted with or without interaction by the observer.
2.4.4 Techniques used in this thesis
This thesis used a multitude of different techniques for data acquisition. A
logbook was upheld during the entirety of writing the thesis to keep track of
important meetings, discussions and decisions. Project documentation and
internal documentation of processes and practices at Axis was pivotal in
understanding the current situation. Observations and insights gained at meetings
and other activities at Axis were documented.
7
2.5 Validity, reliability and objectivity There are commonly three aspects when conducting and evaluating research
studies. These form the basis for evaluating and assessing the quality of scientific
work [3].
2.5.1 Validity
There are two main types of validity in quantitative research, internal and external.
Internal validity is the ability to measure what is intended to be measured.
External validity is the results generalizability to other situations [3].
2.5.2 Reliability
The stability of a measurement is referred to as reliability. If the same value is
obtained using repeated measuring or sampling, said measuring is considered
having good reliability [3].
2.5.3 Objectivity
The concept of objectivity is used in quantitative studies and concerns the
researcher's neutrality. The objective researcher remains impartial to his or her
work, regardless of preconceptions and opinions [3]. This is sometimes referred to
as “researcher bias”.
2.6 Redefining validity, reliability and objectivity into a
qualitative context Defining validity and reliability is highly applicable when working with quantitative
data. When data is qualitative though, it’s problematic to translate these aspects
into a qualitative context. Instead of trying to assimilate these dimensions to fit
the qualitative study’s nature, [3] have redefined these dimensions into two
fundamentally new categories: Trustworthiness and Authenticity. Trustworthiness
is split into four general categories:
2.6.1 Credibility (analogous to internal validity)
How accurately a study or research depicts reality. The researcher can confirm
that his or her interpretation of reality has credibility by letting affected
participants or other stakeholders validate the material. This is referred to as
respondents’ validation [3].
2.6.2 Transferability (analogous to external validity)
How well research conclusions generalize to other contexts. Since qualitative
studies often consist of extensive investigation of a small number of samples
8
(people/organizations), it’s often problematic to transfer the results to any other
context. Researchers should therefore provide an extensive description sufficient
enough to let the reader determine the degree of transferability [3].
2.6.3 Dependability (analogous to reliability)
Ensures that a scientific work is well documented and properly described. An
implication of dependability when conducting research is that someone else will
be able to replicate your work and come to similar conclusions [3].
2.6.4 Conformability (analogous to objectivity)
Conformability aims to measure how well the researcher ensures that personal
valuations or likewise not having affected the results, or the procedures used to
reach it [3].
2.6.5 Authenticity
In addition to these categories there is a fifth category named authenticity. This
relates to aspects such as giving a fair description of reality, but also includes
reasons for contributing to the population or situation being researched [3].
2.7 Trustworthiness and authenticity of this thesis Our approach was to depict the situation at Axis as truthfully and extensively as
possible. This was achieved by applying a theoretical framework to solve
perceived problems. The large amount of available internal documentation and
material with varying quality and relevance made it necessary to adopt a strategy
for information processing and validation. This was performed by gathering
internal information related to the topic and then validate the information by
cross-checking it with managers or affected staff. This assured that the
information was up-to-date and relevant.
The case study was based on Axis internal documentation, informal meetings with
managers and staff as well as our own observations.
We continuously asked managers and affected personnel at Axis to review and
validate our produced material in order to maintain a high level of credibility. The
material was reiterated and weekly meetings were held to confirm our
understanding of roles and practices at Axis. We kept an updated logbook of our
proceedings and activities at Axis to ensure a well-documented and
understandable work process was available.
9
Our approach has been to remain as objective as possible when reviewing internal
material and participating in meetings and daily activities. As a part of the
literature studies we performed bibliometric validation on selected articles by
investigating how many times and in what context they were being cited.
Using case study methodology does not guarantee transferability to other
situations or populations [3]. We cannot claim that the conclusions of this thesis
are generalizable to other contexts, but should reasonably be applicable to
companies similar to Axis.
During the time of the writing of this thesis, managers at Axis decided to take
action and restructure the organization into a team-based arrangement not unlike
the one suggested in our solution proposal. Obviously, we cannot take full credit
for this decision. The fact is however, that through discussions with managers and
short presentations of the principles of BPM we can claim to have at least
contributed to the decision.
10
3 Theory
This section summarizes current research in areas relevant to the master thesis.
Software Product Lines applies to development methodologies, while Business
Process Management relates to business and organizational dimensions.
The chapter is concluded with a short summary of theories within firmware
development, quality assurance and methods for measuring software
development.
Leaving this chapter, the reader should have an understanding of the theoretical
framework used in the thesis.
3.1 Software Product Lines The concept of Software Product Lines (SPL) was developed in the second half of
the 1990s and aims to allow extensive software reuse within organizations. The
benefits of software reuse include decreased development costs and faster time-
to-market. Often the cost-aspect is the prime motivator for implementing SPL
even though the adoption usually brings several other positive effects, including
increased code quality and decrease in total lines of code needed within the
organization. The latter may in turn also lead to decreased software maintenance
costs.
The Software Engineering Institute at Carnegie Mellon University is a driving force
in the continued development and propagation of software product line practices.
The institute organized the Software Product Line Conference in the year of 2000
and has released the book Software Product Lines: Practices and Patterns which
has sold over 4,000 copies.
Several large corporations including Hewlett Packard, Bosch and Nokia have
successfully implemented software product lines in their organizations. The
required investment for implementing SPL in the organization often reaches full
pay-back after about three products (see Figure 1).
11
Figure 1: Economics of software product line engineering [8]
Software product line engineering clearly distinguishes between development for
reuse and development with reuse. Unlike other reuse approaches often focusing
on code assets, product line infrastructure includes reuse of all assets involved in
the software development life-cycle, ranging from requirements to testing.
Software product line engineering enables the various assets themselves to
contain variability. For example, a requirement specification may contain a
description of certain requirements that apply only for a specific group of
products.
Domain engineering focuses on the development of reusable assets for the
software product line. Since domain engineering continues throughout the life of
the SPL it is crucial to take long-term development into account when designing
and realizing new assets.
Application engineering uses the assets developed in the domain engineering to
build product releases. Usually most of the functionality required by new products
already exists within the platform, the rest is handled by new or modified
variations and in some cases also a portion of product-specific functions.
According to [9], as much as 90% of the new product can directly be derived from
the software platform whereas only the remaining 10% must be developed in
further steps.
12
Ideally the different assets in a software product line are interconnected enabling
traceability between e.g. a requirement and the corresponding implemented code
and test cases.
3.1.1 The BAPO model
The BAPO model was developed as a part of the ESAPS3 project conducted by
ITEA4 between 1999 and 2001. The model can be used as a majoring structure
model for software product lines. It is based on the assumption that four concerns
have to be addressed in the context of software engineering: Business,
Architecture, Process and Organization (BAPO).
Business: the costs and profits of the software, the strategy of applying it
and the planning of producing it.
Architecture: the technical means to build the software.
Process: the roles, responsibilities and relationships within software
development.
Organization: the people and organizational structures that execute the
software development. [9]
As shown in Figure 2, the four dimensions are all interlinked which means that
changing one of them affects the others as well. The arrows denote the natural
order to traverse the concerns. Business is the most influential factor and has to
be set up right before the other dimensions can follow [10]. The architecture
reflects the business concerns in software structure and rules. The process is then
set up to build the products determined by the architecture. Finally, the
organization should host the process.
3
Engineering Software Architectures, Processes and Platforms for System-Families; http://www.esi.es/esaps 4 Information Technology for European Advancement; http://www.itea2.org
13
Figure 2: BAPO model of software engineering [10]
3.1.2 Variability management
Software product line engineering aims at supporting a range of products.
Variability management is used in order to achieve this with minimal amount of
product specific code. By identifying differences between the products to be
included in the SPL, variation points can be determined. A variation point
describes the location at which the variation will occur. The variations can take
place in the context of requirement specification, architecture, component or test.
Assets shared by all products in the software product line are called
commonalities and are included in the software platform. As shown in Figure 3,
the platform consists of the commonalities and variations that later can be used in
application engineering to create software for specific products. Functions or
characteristics that need to be developed uniquely for a specific product are not
included in the platform, but the architecture of the platform must be able to
support such functionality.
14
Figure 3: The relation of different types of variability [9]
3.1.2.1 Variability techniques
Variability within the platform can be achieved with different techniques as
depicted in Figure 4. On an abstract level three different techniques can be
identified:
Adaptation: Only a single implementation is available for a certain component,
but it offers interfaces to adjust its behavior. The behavior alteration can for
example be achieved with configuration files, run-time parameterization or source
code patching.
Replacement: Several implementations of a component are available. Each
implementation fulfills the components specifications as described in the
architecture. When developing the product, either one of the available
implementations is chosen or a product-specific implementation is developed.
Extension: The architecture offers interfaces that allow functionality to be added
in a more generic way. The difference from the replacement technique is that the
extension interface enables several components of different types to be added to
it. Extensions are often called plug-ins [9].
15
Figure 4: Three basic techniques for realizing variability in an architecture[9]
3.1.3 Family Evaluation Framework (FEF)
In order to build a successful software product line, all four BAPO concerns must
be addressed properly. The Family Evaluation Framework is a benchmarking and
evaluation tool that uses the BAPO model in an attempt to capture the complete
view of the software engineering process. The FEF evaluation determines to what
extent the measured organization have implemented the practices of SPL.
As seen in Figure 5, each dimension is divided into five levels and has three to four
evaluation aspects assigned to it. Each level has specified criterions which have to
be met by the organization in order to reach that level. To reach the higher levels,
the organization must also meet the criterions for all preceding levels.
16
Figure 5: The Family Evaluation Framework (FEF) [9]
The result of the evaluation is an evaluation profile consisting of four values, one
for each BAPO dimension. The result can be used to identify which of the
dimensions that need some extra attention.
The different dimensions evaluated by the framework are:
Business measures the business involvement in software product line
engineering and variability management.
Architecture deals with the relationship between domain and application
architectures and how they are related via variability.
Process measures which software product line processes are used and
evaluates their maturity.
Organization measures the effectiveness of the distribution of domain
and application engineering over the organization.
3.1.4 Roles and Responsibilities
There are a number of roles defined for establishing a software product line. A
role can contain several responsibilities and can be assigned to one or multiple
employees within the organization. In small organizations the case of one person
17
having multiple roles is not uncommon. The latter requires that all responsibilities
tied to a specific role are well documented and known to all stakeholders.
As seen in Figure 6 the roles are divided between domain engineering and
application engineering. This underlines the previous mentioned fundamental SPL
distinction of development for reuse and development with reuse.
Figure 6: Overview of defined roles in SPL [9]
3.1.4.1 Product Manager
The role of the product manager involves the planning and evolution of the
complete range of products, the portfolio management.
Responsible for the product portfolio.
Plans present and future products in the product line, their features and
business value.
Initializes development of new functionality for products.
18
3.1.4.2 Domain Requirements Engineer
The domain requirements engineer is responsible for creating a variability model
and preparing requirements for the reference architecture (platform).
Develops common and variable requirements including the underlying
variability model (prepared in agreement with the roadmaps and long-
term plans of the product manager).
Provides feedback on the feasibility and costs of features to the product
manager.
Prepares requirements for the reference architecture in cooperation with
the domain architect.
3.1.4.3 Domain Architect
The domain architect is responsible for outlining and designing the reference
architecture as well as validate whether new components conforms to the
guidelines and rules of the platform.
Prepares and outlines the reference architecture in cooperation with the
domain requirements engineer.
Provides feedback on feasibility and costs of features, and the involved
variability.
Validates whether the designs of reusable assets fulfills the reference
architecture.
Determines configuration mechanisms to be used to build the end
products.
3.1.4.4 Domain Developer
The domain developer is responsible for development of the reusable assets as
designed by the domain architect.
Develops reusable assets according to the reference architecture.
Provides feasibility feedback to the domain architect on the development
of common and reusable assets.
3.1.4.5 Domain Tester
The domain tester is responsible for quality assurance on reusable assets.
Performs integration and system tests on domain assets.
19
Prepares common and variable test assets to be used by the application
tester.
Responsible for the domain testing strategy.
Provides information on the testability of requirements and design
choices.
3.1.4.6 Domain Asset Manager
The domain asset manager is responsible for managing reusable assets contained
in the reference architecture.
Responsible for maintaining versions and variants of all domain assets.
Responsible for maintaining traceability among the assets.
3.1.4.7 Application Requirements Engineer
The application requirements engineer is responsible for requirements for a single
product or application.
Responsible for development and maintenance of requirements for a
single product if specific domain requirements are lacking or nonexistent.
Responsible for providing the actual selection of requirements as input for
architecture and testing.
Provides feedback to the product manager on the feasibility and cost of
specific features.
Provides the domain asset manager with the actual selection of
requirements as input for configuration management.
3.1.4.8 Application Architect
The application architect is responsible for developing the architecture for a single
product or application.
Develops and maintains the architecture for a single product.
Provides information to the domain architect regarding which domain
assets have been overridden in application development (and may
candidate for domain inclusion in the future).
Provides feedback on the feasibility and costs of requirements to the
application requirements engineer.
3.1.4.9 Application Developer
The application developer develops product and application specific components.
20
Develops and maintains application-specific components and interfaces in
accordance with the architecture.
Reports components and interfaces that can be promoted to the domain.
3.1.4.10 Application Tester
The application tester is responsible for the testing of single applications.
Responsible for the testing of single applications (using the domain testing
strategy and the reusable test assets provided for by the domain tester).
Performs application integration and system tests.
3.1.5 Organizational Structures
A software development organization often consists of different roles with
individual functions and responsibilities. In large software companies there are
often groups of people sharing the same role, while in small organizations a single
person may have multiple roles. The challenges that organizations face are
different depending on the size of the organization. However, the basic principles
for choosing an organizational structure are the same for organizations regardless
of size [9].
Interaction and communication between the different roles is primarily
determined by the organizational structure. It is important to have a
communication structure that supports both the product line (internal) as well as
external stakeholders. [9] have identified a couple of typical organizations with
individual advantages and disadvantages where the two most common are
described in section 3.1.5.1 - 3.1.5.2. They also list the following typical structural
characteristics that arise in the context of software product lines:
“Domain engineering and application engineering each perform a
software development life-cycle.”
Domain engineering and application engineering are structured as
different units with their respective life-cycles. This is especially true for
the product-oriented organization described below.
“Interactions between domain and application engineering are mainly
‘functional’, i.e. at requirements, design, realization or test level.”
In software product line engineering, the imposed interaction between
domain and application engineering is almost entirely limited to the
21
exchange of functional5 documents such as requirement specifications,
code design and test instructions etc. In order to clearly communicate
long-term goals and purpose, people from domain and application
engineering departments have to meet on a regular basis and discuss
concerns and thoughts on shared assets.
“The tester has interactions with most other phases in the same
development.”
Testers have continuous interaction with the other roles during
development projects.
“Domain asset manager is a specific role. It interacts with most of the
domain engineering roles and with some of the application engineering
roles as well.”
A SPL-specific role which functions as coordinator between the other roles
as well as manages produced reusable assets in the platform and for
application engineering projects.
“The product manager has a special role, in which he provides the input
for domain engineering, and is the initiator of application engineering.”
The product manager role is the natural initiator of new functionality for
products, which in turn is the main input for domain engineering projects.
3.1.5.1 Product-Oriented Organization
The most common way to structure an organization for software product line
engineering is to separate between domain and application engineering. Each
such unit consists of a sub-structure with departments needed for handling the
different development phases. Often there is a single unit responsible for domain
engineering, and several units for application engineering.
One advantage of having separate domain and application engineering units is
that it clearly distributes the main functions and responsibilities. Related software
5 Functional requirements specify particular results of a system, contrasted with non-
functional requirements which specify overall characteristics and desired software qualities of the system. Functional requirements drive the application architecture of a system, while non-functional requirements drive the technical architecture of a system. [11]
22
engineering activities are taking place in the same unit, which improves the
communication and interaction between the different roles inside the units.
However, communication and interaction between roles of different units is
described as being a main challenge with this type of organization according to [9].
3.1.5.2 Process-Oriented Organization
Another typical way of structuring the organization is in a process-oriented
fashion as seen in Figure 7. In this type of organization people can be allocated
more flexibly between projects (both application and domain projects) if needed.
As people work with both domain and application development the incitement of
developing functional reusable assets becomes clearer. It also becomes easier to
ensure the integrity of the architecture as the same person responsible for a part
of it also makes changes to it.
One drawback with this type of organization is that the different phases of
engineering are not close to each other making communication more difficult.
Domain Requirements
Engineer
Domain Architect
Domain Developer
Application Requirements
Engineer
ApplicationArchitect
ApplicationDeveloper
Application Requirements
Engineer
ApplicationArchitect
ApplicationDeveloper
Domain Engineering project
Application Engineering project #1
Application Engineering project #2
RequirementsEngineering
Design Realisation
Figure 7: Process-oriented organization[9]
3.2 Business Process Management Business Process Management can be considered a collective expression for many
different schools of business development, from lean methodologies to six sigma
management. An empirical study by [12] states that there is evidence that BPM
23
helps organizations to gain improved product quality, customer satisfaction,
delivery speed and time-to-market speed.
The common element for this wide range of theories is the focus on the business
process. Considered by many to be the father of the business process paradigm is
management innovator Dr. Michael Hammer. In 1993, Hammer and college James
A Champy laid the foundation for the process approach when they published the
ground breaking Reengineering the Corporation: A manifesto for Business
Revolution (1993). Hammer’s approach to focus on radical change and the
business process was a wakeup call to corporations worldwide and the book
became a best seller, selling over 2.5 million copies and remaining on the New
York Times best-seller list for over a year. Hammer and Champy named called
their new found approach “Business Process Reengineering”, BPR.
Hammers revolutionizing idea was to focus on processes of value creation.
Hammer and Champy defined a business process as a “collection of activities that
takes one or more kinds of input and creates an output that is of value to the
customer.” According to Hammer, business development had to start with radical
change in the organization towards a process based approach. Hammer and
Champy did not focus on traditional business development and improvement
entities such as organization, management and information systems but rather
promoted a radical shift of paradigm to solely focusing on value creating
processes.
Following Reengineering the Corporation, in 1995 Hammer published The
Reengineering Revolution and two years later Beyond Reengineering. BPR had
become an increasingly popular expression, stirring up a buzz in the corporate
management world. Although gaining a lot of momentum, voices had been raised
against the radical methods proposed in BRP; following a number of failures in
BPR projects around the world [13]. The BPR approach was deemed too radical
and narrow to implement. In The Reengineering Revolution Hammer reconsidered
the emphasis of radical changes to instead focus on the process itself. According
to much of the criticism, The Reengineering Revolution had underestimated the
importance of the human factors for the effectiveness of business processes and
proposing a combination of radical reengineering and continuous improvement as
a solution to the shortcomings in the previous book.
Even though BPR has received a lot of critique over the years it continues to
remain an influential approach to business process management in both the
24
public and private sector [14]. At the core of the concept is the focus on value
creation activities and managing the company with a focus on deliverables.
3.2.1 Process identification and management
There are many reasons for identifying the processes in an organization. Primarily,
the identification establishes a base line of the general work practice to manage
and communicate the day-to-day work flow in the company. When processes
have been identified, they can form the foundation for having common systems
(IT, management, CRM) throughout the company.
3.2.2 The Seven Principles of BPR
Hammer and Champy stated seven principles for successful process improvement.
The guidelines are designed to counteract traditional line management methods
to streamline an organization into a process based business approach; and in this
way lower cost, reduce time to market and improve quality [13].
1. Organize around outcomes, not tasks
To keep a focus on actual value creation, companies need to focus on
outcomes rather than tasks that need to be performed. When companies
divide processes into task and assign these to different people, there is an
obvious risk of sub-optimization. Responsibility for an entire process
should therefore be assigned to a single individual if possible. In
reengineered systems, each job is designed around an objective or an
outcome, such as a deliverable in the end of a completed process, rather
than one of the tasks necessary to complete the process.
2. Identify all the processes in an organization
To perform any kind of process management, internal activities need to
be mapped and organized. This is preferably done in a business process
map where internal activities, responsibilities and deliverables are
structured into graphical charts. When this has been performed, a
company can perform optimization and reengineering of processes.
3. Integrate information processing
Many companies have a chain of information where individuals produce
data used by someone else in the organization. For example, a
salesperson enters customer data into an order later processed by the
accounting department. Instead of separating the data acquisition and
25
information processing, the salesperson could process the data himself
into an internal customer relationship management system.
4. Output users should perform the processes
Many companies use a split up structure with separate departments that
specializes in performing a specific task. Each department completes its
particular task and passes its "product" off to another department. The
principle states that the people who use information from the system
should be those who perform the process that produces that information.
This eliminates handovers between different employees and departments.
Handover procedures can be the source of misunderstandings and creates
an unnecessary workload in the process.
5. Link parallel activities in the workflow
Many business processes are complex to an extent where they are divided
up and assigned to independent teams. These teams work in parallel with
each other and then integrate their tasks when they are done. Since
teams are managed independently many companies experience process
sub optimization. By linking and integrating parallel activity workflow
companies can shift the focus from single team performance to the
performance and results of the entire business process.
6. Put the decision point where the work is performed, empower workers
Many companies use complex hierarchical structures to control and
manage work flow and processes. By empowering employees and by
putting the decision point where the actual work is performed, companies
can eliminate unnecessarily complex decision structures and formal
processes.
7. Capture information once and at the source
In many companies, from industrial manufacturing to modern software
companies, there is an abundance of separate information systems such
as accounting systems, CRM systems, project planning systems and so on.
These systems can potentially collect and process partially similar
information. Not only is this often inefficient and expensive, but
redundant data containing discrepancies damages the reliability of the
collected information. These problems can be solved by capturing data
26
once, at its source, storing it in data bases and making the data accessible
to all authorized users. This approach reduces errors and costs and
eliminates data processing delays.
3.2.3 The process organization
Figure 8: The Business process organization
BPM constitutes a general model for work flow and management that is entirely
based around identified main processes. The process organization model depicted
in Figure 8 applies to any type of company, from manufacturing to IT or service
based companies. The goal of it is to align the organization with its core value
creating operations rather than focus on traditional organizational roles and
hierarchies. The central recurring theme in process management is to focus on
value creation. The process model of value creation considers the process as a
chain of events from a specified need to the fulfillment of this. A fulfillment of a
need could be the delivery of a required functionality, a release of a product
platform or the launch of a new product to the end consumer. By working
according to a process-based practice, work tasks typically are more complex
since the value creation processes often range across many different types of
functions and responsibilities. A process organization utilizes a team based
approach where the team also operates on a cross-functional basis. The
competence centers can for simplicity’s sake be considered as the traditional
company departments such as development, QA or maintenance. No actual work
is performed in the competence centers. The role of competence centers is rather
to ensure that individuals' knowledge and skills are maintained and improved,
27
provide education and forming the base for a network where workers can assist
each other and exchange experiences and knowledge.
The process organization in this sense is a balance between horizontal and vertical
forces. The vertical force drives work flow and the value creation process, while
the vertical forces represent decision structures, managers and resource planning
activities.
3.2.4 Roles
When defining the roles in a process organization, [15] apply the theories of BPM
into an applicable format and define clear roles to achieve a process oriented
organization. The purpose of establishing roles is to create ownership of resources
and responsibilities and reward systems that align according to defined processes.
3.2.4.1 Resource Owner
The resource owners’ main responsibility is to manage and allocate resources to
the process teams. It is also in the resource owners’ scope to manage the
knowledge base of the employees and teams in the competence and resource
centers. According to [15], knowledge is the most important resource to achieve
competitiveness. A well planned and structured training and skills development
can therefore be mutual beneficial for the company and its employees, bringing
more satisfied personnel and increased competitiveness. Skills take time to
acquire and build. Knowledge is a strategic business driver, and becoming more
and more critical for long-term sustainable business strategy and survival.
The daily work for a resource owner is to assure that individuals in the
organization are allocated to the right processes, solve resource conflicts, develop
individual development plans and handle staffing processes. The resource owner
hires personnel and performs the role of acting manager. While the process
owner is in charge of the actual value creating process, the resource owner has
the long-term relationship with the individual. A resource owner may also be
responsible for planning, maintenance and development of non-human resources.
3.2.4.2 Process Owner
The role of the process owner according to Hammer is “the manager in charge of
designing the process, building its supporting tools, installing it in the organization,
and ensuring its ongoing high performance”. To be successful in process based
organization development, one of the major steps towards a process oriented
organization is to appoint a process owner. The process owner’s responsibilities
28
are, from a holistic perspective, to coordinate and lead the development of the
appointed process. The process owner defines the process structure and
associated support systems. The output of the mapping of such activities is a
formal document description, a process map. This will allow for a flexible
approach, providing guidance to those involved in the process and show how the
different work phases interact. The result is an increased understanding of the
whole and increased cooperation over the past territorial boundaries.
The long term development of the process can be defined according to the actual
contents and activities in the process. Instead of defining every possible scenario,
[15] determines three goals for a well-developed process.
Appropriate
Process appropriateness relates to the ability to create customer
satisfaction and focus on the right activities. The process owner must
therefore identify the key process activities and analyze customers' needs
and expectations. Customers of a particular process can be both internal
and external.
Effective
Process efficiency is concerned with the ability to produce desired result
at the lowest total cost. The implication of process efficiency is to use the
right resources at the right time and place as well as to analyze the
process of reducing and possibly eliminating activities that do not create
value for the customer.
Flexible
Process flexibility is the ability to change as the business and / or technical
environment changes. It also signifies the ability to satisfy diverse
customer requirements. Process flexibility is partly due to competence
and flexibility of staff working in the process
3.2.4.3 Team Leader
The resource owner has the overall responsibilities for competences in the
organization, and the process owner the overall responsibility for the producing
process. The team leader is the operationally responsible for combining the
provided process structure and the competences of employees into a value
creating process. The team leader assembles the team, delegates responsibilities
and manages the day-to-day work process of the team. The role of a process team
leader resembles the classic role of a project-leader. The team leader performs
29
the role of acting manager in the daily work, so it’s highly important that the team
leader has a deep understanding of the process and utilized components.
3.2.5 Competence teams and Knowledge management
The intangible resource of knowledge is one of the most key production tools in
high technology intensive markets as it does not observe the Law of Diminishing
returns [16]. Knowledge Management can be summarized as delivering the right
knowledge to the right place at the right time. Knowledge is a perishable resource,
and must be continuously maintained. The process of knowledge management
starts at the acquisition and organization of knowledge to the sharing and
utilization of it.
In BPM, the person responsible for knowledge management is the resource owner.
The resource owner should maintain a long term plan for each individual’s
development and skill set. The competence center is virtual, meaning that no
actual work is done in the center but it functions as a common entity for
collaboration, education and knowledge sharing. Competence can use internal
tools for sharing information such as instant messaging programs or simple
mailing lists. The resource owner is responsible for arranging seminars, workshops
and weekly or monthly meetings to secure a common baseline of knowledge and
proficiency within the competence team [17].
3.2.6 Proactive measurement
When processes are mapped, restructured and established, the next step in
process management is to initiate a measuring of the performance of the
processes. According to [15], companies should use a proactive approach when
measuring processes. A proactive measuring system implies using measurements
that enables taking action rather than just using it as a tool for performance
follow up. A proactive measurement predicts future events and performance of
the processes. Measurements should also be tied to the process rather than
specific company divisions or functions.
A simple approach to determine what measurements should be used is to identify
what demands stakeholders have on the processes. This implies looking at
developers, costumers as well as product managers’ demands on for instance
code and function quality. Development has cost, time-to-market as well as
quality requirements and measurement and follow up should be aimed at
reflecting these requirements as closely as possible. A good system for measuring
30
explains where and whither, provides readiness for action and allows for
benchmarking.
3.2.7 Lean, Six Sigma and CMMI
The concept of Business Process Management is found in many forms of business
management, including modern Six Sigma and Lean methodologies.
3.2.8 Capability Maturity Model (CMM)
Capability maturity model (CMM) is an organization evaluation tool that provides
software organizations with a roadmap for continuous process improvement,
helping them identify which process areas need attention to reach a certain
maturity level. The model consists of five levels of maturity with examples of
behavior on each level for easy identification and application. These five maturity
levels define an ordinal scale for measuring the maturity of an organization’s
software process and for evaluating its software process capability. The
established levels are
1. Initial – ad hoc, the starting point for use of a new process.
2. Managed – the process is managed in accordance with agreed conditions.
3. Defined – the process is both defined and managed as a standard
business process.
4. Quantified – measured and managed
5. Optimized – process management includes deliberate process
optimization/improvement and the process output is followed up via
feedback.
The CMM was designed to guide software organizations in selecting process
improvement strategies by determining current process maturity and identifying
what factors most critical to software quality and process improvement [23].
3.3 Firmware and embedded software Firmware is specialized software embedded in a specific hardware environment or
electronic device. The firmware is stored in the memory of the hardware and
generally includes all runtime drivers, applications and runtime libraries needed to
operate the device. Most modern microprocessor-based technologies requires
firmware; from washing machines to the motherboard in your standard personal
computer [24]. Software development in this integrated environment is referred
to as embedded software development. To use an analogy with the human body;
if the body is equated to the device, then the hardware is the brain and the
31
firmware is the intelligence and consciousness. It is important to note that
firmware development constitutes an interaction of software and hardware.
Firmware is always developed with hardware specifications in mind. This makes
testing or debugging firmware a complicated matter. The firmware environment
is inside the device, and it could potentially be problematic to gain access and
monitor such a closed system.
Historically, embedded firmware has been considered a static entity. That is, the
software is embedded into the device or medium at the time of production and is
never altered or updated. This is still the case for many devices, such as most
washing machines and other household appliances. In general though, modern
embedded firmware is becoming increasingly more dynamic in the sense that it is
possible to change and update post production. The rapid growth of the internet
and increasing connectivity enables firmware to be continuously updated and
refined. Furthermore, standardization of connector formats such as USB and
Ethernet has catalyzed the evolution of firmware to the point where it is highly
accessible even for the end user. Today, it is increasingly common that
manufacturers of electronic devices are expected to provide upgradeable
firmware solutions [25].
A firmware upgrade can contain bug fixes for detected issues or even provide
completely new functionality. In essence, firmware upgrades could be considered
both a support/maintenance function and value adding towards the end user. The
strategic importance of continuous quality- and user experience (UX)
improvement should not be underestimated. When comparing two similarly
priced devices, well-functioning and maintained firmware could very well be the
deciding selling point.
3.4 Firmware Quality Firmware time-to-market can be measured as the time for one release cycle; the
time it takes from requirements specification to completed and released software.
In fast changing, technology intensive, markets there is often a requirement of
short time-to-market cycles. This is primarily to maintain flexibility for market
needs and be able to quickly adapt to the competition. Short time-to-market is an
important factor for the entire product development process, but especially in the
case of firmware development as it has some tendencies to create a development
bottleneck. This issue will be addressed later in the thesis. While most parts of the
32
development process such as research and product design can be decentralized,
firmware development tends to be centralized in an in-house department [26].
However, time-to-market is not the only important factor in firmware
development. Even more important is well functioning quality assurance. The
single most important factor of firmware is robustness. If the quality level is not
adequate, rapid time-to-market becomes pointless or even counterproductive.
When addressing product quality in firmware, a distinction is often made between
bugs and performance issues. Software bugs are defined as errors or flaws
introduced into the software that prevents it from operating as intended.
Software bugs can be of varying gravity, from simple esthetical flaws to errors
serious enough to compromise the entire functionality.
Performance issues can be caused by a multitude of different factors such as
hardware shortcomings, poorly written code or interface faults [27]. Performance
issues are most often derived from memory leaks, incorrect memory allocation or
system memory buildup due to faulting release of memory, i.e. garbage collection.
Firmware bugs can be both costly and create significant amount of badwill to the
corporation and its brand. Even though most modern firmware can be updated
and patched, any bugs that affect the end user are potentially damaging. To
secure a bug-free and performance issue-free firmware many companies invest in
specialized testing personnel and procedures. The effort to deliver a high quality
product and continuously test, monitor and ensure that quality standards are met
is often referred to as Quality Assurance (QA). Securing firmware quality requires
expertise in both software and hardware quality as firmware is a cross-functional
development process with interaction between software and hardware [28].
3.5 Software metrics and measurements There are a multitude of techniques for using metrics and measuring methods to
control and oversee software development. What techniques and metrics to use
depend heavily on the type of information that is critical for the success of a
project. Metrics can be used to for example perform design evaluation, code
complexity analysis, fault targeting and to determine general code health. A case
study at HP DeskJet firmware development investigates the use of metrics in the
end-game of a software project [29]. The HP firmware study constructed a system
of measurements specifically targeted at maintaining the schedule for release.
The measurements were designed to enable a quick overview of project status
and thus assessing the progression of the project. The four measurements are:
33
Code Turmoil
Code turmoil indicates the amount of changed lines of code per day. At
lower change rates the project is stabilized and should be close to ready
for delivery. At higher change rates, the project is not stable or ready for
release, and also there is increased risk of introducing additional bugs into
the code.
Open Defects
Open defects measures number of found but still uncorrected defects for
a project. It is mainly a quality and stability measurement, but also serves
as an indicator of remaining code turmoil. Addressing these shortcomings
will require changes in the code, which in turn increase the code turmoil.
Defect Find Rate
Defect find rate measure number of defects found during a week. A high
rate indicates that the code is unstable and that the number of open
tickets can be expected to remain high until the code stabilizes.
Test Passing Rate
Test passing rate determines the passing rate of function and system tests.
If a large proportion of the tests fail to run, it may result in a subsequent
increase in number of open tickets. This because of the increased defect
find rate when previously blocked test are running again.
Find rate
Open defects
Code turmoil
Test passing rate
10
8
6
4
2
0
Figure 9: Measurement diagram
These measurements are standardized into a scale of 1 to 10. To visualize the
result these are plotted into a spider diagram, as seen in Figure 9: Measurement
34
diagram. It is then possible to draw basic conclusions on the status of the project.
Some simplified examples of measurement interpretations can be found in Table
1 below:
Code
Turmoil
Open
Defects
Defect
Find
Rate
Test
Passing
Rate
Indicates
- - High High Large number of defects is
being found by informal
testing activities
High Low - - Lots of changes in the code
are made in addition to
addressing reported
defects
Low Low Low High Product is ready for
release
Table 1: Examples of measurement interpretations
These metrics are only general type examples. In reality, these metrics can be
hard to interpret and changes continuously during the course of the project.
Although almost overly simplified, these metrics proved beneficial in the case of
HP firmware [29]. The case study found that to obtain any real benefits from
these measurements it is critical to create a database of past project metrics. This
supports project management in drawing conclusions of the data material. By
forming this statistical background, potentially risky phases in the development
can be identified. In order to compare one project to the next it’s pivotal to keep
metric definitions consistent.
35
4 Case study, Axis Communications AB
This section summarizes observations, documentation and other material into
actual findings describing the current situation at Axis Communications.
Leaving this chapter, the reader should have a good understanding of roles,
methods and current practices at Axis.
4.1 Product Portfolio Axis current portfolio comprises around 45 products in total. Product types can be
divided into roughly three major segments:
Network video cameras
Encoders and decoders
Software and recording equipment
4.1.1 Network video cameras
Axis Network cameras are marketed as surveillance products for both private and
public use. Camera types range from small and compact surveillance cameras to
high-tech thermal vision CCTV cameras. Prices range from 1500 SEK to 40.000 SEK
and cameras are also available in bundles of 10 and 20 at discounted prices. Axis
also provides a range of camera accessories such as dustcovers, mounts and extra
equipment.
4.1.2 Encoders and decoders
This product segment mainly consists of hardware video encoders and decoders
that convert video signals between analog and digital equivalents. Older types of
surveillance equipment often use analog coaxial cables, while newer camera-
types generally use IP technology and ethernet cables. Encoders and decoders are
needed when customers want to install digital cameras to operate with an older
analog surveillance system or vice versa.
4.1.3 Software and recording equipment
To offer a complete surveillance solution, Axis also provides miscellaneous
equipment such as monitors, monitoring software, recorders and other related
devices.
4.2 Axis Firmware Development All cameras provided by Axis implement firmware based on the Linux Firmware
Platform (LFP). The development process therefore revolves around the LFP and
36
the different versions of the platform. Responsibility is divided between different
internal departments.
4.2.1 Axis Development Organization
Figure 10: Axis firmware development organization
The firmware development process at Axis can be described by explaining the
general procedure of integrating new functionality into the LFP as seen in Figure
10. All new projects start with a master branch of the platform. Department A and
division B1 are responsible for developing new camera functionality, which
iteratively is integrated into the LFP. Department A is product oriented, and the
release of new product is the ultimate goal of development. Division B1 develops
functionality for the platform, firmware that is beneficial for everyone;
Department A as well as the B-divisions. The integration work is done within time
restrained integration slots. A multitude of projects can run in parallel, all based
on the same version of the LFP.
Following is a short description of the company’s divisions and departments
within the scope we have chosen.
4.2.1.1 Department A
Department A develops new products from idea to volume production, developed
in-house or together with partners. The team consists of about 70 employed
engineers at Lund as well as R&D personnel across Axis offices overseas. A new
37
product project starts with the most current master branch of the LFP. New
software functionality is planned by architectural and product specialists. New
hardware requirements are specified and hardware and product design is
implemented into a prototype in cooperation with the electronics and mechanics
department. Before any actual product launches, the project is tested in a full
test-run. After the functionality has been tested and the hardware has been
specified and sourced, the new product is ready for launch. Functionality
developed by department A can be integrated into the LFP prior to product
release or after, depending on the scope of the project.
4.2.1.2 Division B1
To ensure that Axis product portfolio is competitive and updated, new software
functionality for existing products is continuously developed by division B1. The
division consists of 20 software developers. New functionality may consists of a
new streaming functionality for network cameras or a new set of format
capabilities for Axis product line of video encoders. The functionality developed
by Division B1 is always integrated into the platform prior to release. Division B1
uses a software development methodology similar to a classic development
model called waterfall model 6 . The division has tried to use a different
development methodology called Test Driven Development7 (TDD) in a handful of
projects, but eventually decided not to make a shift in methodology due to a
number of reasons. A more extensive description of the Axis current development
process can be found in the 4.3 Tollgates chapter.
4.2.1.3 Division B2
Division B2 is a 15 person unit responsible for integrating newly developed code
into the platform. Platform integration is structured in integration slots, i.e. points
in time where the team performs merging of developed functionality into the LFP.
In order to optimize the process of integration and testing, projects from
department A and division B1 must be synchronized to fit into the same
integration slots. Division B2 works closely with test engineers in department C to
perform testing of integrated functionality. This process is normally time-
6 The waterfall development model is a sequential process in which progress is seen as
flowing downwards (like a waterfall). The model is defined by a series of development phases that are passed during the project life cycle. 7 TDD is a software development process methodology where the developer writes test
cases for a specific function prior to development and code implementation. The methodology is closely related to extreme programming with very short development cycles.
38
consuming. Prior to any functionality testing, division B2 runs tests to detect any
issues regarding the integration process. If deemed successful, the project
proceeds to the system testing phase. If the newly integrated code is limited to a
specific product (most often bug-fix releases), the test suite is run for the affected
products only. If the firmware functionality implies major updates for multiple
products, a full test-run is performed. Using reference products the new version of
the LFP is installed and a series of predefined system tests are carried out.
4.2.1.4 Division B3
The main focus and responsibility of division B3 is finalizing firmware versions for
the individual products. The team comprises about 15 people whose major
activity is correcting and adapting newly integrated software in order to work with
all of the targeted products. Developed functionality is not always directly
compatible with all of Axis products and is sometimes simply too generic to use all
available features in a specific product. Further customization and development
might therefore be necessary before the firmware can be released. Division B3 is
the last instance before the newly developed firmware is being compiled and
made available for download by the end users. The quality aspect is therefore of
highest importance.
4.2.1.5 Department C
Department C is split in two divisions, one working with embedded firmware
testing and the other with application testing. The embedded firmware team is
substantially larger with its 30 person work force; while application testing
consists of about ten people. The embedded firmware testing division is in turn
split in two parts, a product maintenance team and a LFP team. The LFP team
works with division B2 in integration tasks and system testing in embedded
projects. One tool used in that work is stability testing, which measures the
product’s abilities to perform well over prolonged periods of time.
Department C is responsible for creating and updating system and integration
tests as software functionality is added or alternated. The department also has a
unit responsible for hardware testing, but since the focus of this thesis solely is on
software development, this subject will not be further investigated.
4.3 Tollgates The development structure at Axis firmware development is complex and involves
a multitude of stakeholders in each development and testing phase. The project
model is basically a project process divided into phases, with formal handover
39
phases as described in Figure 11. These handover procedures are also known as
tollgates. Tollgates can be of symbolic nature, signifying a point in time where a
project reaches a defined state of maturity; or consist of a formal meeting with a
predefined agenda. Throughout the development process, Axis distinguishes
between three different project types: product projects, feature projects and
platform projects. Product projects involve the creation of new products, while
feature projects are limited to the creation of new functionality for existing
products. A platform project is development of new functionality for the LFP.
Figure 11: Axis project model
Even though the requirements for passing tollgates differ for different types of
projects, some guidelines are shared by all project types. A large quantity of
documentation should be written before projects are allowed to pass the
different tollgates. Below are short descriptions of the project phases in Axis
project model. Axis specific roles such as Technical Lead, Product Specialist etc.
are described in chapter 4.9.
4.3.1.1 Initial phase (pre-startup)
A project is generally initiated with a product proposal from either a product
owner or a product specialist according to the long-term roadmap for the
specified product. Product in this context shall be defined as a desired project
outcome, not necessarily a physical product. New project concepts are instead
discussed within product management, and scheduled according to the roadmap
work for relevant product segment. When a project is to be initiated, the
originator asks the corresponding team manager for a project manager resource,
and a temporary steering group is formed. The steering group decides on budget
constraints and timeframe for the startup phase.
40
Before a project is allowed to enter the startup phase, a technical lead shall be
appointed and made available - particularly during the beginning of a project.
4.3.1.2 Startup phase
During the startup phase, the main purpose of the project is to understand the
project goals and user requirements. To gain this understanding, workshops
typically are held to gather information on user requirements and stakeholder
needs. Feasibility studies and rapid prototyping can also be used to gain further
knowledge.
During the startup phase a lot of the documentation and specifications are
outlined, but the estimates are very preliminary. Also practicalities for the project
are set up, such as an internal project webpage and mailing lists etc. A tollgate
schedule for remaining tollgates is created and legal aspects and potential patent
infringements are investigated. A project requirement specification (PRS) is
created based on the identified user requirements. A document called Project
Management Plan is created and an initial risk analysis is performed and included
in the plan.
Before any actual development takes place, a software overview (SWO) document
is written to clarify where additional analysis and/or design is needed. In the
startup phase, such a document is outlined before the project is allowed to pass
the START!-tollgate. The SWO is a basis for discussion between the product owner,
technical lead, project manager and system architect. It explains the software that
will be developed and modified and draws a system overview diagram. The
system overview describes how the different software blocks in the system
interact.
4.3.1.3 Planning phase
In the planning phase a series of plans and project documentation are created and
the SWO is further specified. Resources are allocated and personnel are assigned.
When all necessary documentation is in place a project planning meeting is held
to review the plans and pass the project through the PLANS REVIEWED!-tollgate
into the Execution phase.
4.3.1.4 Execution phase
The execution phase is where the specified functionality is developed.
Development is performed by project teams in either department A or division B1,
depending on whether the project consist of new product development or
41
existing portfolio products respectively. After the main functionality has been
implemented, additional documentation is written. Developers are responsible for
writing unit tests covering newly developed and altered code. The LFP enables
developers to use previously completed projects as reference and enables new
projects to reuse previous developed code. The usage of a platform also enables
reuse of tests in a higher degree than what’s possible when developing tests for
applications without a shared codebase.
In order to pass the Freeze!-tollgate the developed function must endure 24h of
stability tests without e.g. excessive memory usage or hang-ups. After passing this
tollgate no new functionality is allowed to be added within the current project.
The purpose is to freeze the outcome of the project; stop project changes and
allow the project to concentrate on securing the quality of the project
deliverables.
4.3.1.5 Acceptance phase
The major activity in the acceptance phase is testing and creation of error-tickets
for correction. All tickets are handled via a web-based tool called the Trouble
database (described in chapter 4.7). New code block managers are selected by
the project manager and LFP documentation is created for the future LFP
integration work. Release candidates are prepared and communicated to the B2
division. If the project will result in a new product introduction, new production
test systems are created and a firmware beta version is delivered for prototype
testing. In order to pass the DELIVERY!-tollgate and enter the handover phase, a
formal meeting is held where release notes and platform documentation is
presented. The documentation is evaluated in cooperation with project
responsible from the B2 division.
4.3.1.6 Handover phase
The main focus of this project phase is LFP integration and merging of code and
documentation. Project responsibility now shifts to the B2 division as the code is
merged into the platform.
Following after integration is platform and regression testing where errors related
to the integration process and merging is reported along with other found bugs.
As before, all found errors are reported in the Trouble database. Since the project
is now a part of the LFP all tickets are reported under the LFP-part of the Trouble
database. The project team is now only assisting the LFP team in the subsequent
error correction work.
42
Lastly platform documentation is updated and finalized. Beginning with this
project phase, full scale stability testing is also performed on the platform. The
requirements to pass a certain stability test can for example be to have a
continuous stream of visual image data for at least one full week.
To fully pass the stability tests is considered a major milestone. If the stability
tests fail due to errors introduced before the integration work, major delays are to
be expected. This is because the project then needs to re-iterate with testers and
developers trying to identify and correct possible issues and flaws. Error
correction in this late project state is often cumbersome and time consuming [30].
Best case scenario is that the error is located and corrected followed by a
successful re-run of previously failed stability test. Even so, the nature of the
stability tests makes them very time consuming, which can cause delays in the
firmware release work if several test iterations have to be executed before the
error can be corrected.
To pass the Handover!-tollgate the LFP merge has to be approved without
indications of any new integration problems caused by the project merge. If any
issues still exists, the correction of these are planned and prioritized.
4.3.1.7 Closing phase
The closing phase of the project is defined by addressing issues stated in the
handover tollgate. Preferably, no new issues or bugs are detected in this phase.
Only finalizing measures are to be performed in this final project phase.
4.4 Linux Firmware Platform
4.4.1 Background
The Linux Firmware Platform is a wrapper platform, i.e. it contains other
platforms and components needed to build and test firmware for Axis network
camera products. The contained platform components are referred to as sub-
platforms.
The history of the firmware platform at Axis can be summarized as a series of
iterative projects aiming to create a common platform for all camera and camera
equipment software. It’s difficult to point out a certain point in time where Axis
took the decision to begin developing according to this strategy. Platform
development has, to a certain degree, been practiced at least since 2003 in the
company. The motives behind establishing a shared set of firmware assets were
43
to simplify the development process and enable structured and strategic code
reuse. The primary strategy for organizing the existing code into what came to be
the LFP was based on collecting all components required by the hardware
platform. Axis relies on in-house research for designing and developing the
various hardware platforms and ASICs8 used in the video products. A camera
hardware platform consists of a number of components, such as a processing unit,
memory, ethernet and audio controllers etc., all of which are mounted on a
motherboard and fitted in the product. Axis has throughout the years developed a
range of different hardware platforms, the most commonly used is the ARTPEC-
series9 on which most of the current product portfolio is based.
The project process prior to collecting common components and software in a
platform was to develop products independently of each other. This single-system
engineering sometimes used a previous camera as a reference product if the
camera used similar hardware specifications. The purpose of the LFP release
project was to supply a base software platform for department A and division B2
projects. The currently used platform integration and development process was
developed by an assigned project group representing different parts of the
organization. The goal of the project was to establish a formal process for
platform release and integration work. The scope of the project was also to
establish how quality assurance was to be managed for LFP releases. At the time,
no quality assurance was performed on the actual platform components, although
it was only performed camera wise. The project also established how the LFP
mainline branch would be managed and documented. The LFP release project
formally established the LFP as the core of firmware development at Axis.
4.4.2 The Linux Firmware Platform development process
When new features are developed for inclusion in the LFP a new branch is created
from a certain predefined release version of the LFP. The LFP release is defined by
included packages and their versions, specified in a configuration file named
LFP.conf. When compiled, only the specified version of each package is included.
The strategy of selecting certain packet revisions is done in order to achieve
variability within the platform called compile time configuration. The
configuration enables a way to specify releases for all products using one simple
text document. Each LFP version is defined by a unique LFP.conf file.
8 Application-Specific Integrated Circuit
9 Video processing chip developed by Axis and used in camera products.
44
The process of development and integration is best described from a perspective
of one master branch and several simultaneous development projects. In Figure
12 projects A, B, C and D represent newly developed functionality and a camera
firmware release project (version 5.20 release). The master branch is the baseline
version of all the packets, classes and drivers included in the platform. From this
baseline of classes and packets a certain LFP version is isolated and defined as LFP
3.2. With the use of such a platform version project teams can start development
new functionality when requested. A multitude of projects run simultaneously.
Besides new camera functionality or new product development, Axis also runs
platform projects with the purpose to assure a high level of quality of the platform
itself. It is important to point out that before any new camera functionality can be
released to the end customer, it needs to be integrated into the platform. This
means that the time to market for new functionality heavily relies on when it gets
integrated into the platform. As seen in figure 12, several projects run in parallel
but are all integrated in quick succession. The goal is to synchronize projects in
merge slots with as close proximity in time as possible. The reasoning behind
synchronizing projects to merge into the platform is to minimize testing and
quality assurance efforts. By collecting all functionality developed in a number of
separate projects and then integrating all of it into the platform at once, Axis tries
to limit the amount of needed release work.
5.20 Bugfix branch (Division B3)
Master
Project A (Division B1)
Project B (Department A)
LFP 3
.2
(Divisio
n B2)
Project C (Division B1)
Project D (Department A)
Rel
ease
wor
k (D
ivisio
n B3)
3 weeks of testing
(Department C) Rel
ease
wor
k (D
ivisio
n B3)
Rel
ease
wor
k (D
ivisio
n B3)
Merge slots
Figure 12: LFP integration process. X-axis represents a generic timescale
4.4.3 LFP testing and release work
The actual merging of new functionality into the master branch is usually quick
and painless. Most effort in the integration process is spent on testing and error
correction. After the merging of new features and bug fixes is finished, a new LFP
45
branch is created. This is done to allow for the department C to perform their
tests on an isolated branch unaffected by active projects running in parallel. At
this stage the testers performs a full test-run on the new release candidate. A full
test-run includes the complete set of system tests and stability tests to ensure
that the firmware is stable and free of potential memory leaks. The testers use a
number of reference products to perform the actual testing. Since testing the LFP
release candidate on every single product would be extremely time-consuming,
the team has decided on eight cameras and encoders that together cover the
majority of features and hardware combinations available in Axis product
portfolio. When errors are found, tickets describing the errors and the steps
required to reproduce them are created. All tickets are stored in a web-based
ticket database called Trouble (described in chapter 4.7). Division B2 proceeds
with correcting any found tickets and stability issues. When majority of found
critical bugs are corrected, the B3 division creates separate release-braches for
the different products.
Before the functionality in the new LFP release can reach end customer via
downloadable updates on the product’s support homepages, the B3 division
performs thorough testing and bug correction on the individual release braches. If
a bug is corrected in a branch of the LFP, it is preferable that this is pushed out to
all active branches derived from the master branch as well. Using this strategy the
error is corrected at its origin, rather than in the deriving branches. If not, it is
possible that the same bug is corrected a multitude of times or even be corrected
in one branch only to be reintroduced in the LFP via another branch carrying the
uncorrected code block. The subject of the LFP testing and release process will be
further investigated in the Analysis section of the thesis (chapter 5).
4.4.4 Contents
The LFP can be divided into three sections; Linux Application Platform, Technology
Platform and Axis Media Control (Figure 13). Except these three categories, the
LFP also includes support functionality such as function tests and build tools.
46
Figure 13: The Linux Firmware Platform contents
4.4.4.1 Linux Application Platform (LAP)
The Linux Application Platform contains the latest stable package revisions of
applications, libraries, scripts and configuration files. This section represents the
user space level in Linux software. These packets can for example be network
components such as DHCP client, runtime libraries and various scripts.
4.4.4.2 Technology Product Platform (TPP)
The Technology Product Platform contains the Linux kernel, and components
belonging to the kernel space layer in Linux software. These packets can for
example be drivers for the ARTPEC platform and audio chip.
4.4.4.3 Axis Media Control (AMC)
The Axis Media Control software is included in the Linux platform, but unlike the
LAP and TPP sections of the LFP the AMC software is executed on client side in the
user’s browser. When a camera is remotely accessed through a compatible web-
browser, the AMC package provides the user with a multitude of functionality.
This includes user controlled streaming like Pan-Tilt-Zoom, different privacy masks
and streaming options.
4.5 Software Product Lines at Axis When reviewing information in Axis intranet, documents indicating that Axis had
made efforts to implement SPL were found. The projects had been carried out in
isolated parts of the development organization. The main goals of these projects
were to obtain a higher degree of code reuse and to establish a common code
base. There were also academic pre-studies and case studies strongly indicating
that Axis had a long-term commitment for the practices and processes of SPL. A
quote from [31] “The company has employed product-Iine architecture based
software development since the beginning of the 1990s and uses object-oriented
frameworks as the components in the product-Iine.” But there is not only
47
historical evidence of previous implementations of SPL, Axis also use SPL specific
terminology when describing certain roles and titles within the company. Even
though there are strong indicatives of SPL at Axis, efforts finding information
concerning the implementation, and people responsible for it, failed. The SPL was
mainly implemented back in time when the company developed print servers.
Parts of management have changed when the company started the development
of network cameras and the company has grown.
Despite the lack of awareness and use of SPL at Axis, we decided that there were
strong enough similarities between the organizational structure and practices to
those described in the SPL theories to motivate the use of SPL specific theories
and tools in the case study.
4.6 Global Information Tracker (GIT) Modern firmware development projects in large-scale corporation typically
consist of many million lines of code. To synchronize projects, resources and data,
companies rely heavily on version control systems. These systems provide a
repository for current projects that enable multiple developers to edit code
simultaneously. These systems often have built-in intelligence to handle code
merging conflicts that may occur when multiple developers upload divergent code
[32]. Axis uses GIT, a version control system developed by Linus Torvalds, the
creator of Linux. The system is a distributed revision control system, meaning that
every GIT directory is a repository with complete history and revision tracking
capabilities. This means that GIT based development is not dependent on network
access or a central server. Axis previously used Concurrent Versions System (CVS)
to manage and synchronize developed source code. In late 2010 development
managers decided that CVS was unable to fulfill the version control needs at Axis
and a transition to GIT was initialized.
The transition was relatively seamless but GIT implementation meant developers
had to adopt new merging strategies. Merging code, in particular bug fixes, into
large scale projects demand a well-defined merge handling process. Axis
development is based on a main branch of the LFP (as previously described in
section 4.4.2). New projects are often based on the latest version of the LFP. At
any given time, multiple projects branch out from the main branch. When
implementing a bug-fix and merging it into the mainline, it is of vital importance
that it also is distributed out to all of the project branches. If not, bugs that have
been previously fixed can be reintroduced when the same code block residing in
48
other projects eventually are merged back into the main branch. GIT provided the
opportunity to achieve such a code distribution to derived branches which
previously had been impossible while using CVS.
4.7 Trouble To manage bug reports and errors in the firmware, Axis relies on an in-house
developed SQL database system named Trouble. Users access the system via a
web based interface where all relevant information is available. Tickets can be
added by both testers and developers. Mandatory fields are ticket status, affected
products or category, how to reproduce the error, what project the bug is related
to and an assessment of the degree of severity of the error. Preferably there is
also a short description of how the bug manifests and how it was discovered. The
ticket is time-stamped when added to the database.
When a ticket has been handled and a fix has been implemented in a satisfactorily
way, the fix is submitted for review by the code block manager. If the fix is
approved the ticket is marked as corrected. There is currently no measuring made
on the amount of time spent on specific tickets. Through the user interface
developers, testers and management have the option of subscribing to tickets
that are of special interest to their current project or assignment. The user
interface offers a graphical tool to extract statistics regarding remaining tickets in
certain projects or releases, but the tool itself doesn’t offer any extractable data.
4.8 QACE To aid project planning and management Axis use an internally developed
resource allocation tool (RAT), known as QACE, which offers a graphical web
interface for project planning as well as a time planning tool. QACE is used solely
by department C while other departments at Axis use a multitude of different
planning aids ranging from simple Microsoft Excel sheets to more advanced
project tools. QACE differentiates between projects directly linked to products
and firmware; and business development oriented line work projects. The QACE
system offers a very effective way of allocate and manage man-hours for current
projects, but is not suitable for planning daily activities.
Except project planning and time allocation, QACE also offers full test suite
capabilities. All existing test cases are available through a web-based interface.
Testers from department C use the same interface to perform test-runs. A test-
run consists of a set of test cases relevant to the function or system being tested.
49
A lead tester is responsible for collecting the relevant test cases and assigning a
tester to perform the actual testing. The assigned tester then executes the test
cases and reports the results directly in QACE. If an error is found, the tester
creates a ticket describing the error using Trouble described in previous chapter.
4.9 Roles at Axis To be able to manage and control the evolution of the LFP and to ensure a
sufficient knowledge base, the development at Axis is divided into several roles
and responsibilities. These roles exists in parallel to the more traditional
development organization which defines e.g. line managers and development
processes. The role system is hierarchical with system architects at the top. The
following roles are utilized in the software development process at Axis.
4.9.1 System Architect (SA)
At the top of the hierarchy, system architects (SA) have the overall responsibility
of the platform architecture and top-level design. System architects take part in
the roadmap work and are a driving force for feature architectural
implementations. They often act as orderers of projects and activities in line with
decided roadmap plans. SA is the final approver of architectural design and
therefore has to be tightly involved throughout platform projects. In many ways,
the SA acts as a project manager monitoring and managing the project processes
and progress.
SAs are normally selected among the most experienced software engineers in the
company. A significant part of the responsibility is also to ensure a continuous
evolution and improvement of the LFP and to set guidelines regarding coding and
documentation standards. The SA has the authority to refuse inclusion of
suggested changes or new code into the main branch of the LFP. SA have the
overall responsibility of appointing code block architects and code block
maintainers for the LFP but can still veto their decisions.
4.9.2 Code Block Architect (CBA)
A code block architect is the architect for a subset of the LFP. Above the CBA are
the SAs, responsible for the platform, and underneath are the code block
maintainers, responsible for individual subsystems. The platform is divided into
subsets based on functionality; each functionality-subset has a responsible CBA.
This partition is collectively done by the SAs and is a dynamic process in terms on
how the LFP is split. In development projects the CBA act as a gatekeeper for
changes to the functionality in the specific subset of the LFP. If any developer
50
wants to edit or add functionality this must first be approved by the appointed
CBA. The CBA also have the overall responsibility to ensure a clean and properly
commented code structure and functionality. Lastly, the CBA in collaboration with
CBMs is also responsible for reviewing the code trying to identify any potential
security flaws. CBAs appoint CBMs for the code blocks existing in the functionality
subsets.
4.9.3 Code Block Maintainer (CBM)
A Code Block Maintainer has similar responsibilities as a CBA but in an even more
limited block of code. CBMs manage single packages in the platform and review
updates or edits of the code before they are merged into the mainline. The CMB
functions as a specialist and code administrator in the day-to-day development
process. Developers are instructed to primarily contact the CBM with questions or
code reviews, and only contact the CBA in case of emergency.
4.9.4 Technical Lead (TL)
This role is appointed on a per project basis. The technical lead contributes with
technical assistance to the project manager. The TL also represents the system
architect within a project and manages technical issues and difficulties related to
the project. While other roles are tied to maintenance and expertise of the LFP,
the TL operates within the constraints of a specific project. The TL monitors the
entire project as well as planning and coordination of building the project releases.
The TL also organizes resources for quality assurance and has the overall
responsibility of project tickets added into the trouble database. Quality
responsibilities also include risk management and the elimination of potential
security flaws within the scope of the project.
4.9.5 Product Specialist
A product specialist role is defined as an expert on customer needs and possible
problem areas. He or she takes part in development projects as an expert on end-
user problems, needs and expectations. Other tasks include creating product
training material, hold training sessions and write technical notes and FAQs.
4.9.6 Product owner
The product owner is responsible for the long term road map and other strategic
product issues regarding a single, or a series of, products. The product owner
works closely with a technical product owner, who has deeper technical
knowledge of the product. He or she also functions as an interface to the sales
organization. This role is sometimes also referred to as the product manager.
52
5 Analysis
This section concludes findings from the case study and highlights the main issues
addressed in chapter 6 (Solution proposal). Findings from chapter 4 (Case study),
are evaluated using theories in chapter 3 (Theory).
Leaving this chapter, the reader should have an understanding of these issues and
how described theories apply to the situation at Axis.
5.1 Quality Assurance In late 2010, Axis performed a worldwide customer satisfaction survey targeting
retail suppliers, distribution and end costumers. When querying costumers how
they perceived Axis product quality overall they generally rated very high scores10.
This was especially evident when it came to product quality where 95% of the
customers agreed that Axis products are of high quality. Even though the scores
concern the entire product package including hardware and other factors, the
result stated a clear message; quality is in focus at Axis. It is deeply embedded in
the corporate culture and a source of great pride amongst the employees as well
as management.
Nevertheless our findings indicate that the testing process is somewhat
fragmented and the testing results are sometimes unpredictable. When
performing an internal review of tickets in the Trouble database, managers came
to the conclusion that only about half of the tickets were actually related to an
actual test case. This is a result of using both unplanned experience-based testing
as well as formally structured testing techniques. From these figures, the
experience based ad-hoc methods seem to be equally effective as the formally
structured testing. The comparison is however not completely fair, since in some
cases the insights given when executing the structured test may have led to
finding potential related flaws and defects. These flaws would then not be directly
tied to any specific test case, but could still be considered detected by structured
testing methods.
5.2 Software product lines The practices of Software Product Lines have been implemented in the firmware
development organization at Axis years ago. Today the organization differs from
the theories at several points, something that became clear when performing the
10
Averaging 6.0 on a scale of 1-7
53
SPL-evaluation. In section 5.2.2 SPL-roles are compared to existing roles at Axis to
further highlight the differences between theory and reality. The SPL-part of the
analysis chapter is concluded with an overview of available software tools
designed for organizations using the SPL-methodology.
5.2.1 SPL evaluation with Family Evaluation Framework
As previously mentioned, findings from the case study justifies the use of SPL
specific evaluation tools. The Family Evaluation Framework described in chapter
3.1.3 was used to evaluate to what extent and how well Axis implements SPL.
5.2.1.1 Business Dimension (Score: 2)
The business dimension consists of four categories aiming to capture the business
and financially tangible aspects of SPL. The framework focuses on those business
aspects that are specific to software product line organizations regarding
investment decisions, measuring the costs and profits of product line engineering
and funding of domain engineering.
Axis utilizes the platform to obtain high code reuse and decreased time-to-market.
The largest part of development is focused on domain engineering, which means
the amount of investment on reusable assets and the platform is substantial.
Since there is little or no awareness of the theories of SPL, the concepts and
methods are not communicated either inside or outside of Axis. The evolution of
the shared resources in the LFP are planned based on the demand of application
engineering projects and proposed product features.
Commercial (level 2)
Department A is well aware of the benefits that the platform presents today, e.g.
in the form of code reuse and decreased time to market for new products. An
example of this is the recent development of a widescreen-variant of the M31
camera. The development time for the new camera model could be kept down
due to extensive code reuse which was enabled by the platform.
However, there is no apparent strategy available for using software product line
engineering in marketing, sales and product planning. The commercial aspect is
evaluated at level 2: Aware.
Financial (level 2)
The vast majority of development done at the B-divisions is dedicated to domain
engineering, so the investment in reusable assets is substantial. The application
54
engineering process is focused on using the platform and hence, there has been
no need of defining budgeting consequences for not using it.
Axis doesn’t measure the costs of developing and profits from using the reusable
assets in the platform. The financial aspect is therefore evaluated at level 2:
Aware.
Vision and business objectives (level 1)
The existence and use of software product line in firmware development is not
clearly communicated either inside or outside of Axis and is not mentioned in the
vision or business objectives. The vision aspect is therefore evaluated at level 1:
Project based.
Strategic planning (level 3)
The plans for future LFP versions are derived from the demand of application
engineering projects and new product features. The LFP has a long term roadmap,
so does the individual projects. The strategic planning aspect is therefore
evaluated at level 3: Managed.
5.2.1.2 Architecture Dimension (Score: 3)
The architecture dimension contains three dimensions to evaluate the technical
aspects of building software in a software product line organization: assets reuse
level, reference architecture and variability management.
Axis has a high level of code reuse and uses an explicit reference architecture that
defines variation points. The variation points are however not fully managed or
controlled but rather used in a decentralized manner directly by the developers.
Asset reuse level (level 3)
The LFP developed by Axis can be defined as a collection of common assets
located in a domain repository (GIT and CVS). Since almost all firmware
development at Axis is done using domain engineering, the code base is
continuously growing and the reuse level high. Nevertheless, the variation points
in the LFP are not explicitly defined and managed on any higher level. The asset
reuse aspect is therefore evaluated at level 3: Software platform.
Reference architecture (level 4)
The LFP is an explicit reference architecture, which clearly defines a set of
configuration parameters to derive application specific variants from the shared
domain resources. The process of variability management is however not fully
55
automated and therefore the aspect of reference architecture is evaluated at
level 4: Variant products.
Variability management (level 3)
Variability in the LFP is managed with a menu based configuration system
accessed via a console terminal. It’s easy to select or deselect parameters but
there is no clear relation between the configuration parameter and what part of
the code that is affected. Furthermore, the reference architecture doesn’t
determine rules that application-specific variants have to obey. The aspect of
variability management is evaluated at level 3: Software platform.
5.2.1.3 Process Dimension (Score: 1)
The process dimension deals with the roles, responsibilities and relationships
within a software development organization. The dimension is split between the
three aspects of domain, application and collaboration processes. The aspect of
domain and application engineering relates to the traceability and to what extent
shared resource usage is measured and managed. The process dimension is
evaluated to level 1: Initial in all aspects. The key reasons for this are the lack of
traceability, limited project control and the absence of a requirements database.
Domain engineering (level 1)
There is no traceability between variants (releases) and variation points (code
blocks). If a bug is found in a specific block of code it is not immediately clear
which products are affected by the bug.
Axis currently lacks a comprehensive database and management of requirements
and a tool to link requirements to specific projects or functionality. There is no
monitoring of the use of reusable assets per application. Axis doesn’t conform to
the level two criteria of the domain engineering aspect, and is therefore evaluated
to level 1: Initial.
Application engineering (level 1)
Axis doesn’t distinguish between domain engineering and application engineering
in the way determined by the SPL framework. This implies that the domain
engineering evaluation also applies to application engineering, level 1: Initial.
Collaboration (level 1)
There is no deliberate separation between application and domain engineering at
Axis, making it hard to evaluate the quality of any collaboration between the two
entities. Nevertheless, collaboration between entities working solely on LFP
56
development and product release work is crucial. This aspect will remain
evaluated at level 1: Initial, until there is a clearly defined separation between
domain and application engineering.
5.2.1.4 Organization Dimension (Score: 3)
This dimension relates to the internal collaboration and general structure of the
product line. Even though Axis doesn’t distinguish between domain and
application engineering, department A could be considered working mainly with
application engineering. For the sake of the evaluation, we will consider this to be
the case. Given this, domain and application responsibilities are distributed
throughout the organization. Collaboration schemes at Axis are largely based on
documentation exchange and problem reports reviewed in internal meetings.
Roles and responsibilities (level 3)
Most of the development is focused towards developing common and reusable
assets. There are domain engineering roles and responsibilities defined; the B-
divisions. The interaction between domain and application engineering entities is
limited to the early initial and late release stages of development projects. The
aspect of roles and responsibilities is therefore evaluated to level 3: Weakly
connected.
Structure (level 3)
The domain and application roles are distributed throughout the organization.
There is a separate domain engineering department. Both domain and application
engineering have mostly project-oriented structures. The processes of
Department A and the B-divisions are separately defined and it could therefore be
argued that domain and application tasks have different descriptions. This implies
that structure is evaluated to level 3: Weakly connected.
Collaboration schemes (level 3)
Collaboration schemes are largely based on documentation exchange and
problem reports reviewed in internal meetings. Problem reports in the form of
tickets is the main collaboration tool used in the late project stages before release.
Formal change request are used to ensure that dependent functionality is not
affected negatively by the proposed changes. Since the collaboration schemes are
mostly document based, the collaboration schemes are evaluated to level 3:
Weakly connected.
57
5.2.1.5 Evaluation results
Figure 14 shows the evaluation result plotted in a spider-diagram. It was difficult
to evaluate the interaction between domain and application engineering as there
is no clear distinguish between the two at Axis. We have however considered
department A to correspond with application engineering and the B-divisions to
be mostly concerned with domain engineering. Some measurements fall between
several categories, in these cases we have made assessments in consultation with
our supervisors and other concerned personnel at Axis. The result shows that
there is an established and functioning software product line in the firmware
department of Axis. However, a broader perspective and companywide
awareness of the impact and results of the product line is low. This could
potentially result in the loss of synergy effects on broader scale throughout the
company.
Axis does not have any systems providing traceability among domain assets or
monitoring of the actual platform usage. Also there is no explicit database
collecting requirements according to the theories of software product lines. The
process dimension dealing with these matters are the one of the four dimensions
with lowest evaluated score. The result of the FEF-evaluation suggests that efforts
should be made primarily in the process dimension in order to increase the
aggregated maturity of the firmware development organization at Axis.
Figure 14: Axis Evaluation Profile
58
5.2.2 Mapping of SPL roles to existing roles at Axis
In the left column of Figure 15 all SPL specific roles according to the theory is
listed and in the right column is a mapping of their counterpart at Axis. One of the
biggest differences is that Axis has no separate application engineering roles. The
reason for this is that today everything is eventually integrated into the LFP which
is a domain engineering activity according to SPL-theories.
No clear counterpart to the role of Domain Asset Manager could be identified at
Axis. This role is responsible for maintaining versions and variants of all domain
assets. This role is also responsible for establishing and ensuring traceability of all
components.
Figure 15: Mapping of SPL-roles to existing roles at Axis
5.2.3 Third party SPL development tools
There exist a lot of development tools, and even some designed solely for SPL
organizations. The tools investigated can be used to achieve a clear requirements
management and traceability between different assets. IBM Jazz is one of the
tools investigated, which also support e.g. project management and outcome
reporting.
Gears from the company BigLever is a tool for SPL aiming at supporting SPL
specific entities such as reference architecture, variation points and variants.
59
5.2.3.1 IBM Jazz Platform och IBM Rational™
Most important features11:
Requirement handling and storage.
Enables traceability between requirements, function design, code and
tests.
Supports version control with gated commits where specific conditions
can be set (e.g. all unit tests must pass before developers are able to
check in code).
Change management capabilities enabling feedback how changes in
requirements or code impact linked entities.
Collaboration tools such as internal instant messaging (IM), code and
project annotation etc.
Compatible with development tools such as Eclipse and Netbeans.
5.2.3.2 BigLever Gears™
The tool manages all SPL specific components such as variation points, variants,
product portfolio with product properties and all platform assets.
BigLever and IBM are working closely together, enabling Gears to be combined
with Rational and the Jazz platform. Most of the features in Gears are available
directly within the interface of used development environment (e.g. Eclipse or
Netbeans).
Most important features12:
Manages variation points, variants and configuration parameters
Integrates with e.g. IBM Rational, Microsoft Visual Studio and Eclipse
Handles the reference architecture including components, shared assets
etc.
11
Information gathered from http://jazz.net 2011-05-30 12
Information gathered from http://www.biglever.com2011-05-30
60
5.3 Business Process Management The BPM parts of the situation analysis clearly underline the need for
organizational change in order to ensure efficiency in skills and resources usage.
The situation analysis has formed the basis for determining how Axis work
methods and organizational structure differs from the structures and theories
proposed by the BPM framework.
5.3.1 Process based approach
Axis internal structure and development methods indicate that a traditional
functional organization approach is being used. The seven principles of BPR state
a clear model of how to restructure a company into a process organizational
structure. To clarify how Axis differs from the theories of BPR, three of the seven
principles from chapter 3.2.2 are highlighted and compared to actual practices.
Organize around outcomes, not tasks
“To keep a focus on actual value creation, companies need to focus on
outcomes rather than tasks that need to be performed. When companies
divide processes into task and assign these to different people, there is an
obvious risk of sub optimization. In reengineered systems each job is
designed around an objective or an outcome, such as a deliverable in the
end of a completed process…”
Departments have strict tollgates and responsibilities are tied to the function
rather than the completion of a certain project. The handover is structured in a
formal way to handover responsibility of the code, after this handover division B
and department A considers projects to be completed. This despite the fact that a
lot of effort remains before developed functionality actually can reach end
costumer. The handover process also makes it more difficult to measure and
follow up on a project basis. The tollgate model clearly and in a very detailed
manner specifies what specific tasks need be completed in each project phase.
This could potentially divert the attention from the actual value creating process -
development of high quality functionality.
Output users should perform the processes
“Many companies use a split structure with separate departments that
specializes in performing a specific task. Each department completes its
particular task and passes its ‘product’ off to another department. The
principle states that the people who use information from the system
should be those who perform the process that produces that information.
61
This eliminates “handovers” between different employees and
departments. Handover procedures can be the source of
misunderstandings and creates an unnecessary workload in the process.”
The development process is partially fragmented. Departments developing
functionality doesn’t fully implement and test their own code. In practice,
department A and division B1 develop functionality up to a point where the
project code is to be integrated into the LFP. The code is then integrated and
department C executes a full test-run on one product. The development team
then solves the most critical errors before the code is released into the latest LFP.
It remains then to run a full test-run towards all reference products. Once several
development projects are done and integrated into the LFP, the platform is
frozen 13 and Department C performs testing and creates tickets describing
identified errors. Division B2 and B3 then have to correct errors residing in code
that for them is unfamiliar. The fact that all divisions of the company; B1, B2 and
B3 use a common development platform creates an even clearer image that all
departments are involved in a common process with a common goal. The tollgate
model with strict handovers fortifies the invisible walls that separate the different
divisions of the company.
Link parallel activities in the workflow
“Many business processes are complex to an extent where they are
divided up and assigned to independent teams. These teams work in
parallel with each other and then integrate their tasks when they are
done. Since teams are managed independently many companies
experience process sub optimization.”
The tollgate model used in software development at Axis explicitly defines the
process of which all the projects must pass to reach completion. Some parts of
this process could be seen as overly complicated and formal. However, a far more
pressing issue is the fact that the development is split up in a number of
independently working teams. These teams have, in some instances, insufficient
cross-communication. Even though the teams work towards a common goal they
are managed as separate entities and herein lay the risk of sub optimization.
13
When projects enter the Freeze-tollgate no additional features are allowed to be implemented; only error correction is performed. For further descriptions of the tollgates and project phases, see section 4.3.
62
5.3.2 Measurement
Another benefit of switching to a process organization methodology is the
facilitation of measuring performance and project follow up. Measuring and
project follow up is something that is not currently implemented in the Axis
development organization. The reason for this is simple; the development is
simply too fragmented. According to the principles of BPM, measurement should
be proactive and reflect the demands of stakeholders of the process. After a
process based, team-structured organization has been established, project
performance can be easily measured on a team basis. Each project then has a
defined set of resources and personnel tied to it, which greatly simplifies the
implementation of measurement system and follow up routines.
5.3.3 Refocusing on value creation
BPM offers a method for breaking down walls and refocusing the development
organization on the actual value creating process. The team-based approach
suggested by [15] also addresses one of the core issues expressed by managers
and project leaders at Axis, namely the topic of code knowledge and responsibility.
Restructuring into a process based organization refocuses the emphasis on
teamwork and focus the process using a horizontal development approach.
In the current situation all departments and divisions work independently of each
other with formal handover processes. There is really no clearly stated reason
justifying why the process is divided in the manner that it currently is. Specifically,
division B1 develop functionality that division B2 integrates into the LFP which
division B3 correct errors in. This creates inefficiency in the development process
in the sense that division B3 need to work with code developed by another
division of the company. Division B1 and B3 operate as two independent entities
and the communication between these are also impaired by the fact that they are
separated in time by integration and testing phases. At the time when the
functionality reaches B3, B1 considers the project to be finished which leads to
difficulties in determining responsibilities of the actual code. This also relates to
the fact that there is no clear connection between product requirements and the
integrated code of the LFP.
5.4 Packet distribution analysis The LFP contains a lot of functionality divided into separate packages.
Functionality can be configured for inclusion or exclusion from a menu
configuration tool. At compile-time the selected functionality and their
63
dependencies are included into the generated platform variant which will later
become the product firmware made available for download by end consumer.
As the functionality of the platform and the number of products it supports is
increasing, so does the risk that old and outdated features remaining in the
platform as a result of nonexistent traceability between variation points and the
variants it supports.
As part of our investigation of how the platform and its usage should develop,
there was a need to find out how the package usage was distributed among the
products. Since developing functionality to be included in the platform is more
expensive than developing for individual products, ideally the packages included
in the platform should be used by several products. Since the packages contained
in the platform differ greatly in size and functionality, lines of code (LOC) for the
individual packages must be considered when analyzing the distribution.
There is currently no data available describing the size of the platform or the size
of the individual packages. In order to analyze the distribution we had to write a
script collecting and aggregating the data for each package. A schematic outline of
the data collection process is described in Figure 16. The result was imported into
Microsoft Excel and was compiled in a histogram.
The distribution analysis was limited to products that had a firmware version
based on the latest LFP. The reason for this was to eliminate errors in the data
caused by discrepancies in the set of available packages between the different LFP
versions.
The compiled data shows that a great part of the available packages are being
used by a majority of the products. This observation holds true both measured as
number of imports and according to distribution of LOC. This indicates that the
platform is being used according to the guidelines; only functions shared by
multiple products should be included in the platform. However, there is a risk that
the results are misleading because of the undocumented relationship between
the different setting in available configuration parameters and the code
controlled by the parameter settings. Obsolete parameter settings or even unused
parameters can hide an unknown amount of unused or “dead” code, being
difficult to measure.
64
One of the criteria to reach level 2 in the process dimension of the SPL-evaluation
is to implement monitoring of the platform usage in terms of packages and code
books. The work we did to gather and compile usage-data shows that there exists
more data than today is being used by Axis. To reach higher maturity in the FEF-
evaluation and improve monitoring and control of the platform, Axis would have
to start collecting and analyze that data.
Figure 16: Schematic flowchart describing data collection principles for creating package usage and code distribution chart
65
6 Solution proposal
This solution proposal is structured into two sections, Software Product Lines and
Business Process Management. These two proposals are complementary and
together form a comprehensive solution to issues addressed in chapter 5 (Analysis).
Leaving this chapter, the reader should have an understanding of proposed
solutions and implications of these.
6.1 Software Product Lines Based on the FEF-evaluation carried out in consultation with managers at Axis, we
have developed the following plan of action based on theories from SPL. The
evaluation resulted in the process dimension receiving lowest grade of the four
evaluated dimensions. The areas that require attention within the process
dimension are also perceived as being among the more central areas of concern
according to our case study analysis.
6.1.1 Manage versions and variants
As the platform grows and number of products increase, the need of monitor and
manage the different aspects of the platform increases as well. An important step
in managing the platform is to create a model describing variation points and
variants as well as introduce traceability between the two. By monitor which parts
of the platform that are no longer used by any products, the platform can be kept
as compact as possible.
When a clear connection between variants (specific product firmware releases)
and variation points is established, it will become possible to list all products
affected by a found software bug or security flaw. The extent of the problem will
be apparent and necessary actions can be initiated and prioritized for the affected
releases.
Monitoring variation points may also simplify troubleshooting abnormalities that
only occur in certain products. A cross-check of shared variation points between
the faulty products can help identify and isolate the source of the problem.
A prerequisite for managing traceability is the introduction of a role similar to the
one described in section 3.1.4.6 called Domain Asset Manager, responsible for
maintaining versions and variants of all domain assets.
66
Several tools exists (of which some are listed in section 5.2.3) that enable
modeling of variation points and variants as well as versioning of other domain
assets. Used in conjunction with the already implemented GIT-system for code
versioning, asset management and traceability would increase the possibilities to
monitor and control platform usage and propose changes and prioritization of
code and functions accordingly.
6.1.2 Improve requirement management
The use of requirements in software engineering is important in all phases of the
development, ensuring that intended functionality is delivered. Software
architects needs requirements when designing functionality, developers when
implementing the design and testers when writing and executing tests for the
implemented code.
Requirements have to be testable and verifiable in order to subsequently
determine if a requirement is met. To ensure this we propose that Axis seek to
include testers much earlier in the project phase. Ideally, testers should be
involved when the requirements are formulated to be able to give direct
testability feedback.
Today, Axis collect project specific requirements in a document called Product
Requirements Specification. One of the cornerstones in the SPL-methodology is
striving to achieve full traceability among all shared assets in the organization. We
propose collecting all requirements on functionality level in a database with web-
interface, much like the QACE database. This would enable for easy search and
review of all requirements from a common interface. But more importantly this
would enable for traceability between requirements and functions. Testers would
be able to track changing requirements between two versions of a function,
assisting them in writing new and updated tests for the function. The easy
requirements overview could also prove beneficial in the earlier stage of the
process when creating roadmaps and long term product plans. This would enable
managers to see which requirements that are already implemented in the
platform and what needs to be added.
6.2 Business Process Management The solution proposal of this thesis in regards to BPM is a four-step process. Every
step contributes to the transition from a fragmented development structure to a
team based organization. The four steps are:
67
1. Develop a process map to reprioritize and restructure processes
2. Restructure organization to align with processes
3. Establish competence centers
4. Determine appropriate quantifiable measurements
6.2.1 Develop a process map to reprioritize and restructure processes
The process map is the most viable starting point of a BPM restructuring process.
This is done to acquire a starting point of improvement and to create
understanding of the current situation. The creation of a process map includes
mapping all current processes and then organizing them according to importance.
The most critical processes then form the basis for the main process map. A
process owner then restructures the process and begins delegating resources. The
methodology to perform such a mapping is for instance to perform a series of
structured interviews and workshops with key individuals in the organization. It is
important to get input from the entire organization when constructing the
process map. Managers and team leaders as well as developers and testers should
have their say in how the day to day development is structured, without prejudice
and absent of company politics. To streamline this process, we recommend Axis
to take help from a business process consultant. The research and structuring of a
process map is outside the scope of the thesis. However in Figure 17 we suggest a
simple outline for the mapping of the development process.
Figure 17: Axis Process Map
In Figure 17, the left column represents the internal customer e.g. orderer of
functionality. The right column represents the external costumer of the delivered
68
functionality. Costumer focus is one of the cornerstones of BPM, this format
underlines that the ultimate goal of any business process is to deliver customer
satisfaction. We have chosen to focus on the purely work organizational aspects
of BPM, since these are more easily distinguishable and applicable for the current
situation at Axis.
6.2.2 Restructure organization to align with processes
The core of this solution proposal is to restructure the development organization
from its current tollgate structure to a process-oriented team based structure. To
clarify how this organizational structure differs from the current situation,
consider the simplified depiction of the organizational structure in Figure 18.
Figure 18: Current organizational structure
Divisions B1 to B3 work independently of each other with separate management
and routines. The LFP functions as the intermediary as functionality is integrated
and versions iterated in release cycles. Division B1 and B3 have little direct
communication and department C is not involved with the development process
until division B2 initialize the platform integration process.
69
Figure 19: Principle structure Business Process Organization
Figure 19: Principle structure Business Process Organization is a representation of
our restructuring proposal. Key changes are:
Divisions B1, B2 and B3 operate as one unit. Product owner is the
stakeholder of the desired functionality. The role and responsibility of the
product owner remains unchanged. The development team is composed
of personnel from the different competence teams on project basis.
Resources owners are appointed according to area of expertise as per to
the role described in section 3.2.4.1. The resource owners may be
responsible for more than one competence center team each. He or she
delegates available personnel to current projects in collaboration with the
team leader and the product owner. The role of the resource owner
would most closely coincide with the current role of manager or possibly
project manager at Axis.
The team leader takes the operational lead and responsibility; a role
similar to the current role of the project manager with addition of the
responsibilities described in 3.2.4.3 Team Leader.
A process owner is appointed with the overall responsibility for managing
and improving the development process. The role of the process owner is
not equivalent to any role in the current organization; preferably this role
will be filled by personnel on manager level within Axis. To undertake the
responsibilities of process owner there should be a high level of
70
understanding of processes and methodology within the company. This
role is further described in 3.2.4.2 Process Owner.
6.2.3 Establish competence centers
Another key entity of the solution proposal is the establishment of competence
centers. As the company moves from a functional organization to a team based
structure, competence centers could be considered the new “divisions” of the
company, but without the implications of splitting up the organizational structure.
According to the principles of BPM, it is up to the resource owner to follow up on
team performance and long term knowledge improvement. The competence
centers do not limit developers to work on projects only related to their center,
any developer should be able to undertake any type of project. The competence
teams merely ensure knowledge sharing and growth. It is recommended that each
individual developer and tester becomes part of at least one competence team.
At the time being, Axis does not document or secure competence in a consistent
way, which can lead to that irreplaceable expertise is lost when individuals shift
employment. One of the goals with competence centers is to spread important
knowledge and skills to other members of the team in order to decrease
dependency on specific individuals.
The single most important task to perform when implementing competence
teams at Axis is to decide how to split the collective competence into specific
areas. Based on how the platform is currently structured we suggest that the
competence centers should consist of: Pan Tilt Zoom, Streaming, Linux and Web
applications.
6.2.4 Determine appropriate quantifiable measurements
We chose the format suggested in section 3.5, based on a case study performed in
the firmware department of HP DeskJet printers. The motivation for this choice is
simple: the printer firmware department of the company is similar to Axis in size
and technology and therefore should face the same types of challenges. Four
simple measurements are combined in a plotted graph which is then interpreted.
These measurements are iteratively sampled during a specific project. The
measurements are:
Code Turmoil
Open Defects
71
Defect Find Rate
Test Passing Rate
As stated by [15], a cardinal factor to successfully establishing a process based
organization is to ensure a well functioning system of measurements. The reason
for implementing a system of measurements should be to enable a proactive tool
for project leaders and other stakeholders to gain insight in the development
process. There exists, however, not only a single correct answer how to measure
and evaluate the performance and progression of a given project.
72
7 Conclusions This chapter presents the most important experiences and findings and concludes
these. Fulfillment of goals and purposes of the thesis are described and evaluated.
Possible evaluation of the proposal is suggested.
Leaving this chapter, the reader should have an understanding of the results and
limitations of this thesis.
7.1 Implementation
7.1.1 SPL
For the implementation of the two-part SPL improvement suggestion we
recommend the following implementation strategies.
7.1.1.1 Manage versions and variants
This improvement suggestion is centered on a system for managing variation
points, variants and to enable traceability between the two. This is preferably
done by purchasing an already existing tool for this specific purpose, presented in
section 5.2.3. We have also provided managers at Axis with our subjective opinion
and recommendation of most suitable tool, but the actual selection of this lies
outside the scope of this thesis. For us to estimate how much time an
implementation of such a tool would take is impossible as it’s depending on what
tool ultimately is selected.
7.1.1.2 Improve requirement management
The core of this suggestion is to collect all requirements on functionality level in a
database with web-interface, similar to the already existing QACE database. This
would preferably be developed in-house, under the same premises as QACE. A
requirements database would impact the entire organization, from development
units to testers and even product managers. Managers from respective units need
to meet and discuss the implications and possibilities of the requirements
database. It’s hard to say how long it would take to develop such a system.
7.1.2 BPM
For the implementation of the four-step BPM improvement suggestion we
recommend the following implementation strategies.
73
7.1.2.1 Develop a process map to reprioritize and restructure processes
When developing a process map it is critical to involve all stakeholders in the
process. For the technical expertise needed when constructing the process map
we recommend an external consultant with expert knowledge in BPM and process
improvement. The procedure of creating a process map can be both time-
consuming and complicated. It’s important to be persistent though, the process
map is at the core of the BPM restructuring process.
7.1.2.2 Restructure organization to align with processes
Before any actual change takes place, the organization restructuring need to be
acknowledged and motivated throughout the organization. Top level
management need to be involved and give sufficient mandate to the restructuring
process. Only if the restructuring is understood and its potential benefits are
apparent all parties will truly embrace the change. This is closely related to
concepts such as change management.
Axis has previously implemented organizational restructurings and managers
continuously implement improvements related to development. Historically, the
actual change process has proven relatively painless. We therefore think that the
core changes in the implementation phase of a restructuring could be performed
relatively quickly.
7.1.2.3 Establish competence centers
The basic components of the competence centers are already in place. Axis have
appointed code block maintainers and code block administrators, these have
deep knowledge in their respective fields. To minimalize impingement on the daily
work the competence centers should be established along with the organizational
restructuring. It would be problematic to restructure the organization to align
with processes without establishing competence centers, given their central role
in process based organizations. The formation of competence centers is a
separate activity in the restructuring and could therefore be implemented
relatively early in the process.
7.1.2.4 Determine appropriate quantifiable measurements
Hereafter follows a short description of how to obtain the suggested
measurements.
Code Turmoil This measurement can be sampled by using scripting commands available
74
in the global information tracker, GIT. An example of such a script has been presented to Axis.
Open Defects This measurement is already available to extract from the Trouble
database.
Defect Find Rate This measurement is already available to extract from the Trouble
database.
Test Passing Rate This measurement is already available to extract from the QACE test suite
environment.
It’s important to underline that these measurements need to be documented and
historically examined to calibrate the scale for a fair and balanced picture of the
project status. Since most of the measurements already are available, the
implementation of a measurement system ought to be done relatively quickly. It
should preferably include someone at manager or project manager level at Axis.
7.2 Results The overall goal of the SPL section of the solution proposal is to better manage
versions and variants and to systematize the use of requirements. Potential
benefits are:
Gain deeper understanding of the platform components and how they
relate to products. Enabling identification of functionality not longer being
used.
Simplify troubleshooting and maintenance of code.
Make requirements visible and up to date, accessible for all stakeholders.
Establish traceability between requirements and functions throughout the
development and testing process.
The overall goal of the four step process improvement suggestion is to shift
paradigm from functional teams towards project teams that follow the process
from requirement to firmware release. Potential benefits are:
Significant reduction of documentation and bureaucracy needed to
control and ensure sufficient quality.
75
To greatly reduce the need for handover procedures and formal meetings,
giving way for more informal and cooperative communication and
collaboration.
Elimination of the knowledge gap; code producers review and take
responsibility for own code.
Better knowledge management in managed competence centers.
These two different proposals together represent a combined solution for
identified problems. Potential drawbacks of this solution could be:
Difficulties in predicting actual impact of suggested changes.
Organizational changes are sometimes complex and require full
dedication by all involved parties to be fully embraced and successful.
7.3 Limitations The solution proposal of this thesis underline how Axis differ from methodology
and practices described in the literature. We have chosen to select merely a few
but highly important suggestions for improvement. We find that a too extensive
solution proposal would not be feasible, neither in terms of cost or time needed
to implement such changes. The multitude of proposed changes threatens to
interfere somewhat with each other, but this risk can be minimized by introducing
changes in evolutionary steps rather than change everything at once.
We find the process of appointing roles at individual level is best handled by
managers at Axis given their deeper understanding of the detailed work processes
and organization. The restructure of the development organization also renders
the current tollgate model insufficient. The current format of handover
procedures simply does not apply in a team based project structure. When
performing the organizational restructuring, the current documentation should be
updated and in some cases rationalized.
7.4 Evaluation An initial evaluation of the proposed solutions was performed subsequently to the
thesis presentation for managers and staff at Axis. We received feedback on the
feasibility of certain suggestions as well as guidance to what areas that needed
clarification. These improvement suggestions were important in refining and
finalizing the thesis.
76
To fully understand the impact of our suggested changes, one would have to
establish a “baseline” status before any implementation takes place. One way of
establishing a baseline could be to implement the system of measurements
suggested in this thesis. These would then be used to compare project outcome
before and after implementing the proposed changes.
To fully grasp the impact of the changes and to capture opinions and suggestions,
we suggest that a small questionnaire will be sent to personnel at Axis. Since the
purpose of the questionnaire is to track how the changes are perceived over time
it should be sent to the employees on a regular basis. Examples of relevant
questions is for instance:
What benefits do you see with these changes?
What are the drawbacks working in the new process?
How has your daily work changed since the implementation of these changes?
Do you have any ideas or suggestions how to improve your daily work?
7.5 Fulfillment of purpose
7.5.1 Goals
In the introduction chapter of this thesis we identified certain goals to be reached.
These were:
G1: Compilation of current academic work in the area.
G2: Investigation of current firmware development processes at Axis
Communications.
G3: Suggest possible improvements based on academic studies.
G4: Evaluation of our suggested improvements and their validity.
G1 is completed and summarized in chapter 3. The basis of this chapter was an
extensive literature search.
G2 is completed in chapter 4. The basis of this chapter was internal process and
role documentations, meetings and other material at Axis.
G3 is completed in chapter 6. The solutions proposal was created by comparing
findings in the case study to theories and practices described in chapter 3 and
suggesting possible changes.
77
G4 is completed in this chapter.
7.5.2 Research questions
In the introduction chapter of this thesis we stated the two main research
questions to be addressed.
RQ1: How should Axis firmware development organization be
restructured to better utilize the benefits from using a software platform?
RQ2: How can firmware development at Axis be measured and managed
using a process based management approach?
RQ1 is addressed in chapter 6.1. The restructuring proposal is based on findings
from the FEF-evaluation in section 5.2.1. The suggestion is focused on managing
traceability in the development process. This can be done by introducing roles,
tools and methodologies from practices established in Software Product Lines.
RQ2 is addressed in chapter 6.2. The solution proposal is a four step model based
on theories and practices found in BPM. The aim of this restructuring is essentially
to align the organizational structure to the actual development, and to refocus on
value creating processes.
By using two fundamentally different approaches, BPM and SPL, to reach the
conclusion for these research questions we feel that we have addressed these
questions as extensively as possible. The two different practices does in many
ways cover different parts of the company spectrum, to create one uniform
solution proposal.
78
8 Discussion
This chapter concludes the thesis with a discussion of generalizability and
suggestions for future work.
8.1 Generalizability The methodology we have chosen to use in this thesis follows the action research
format. This format does not in itself generate any immediate generalizability to
other contexts than described in this thesis. The solutions proposed are applicable
to the current situation of Axis, but should reasonably be transferable to
companies similar in size and practices.
During our literature studies, we found that there are other companies similar in
size and technology that in one way or another use platform based development.
We have based the first part of our solution proposal on established theories in
SPL. These have in turn been tested and proven effective in environments similar
to one in which Axis operates.
The practices described in BPM are very general and applies to almost any type of
company, from the strictly service oriented to production and product based.
Because the practices are general to this degree the solution proposal of the four
step BPM restructuring should be reasonably transferable to other contexts, given
that such a solution is further tailored to fit given preconditions.
8.2 Reflections The conclusions reached in this thesis apply to the firmware development
divisions at Axis. We cannot claim to fully grasp the processes and practices of
product development at department A. However, we do feel that there is little
reason to have split the organization in two departments, one dedicated to
developing firmware, and the other one responsible for product development.
Department A develops products independent of the platform, but still integrate
most of the functionality eventually.
It is clear that Axis uses certain elements of SPL methodology, but the FEF clearly
showed that there is lack of awareness of how to utilize the benefits of using a
platform. There are clearly stated solutions of how to restructure the organization
to reach all benefits of SPL engineering, even further than described in our
solution proposal.
79
During the time of writing the thesis Axis took measures to implement a team
based organization with a structure similar to the competence center model
described in the BPM solution proposal. We find this to be an exciting change and
hope to have contributed to the decision.
The incentive for change is strong; many of the perceived difficulties we have
observed during our time writing the thesis has been of simple communicative or
collaborative nature. We believe these shortcomings to greatly decrease with a
team-based development structure.
80
8.3 Future work During the course of writing this thesis we encountered several areas substantial
enough for serving as the basis for new theses. Rework of the daily build system
and improving the stability tests could be a good candidate for future projects.
Our suggestion of measurement dimensions and their follow up is only an initial
approach. The implementation of the suggested dimensions along with further
improvement of how to utilize collected data could be an appropriate scope for
yet another thesis.
The suggested adoption of a tool for managing versions and variants constitute a
project life cycle in its own, with planning, implementation and follow up phases.
The suggestions in the BPM improvement proposal are partly implemented, with
the exception of creating and reprioritizing a process map. If Axis chooses to
realize a fully process based organization, we recommend the first step to be
creating a process map. This could be done in cooperation with experts or as part
of another thesis. When a process based organization is in place, it would also be
feasible to look into other closely related management techniques such as Lean
and Six Sigma.
81
9 References 1. AXIS, Annual Report 2010. 2011. 2. IMS, The World Market CCTV and Video Surveillance Equipment - 2010
Edition. 2010. 3. Bryman, A., E. Bell, and B. Nilsson, Företagsekonomiska
forskningsmetoder. 2005: Liber ekonomi. 4. Holme, Forskningsmetodik: Om kvalitativa och kvantitativa metoder. 3 ed.
1996. 5. Höst, M., B. Regnell, and P. Runeson, Att genomföra examensarbete. 2006:
Studentlitteratur. 6. Sarah, R., et al., Business Action Research in Practice-A Strategic
Conversation About Conducting Action Research in Business Organizations. Systemic Practice and Action Research, 2002. 15(6): p. 535-546.
7. Denscombe, The Good Research Guide - For small-scale social research projects. 1998: Open University Press, Buckingham.
8. Ardis, M., et al., Software product lines: a case study. Software: Practice and Experience, 2000. 30(7): p. 825-847.
9. Linden, F., K. Schmid, and E. Rommes, Software product lines in action: the best industrial practice in product line engineering. 2007: Springer.
10. van der Linden, F., et al., Software Product Family Evaluation, in Software Product Lines, R. Nord, Editor. 2004, Springer Berlin / Heidelberg. p. 107-109.
11. Chopra, A.K., et al., Requirements as Goals and Commitments Too, in Intentional Perspectives on Information Systems Engineering, S. Nurcan, et al., Editors. 2010, Springer Berlin Heidelberg. p. 137-153.
12. Soliman, K.S. and J.F. Affisco, Business process management journal: E-government. 2006: Emerald Group Publishing.
13. Romney, M., Business process reengineering. (cover story). Internal Auditor, 1995. 52(3): p. 24.
14. MacIntosh, R., BPR: alive and well in the public sector. International Journal of Operations & Production Management, 2003. 23(3): p. 327-345.
15. Ljungberg, A.L., Everth, Processbaserad verksamhetsutveckling. 2001. 16. Tao, Y., et al., A research on BPM system based on process knowledge.
2008 IEEE Conference on Cybernetics and Intelligent Systems, 2008: p. 69-75.
17. Kah-Hin, C. and Y. Chee-Meng, Effective knowledge transfer in virtual teams: linking contents and mechanisms. International Journal of Networking and Virtual Organisations, 2004. 2(4): p. 312-322.
18. Mahanti, R. and J. Antony, Confluence of six sigma, simulation and software development. Managerial Auditing Journal, 2005. 20(7): p. 739-762.
82
19. Pantano, V., et al., Cluster-based Six Sigma Deployment in Small and Medium Sized Enterprises. 2006 IEEE International Conference on Management of Innovation and Technology, 2006. 2: p. 788-792.
20. Staats, B.R., D.J. Brunner, and D.M. Upton, Lean principles, learning, and knowledge work: Evidence from a software services provider. Journal of Operations Management, 2011. 29(5): p. 376-390.
21. Middleton, P., et al., Lean principles and techniques for improving the quality and productivity of software development projects: a case study. International Journal of Productivity and Quality Management, 2007. 2(4): p. 387-403.
22. Scherrer-Rathje, M., T.A. Boyle, and P. Deflorin, Lean, take two! Reflections from the second attempt at lean implementation. Business Horizons, 2009. 52(1): p. 79-88.
23. Santos, R.d., K.d. Oliveira, and W.d. Silva, Evaluating the service quality of software providers appraised in CMM/CMMI. Software Quality Journal, 2009. 17(3): p. 283-301.
24. Behr, P., W.K. Giloi, and R. Gueth, Education in firmware engineering and microprogramming. Microprocessing and Microprogramming, 1981. 8(3-5): p. 153-166.
25. Edward, A.L., Embedded Software. 2009. 26. Greene, B., Agile methods applied to embedded firmware development.
Agile Development Conference, 2004, 2004: p. 71-77. 27. D.E. Perry, W.M.E., An Empirical Study of Software Interface Faults.
Proceedings of the Twentieth Annual Hawaii International Conference on Systems Sciences, January 1987, Volume II 1985.
28. Hecht, M., Products liability issues for embedded software in consumer applications. Product Safety Engineering, 2005 IEEE Symposium on, 2005: p. 42-48.
29. Pearse, T., T. Freeman, and P. Oman, Using metrics to manage the end-game of a software project. Software Metrics Symposium, 1999. Proceedings. Sixth International, 1999: p. 207-215.
30. McConnell, S., Code Complete. 2009: O'Reilly Media, Inc. 31. Mikael, S. and B. Jan, A Case Study on Product Line Architecture Evolution.
2009. 32. Louridas, P., Version control. IEEE Software, 2006. 23(1): p. 104-107.
83
10 List of abbreviations API – Application Programming Interface
CBA – Code Block Administrator
CBM – Code Block Manager
CCTV–Closed Circuit Television
GIT - Global Information Tracker
LFP – Linux Firmware Platform
PTZ – Pan Tilt Zoom
RAT - Resource Allocation Tool
SA – Software Architect
SWO – Software Overview
TL – Technical Lead
Top Related